Nov 03 19:36:02.799927 ip-10-0-132-83 systemd[1]: Starting Kubernetes Kubelet... Nov 03 19:36:03.188559 ip-10-0-132-83 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:36:03.188559 ip-10-0-132-83 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 19:36:03.188559 ip-10-0-132-83 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:36:03.188559 ip-10-0-132-83 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:36:03.193149 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193074 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:36:03.193149 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193150 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:36:03.193225 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193156 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:36:03.193225 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193160 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:36:03.193225 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193163 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:36:03.193225 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193166 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:36:03.193225 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193169 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:36:03.193225 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193171 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:36:03.193225 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193174 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:36:03.193225 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193177 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:36:03.193225 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193180 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:36:03.193225 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193183 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:36:03.193225 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193185 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:36:03.193225 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193193 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:36:03.193225 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193196 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:36:03.193225 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193199 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:36:03.193225 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193201 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:36:03.193225 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193204 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:36:03.193225 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193207 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:36:03.193225 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193209 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:36:03.193225 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193212 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:36:03.193682 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193215 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:36:03.193682 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193217 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:36:03.193682 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193221 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:36:03.193682 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193225 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:36:03.193682 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193229 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:36:03.193682 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193232 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:36:03.193682 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193236 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:36:03.193682 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193239 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:36:03.193682 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193241 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:36:03.193682 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193244 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:36:03.193682 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193247 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:36:03.193682 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193257 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:36:03.193682 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193260 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:36:03.193682 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193263 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:36:03.193682 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193266 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:36:03.193682 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193269 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:36:03.193682 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193271 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:36:03.193682 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193274 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:36:03.193682 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193276 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:36:03.194117 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193279 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:36:03.194117 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193282 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:36:03.194117 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193284 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:36:03.194117 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193287 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:36:03.194117 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193289 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:36:03.194117 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193292 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:36:03.194117 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193295 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:36:03.194117 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193298 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:36:03.194117 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193300 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:36:03.194117 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193303 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:36:03.194117 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193305 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:36:03.194117 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193309 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:36:03.194117 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193311 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:36:03.194117 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193314 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:36:03.194117 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193317 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:36:03.194117 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193320 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:36:03.194117 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193323 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:36:03.194117 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193326 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:36:03.194117 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193328 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:36:03.194117 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193331 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:36:03.194583 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193333 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:36:03.194583 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193336 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:36:03.194583 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193339 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:36:03.194583 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193341 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:36:03.194583 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193344 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:36:03.194583 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193346 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:36:03.194583 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193349 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:36:03.194583 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193351 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:36:03.194583 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193354 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:36:03.194583 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193356 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:36:03.194583 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193359 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:36:03.194583 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193362 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:36:03.194583 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193364 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:36:03.194583 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193367 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:36:03.194583 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193370 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:36:03.194583 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193372 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:36:03.194583 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193375 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:36:03.194583 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193377 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:36:03.194583 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193381 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:36:03.194583 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193384 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:36:03.195071 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193387 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:36:03.195071 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193389 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:36:03.195071 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193392 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:36:03.195071 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193394 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:36:03.195071 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193397 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:36:03.195071 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193400 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:36:03.195071 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193403 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:36:03.195071 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193405 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:36:03.195071 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193408 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:36:03.195071 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193410 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:36:03.195071 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193413 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:36:03.195071 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193416 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:36:03.195071 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193418 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:36:03.195071 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193421 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:36:03.195071 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193424 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:36:03.195071 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193426 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:36:03.195071 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193429 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:36:03.195071 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193432 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:36:03.195071 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193434 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:36:03.195071 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193437 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:36:03.195535 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193439 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:36:03.195535 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193443 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:36:03.195535 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193446 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:36:03.195535 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193448 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:36:03.195535 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193451 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:36:03.195535 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.193453 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:36:03.195535 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194251 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:36:03.195535 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194259 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:36:03.195535 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194262 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:36:03.195535 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194265 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:36:03.195535 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194270 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:36:03.195535 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194273 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:36:03.195535 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194276 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:36:03.195535 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194279 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:36:03.195535 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194281 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:36:03.195535 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194284 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:36:03.195535 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194287 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:36:03.195535 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194290 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:36:03.195535 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194293 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:36:03.196029 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194295 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:36:03.196029 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194298 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:36:03.196029 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194301 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:36:03.196029 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194303 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:36:03.196029 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194306 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:36:03.196029 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194309 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:36:03.196029 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194311 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:36:03.196029 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194314 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:36:03.196029 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194316 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:36:03.196029 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194319 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:36:03.196029 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194322 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:36:03.196029 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194324 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:36:03.196029 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194327 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:36:03.196029 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194329 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:36:03.196029 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194332 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:36:03.196029 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194335 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:36:03.196029 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194337 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:36:03.196029 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194340 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:36:03.196029 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194343 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:36:03.196029 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194345 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:36:03.196502 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194348 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:36:03.196502 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194350 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:36:03.196502 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194353 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:36:03.196502 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194356 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:36:03.196502 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194358 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:36:03.196502 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194362 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:36:03.196502 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194364 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:36:03.196502 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194367 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:36:03.196502 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194370 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:36:03.196502 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194372 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:36:03.196502 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194376 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:36:03.196502 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194379 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:36:03.196502 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194382 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:36:03.196502 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194384 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:36:03.196502 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194387 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:36:03.196502 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194389 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:36:03.196502 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194392 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:36:03.196502 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194395 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:36:03.196502 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194397 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:36:03.196502 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194400 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:36:03.196990 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194402 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:36:03.196990 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194405 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:36:03.196990 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194407 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:36:03.196990 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194411 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:36:03.196990 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194415 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:36:03.196990 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194419 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:36:03.196990 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194422 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:36:03.196990 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194426 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:36:03.196990 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194429 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:36:03.196990 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194431 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:36:03.196990 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194434 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:36:03.196990 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194437 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:36:03.196990 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194439 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:36:03.196990 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194442 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:36:03.196990 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194444 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:36:03.196990 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194448 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:36:03.196990 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194451 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:36:03.196990 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194454 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:36:03.196990 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194457 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:36:03.197449 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194461 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:36:03.197449 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194466 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:36:03.197449 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194469 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:36:03.197449 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194472 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:36:03.197449 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194475 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:36:03.197449 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194477 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:36:03.197449 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194480 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:36:03.197449 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194482 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:36:03.197449 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194485 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:36:03.197449 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194488 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:36:03.197449 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194491 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:36:03.197449 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194493 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:36:03.197449 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194496 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:36:03.197449 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194498 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:36:03.197449 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194501 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:36:03.197449 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194503 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:36:03.197449 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194506 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:36:03.197449 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194509 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:36:03.197449 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194511 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:36:03.197909 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194514 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:36:03.197909 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194517 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:36:03.197909 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194519 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:36:03.197909 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194521 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:36:03.197909 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194524 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:36:03.197909 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194527 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:36:03.197909 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194530 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:36:03.197909 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194532 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:36:03.197909 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194535 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:36:03.197909 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194537 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:36:03.197909 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194540 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:36:03.197909 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194543 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:36:03.197909 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194546 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:36:03.197909 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194550 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:36:03.197909 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.194553 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:36:03.197909 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195050 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:36:03.197909 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195055 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:36:03.197909 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195058 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:36:03.197909 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195061 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:36:03.197909 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195064 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:36:03.198371 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195066 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:36:03.198371 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195069 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:36:03.198371 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195072 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:36:03.198371 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195075 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:36:03.198371 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195077 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:36:03.198371 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195080 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:36:03.198371 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195083 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:36:03.198371 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195086 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:36:03.198371 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195088 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:36:03.198371 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195091 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:36:03.198371 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195094 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:36:03.198371 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195097 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:36:03.198371 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195099 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:36:03.198371 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195102 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:36:03.198371 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195105 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:36:03.198371 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195107 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:36:03.198371 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195110 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:36:03.198371 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195113 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:36:03.198371 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195115 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:36:03.198846 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195118 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:36:03.198846 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195120 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:36:03.198846 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195123 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:36:03.198846 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195125 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:36:03.198846 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195128 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:36:03.198846 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195131 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:36:03.198846 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195134 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:36:03.198846 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195136 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:36:03.198846 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195139 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:36:03.198846 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195141 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:36:03.198846 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195144 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:36:03.198846 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195146 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:36:03.198846 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195149 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:36:03.198846 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195151 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:36:03.198846 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195154 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:36:03.198846 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195156 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:36:03.198846 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195159 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:36:03.198846 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195161 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:36:03.198846 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195164 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:36:03.198846 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195166 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:36:03.199322 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195169 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:36:03.199322 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195172 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:36:03.199322 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195174 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:36:03.199322 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195177 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:36:03.199322 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195180 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:36:03.199322 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195182 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:36:03.199322 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195185 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:36:03.199322 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195187 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:36:03.199322 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195190 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:36:03.199322 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195193 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:36:03.199322 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195195 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:36:03.199322 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195199 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:36:03.199322 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195203 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:36:03.199322 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195205 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:36:03.199322 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195208 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:36:03.199322 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195212 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:36:03.199322 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195214 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:36:03.199322 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195217 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:36:03.199322 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195219 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:36:03.199778 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195223 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:36:03.199778 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195225 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:36:03.199778 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195229 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:36:03.199778 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195231 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:36:03.199778 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195234 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:36:03.199778 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195236 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:36:03.199778 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195239 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:36:03.199778 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195242 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:36:03.199778 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195244 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:36:03.199778 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195246 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:36:03.199778 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195249 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:36:03.199778 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195252 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:36:03.199778 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195254 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:36:03.199778 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195257 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:36:03.199778 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195259 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:36:03.199778 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195261 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:36:03.199778 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195264 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:36:03.199778 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195267 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:36:03.199778 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195269 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:36:03.199778 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195275 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:36:03.200248 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195277 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:36:03.200248 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195280 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:36:03.200248 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195283 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:36:03.200248 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195285 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:36:03.200248 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195288 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:36:03.200248 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195291 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:36:03.200248 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195294 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:36:03.200248 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195297 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:36:03.200248 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195300 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:36:03.200248 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195304 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:36:03.200248 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195308 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:36:03.200248 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195312 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:36:03.200248 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195315 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:36:03.200248 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195319 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:36:03.200248 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195322 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:36:03.200248 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195325 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:36:03.200248 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195327 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:36:03.200248 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195330 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:36:03.200248 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195333 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:36:03.200248 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195335 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:36:03.200770 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195338 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:36:03.200770 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195340 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:36:03.200770 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.195344 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:36:03.200770 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195438 2580 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 19:36:03.200770 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195448 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 19:36:03.200770 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195458 2580 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 19:36:03.200770 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195464 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 19:36:03.200770 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195471 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 19:36:03.200770 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195477 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 19:36:03.200770 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195484 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 19:36:03.200770 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195488 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 19:36:03.200770 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195494 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 19:36:03.200770 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195497 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 19:36:03.200770 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195501 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 19:36:03.200770 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195504 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 19:36:03.200770 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195507 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 19:36:03.200770 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195510 2580 flags.go:64] FLAG: --cgroup-root="" Nov 03 19:36:03.200770 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195514 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 19:36:03.200770 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195517 2580 flags.go:64] FLAG: --client-ca-file="" Nov 03 19:36:03.200770 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195520 2580 flags.go:64] FLAG: --cloud-provider="external" Nov 03 19:36:03.200770 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195523 2580 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 19:36:03.200770 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195527 2580 flags.go:64] FLAG: --cluster-domain="" Nov 03 19:36:03.200770 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195530 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 19:36:03.200770 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195533 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 19:36:03.201327 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195536 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 19:36:03.201327 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195540 2580 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 19:36:03.201327 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195545 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 19:36:03.201327 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195548 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 19:36:03.201327 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195552 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 19:36:03.201327 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195555 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 19:36:03.201327 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195558 2580 flags.go:64] FLAG: --contention-profiling="false" Nov 03 19:36:03.201327 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195561 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 19:36:03.201327 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195564 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 19:36:03.201327 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195568 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 19:36:03.201327 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195571 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 19:36:03.201327 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195575 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 19:36:03.201327 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195578 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 19:36:03.201327 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195581 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 19:36:03.201327 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195585 2580 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 19:36:03.201327 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195588 2580 flags.go:64] FLAG: --enable-server="true" Nov 03 19:36:03.201327 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195591 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 19:36:03.201327 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195599 2580 flags.go:64] FLAG: --event-burst="100" Nov 03 19:36:03.201327 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195602 2580 flags.go:64] FLAG: --event-qps="50" Nov 03 19:36:03.201327 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195606 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 19:36:03.201327 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195609 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 19:36:03.201327 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195613 2580 flags.go:64] FLAG: --eviction-hard="" Nov 03 19:36:03.201327 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195617 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 19:36:03.201327 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195620 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 19:36:03.201926 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195623 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 19:36:03.201926 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195626 2580 flags.go:64] FLAG: --eviction-soft="" Nov 03 19:36:03.201926 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195629 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 19:36:03.201926 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195633 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 19:36:03.201926 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195636 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 19:36:03.201926 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195639 2580 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 19:36:03.201926 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195642 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 19:36:03.201926 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195645 2580 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 19:36:03.201926 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195648 2580 flags.go:64] FLAG: --feature-gates="" Nov 03 19:36:03.201926 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195652 2580 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 19:36:03.201926 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195655 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 19:36:03.201926 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195672 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 19:36:03.201926 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195675 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 19:36:03.201926 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195679 2580 flags.go:64] FLAG: --healthz-port="10248" Nov 03 19:36:03.201926 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195682 2580 flags.go:64] FLAG: --help="false" Nov 03 19:36:03.201926 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195685 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:03.201926 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195688 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 19:36:03.201926 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195691 2580 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 19:36:03.201926 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195694 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 19:36:03.201926 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195698 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 19:36:03.201926 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195701 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 19:36:03.201926 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195704 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 19:36:03.201926 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195707 2580 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195710 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195713 2580 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195716 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195719 2580 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195724 2580 flags.go:64] FLAG: --kube-reserved="" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195727 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195730 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195733 2580 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195736 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195739 2580 flags.go:64] FLAG: --lock-file="" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195742 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195745 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195750 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195756 2580 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195759 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195762 2580 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195765 2580 flags.go:64] FLAG: --logging-format="text" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195768 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195772 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195775 2580 flags.go:64] FLAG: --manifest-url="" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195778 2580 flags.go:64] FLAG: --manifest-url-header="" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195783 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195786 2580 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195790 2580 flags.go:64] FLAG: --max-pods="110" Nov 03 19:36:03.202467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195793 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195797 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195800 2580 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195803 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195806 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195809 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195812 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195819 2580 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195822 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195825 2580 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195828 2580 flags.go:64] FLAG: --pod-cidr="" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195831 2580 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195836 2580 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195839 2580 flags.go:64] FLAG: --pods-per-core="0" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195842 2580 flags.go:64] FLAG: --port="10250" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195845 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195848 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-094694846601f51d3" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195851 2580 flags.go:64] FLAG: --qos-reserved="" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195854 2580 flags.go:64] FLAG: --read-only-port="10255" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195858 2580 flags.go:64] FLAG: --register-node="true" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195862 2580 flags.go:64] FLAG: --register-with-taints="" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195866 2580 flags.go:64] FLAG: --registry-burst="10" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195869 2580 flags.go:64] FLAG: --registry-qps="5" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195872 2580 flags.go:64] FLAG: --reserved-cpus="" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195876 2580 flags.go:64] FLAG: --reserved-memory="" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195879 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 19:36:03.203190 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195883 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195886 2580 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195889 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195892 2580 flags.go:64] FLAG: --runonce="false" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195896 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195899 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195903 2580 flags.go:64] FLAG: --seccomp-default="false" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195906 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195908 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195912 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195915 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195918 2580 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195921 2580 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195924 2580 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195927 2580 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195930 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195933 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195936 2580 flags.go:64] FLAG: --system-cgroups="" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195940 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195946 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195949 2580 flags.go:64] FLAG: --tls-cert-file="" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195952 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195956 2580 flags.go:64] FLAG: --tls-min-version="" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195959 2580 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195962 2580 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 19:36:03.203894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195965 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 19:36:03.204515 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195969 2580 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 19:36:03.204515 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195973 2580 flags.go:64] FLAG: --v="2" Nov 03 19:36:03.204515 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195977 2580 flags.go:64] FLAG: --version="false" Nov 03 19:36:03.204515 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195982 2580 flags.go:64] FLAG: --vmodule="" Nov 03 19:36:03.204515 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195986 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 19:36:03.204515 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.195989 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 19:36:03.204515 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196123 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:36:03.204515 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196127 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:36:03.204515 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196130 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:36:03.204515 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196133 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:36:03.204515 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196135 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:36:03.204515 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196139 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:36:03.204515 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196142 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:36:03.204515 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196144 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:36:03.204515 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196147 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:36:03.204515 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196150 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:36:03.204515 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196154 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:36:03.204515 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196158 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:36:03.204515 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196161 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:36:03.204515 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196164 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:36:03.204515 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196167 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:36:03.205026 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196171 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:36:03.205026 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196174 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:36:03.205026 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196177 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:36:03.205026 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196181 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:36:03.205026 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196184 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:36:03.205026 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196186 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:36:03.205026 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196189 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:36:03.205026 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196193 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:36:03.205026 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196195 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:36:03.205026 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196197 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:36:03.205026 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196201 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:36:03.205026 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196206 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:36:03.205026 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196209 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:36:03.205026 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196212 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:36:03.205026 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196214 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:36:03.205026 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196217 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:36:03.205026 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196220 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:36:03.205026 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196222 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:36:03.205026 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196225 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:36:03.205486 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196228 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:36:03.205486 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196231 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:36:03.205486 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196233 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:36:03.205486 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196236 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:36:03.205486 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196239 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:36:03.205486 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196242 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:36:03.205486 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196244 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:36:03.205486 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196247 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:36:03.205486 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196250 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:36:03.205486 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196252 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:36:03.205486 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196255 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:36:03.205486 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196257 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:36:03.205486 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196260 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:36:03.205486 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196263 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:36:03.205486 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196266 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:36:03.205486 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196268 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:36:03.205486 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196272 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:36:03.205486 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196275 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:36:03.205486 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196277 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:36:03.205952 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196280 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:36:03.205952 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196282 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:36:03.205952 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196285 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:36:03.205952 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196288 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:36:03.205952 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196290 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:36:03.205952 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196295 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:36:03.205952 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196298 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:36:03.205952 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196300 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:36:03.205952 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196303 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:36:03.205952 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196305 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:36:03.205952 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196308 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:36:03.205952 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196311 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:36:03.205952 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196313 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:36:03.205952 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196316 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:36:03.205952 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196318 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:36:03.205952 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196321 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:36:03.205952 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196323 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:36:03.205952 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196326 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:36:03.205952 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196329 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:36:03.205952 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196332 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:36:03.206492 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196334 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:36:03.206492 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196337 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:36:03.206492 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196339 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:36:03.206492 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196342 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:36:03.206492 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196344 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:36:03.206492 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196347 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:36:03.206492 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196349 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:36:03.206492 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196352 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:36:03.206492 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196355 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:36:03.206492 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196358 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:36:03.206492 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196361 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:36:03.206492 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196364 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:36:03.206492 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196367 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:36:03.206492 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196369 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:36:03.206492 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196372 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:36:03.206492 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196375 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:36:03.206492 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196377 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:36:03.206492 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196381 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:36:03.206492 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196384 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:36:03.206492 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196386 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:36:03.206991 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196389 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:36:03.206991 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196392 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:36:03.206991 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196395 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:36:03.206991 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196398 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:36:03.206991 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196400 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:36:03.206991 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196403 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:36:03.206991 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196405 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:36:03.206991 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196408 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:36:03.206991 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196410 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:36:03.206991 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196413 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:36:03.206991 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196415 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:36:03.206991 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196418 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:36:03.206991 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196421 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:36:03.206991 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.196447 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:36:03.206991 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196484 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:36:03.206991 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196488 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:36:03.207367 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196492 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:36:03.207367 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196497 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:36:03.207367 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196499 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:36:03.207367 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196502 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:36:03.207367 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196505 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:36:03.207367 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196508 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:36:03.207367 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196512 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:36:03.207367 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196515 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:36:03.207367 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196518 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:36:03.207367 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196520 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:36:03.207367 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196523 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:36:03.207367 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196526 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:36:03.207367 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196528 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:36:03.207367 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196532 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:36:03.207367 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196535 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:36:03.207367 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196539 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:36:03.207367 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196542 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:36:03.207367 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196545 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:36:03.207367 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196548 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:36:03.207827 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196551 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:36:03.207827 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196553 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:36:03.207827 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196556 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:36:03.207827 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196559 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:36:03.207827 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196562 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:36:03.207827 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196565 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:36:03.207827 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196567 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:36:03.207827 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196570 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:36:03.207827 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196572 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:36:03.207827 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196575 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:36:03.207827 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196578 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:36:03.207827 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196580 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:36:03.207827 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196583 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:36:03.207827 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196586 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:36:03.207827 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196588 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:36:03.207827 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196591 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:36:03.207827 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196594 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:36:03.207827 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196596 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:36:03.207827 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196599 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:36:03.207827 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196603 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:36:03.208291 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196605 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:36:03.208291 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196608 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:36:03.208291 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196610 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:36:03.208291 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196613 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:36:03.208291 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196616 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:36:03.208291 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196619 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:36:03.208291 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196623 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:36:03.208291 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196625 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:36:03.208291 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196628 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:36:03.208291 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196631 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:36:03.208291 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196634 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:36:03.208291 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196636 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:36:03.208291 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196639 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:36:03.208291 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196641 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:36:03.208291 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196644 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:36:03.208291 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196647 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:36:03.208291 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196649 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:36:03.208291 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196652 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:36:03.208291 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196654 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:36:03.208291 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196657 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:36:03.208765 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196671 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:36:03.208765 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196673 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:36:03.208765 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196677 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:36:03.208765 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196679 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:36:03.208765 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196682 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:36:03.208765 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196685 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:36:03.208765 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196687 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:36:03.208765 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196690 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:36:03.208765 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196692 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:36:03.208765 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196695 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:36:03.208765 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196697 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:36:03.208765 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196702 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:36:03.208765 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196704 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:36:03.208765 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196707 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:36:03.208765 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196709 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:36:03.208765 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196712 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:36:03.208765 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196715 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:36:03.208765 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196718 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:36:03.208765 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196722 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:36:03.209210 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196725 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:36:03.209210 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196728 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:36:03.209210 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196730 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:36:03.209210 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196733 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:36:03.209210 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196736 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:36:03.209210 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196739 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:36:03.209210 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196741 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:36:03.209210 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196745 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:36:03.209210 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196748 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:36:03.209210 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196751 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:36:03.209210 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196753 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:36:03.209210 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196756 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:36:03.209210 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196759 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:36:03.209210 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196761 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:36:03.209210 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196764 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:36:03.209210 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196767 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:36:03.209210 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196770 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:36:03.209210 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196772 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:36:03.209210 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196775 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:36:03.209210 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196777 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:36:03.209739 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196780 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:36:03.209739 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196783 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:36:03.209739 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196785 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:36:03.209739 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196788 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:36:03.209739 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196791 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:36:03.209739 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.196794 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:36:03.209739 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.205750 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 19:36:03.209739 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.205767 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 19:36:03.209739 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.205780 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 19:36:03.209739 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.205784 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 19:36:03.209739 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205859 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:36:03.209739 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205864 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:36:03.209739 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205873 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:36:03.209739 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205878 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:36:03.209739 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205881 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:36:03.209739 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205884 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:36:03.209739 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205886 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:36:03.209739 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205889 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:36:03.209739 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205892 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:36:03.210174 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205895 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:36:03.210174 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205897 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:36:03.210174 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205900 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:36:03.210174 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205903 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:36:03.210174 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205906 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:36:03.210174 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205908 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:36:03.210174 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205911 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:36:03.210174 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205913 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:36:03.210174 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205916 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:36:03.210174 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205918 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:36:03.210174 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205921 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:36:03.210174 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205924 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:36:03.210174 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205927 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:36:03.210174 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205930 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:36:03.210174 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205933 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:36:03.210174 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205936 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:36:03.210174 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205938 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:36:03.210174 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205941 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:36:03.210174 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205944 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:36:03.210174 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205946 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:36:03.210639 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205955 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:36:03.210639 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205957 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:36:03.210639 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205960 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:36:03.210639 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205963 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:36:03.210639 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205966 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:36:03.210639 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205969 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:36:03.210639 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205973 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:36:03.210639 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205977 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:36:03.210639 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205979 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:36:03.210639 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205982 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:36:03.210639 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205985 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:36:03.210639 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205987 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:36:03.210639 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205990 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:36:03.210639 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205993 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:36:03.210639 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205995 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:36:03.210639 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.205998 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:36:03.210639 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206001 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:36:03.210639 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206003 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:36:03.210639 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206006 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:36:03.210639 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206008 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:36:03.211119 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206011 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:36:03.211119 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206014 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:36:03.211119 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206016 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:36:03.211119 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206019 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:36:03.211119 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206022 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:36:03.211119 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206025 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:36:03.211119 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206027 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:36:03.211119 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206030 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:36:03.211119 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206033 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:36:03.211119 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206035 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:36:03.211119 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206038 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:36:03.211119 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206040 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:36:03.211119 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206043 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:36:03.211119 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206051 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:36:03.211119 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206054 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:36:03.211119 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206057 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:36:03.211119 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206059 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:36:03.211119 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206062 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:36:03.211119 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206065 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:36:03.211567 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206067 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:36:03.211567 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206070 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:36:03.211567 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206072 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:36:03.211567 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206075 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:36:03.211567 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206078 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:36:03.211567 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206080 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:36:03.211567 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206083 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:36:03.211567 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206086 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:36:03.211567 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206088 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:36:03.211567 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206091 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:36:03.211567 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206093 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:36:03.211567 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206096 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:36:03.211567 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206098 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:36:03.211567 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206101 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:36:03.211567 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206103 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:36:03.211567 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206107 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:36:03.211567 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206110 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:36:03.211567 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206112 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:36:03.211567 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206115 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:36:03.211567 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206117 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:36:03.212050 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206120 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:36:03.212050 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206122 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:36:03.212050 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206125 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:36:03.212050 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206127 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:36:03.212050 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206130 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:36:03.212050 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206133 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:36:03.212050 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206135 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:36:03.212050 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206144 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:36:03.212050 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206147 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:36:03.212050 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206150 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:36:03.212050 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206152 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:36:03.212050 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206155 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:36:03.212050 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206158 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:36:03.212050 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206160 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:36:03.212050 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206163 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:36:03.212050 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206166 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:36:03.212050 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206170 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:36:03.212050 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206174 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:36:03.212460 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.206202 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:36:03.212460 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206381 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:36:03.212460 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206385 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:36:03.212460 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206388 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:36:03.212460 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206391 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:36:03.212460 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206393 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:36:03.212460 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206396 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:36:03.212460 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206399 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:36:03.212460 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206401 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:36:03.212460 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206404 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:36:03.212460 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206407 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:36:03.212460 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206411 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:36:03.212460 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206414 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:36:03.212460 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206417 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:36:03.212460 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206420 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:36:03.212460 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206423 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:36:03.212916 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206425 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:36:03.212916 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206428 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:36:03.212916 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206430 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:36:03.212916 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206433 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:36:03.212916 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206435 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:36:03.212916 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206438 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:36:03.212916 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206446 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:36:03.212916 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206449 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:36:03.212916 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206452 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:36:03.212916 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206455 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:36:03.212916 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206457 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:36:03.212916 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206460 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:36:03.212916 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206462 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:36:03.212916 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206465 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:36:03.212916 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206467 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:36:03.212916 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206470 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:36:03.212916 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206473 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:36:03.212916 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206475 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:36:03.212916 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206478 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:36:03.213368 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206480 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:36:03.213368 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206482 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:36:03.213368 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206485 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:36:03.213368 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206488 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:36:03.213368 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206490 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:36:03.213368 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206492 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:36:03.213368 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206495 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:36:03.213368 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206498 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:36:03.213368 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206501 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:36:03.213368 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206506 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:36:03.213368 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206509 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:36:03.213368 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206512 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:36:03.213368 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206515 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:36:03.213368 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206518 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:36:03.213368 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206521 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:36:03.213368 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206523 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:36:03.213368 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206526 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:36:03.213368 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206528 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:36:03.213368 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206531 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:36:03.213968 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206533 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:36:03.213968 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206541 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:36:03.213968 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206544 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:36:03.213968 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206546 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:36:03.213968 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206549 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:36:03.213968 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206551 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:36:03.213968 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206553 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:36:03.213968 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206556 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:36:03.213968 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206558 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:36:03.213968 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206561 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:36:03.213968 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206563 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:36:03.213968 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206566 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:36:03.213968 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206568 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:36:03.213968 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206571 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:36:03.213968 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206574 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:36:03.213968 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206576 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:36:03.213968 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206579 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:36:03.213968 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206581 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:36:03.213968 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206584 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:36:03.213968 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206586 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:36:03.214495 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206589 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:36:03.214495 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206593 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:36:03.214495 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206595 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:36:03.214495 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206598 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:36:03.214495 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206600 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:36:03.214495 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206603 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:36:03.214495 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206606 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:36:03.214495 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206608 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:36:03.214495 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206611 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:36:03.214495 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206613 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:36:03.214495 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206616 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:36:03.214495 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206618 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:36:03.214495 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206620 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:36:03.214495 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206623 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:36:03.214495 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206631 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:36:03.214495 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206634 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:36:03.214495 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206636 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:36:03.214495 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206639 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:36:03.214495 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206641 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:36:03.214971 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206644 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:36:03.214971 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206647 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:36:03.214971 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206649 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:36:03.214971 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206651 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:36:03.214971 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206654 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:36:03.214971 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206656 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:36:03.214971 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206671 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:36:03.214971 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206674 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:36:03.214971 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206677 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:36:03.214971 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206679 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:36:03.214971 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206682 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:36:03.214971 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206684 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:36:03.214971 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206687 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:36:03.214971 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206690 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:36:03.214971 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.206713 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:36:03.214971 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206751 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:36:03.215352 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206755 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:36:03.215352 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206758 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:36:03.215352 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206761 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:36:03.215352 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206763 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:36:03.215352 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206766 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:36:03.215352 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206769 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:36:03.215352 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206771 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:36:03.215352 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206774 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:36:03.215352 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206777 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:36:03.215352 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206779 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:36:03.215352 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206781 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:36:03.215352 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206784 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:36:03.215352 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206803 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:36:03.215352 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206805 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:36:03.215352 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206808 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:36:03.215352 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206811 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:36:03.215352 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206813 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:36:03.215352 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206816 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:36:03.215352 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206818 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:36:03.215352 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206821 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:36:03.215837 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206825 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:36:03.215837 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206828 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:36:03.215837 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206832 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:36:03.215837 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206835 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:36:03.215837 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206838 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:36:03.215837 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206841 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:36:03.215837 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206843 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:36:03.215837 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206846 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:36:03.215837 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206848 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:36:03.215837 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206851 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:36:03.215837 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206853 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:36:03.215837 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206855 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:36:03.215837 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206858 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:36:03.215837 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206861 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:36:03.215837 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206863 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:36:03.215837 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206865 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:36:03.215837 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206868 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:36:03.215837 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206871 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:36:03.215837 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206873 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:36:03.215837 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206875 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:36:03.216319 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206878 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:36:03.216319 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206880 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:36:03.216319 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206883 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:36:03.216319 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206885 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:36:03.216319 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206888 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:36:03.216319 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206896 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:36:03.216319 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206899 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:36:03.216319 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206902 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:36:03.216319 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206904 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:36:03.216319 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206907 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:36:03.216319 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206909 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:36:03.216319 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206912 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:36:03.216319 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206914 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:36:03.216319 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206920 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:36:03.216319 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206923 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:36:03.216319 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206926 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:36:03.216319 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206928 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:36:03.216319 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206931 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:36:03.216319 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206933 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:36:03.216826 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206936 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:36:03.216826 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206939 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:36:03.216826 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206941 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:36:03.216826 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206943 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:36:03.216826 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206946 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:36:03.216826 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206949 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:36:03.216826 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206951 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:36:03.216826 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206954 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:36:03.216826 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206956 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:36:03.216826 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206959 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:36:03.216826 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206961 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:36:03.216826 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206964 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:36:03.216826 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206967 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:36:03.216826 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206970 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:36:03.216826 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206972 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:36:03.216826 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206975 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:36:03.216826 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206977 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:36:03.216826 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206980 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:36:03.216826 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206982 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:36:03.217260 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206990 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:36:03.217260 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206993 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:36:03.217260 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206996 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:36:03.217260 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.206998 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:36:03.217260 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.207001 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:36:03.217260 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.207004 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:36:03.217260 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.207007 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:36:03.217260 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.207011 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:36:03.217260 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.207014 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:36:03.217260 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.207016 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:36:03.217260 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.207019 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:36:03.217260 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.207021 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:36:03.217260 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.207024 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:36:03.217260 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.207026 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:36:03.217260 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.207029 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:36:03.217260 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.207033 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:36:03.217260 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.207037 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:36:03.217260 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.207040 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:36:03.217260 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.207043 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:36:03.217754 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.207046 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:36:03.217754 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.207048 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:36:03.217754 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.207051 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:36:03.217754 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.207053 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:36:03.217754 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.207056 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:36:03.217754 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.207058 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:36:03.217754 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.207061 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:36:03.217754 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:03.207063 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:36:03.217754 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.207122 2580 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 19:36:03.217754 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.209561 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 19:36:03.217754 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.210702 2580 server.go:1014] "Starting client certificate rotation" Nov 03 19:36:03.217754 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.210804 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 19:36:03.217754 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.210842 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 19:36:03.217754 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.214359 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:36:03.217754 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.216459 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:36:03.228169 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.228147 2580 log.go:25] "Validated CRI v1 runtime API" Nov 03 19:36:03.231925 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.231911 2580 log.go:25] "Validated CRI v1 image API" Nov 03 19:36:03.232773 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.232751 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 19:36:03.238310 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.238285 2580 fs.go:135] Filesystem UUIDs: map[160a260a-1b31-4b52-8c5a-0749e0241f24:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 98c29d79-7c2a-4cc8-bcbf-d3d2f24eb05b:/dev/nvme0n1p4] Nov 03 19:36:03.238383 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.238310 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 19:36:03.244099 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.243993 2580 manager.go:217] Machine: {Timestamp:2026-11-03 19:36:03.242510363 +0000 UTC m=+0.335639794 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099924 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26ff604744fdec5b889a367c7545e9 SystemUUID:ec26ff60-4744-fdec-5b88-9a367c7545e9 BootID:2904ca26-dd83-4680-b3d7-33f1ad2aa3e3 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c4:77:56:bb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c4:77:56:bb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:dd:de:9f:65:4c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 19:36:03.244099 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.244100 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 19:36:03.244210 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.244174 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 19:36:03.247435 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.247408 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 19:36:03.247568 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.247436 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-83.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 19:36:03.248289 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.248279 2580 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 19:36:03.248329 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.248291 2580 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 19:36:03.248329 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.248316 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:36:03.248386 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.248330 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:36:03.249782 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.249771 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 19:36:03.249829 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.249793 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 19:36:03.254714 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.254702 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 19:36:03.254834 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.254824 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 19:36:03.254874 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.254854 2580 kubelet.go:493] "Attempting to sync node with API server" Nov 03 19:36:03.254874 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.254864 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 19:36:03.254874 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.254874 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 19:36:03.254981 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.254882 2580 kubelet.go:405] "Adding apiserver pod source" Nov 03 19:36:03.254981 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.254890 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 19:36:03.255267 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.255252 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:36:03.257838 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.257822 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 19:36:03.266585 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.266557 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:36:03.267402 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.267388 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 19:36:03.267458 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.267420 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 19:36:03.267595 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.267584 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 19:36:03.267633 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.267600 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 19:36:03.267633 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.267606 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 19:36:03.267633 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.267612 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 19:36:03.267633 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.267618 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 19:36:03.267633 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.267624 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 19:36:03.268937 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.268927 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 19:36:03.268969 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.268937 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 19:36:03.268969 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.268945 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 19:36:03.268969 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.268951 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 19:36:03.268969 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.268960 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 19:36:03.268969 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.268969 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 19:36:03.270049 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.270026 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 19:36:03.270049 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.270043 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 19:36:03.273848 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.273832 2580 server.go:1263] "Started kubelet" Nov 03 19:36:03.273950 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.273928 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 19:36:03.274649 ip-10-0-132-83 systemd[1]: Started Kubernetes Kubelet. Nov 03 19:36:03.275625 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.275193 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 19:36:03.275718 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.275654 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 19:36:03.276191 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.276174 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-83.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 19:36:03.276383 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.276369 2580 server.go:317] "Adding debug handlers to kubelet server" Nov 03 19:36:03.276756 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.276727 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-83.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 19:36:03.276834 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.276759 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 19:36:03.278811 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.278756 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 19:36:03.280106 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.280084 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 19:36:03.285154 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.285135 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 19:36:03.285272 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.285136 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 19:36:03.285454 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.285432 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-83.us-east-2.compute.internal\" not found" Nov 03 19:36:03.286006 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.285984 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 19:36:03.286125 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.286108 2580 factory.go:55] Registering systemd factory Nov 03 19:36:03.286192 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.286130 2580 factory.go:223] Registration of the systemd container factory successfully Nov 03 19:36:03.286192 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.286113 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 19:36:03.286192 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.286029 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 19:36:03.286317 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.286218 2580 reconstruct.go:97] "Volume reconstruction finished" Nov 03 19:36:03.286317 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.286227 2580 reconciler.go:29] "Reconciler: start to sync state" Nov 03 19:36:03.286472 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.286452 2580 factory.go:153] Registering CRI-O factory Nov 03 19:36:03.286472 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.286473 2580 factory.go:223] Registration of the crio container factory successfully Nov 03 19:36:03.286643 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.286548 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 19:36:03.286643 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.286571 2580 factory.go:103] Registering Raw factory Nov 03 19:36:03.286643 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.286589 2580 manager.go:1199] Started watching for new ooms in manager Nov 03 19:36:03.287238 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.286067 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-83.us-east-2.compute.internal.18e4a014e9466f14 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-83.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-83.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:36:03.273797396 +0000 UTC m=+0.366926828,LastTimestamp:2026-11-03 19:36:03.273797396 +0000 UTC m=+0.366926828,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-83.us-east-2.compute.internal,}" Nov 03 19:36:03.287616 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.287601 2580 manager.go:319] Starting recovery of all containers Nov 03 19:36:03.293643 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.293616 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-83.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 19:36:03.300499 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.300471 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z8nps" Nov 03 19:36:03.300724 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.300646 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 19:36:03.302459 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.302445 2580 manager.go:324] Recovery completed Nov 03 19:36:03.306478 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.306466 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:36:03.308067 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.308049 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:36:03.308136 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.308073 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:36:03.308136 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.308083 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:36:03.308583 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.308569 2580 cpu_manager.go:226] "Starting" policy="none" Nov 03 19:36:03.308583 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.308582 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 19:36:03.308701 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.308601 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 19:36:03.308701 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.308610 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:36:03.309335 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.309319 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:36:03.309389 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.309342 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:36:03.309389 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.309352 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:36:03.312028 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.311938 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-83.us-east-2.compute.internal.18e4a014eb514e5e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-83.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-83.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-83.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:36:03.30806435 +0000 UTC m=+0.401193781,LastTimestamp:2026-11-03 19:36:03.30806435 +0000 UTC m=+0.401193781,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-83.us-east-2.compute.internal,}" Nov 03 19:36:03.319964 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.319937 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z8nps" Nov 03 19:36:03.321793 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.321734 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-83.us-east-2.compute.internal.18e4a014eb5184b4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-83.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-83.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-83.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:36:03.30807826 +0000 UTC m=+0.401207691,LastTimestamp:2026-11-03 19:36:03.30807826 +0000 UTC m=+0.401207691,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-83.us-east-2.compute.internal,}" Nov 03 19:36:03.327558 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.327484 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-83.us-east-2.compute.internal.18e4a014eb51a8fd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-83.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-83.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-83.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:36:03.308087549 +0000 UTC m=+0.401216981,LastTimestamp:2026-11-03 19:36:03.308087549 +0000 UTC m=+0.401216981,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-83.us-east-2.compute.internal,}" Nov 03 19:36:03.339030 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.338974 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-83.us-east-2.compute.internal.18e4a014eb514e5e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-83.us-east-2.compute.internal.18e4a014eb514e5e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-83.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-83.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-83.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:36:03.30806435 +0000 UTC m=+0.401193781,LastTimestamp:2026-11-03 19:36:03.309334279 +0000 UTC m=+0.402463710,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-83.us-east-2.compute.internal,}" Nov 03 19:36:03.358160 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.358106 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-83.us-east-2.compute.internal.18e4a014eb5184b4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-83.us-east-2.compute.internal.18e4a014eb5184b4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-83.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-83.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-83.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:36:03.30807826 +0000 UTC m=+0.401207691,LastTimestamp:2026-11-03 19:36:03.309346291 +0000 UTC m=+0.402475723,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-83.us-east-2.compute.internal,}" Nov 03 19:36:03.374846 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.374787 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-83.us-east-2.compute.internal.18e4a014eb51a8fd\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-83.us-east-2.compute.internal.18e4a014eb51a8fd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-83.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-83.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-83.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:36:03.308087549 +0000 UTC m=+0.401216981,LastTimestamp:2026-11-03 19:36:03.309355359 +0000 UTC m=+0.402484790,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-83.us-east-2.compute.internal,}" Nov 03 19:36:03.385973 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.385954 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-83.us-east-2.compute.internal\" not found" Nov 03 19:36:03.445803 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.445758 2580 policy_none.go:50] "Start" Nov 03 19:36:03.445803 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.445777 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 19:36:03.445803 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.445791 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 19:36:03.447813 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.447801 2580 policy_none.go:44] "Start" Nov 03 19:36:03.473102 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.469033 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 19:36:03.473102 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.470194 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 19:36:03.473102 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.470216 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 19:36:03.473102 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.470236 2580 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 19:36:03.473102 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.470267 2580 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 19:36:03.475270 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.475075 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 19:36:03.485210 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.485193 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:36:03.486300 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.486285 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-83.us-east-2.compute.internal\" not found" Nov 03 19:36:03.496023 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.496011 2580 manager.go:341] "Starting Device Plugin manager" Nov 03 19:36:03.496749 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.496735 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 19:36:03.496749 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.496748 2580 server.go:87] "Starting device plugin registration server" Nov 03 19:36:03.496970 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.496960 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 19:36:03.497004 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.496972 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 19:36:03.497101 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.497082 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 19:36:03.497200 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.497177 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 19:36:03.497200 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.497194 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 19:36:03.497730 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.497708 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 19:36:03.497821 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.497763 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-83.us-east-2.compute.internal\" not found" Nov 03 19:36:03.501014 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.500998 2580 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-83.us-east-2.compute.internal\" not found" node="ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:03.571161 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.571133 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal"] Nov 03 19:36:03.571220 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.571194 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:36:03.572185 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.572159 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:36:03.572270 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.572204 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:36:03.572270 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.572231 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:36:03.574337 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.574325 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:36:03.574506 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.574491 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:03.574565 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.574525 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:36:03.575155 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.575142 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:36:03.575235 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.575164 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:36:03.575235 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.575168 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:36:03.575235 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.575174 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:36:03.575378 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.575199 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:36:03.575378 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.575268 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:36:03.577367 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.577351 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:03.577461 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.577379 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:36:03.582441 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.582426 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:36:03.582510 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.582452 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:36:03.582510 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.582461 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:36:03.588287 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.588273 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ea3063cb5f7f736e0060bab60ff36a5e-config\") pod \"kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal\" (UID: \"ea3063cb5f7f736e0060bab60ff36a5e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:03.588339 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.588293 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e0047dbb7ee8d4e56f23e3bd51214c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal\" (UID: \"3e0047dbb7ee8d4e56f23e3bd51214c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:03.588339 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.588309 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e0047dbb7ee8d4e56f23e3bd51214c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal\" (UID: \"3e0047dbb7ee8d4e56f23e3bd51214c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:03.593850 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.593830 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-83.us-east-2.compute.internal\" not found" node="ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:03.597137 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.597114 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:36:03.597859 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.597839 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:36:03.597945 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.597867 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:36:03.597945 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.597882 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:36:03.597945 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.597912 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:03.598074 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.598058 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-83.us-east-2.compute.internal\" not found" node="ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:03.606947 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.606933 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:03.607017 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.606957 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-83.us-east-2.compute.internal\": node \"ip-10-0-132-83.us-east-2.compute.internal\" not found" Nov 03 19:36:03.634933 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.634916 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-83.us-east-2.compute.internal\" not found" Nov 03 19:36:03.688870 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.688846 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ea3063cb5f7f736e0060bab60ff36a5e-config\") pod \"kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal\" (UID: \"ea3063cb5f7f736e0060bab60ff36a5e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:03.688870 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.688870 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e0047dbb7ee8d4e56f23e3bd51214c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal\" (UID: \"3e0047dbb7ee8d4e56f23e3bd51214c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:03.688990 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.688887 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e0047dbb7ee8d4e56f23e3bd51214c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal\" (UID: \"3e0047dbb7ee8d4e56f23e3bd51214c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:03.688990 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.688935 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e0047dbb7ee8d4e56f23e3bd51214c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal\" (UID: \"3e0047dbb7ee8d4e56f23e3bd51214c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:03.688990 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.688950 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ea3063cb5f7f736e0060bab60ff36a5e-config\") pod \"kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal\" (UID: \"ea3063cb5f7f736e0060bab60ff36a5e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:03.688990 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.688961 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e0047dbb7ee8d4e56f23e3bd51214c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal\" (UID: \"3e0047dbb7ee8d4e56f23e3bd51214c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:03.735913 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.735862 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-83.us-east-2.compute.internal\" not found" Nov 03 19:36:03.836490 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.836475 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-83.us-east-2.compute.internal\" not found" Nov 03 19:36:03.899003 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.898988 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:03.902458 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:03.902444 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:03.936695 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:03.936674 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-83.us-east-2.compute.internal\" not found" Nov 03 19:36:04.037215 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:04.037163 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-83.us-east-2.compute.internal\" not found" Nov 03 19:36:04.138021 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:04.137991 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-83.us-east-2.compute.internal\" not found" Nov 03 19:36:04.176006 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:04.175985 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:36:04.209900 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:04.209873 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 19:36:04.210319 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:04.210022 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 19:36:04.210319 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:04.210044 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 19:36:04.239317 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:04.239290 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-83.us-east-2.compute.internal\" not found" Nov 03 19:36:04.339913 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:04.339891 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-83.us-east-2.compute.internal\" not found" Nov 03 19:36:04.344330 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:04.344315 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:36:04.345280 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:04.345264 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:36:04.345342 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:04.345295 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:36:04.345342 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:04.345307 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:36:04.440799 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:04.440776 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-83.us-east-2.compute.internal\" not found" Nov 03 19:36:04.450662 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:04.450615 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:31:03 +0000 UTC" deadline="2028-07-20 09:09:26.866160628 +0000 UTC" Nov 03 19:36:04.450764 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:04.450660 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14989h33m22.415502917s" Nov 03 19:36:04.457020 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:04.457004 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:36:04.541528 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:04.541493 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-83.us-east-2.compute.internal\" not found" Nov 03 19:36:04.549196 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:04.549172 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:36:04.586367 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:04.586342 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:04.593808 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:04.593731 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e0047dbb7ee8d4e56f23e3bd51214c3.slice/crio-0c4f7b61fcd4ba2db6accad32dd25c0948b92f9ceb71408c97376cd46a8690c4 WatchSource:0}: Error finding container 0c4f7b61fcd4ba2db6accad32dd25c0948b92f9ceb71408c97376cd46a8690c4: Status 404 returned error can't find the container with id 0c4f7b61fcd4ba2db6accad32dd25c0948b92f9ceb71408c97376cd46a8690c4 Nov 03 19:36:04.597992 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:04.597964 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3063cb5f7f736e0060bab60ff36a5e.slice/crio-d74276b7e45093444f92ecc5311a048e8ade116d042b902e48a8c8dd526a8f79 WatchSource:0}: Error finding container d74276b7e45093444f92ecc5311a048e8ade116d042b902e48a8c8dd526a8f79: Status 404 returned error can't find the container with id d74276b7e45093444f92ecc5311a048e8ade116d042b902e48a8c8dd526a8f79 Nov 03 19:36:04.599601 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:04.599586 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:36:04.605199 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:04.605182 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:36:04.606377 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:04.606362 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:04.619598 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:04.619575 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:36:05.256044 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.256019 2580 apiserver.go:52] "Watching apiserver" Nov 03 19:36:05.265940 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.265912 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 19:36:05.266336 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.266314 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-mkm7n","openshift-image-registry/node-ca-kzr2l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-qvxjq","openshift-network-diagnostics/network-check-target-b4x9q","openshift-ovn-kubernetes/ovnkube-node-2m4mq","kube-system/global-pull-secret-syncer-wh9tj","kube-system/konnectivity-agent-q4zkg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8","openshift-dns/node-resolver-4d4ll","openshift-multus/multus-additional-cni-plugins-cxdrc","openshift-multus/multus-xlfb8","openshift-network-operator/iptables-alerter-7zx7h"] Nov 03 19:36:05.269073 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.269047 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:05.271368 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.271345 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.273280 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.273258 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 19:36:05.273532 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.273509 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xv4gr\"" Nov 03 19:36:05.273656 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.273631 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 19:36:05.273784 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.273740 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 19:36:05.275312 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.274975 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kzr2l" Nov 03 19:36:05.275312 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.275042 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 19:36:05.275312 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.275248 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qhpjg\"" Nov 03 19:36:05.275966 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.275729 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 19:36:05.275966 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.275852 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 19:36:05.275966 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.275935 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 19:36:05.276168 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.276089 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7zx7h" Nov 03 19:36:05.278430 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.278409 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:05.278561 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:05.278536 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxjq" podUID="caf7d6c9-0bb2-4da0-be2d-caea7734bbcb" Nov 03 19:36:05.279408 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.278981 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 19:36:05.279408 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.279020 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 19:36:05.279408 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.279085 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ldllm\"" Nov 03 19:36:05.279408 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.279170 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 19:36:05.281007 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.280983 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rs44g\"" Nov 03 19:36:05.281101 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.281055 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:36:05.281242 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.281221 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 19:36:05.281480 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.281463 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:05.281577 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:05.281553 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4x9q" podUID="c95249a6-899f-4faf-82dc-6e553453ff80" Nov 03 19:36:05.281656 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.281640 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 19:36:05.283569 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.283549 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q4zkg" Nov 03 19:36:05.285881 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.285863 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 19:36:05.287841 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.287345 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.290856 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.290804 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.293995 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.293960 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:05.294092 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:05.294041 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wh9tj" podUID="116af81a-d3a0-4558-93c4-8bf061b6815f" Nov 03 19:36:05.296268 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.296252 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4d4ll" Nov 03 19:36:05.298135 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.298113 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-etc-kubernetes\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.298219 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.298148 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs\") pod \"network-metrics-daemon-qvxjq\" (UID: \"caf7d6c9-0bb2-4da0-be2d-caea7734bbcb\") " pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:05.298219 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.298176 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-run-openvswitch\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.298219 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.298200 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p75v\" (UniqueName: \"kubernetes.io/projected/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-kube-api-access-7p75v\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.298326 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.298226 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-ovnkube-config\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.298326 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.298250 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4fdbd17a-1e15-4442-b715-aaec1087da11-konnectivity-ca\") pod \"konnectivity-agent-q4zkg\" (UID: \"4fdbd17a-1e15-4442-b715-aaec1087da11\") " pod="kube-system/konnectivity-agent-q4zkg" Nov 03 19:36:05.298326 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.298276 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-socket-dir\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:05.298326 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.298314 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55f124eb-2ebf-4675-a08a-bad3757061b3-iptables-alerter-script\") pod \"iptables-alerter-7zx7h\" (UID: \"55f124eb-2ebf-4675-a08a-bad3757061b3\") " pod="openshift-network-operator/iptables-alerter-7zx7h" Nov 03 19:36:05.298467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.298341 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-device-dir\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:05.298467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.298366 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2398ba99-35d3-4461-b66e-f44ac013b2d2-serviceca\") pod \"node-ca-kzr2l\" (UID: \"2398ba99-35d3-4461-b66e-f44ac013b2d2\") " pod="openshift-image-registry/node-ca-kzr2l" Nov 03 19:36:05.298467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.298391 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-run-netns\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.298467 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.298438 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-run-ovn-kubernetes\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.298649 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.298520 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-ovn-node-metrics-cert\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.298649 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.298569 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx7kl\" (UniqueName: \"kubernetes.io/projected/2398ba99-35d3-4461-b66e-f44ac013b2d2-kube-api-access-jx7kl\") pod \"node-ca-kzr2l\" (UID: \"2398ba99-35d3-4461-b66e-f44ac013b2d2\") " pod="openshift-image-registry/node-ca-kzr2l" Nov 03 19:36:05.298649 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.298602 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-etc-openvswitch\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.298795 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.298714 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-system-cni-dir\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.298795 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.298749 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-var-lib-cni-bin\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.298795 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.298791 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-run-multus-certs\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.298945 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.298842 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tpvj\" (UniqueName: \"kubernetes.io/projected/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-kube-api-access-2tpvj\") pod \"network-metrics-daemon-qvxjq\" (UID: \"caf7d6c9-0bb2-4da0-be2d-caea7734bbcb\") " pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:05.298945 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.298886 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hszb\" (UniqueName: \"kubernetes.io/projected/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-kube-api-access-9hszb\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.298945 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.298916 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4fdbd17a-1e15-4442-b715-aaec1087da11-agent-certs\") pod \"konnectivity-agent-q4zkg\" (UID: \"4fdbd17a-1e15-4442-b715-aaec1087da11\") " pod="kube-system/konnectivity-agent-q4zkg" Nov 03 19:36:05.299098 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.298947 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-etc-selinux\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:05.299098 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.298972 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-slash\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.299098 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299008 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-run-systemd\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.299098 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299060 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-node-log\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.299098 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299088 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-cni-binary-copy\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.299351 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299114 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-multus-daemon-config\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.299351 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299140 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-cnibin\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.299351 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299198 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-multus-socket-dir-parent\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.299351 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299238 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:05.299351 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299272 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-log-socket\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.299351 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299289 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-ovnkube-script-lib\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.299351 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299309 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx2l9\" (UniqueName: \"kubernetes.io/projected/55f124eb-2ebf-4675-a08a-bad3757061b3-kube-api-access-lx2l9\") pod \"iptables-alerter-7zx7h\" (UID: \"55f124eb-2ebf-4675-a08a-bad3757061b3\") " pod="openshift-network-operator/iptables-alerter-7zx7h" Nov 03 19:36:05.299351 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299330 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-kubelet\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.299653 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299368 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-run-ovn\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.299653 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299386 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:05.299653 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299404 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-multus-cni-dir\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.299653 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299449 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-run-k8s-cni-cncf-io\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.299653 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299474 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-run-netns\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.299653 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299521 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf5tn\" (UniqueName: \"kubernetes.io/projected/c95249a6-899f-4faf-82dc-6e553453ff80-kube-api-access-lf5tn\") pod \"network-check-target-b4x9q\" (UID: \"c95249a6-899f-4faf-82dc-6e553453ff80\") " pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:05.299653 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299551 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2398ba99-35d3-4461-b66e-f44ac013b2d2-host\") pod \"node-ca-kzr2l\" (UID: \"2398ba99-35d3-4461-b66e-f44ac013b2d2\") " pod="openshift-image-registry/node-ca-kzr2l" Nov 03 19:36:05.299653 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299582 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-cni-netd\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.299653 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299621 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn9fc\" (UniqueName: \"kubernetes.io/projected/e64f9d9f-c868-470e-bd78-4ce8baabd97a-kube-api-access-mn9fc\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:05.300082 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299694 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-multus-conf-dir\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.300082 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299721 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-cni-bin\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.300082 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299768 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.300082 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299843 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-var-lib-openvswitch\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.300082 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299870 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-var-lib-cni-multus\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.300082 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.299914 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-var-lib-kubelet\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.300082 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.300023 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-hostroot\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.300082 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.300080 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55f124eb-2ebf-4675-a08a-bad3757061b3-host-slash\") pod \"iptables-alerter-7zx7h\" (UID: \"55f124eb-2ebf-4675-a08a-bad3757061b3\") " pod="openshift-network-operator/iptables-alerter-7zx7h" Nov 03 19:36:05.300459 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.300125 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-sys-fs\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:05.300459 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.300166 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-systemd-units\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.300459 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.300194 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-registration-dir\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:05.300459 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.300249 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-env-overrides\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.302146 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.301877 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 19:36:05.333771 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.333704 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hqv2h" Nov 03 19:36:05.346497 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.346471 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hqv2h" Nov 03 19:36:05.383558 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.383528 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lrbp4\"" Nov 03 19:36:05.383696 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.383641 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 19:36:05.383794 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.383711 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xbkg2\"" Nov 03 19:36:05.383858 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.383850 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 19:36:05.385409 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.383948 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bkd9f\"" Nov 03 19:36:05.385409 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.384055 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 19:36:05.385409 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.384174 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:36:05.385409 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.384237 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 19:36:05.385409 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.384407 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 19:36:05.385409 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.384501 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 19:36:05.385409 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.384680 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 19:36:05.385409 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.384801 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 19:36:05.385409 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.384857 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ndfpg\"" Nov 03 19:36:05.385409 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.384914 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jzp4r\"" Nov 03 19:36:05.385409 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.384952 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 19:36:05.385409 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.384991 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 19:36:05.385409 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.385006 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 19:36:05.385409 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.385075 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 19:36:05.385409 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.385343 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 19:36:05.387161 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.387142 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 19:36:05.401280 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.400871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-run-multus-certs\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.401280 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.400921 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tpvj\" (UniqueName: \"kubernetes.io/projected/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-kube-api-access-2tpvj\") pod \"network-metrics-daemon-qvxjq\" (UID: \"caf7d6c9-0bb2-4da0-be2d-caea7734bbcb\") " pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:05.401280 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.401001 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-run-multus-certs\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.401280 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.401150 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hszb\" (UniqueName: \"kubernetes.io/projected/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-kube-api-access-9hszb\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.401280 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.401215 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-kubernetes\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.401280 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.401254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-sysctl-conf\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.401644 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.401286 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-system-cni-dir\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:05.401644 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.401318 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4fdbd17a-1e15-4442-b715-aaec1087da11-agent-certs\") pod \"konnectivity-agent-q4zkg\" (UID: \"4fdbd17a-1e15-4442-b715-aaec1087da11\") " pod="kube-system/konnectivity-agent-q4zkg" Nov 03 19:36:05.401644 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.401380 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-etc-selinux\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:05.401644 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.401474 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-slash\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.401644 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.401548 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-etc-selinux\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:05.401958 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.401638 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-run-systemd\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.401958 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.401662 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-slash\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.401958 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.401733 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-run-systemd\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.401958 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.401736 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-node-log\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.401958 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.401779 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-systemd\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.401958 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.401785 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-node-log\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.401958 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.401854 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-lib-modules\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.401958 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.401902 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-cni-binary-copy\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.401958 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.401906 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 19:36:05.402441 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402085 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-multus-daemon-config\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.402441 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402120 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-cnibin\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.402441 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402145 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-multus-socket-dir-parent\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.402441 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402172 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:05.402441 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402200 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-log-socket\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.402441 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402228 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-ovnkube-script-lib\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.402441 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402322 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:05.402441 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402361 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-multus-socket-dir-parent\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.402441 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402410 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/94d082d5-1c33-4394-9d93-67645f9b60cd-hosts-file\") pod \"node-resolver-4d4ll\" (UID: \"94d082d5-1c33-4394-9d93-67645f9b60cd\") " pod="openshift-dns/node-resolver-4d4ll" Nov 03 19:36:05.402441 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402414 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-cnibin\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.402441 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402440 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx2l9\" (UniqueName: \"kubernetes.io/projected/55f124eb-2ebf-4675-a08a-bad3757061b3-kube-api-access-lx2l9\") pod \"iptables-alerter-7zx7h\" (UID: \"55f124eb-2ebf-4675-a08a-bad3757061b3\") " pod="openshift-network-operator/iptables-alerter-7zx7h" Nov 03 19:36:05.403022 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402486 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-cni-binary-copy\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.403022 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402490 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-kubelet\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.403022 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402567 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-run-ovn\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.403022 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402615 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-run-ovn\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.403022 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402619 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-multus-daemon-config\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.403022 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402636 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5xvg\" (UniqueName: \"kubernetes.io/projected/53b14f9a-431e-4dbb-a725-4c8470262ca7-kube-api-access-m5xvg\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.403022 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402659 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-kubelet\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.403022 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402687 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-multus-cni-dir\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.403022 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402719 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-run-k8s-cni-cncf-io\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.403022 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402769 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-run-netns\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.403022 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402801 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf5tn\" (UniqueName: \"kubernetes.io/projected/c95249a6-899f-4faf-82dc-6e553453ff80-kube-api-access-lf5tn\") pod \"network-check-target-b4x9q\" (UID: \"c95249a6-899f-4faf-82dc-6e553453ff80\") " pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:05.403022 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402824 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-run-k8s-cni-cncf-io\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.403022 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402803 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-multus-cni-dir\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.403022 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402883 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2398ba99-35d3-4461-b66e-f44ac013b2d2-host\") pod \"node-ca-kzr2l\" (UID: \"2398ba99-35d3-4461-b66e-f44ac013b2d2\") " pod="openshift-image-registry/node-ca-kzr2l" Nov 03 19:36:05.403022 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.402980 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-run-netns\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.408761 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.405841 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-log-socket\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.408761 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.405905 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-cni-netd\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.408761 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.406582 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-run\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.408761 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.405953 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-cni-netd\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.408761 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.406493 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-ovnkube-script-lib\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.408761 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.406006 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2398ba99-35d3-4461-b66e-f44ac013b2d2-host\") pod \"node-ca-kzr2l\" (UID: \"2398ba99-35d3-4461-b66e-f44ac013b2d2\") " pod="openshift-image-registry/node-ca-kzr2l" Nov 03 19:36:05.408761 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.407342 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn9fc\" (UniqueName: \"kubernetes.io/projected/e64f9d9f-c868-470e-bd78-4ce8baabd97a-kube-api-access-mn9fc\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:05.408761 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.407397 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/116af81a-d3a0-4558-93c4-8bf061b6815f-kubelet-config\") pod \"global-pull-secret-syncer-wh9tj\" (UID: \"116af81a-d3a0-4558-93c4-8bf061b6815f\") " pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:05.408761 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.407432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-multus-conf-dir\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.408761 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.407492 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-cni-bin\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.408761 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.407531 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.408761 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.407572 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:05.408761 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.407602 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-var-lib-openvswitch\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.408761 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.407637 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-var-lib-cni-multus\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.408761 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.407676 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.408761 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.407653 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-cni-bin\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.408761 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.407894 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-var-lib-cni-multus\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.409619 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.407914 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-var-lib-kubelet\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.409619 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.407940 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-multus-conf-dir\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.409619 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.407952 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-hostroot\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.409619 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.407943 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-var-lib-openvswitch\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.409619 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.408039 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55f124eb-2ebf-4675-a08a-bad3757061b3-host-slash\") pod \"iptables-alerter-7zx7h\" (UID: \"55f124eb-2ebf-4675-a08a-bad3757061b3\") " pod="openshift-network-operator/iptables-alerter-7zx7h" Nov 03 19:36:05.409619 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.408127 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-sys\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.409619 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.408198 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55f124eb-2ebf-4675-a08a-bad3757061b3-host-slash\") pod \"iptables-alerter-7zx7h\" (UID: \"55f124eb-2ebf-4675-a08a-bad3757061b3\") " pod="openshift-network-operator/iptables-alerter-7zx7h" Nov 03 19:36:05.409619 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.408188 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:05.409619 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.408234 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-sys-fs\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:05.409619 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.408239 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-var-lib-kubelet\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.409619 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.408311 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-hostroot\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.409619 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.408370 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-sys-fs\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:05.409619 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.408390 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-systemd-units\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.409619 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.408473 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-modprobe-d\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.409619 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.408517 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-systemd-units\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.409619 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.408594 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-host\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.409619 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.408623 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-cnibin\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:05.410396 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.408701 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-registration-dir\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:05.410396 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.408758 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-env-overrides\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.410396 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.408787 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/116af81a-d3a0-4558-93c4-8bf061b6815f-dbus\") pod \"global-pull-secret-syncer-wh9tj\" (UID: \"116af81a-d3a0-4558-93c4-8bf061b6815f\") " pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:05.410396 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.408820 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-sysctl-d\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.410396 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.408898 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:05.410396 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.408935 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnslt\" (UniqueName: \"kubernetes.io/projected/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-kube-api-access-lnslt\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:05.410396 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.408976 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-etc-kubernetes\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.410396 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.409076 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs\") pod \"network-metrics-daemon-qvxjq\" (UID: \"caf7d6c9-0bb2-4da0-be2d-caea7734bbcb\") " pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:05.410396 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.409108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-run-openvswitch\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.410396 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.409140 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-tuned\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.410396 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.409168 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p75v\" (UniqueName: \"kubernetes.io/projected/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-kube-api-access-7p75v\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.410396 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.409200 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-ovnkube-config\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.410396 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.409332 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-var-lib-kubelet\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.410396 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.409366 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsppf\" (UniqueName: \"kubernetes.io/projected/94d082d5-1c33-4394-9d93-67645f9b60cd-kube-api-access-rsppf\") pod \"node-resolver-4d4ll\" (UID: \"94d082d5-1c33-4394-9d93-67645f9b60cd\") " pod="openshift-dns/node-resolver-4d4ll" Nov 03 19:36:05.410396 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.409400 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4fdbd17a-1e15-4442-b715-aaec1087da11-konnectivity-ca\") pod \"konnectivity-agent-q4zkg\" (UID: \"4fdbd17a-1e15-4442-b715-aaec1087da11\") " pod="kube-system/konnectivity-agent-q4zkg" Nov 03 19:36:05.410396 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.409432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-socket-dir\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:05.410396 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.409456 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-sysconfig\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.411077 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.409510 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-cni-binary-copy\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:05.411077 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.409544 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55f124eb-2ebf-4675-a08a-bad3757061b3-iptables-alerter-script\") pod \"iptables-alerter-7zx7h\" (UID: \"55f124eb-2ebf-4675-a08a-bad3757061b3\") " pod="openshift-network-operator/iptables-alerter-7zx7h" Nov 03 19:36:05.411077 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.409578 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-device-dir\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:05.411077 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.409607 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2398ba99-35d3-4461-b66e-f44ac013b2d2-serviceca\") pod \"node-ca-kzr2l\" (UID: \"2398ba99-35d3-4461-b66e-f44ac013b2d2\") " pod="openshift-image-registry/node-ca-kzr2l" Nov 03 19:36:05.411077 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.409640 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-run-netns\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.411077 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.409673 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-run-ovn-kubernetes\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.411077 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.409705 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-ovn-node-metrics-cert\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.411077 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.409759 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/116af81a-d3a0-4558-93c4-8bf061b6815f-original-pull-secret\") pod \"global-pull-secret-syncer-wh9tj\" (UID: \"116af81a-d3a0-4558-93c4-8bf061b6815f\") " pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:05.411077 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.409794 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx7kl\" (UniqueName: \"kubernetes.io/projected/2398ba99-35d3-4461-b66e-f44ac013b2d2-kube-api-access-jx7kl\") pod \"node-ca-kzr2l\" (UID: \"2398ba99-35d3-4461-b66e-f44ac013b2d2\") " pod="openshift-image-registry/node-ca-kzr2l" Nov 03 19:36:05.411077 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.409843 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-etc-openvswitch\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.411077 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.409911 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53b14f9a-431e-4dbb-a725-4c8470262ca7-tmp\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.411077 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.409967 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94d082d5-1c33-4394-9d93-67645f9b60cd-tmp-dir\") pod \"node-resolver-4d4ll\" (UID: \"94d082d5-1c33-4394-9d93-67645f9b60cd\") " pod="openshift-dns/node-resolver-4d4ll" Nov 03 19:36:05.411077 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.410015 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-system-cni-dir\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.411077 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.410086 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-var-lib-cni-bin\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.411077 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.410193 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-var-lib-cni-bin\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.411077 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.410297 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-etc-kubernetes\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.411077 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.410401 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55f124eb-2ebf-4675-a08a-bad3757061b3-iptables-alerter-script\") pod \"iptables-alerter-7zx7h\" (UID: \"55f124eb-2ebf-4675-a08a-bad3757061b3\") " pod="openshift-network-operator/iptables-alerter-7zx7h" Nov 03 19:36:05.411077 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:05.410425 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:36:05.411900 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.410488 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-device-dir\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:05.411900 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:05.410536 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs podName:caf7d6c9-0bb2-4da0-be2d-caea7734bbcb nodeName:}" failed. No retries permitted until 2026-11-03 19:36:05.910498409 +0000 UTC m=+3.003627847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs") pod "network-metrics-daemon-qvxjq" (UID: "caf7d6c9-0bb2-4da0-be2d-caea7734bbcb") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:36:05.411900 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.410588 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-run-openvswitch\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.411900 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.410800 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4fdbd17a-1e15-4442-b715-aaec1087da11-konnectivity-ca\") pod \"konnectivity-agent-q4zkg\" (UID: \"4fdbd17a-1e15-4442-b715-aaec1087da11\") " pod="kube-system/konnectivity-agent-q4zkg" Nov 03 19:36:05.411900 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.410831 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4fdbd17a-1e15-4442-b715-aaec1087da11-agent-certs\") pod \"konnectivity-agent-q4zkg\" (UID: \"4fdbd17a-1e15-4442-b715-aaec1087da11\") " pod="kube-system/konnectivity-agent-q4zkg" Nov 03 19:36:05.411900 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.410872 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-socket-dir\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:05.411900 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.410893 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-registration-dir\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:05.411900 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.410955 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-etc-openvswitch\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.411900 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.411003 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2398ba99-35d3-4461-b66e-f44ac013b2d2-serviceca\") pod \"node-ca-kzr2l\" (UID: \"2398ba99-35d3-4461-b66e-f44ac013b2d2\") " pod="openshift-image-registry/node-ca-kzr2l" Nov 03 19:36:05.411900 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.411100 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-system-cni-dir\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.411900 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.411106 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-env-overrides\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.411900 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.411104 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-run-netns\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.411900 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.411146 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-run-ovn-kubernetes\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.412385 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.412135 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-ovnkube-config\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.416329 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.416266 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-ovn-node-metrics-cert\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.416503 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.416484 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx2l9\" (UniqueName: \"kubernetes.io/projected/55f124eb-2ebf-4675-a08a-bad3757061b3-kube-api-access-lx2l9\") pod \"iptables-alerter-7zx7h\" (UID: \"55f124eb-2ebf-4675-a08a-bad3757061b3\") " pod="openshift-network-operator/iptables-alerter-7zx7h" Nov 03 19:36:05.416797 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.416750 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hszb\" (UniqueName: \"kubernetes.io/projected/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-kube-api-access-9hszb\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.417089 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.417068 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tpvj\" (UniqueName: \"kubernetes.io/projected/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-kube-api-access-2tpvj\") pod \"network-metrics-daemon-qvxjq\" (UID: \"caf7d6c9-0bb2-4da0-be2d-caea7734bbcb\") " pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:05.417549 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:05.417528 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:36:05.417634 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:05.417553 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:36:05.417634 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:05.417564 2580 projected.go:196] Error preparing data for projected volume kube-api-access-lf5tn for pod openshift-network-diagnostics/network-check-target-b4x9q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:36:05.417634 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:05.417615 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c95249a6-899f-4faf-82dc-6e553453ff80-kube-api-access-lf5tn podName:c95249a6-899f-4faf-82dc-6e553453ff80 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:05.917597703 +0000 UTC m=+3.010727139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lf5tn" (UniqueName: "kubernetes.io/projected/c95249a6-899f-4faf-82dc-6e553453ff80-kube-api-access-lf5tn") pod "network-check-target-b4x9q" (UID: "c95249a6-899f-4faf-82dc-6e553453ff80") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:36:05.421120 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.421100 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn9fc\" (UniqueName: \"kubernetes.io/projected/e64f9d9f-c868-470e-bd78-4ce8baabd97a-kube-api-access-mn9fc\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:05.423614 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.423591 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p75v\" (UniqueName: \"kubernetes.io/projected/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-kube-api-access-7p75v\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.427962 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.427937 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx7kl\" (UniqueName: \"kubernetes.io/projected/2398ba99-35d3-4461-b66e-f44ac013b2d2-kube-api-access-jx7kl\") pod \"node-ca-kzr2l\" (UID: \"2398ba99-35d3-4461-b66e-f44ac013b2d2\") " pod="openshift-image-registry/node-ca-kzr2l" Nov 03 19:36:05.476234 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.476170 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal" event={"ID":"ea3063cb5f7f736e0060bab60ff36a5e","Type":"ContainerStarted","Data":"d74276b7e45093444f92ecc5311a048e8ade116d042b902e48a8c8dd526a8f79"} Nov 03 19:36:05.477470 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.477443 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" event={"ID":"3e0047dbb7ee8d4e56f23e3bd51214c3","Type":"ContainerStarted","Data":"0c4f7b61fcd4ba2db6accad32dd25c0948b92f9ceb71408c97376cd46a8690c4"} Nov 03 19:36:05.511765 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.511201 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-sysconfig\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.511765 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.511244 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-cni-binary-copy\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:05.511765 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.511277 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/116af81a-d3a0-4558-93c4-8bf061b6815f-original-pull-secret\") pod \"global-pull-secret-syncer-wh9tj\" (UID: \"116af81a-d3a0-4558-93c4-8bf061b6815f\") " pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:05.511765 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.511342 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-sysconfig\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.511765 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:05.511375 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:36:05.511765 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:05.511432 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/116af81a-d3a0-4558-93c4-8bf061b6815f-original-pull-secret podName:116af81a-d3a0-4558-93c4-8bf061b6815f nodeName:}" failed. No retries permitted until 2026-11-03 19:36:06.011414162 +0000 UTC m=+3.104543597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/116af81a-d3a0-4558-93c4-8bf061b6815f-original-pull-secret") pod "global-pull-secret-syncer-wh9tj" (UID: "116af81a-d3a0-4558-93c4-8bf061b6815f") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:36:05.511765 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.511475 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53b14f9a-431e-4dbb-a725-4c8470262ca7-tmp\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.511765 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.511506 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94d082d5-1c33-4394-9d93-67645f9b60cd-tmp-dir\") pod \"node-resolver-4d4ll\" (UID: \"94d082d5-1c33-4394-9d93-67645f9b60cd\") " pod="openshift-dns/node-resolver-4d4ll" Nov 03 19:36:05.511765 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.511536 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-kubernetes\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.511765 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.511561 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-sysctl-conf\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.511765 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.511588 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-system-cni-dir\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:05.511765 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.511618 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-systemd\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.511765 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.511642 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-lib-modules\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.511765 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.511674 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/94d082d5-1c33-4394-9d93-67645f9b60cd-hosts-file\") pod \"node-resolver-4d4ll\" (UID: \"94d082d5-1c33-4394-9d93-67645f9b60cd\") " pod="openshift-dns/node-resolver-4d4ll" Nov 03 19:36:05.511765 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.511722 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5xvg\" (UniqueName: \"kubernetes.io/projected/53b14f9a-431e-4dbb-a725-4c8470262ca7-kube-api-access-m5xvg\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.511765 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.511763 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-run\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.512427 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.511789 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/116af81a-d3a0-4558-93c4-8bf061b6815f-kubelet-config\") pod \"global-pull-secret-syncer-wh9tj\" (UID: \"116af81a-d3a0-4558-93c4-8bf061b6815f\") " pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:05.512427 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.511823 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:05.512427 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.511871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-sys\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.512427 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.511899 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:05.512427 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.511907 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-cni-binary-copy\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:05.512427 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.511928 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-modprobe-d\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.512427 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.511964 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-host\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.512427 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.511984 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/94d082d5-1c33-4394-9d93-67645f9b60cd-hosts-file\") pod \"node-resolver-4d4ll\" (UID: \"94d082d5-1c33-4394-9d93-67645f9b60cd\") " pod="openshift-dns/node-resolver-4d4ll" Nov 03 19:36:05.512427 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.512001 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-cnibin\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:05.512427 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.512030 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/116af81a-d3a0-4558-93c4-8bf061b6815f-dbus\") pod \"global-pull-secret-syncer-wh9tj\" (UID: \"116af81a-d3a0-4558-93c4-8bf061b6815f\") " pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:05.512427 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.512027 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-kubernetes\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.512427 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.512056 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-sysctl-d\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.512427 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.512135 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-sysctl-conf\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.512427 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.512134 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-systemd\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.512427 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.512161 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:05.512427 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.512187 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-sysctl-d\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.512427 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.512199 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnslt\" (UniqueName: \"kubernetes.io/projected/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-kube-api-access-lnslt\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:05.513217 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.512249 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-tuned\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.513217 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.512253 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-lib-modules\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.513217 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.512280 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-var-lib-kubelet\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.513217 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.512309 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsppf\" (UniqueName: \"kubernetes.io/projected/94d082d5-1c33-4394-9d93-67645f9b60cd-kube-api-access-rsppf\") pod \"node-resolver-4d4ll\" (UID: \"94d082d5-1c33-4394-9d93-67645f9b60cd\") " pod="openshift-dns/node-resolver-4d4ll" Nov 03 19:36:05.513217 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.512331 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-run\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.513217 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.512381 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/116af81a-d3a0-4558-93c4-8bf061b6815f-kubelet-config\") pod \"global-pull-secret-syncer-wh9tj\" (UID: \"116af81a-d3a0-4558-93c4-8bf061b6815f\") " pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:05.513217 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.512425 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-system-cni-dir\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:05.513217 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.512435 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-modprobe-d\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.513217 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.512480 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-host\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.513217 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.512508 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-cnibin\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:05.513217 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.512534 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:05.513217 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.512544 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/116af81a-d3a0-4558-93c4-8bf061b6815f-dbus\") pod \"global-pull-secret-syncer-wh9tj\" (UID: \"116af81a-d3a0-4558-93c4-8bf061b6815f\") " pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:05.513217 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.512602 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-sys\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.513217 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.512635 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-var-lib-kubelet\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.513812 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.513730 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:05.514134 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.514057 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94d082d5-1c33-4394-9d93-67645f9b60cd-tmp-dir\") pod \"node-resolver-4d4ll\" (UID: \"94d082d5-1c33-4394-9d93-67645f9b60cd\") " pod="openshift-dns/node-resolver-4d4ll" Nov 03 19:36:05.514219 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.514195 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:05.515786 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.515755 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-tuned\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.516351 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.516328 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53b14f9a-431e-4dbb-a725-4c8470262ca7-tmp\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.529206 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.529184 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsppf\" (UniqueName: \"kubernetes.io/projected/94d082d5-1c33-4394-9d93-67645f9b60cd-kube-api-access-rsppf\") pod \"node-resolver-4d4ll\" (UID: \"94d082d5-1c33-4394-9d93-67645f9b60cd\") " pod="openshift-dns/node-resolver-4d4ll" Nov 03 19:36:05.529783 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.529761 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnslt\" (UniqueName: \"kubernetes.io/projected/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-kube-api-access-lnslt\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:05.532351 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.532280 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5xvg\" (UniqueName: \"kubernetes.io/projected/53b14f9a-431e-4dbb-a725-4c8470262ca7-kube-api-access-m5xvg\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.591905 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.591879 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:05.599407 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.599363 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kzr2l" Nov 03 19:36:05.603434 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:05.603405 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64f9d9f_c868_470e_bd78_4ce8baabd97a.slice/crio-03461b119eb96fc911cff538b5a6be0cb57a44cba669682e00c64daa7dd146fd WatchSource:0}: Error finding container 03461b119eb96fc911cff538b5a6be0cb57a44cba669682e00c64daa7dd146fd: Status 404 returned error can't find the container with id 03461b119eb96fc911cff538b5a6be0cb57a44cba669682e00c64daa7dd146fd Nov 03 19:36:05.609659 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.609639 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xlfb8" Nov 03 19:36:05.610124 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:05.610100 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2398ba99_35d3_4461_b66e_f44ac013b2d2.slice/crio-378e0842a4e27216e6788143c711761cf336eabefb9faface56ff15da16e8218 WatchSource:0}: Error finding container 378e0842a4e27216e6788143c711761cf336eabefb9faface56ff15da16e8218: Status 404 returned error can't find the container with id 378e0842a4e27216e6788143c711761cf336eabefb9faface56ff15da16e8218 Nov 03 19:36:05.616925 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.616899 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7zx7h" Nov 03 19:36:05.619922 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:05.619902 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6025fec_2ce4_4638_9a74_1f58c7ba7e8b.slice/crio-9ba62026c6d8111a62ecf49684a219609734418cf6b76e7f93021ff6e6d26ddc WatchSource:0}: Error finding container 9ba62026c6d8111a62ecf49684a219609734418cf6b76e7f93021ff6e6d26ddc: Status 404 returned error can't find the container with id 9ba62026c6d8111a62ecf49684a219609734418cf6b76e7f93021ff6e6d26ddc Nov 03 19:36:05.624433 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.624414 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q4zkg" Nov 03 19:36:05.627307 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:05.627268 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f124eb_2ebf_4675_a08a_bad3757061b3.slice/crio-8a115502e7ad24bf1026d336ac422c9fc0e0473373c5d1488bcfc2fc861f4aec WatchSource:0}: Error finding container 8a115502e7ad24bf1026d336ac422c9fc0e0473373c5d1488bcfc2fc861f4aec: Status 404 returned error can't find the container with id 8a115502e7ad24bf1026d336ac422c9fc0e0473373c5d1488bcfc2fc861f4aec Nov 03 19:36:05.635031 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:05.635007 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fdbd17a_1e15_4442_b715_aaec1087da11.slice/crio-0ff53ee8da3120ef2b5adedd44d8db06f9192eab24e2f05017e50093b4b8bb2d WatchSource:0}: Error finding container 0ff53ee8da3120ef2b5adedd44d8db06f9192eab24e2f05017e50093b4b8bb2d: Status 404 returned error can't find the container with id 0ff53ee8da3120ef2b5adedd44d8db06f9192eab24e2f05017e50093b4b8bb2d Nov 03 19:36:05.649228 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.649201 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:05.653272 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.653252 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:05.656072 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.655902 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4d4ll" Nov 03 19:36:05.660760 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:05.660728 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b14f9a_431e_4dbb_a725_4c8470262ca7.slice/crio-7cce1f0196018b657444d70f50eeb90812a0b191dbcc836b00b0813599e43922 WatchSource:0}: Error finding container 7cce1f0196018b657444d70f50eeb90812a0b191dbcc836b00b0813599e43922: Status 404 returned error can't find the container with id 7cce1f0196018b657444d70f50eeb90812a0b191dbcc836b00b0813599e43922 Nov 03 19:36:05.668496 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:05.668472 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f8e09a_ee3b_4c5e_acd1_bce32adc0a42.slice/crio-68f02c50c3562aee1180e793d24de99a71dde738bf562bb7e9f0dc8649d1eab4 WatchSource:0}: Error finding container 68f02c50c3562aee1180e793d24de99a71dde738bf562bb7e9f0dc8649d1eab4: Status 404 returned error can't find the container with id 68f02c50c3562aee1180e793d24de99a71dde738bf562bb7e9f0dc8649d1eab4 Nov 03 19:36:05.670413 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:05.670283 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d082d5_1c33_4394_9d93_67645f9b60cd.slice/crio-c2b962f17b61966cbbfd2bd9acd8c1d47071750979c7e64374959e75a9332477 WatchSource:0}: Error finding container c2b962f17b61966cbbfd2bd9acd8c1d47071750979c7e64374959e75a9332477: Status 404 returned error can't find the container with id c2b962f17b61966cbbfd2bd9acd8c1d47071750979c7e64374959e75a9332477 Nov 03 19:36:05.671826 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.671806 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:05.682112 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:05.682087 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f0d931_3a92_424f_a5fc_d9ebef985dfd.slice/crio-b164ebe179b4dc809db2208bdd9376de1000c8be52a466df2cf5cc5c8cdf45e9 WatchSource:0}: Error finding container b164ebe179b4dc809db2208bdd9376de1000c8be52a466df2cf5cc5c8cdf45e9: Status 404 returned error can't find the container with id b164ebe179b4dc809db2208bdd9376de1000c8be52a466df2cf5cc5c8cdf45e9 Nov 03 19:36:05.732638 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.732616 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:36:05.915441 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:05.915400 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs\") pod \"network-metrics-daemon-qvxjq\" (UID: \"caf7d6c9-0bb2-4da0-be2d-caea7734bbcb\") " pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:05.915651 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:05.915590 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:36:05.915792 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:05.915699 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs podName:caf7d6c9-0bb2-4da0-be2d-caea7734bbcb nodeName:}" failed. No retries permitted until 2026-11-03 19:36:06.915662825 +0000 UTC m=+4.008792251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs") pod "network-metrics-daemon-qvxjq" (UID: "caf7d6c9-0bb2-4da0-be2d-caea7734bbcb") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:36:06.016180 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:06.016142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf5tn\" (UniqueName: \"kubernetes.io/projected/c95249a6-899f-4faf-82dc-6e553453ff80-kube-api-access-lf5tn\") pod \"network-check-target-b4x9q\" (UID: \"c95249a6-899f-4faf-82dc-6e553453ff80\") " pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:06.016350 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:06.016212 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/116af81a-d3a0-4558-93c4-8bf061b6815f-original-pull-secret\") pod \"global-pull-secret-syncer-wh9tj\" (UID: \"116af81a-d3a0-4558-93c4-8bf061b6815f\") " pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:06.016350 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:06.016328 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:36:06.016462 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:06.016375 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/116af81a-d3a0-4558-93c4-8bf061b6815f-original-pull-secret podName:116af81a-d3a0-4558-93c4-8bf061b6815f nodeName:}" failed. No retries permitted until 2026-11-03 19:36:07.016362355 +0000 UTC m=+4.109491777 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/116af81a-d3a0-4558-93c4-8bf061b6815f-original-pull-secret") pod "global-pull-secret-syncer-wh9tj" (UID: "116af81a-d3a0-4558-93c4-8bf061b6815f") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:36:06.016772 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:06.016755 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:36:06.016874 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:06.016776 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:36:06.016874 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:06.016786 2580 projected.go:196] Error preparing data for projected volume kube-api-access-lf5tn for pod openshift-network-diagnostics/network-check-target-b4x9q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:36:06.016874 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:06.016846 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c95249a6-899f-4faf-82dc-6e553453ff80-kube-api-access-lf5tn podName:c95249a6-899f-4faf-82dc-6e553453ff80 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:07.016831389 +0000 UTC m=+4.109960828 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lf5tn" (UniqueName: "kubernetes.io/projected/c95249a6-899f-4faf-82dc-6e553453ff80-kube-api-access-lf5tn") pod "network-check-target-b4x9q" (UID: "c95249a6-899f-4faf-82dc-6e553453ff80") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:36:06.350203 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:06.350094 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:31:05 +0000 UTC" deadline="2028-05-11 05:32:39.48727675 +0000 UTC" Nov 03 19:36:06.350203 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:06.350153 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13305h56m33.137129017s" Nov 03 19:36:06.470662 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:06.470616 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:06.470839 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:06.470768 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wh9tj" podUID="116af81a-d3a0-4558-93c4-8bf061b6815f" Nov 03 19:36:06.499593 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:06.499502 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7zx7h" event={"ID":"55f124eb-2ebf-4675-a08a-bad3757061b3","Type":"ContainerStarted","Data":"8a115502e7ad24bf1026d336ac422c9fc0e0473373c5d1488bcfc2fc861f4aec"} Nov 03 19:36:06.509761 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:06.509717 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xlfb8" event={"ID":"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b","Type":"ContainerStarted","Data":"9ba62026c6d8111a62ecf49684a219609734418cf6b76e7f93021ff6e6d26ddc"} Nov 03 19:36:06.520248 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:06.520145 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kzr2l" event={"ID":"2398ba99-35d3-4461-b66e-f44ac013b2d2","Type":"ContainerStarted","Data":"378e0842a4e27216e6788143c711761cf336eabefb9faface56ff15da16e8218"} Nov 03 19:36:06.531839 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:06.531736 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" event={"ID":"e64f9d9f-c868-470e-bd78-4ce8baabd97a","Type":"ContainerStarted","Data":"03461b119eb96fc911cff538b5a6be0cb57a44cba669682e00c64daa7dd146fd"} Nov 03 19:36:06.542395 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:06.542316 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxdrc" event={"ID":"b6f0d931-3a92-424f-a5fc-d9ebef985dfd","Type":"ContainerStarted","Data":"b164ebe179b4dc809db2208bdd9376de1000c8be52a466df2cf5cc5c8cdf45e9"} Nov 03 19:36:06.554708 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:06.554679 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4d4ll" event={"ID":"94d082d5-1c33-4394-9d93-67645f9b60cd","Type":"ContainerStarted","Data":"c2b962f17b61966cbbfd2bd9acd8c1d47071750979c7e64374959e75a9332477"} Nov 03 19:36:06.572640 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:06.572528 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" event={"ID":"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42","Type":"ContainerStarted","Data":"68f02c50c3562aee1180e793d24de99a71dde738bf562bb7e9f0dc8649d1eab4"} Nov 03 19:36:06.586748 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:06.586713 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" event={"ID":"53b14f9a-431e-4dbb-a725-4c8470262ca7","Type":"ContainerStarted","Data":"7cce1f0196018b657444d70f50eeb90812a0b191dbcc836b00b0813599e43922"} Nov 03 19:36:06.599350 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:06.599092 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q4zkg" event={"ID":"4fdbd17a-1e15-4442-b715-aaec1087da11","Type":"ContainerStarted","Data":"0ff53ee8da3120ef2b5adedd44d8db06f9192eab24e2f05017e50093b4b8bb2d"} Nov 03 19:36:06.926218 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:06.926135 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs\") pod \"network-metrics-daemon-qvxjq\" (UID: \"caf7d6c9-0bb2-4da0-be2d-caea7734bbcb\") " pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:06.926402 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:06.926310 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:36:06.926402 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:06.926377 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs podName:caf7d6c9-0bb2-4da0-be2d-caea7734bbcb nodeName:}" failed. No retries permitted until 2026-11-03 19:36:08.926356579 +0000 UTC m=+6.019486003 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs") pod "network-metrics-daemon-qvxjq" (UID: "caf7d6c9-0bb2-4da0-be2d-caea7734bbcb") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:36:06.957318 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:06.957286 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:36:07.027309 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:07.027277 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf5tn\" (UniqueName: \"kubernetes.io/projected/c95249a6-899f-4faf-82dc-6e553453ff80-kube-api-access-lf5tn\") pod \"network-check-target-b4x9q\" (UID: \"c95249a6-899f-4faf-82dc-6e553453ff80\") " pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:07.027471 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:07.027355 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/116af81a-d3a0-4558-93c4-8bf061b6815f-original-pull-secret\") pod \"global-pull-secret-syncer-wh9tj\" (UID: \"116af81a-d3a0-4558-93c4-8bf061b6815f\") " pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:07.027471 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:07.027445 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:36:07.027602 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:07.027476 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:36:07.027602 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:07.027491 2580 projected.go:196] Error preparing data for projected volume kube-api-access-lf5tn for pod openshift-network-diagnostics/network-check-target-b4x9q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:36:07.027602 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:07.027554 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c95249a6-899f-4faf-82dc-6e553453ff80-kube-api-access-lf5tn podName:c95249a6-899f-4faf-82dc-6e553453ff80 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:09.027533857 +0000 UTC m=+6.120663291 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lf5tn" (UniqueName: "kubernetes.io/projected/c95249a6-899f-4faf-82dc-6e553453ff80-kube-api-access-lf5tn") pod "network-check-target-b4x9q" (UID: "c95249a6-899f-4faf-82dc-6e553453ff80") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:36:07.027960 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:07.027451 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:36:07.028066 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:07.028053 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/116af81a-d3a0-4558-93c4-8bf061b6815f-original-pull-secret podName:116af81a-d3a0-4558-93c4-8bf061b6815f nodeName:}" failed. No retries permitted until 2026-11-03 19:36:09.028018696 +0000 UTC m=+6.121148133 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/116af81a-d3a0-4558-93c4-8bf061b6815f-original-pull-secret") pod "global-pull-secret-syncer-wh9tj" (UID: "116af81a-d3a0-4558-93c4-8bf061b6815f") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:36:07.350446 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:07.350382 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:31:05 +0000 UTC" deadline="2028-07-27 18:18:53.337070362 +0000 UTC" Nov 03 19:36:07.350446 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:07.350421 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15166h42m45.986655964s" Nov 03 19:36:07.473219 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:07.470782 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:07.473219 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:07.470931 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxjq" podUID="caf7d6c9-0bb2-4da0-be2d-caea7734bbcb" Nov 03 19:36:07.473219 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:07.473010 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:07.473219 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:07.473116 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4x9q" podUID="c95249a6-899f-4faf-82dc-6e553453ff80" Nov 03 19:36:07.620481 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:07.620441 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal" event={"ID":"ea3063cb5f7f736e0060bab60ff36a5e","Type":"ContainerStarted","Data":"c9c1860be7dfaad8a21ff56677a08bb380d30d2c7b48c834c50e89b4e2eb7555"} Nov 03 19:36:07.628316 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:07.628284 2580 generic.go:356] "Generic (PLEG): container finished" podID="3e0047dbb7ee8d4e56f23e3bd51214c3" containerID="c6978819fa7c37d1667e66b210c3e508059c55dfaaee2d872abd178cd61f6c5d" exitCode=0 Nov 03 19:36:07.628536 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:07.628333 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" event={"ID":"3e0047dbb7ee8d4e56f23e3bd51214c3","Type":"ContainerDied","Data":"c6978819fa7c37d1667e66b210c3e508059c55dfaaee2d872abd178cd61f6c5d"} Nov 03 19:36:07.640388 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:07.640337 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal" podStartSLOduration=2.640319499 podStartE2EDuration="2.640319499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:07.63951194 +0000 UTC m=+4.732641397" watchObservedRunningTime="2026-11-03 19:36:07.640319499 +0000 UTC m=+4.733448947" Nov 03 19:36:08.471122 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:08.471088 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:08.471617 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:08.471244 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wh9tj" podUID="116af81a-d3a0-4558-93c4-8bf061b6815f" Nov 03 19:36:08.653779 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:08.653724 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" event={"ID":"3e0047dbb7ee8d4e56f23e3bd51214c3","Type":"ContainerStarted","Data":"88ccefea5dcfebefffd8248d43bf2bd987ea1b0002b05b75afaa845ea72bd60e"} Nov 03 19:36:08.672785 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:08.672708 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" podStartSLOduration=3.672629248 podStartE2EDuration="3.672629248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:08.672019572 +0000 UTC m=+5.765149016" watchObservedRunningTime="2026-11-03 19:36:08.672629248 +0000 UTC m=+5.765758696" Nov 03 19:36:08.946681 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:08.946300 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs\") pod \"network-metrics-daemon-qvxjq\" (UID: \"caf7d6c9-0bb2-4da0-be2d-caea7734bbcb\") " pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:08.946681 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:08.946535 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:36:08.946681 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:08.946607 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs podName:caf7d6c9-0bb2-4da0-be2d-caea7734bbcb nodeName:}" failed. No retries permitted until 2026-11-03 19:36:12.946587608 +0000 UTC m=+10.039717048 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs") pod "network-metrics-daemon-qvxjq" (UID: "caf7d6c9-0bb2-4da0-be2d-caea7734bbcb") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:36:09.048104 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:09.048072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf5tn\" (UniqueName: \"kubernetes.io/projected/c95249a6-899f-4faf-82dc-6e553453ff80-kube-api-access-lf5tn\") pod \"network-check-target-b4x9q\" (UID: \"c95249a6-899f-4faf-82dc-6e553453ff80\") " pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:09.048259 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:09.048157 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/116af81a-d3a0-4558-93c4-8bf061b6815f-original-pull-secret\") pod \"global-pull-secret-syncer-wh9tj\" (UID: \"116af81a-d3a0-4558-93c4-8bf061b6815f\") " pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:09.048327 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:09.048274 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:36:09.048327 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:09.048284 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:36:09.048327 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:09.048307 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:36:09.048327 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:09.048323 2580 projected.go:196] Error preparing data for projected volume kube-api-access-lf5tn for pod openshift-network-diagnostics/network-check-target-b4x9q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:36:09.048539 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:09.048334 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/116af81a-d3a0-4558-93c4-8bf061b6815f-original-pull-secret podName:116af81a-d3a0-4558-93c4-8bf061b6815f nodeName:}" failed. No retries permitted until 2026-11-03 19:36:13.048315125 +0000 UTC m=+10.141444546 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/116af81a-d3a0-4558-93c4-8bf061b6815f-original-pull-secret") pod "global-pull-secret-syncer-wh9tj" (UID: "116af81a-d3a0-4558-93c4-8bf061b6815f") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:36:09.048539 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:09.048389 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c95249a6-899f-4faf-82dc-6e553453ff80-kube-api-access-lf5tn podName:c95249a6-899f-4faf-82dc-6e553453ff80 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:13.04837121 +0000 UTC m=+10.141500633 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lf5tn" (UniqueName: "kubernetes.io/projected/c95249a6-899f-4faf-82dc-6e553453ff80-kube-api-access-lf5tn") pod "network-check-target-b4x9q" (UID: "c95249a6-899f-4faf-82dc-6e553453ff80") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:36:09.471847 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:09.471810 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:09.472348 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:09.471962 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxjq" podUID="caf7d6c9-0bb2-4da0-be2d-caea7734bbcb" Nov 03 19:36:09.472431 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:09.472366 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:09.472495 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:09.472458 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4x9q" podUID="c95249a6-899f-4faf-82dc-6e553453ff80" Nov 03 19:36:09.660707 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:09.660660 2580 generic.go:356] "Generic (PLEG): container finished" podID="b6f0d931-3a92-424f-a5fc-d9ebef985dfd" containerID="8375618a0150cf854feba7f5b17ca368bd2306889a76ded11e3299f2f6eba684" exitCode=0 Nov 03 19:36:09.660900 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:09.660871 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxdrc" event={"ID":"b6f0d931-3a92-424f-a5fc-d9ebef985dfd","Type":"ContainerDied","Data":"8375618a0150cf854feba7f5b17ca368bd2306889a76ded11e3299f2f6eba684"} Nov 03 19:36:10.470884 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:10.470854 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:10.471049 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:10.470991 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wh9tj" podUID="116af81a-d3a0-4558-93c4-8bf061b6815f" Nov 03 19:36:10.668216 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:10.668176 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q4zkg" event={"ID":"4fdbd17a-1e15-4442-b715-aaec1087da11","Type":"ContainerStarted","Data":"ea58d8caa58e86ec3d0b9ec85b35c3fab2bfff534397c59fa2c7715e20cad2f2"} Nov 03 19:36:11.473900 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:11.473214 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:11.473900 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:11.473351 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxjq" podUID="caf7d6c9-0bb2-4da0-be2d-caea7734bbcb" Nov 03 19:36:11.473900 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:11.473720 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:11.473900 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:11.473805 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4x9q" podUID="c95249a6-899f-4faf-82dc-6e553453ff80" Nov 03 19:36:11.673379 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:11.673345 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kzr2l" event={"ID":"2398ba99-35d3-4461-b66e-f44ac013b2d2","Type":"ContainerStarted","Data":"92d86a0ec18b471bfd20d1c8cbf6c635ca73de9dfa3fa2018cf0bce7364d858d"} Nov 03 19:36:11.690455 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:11.690401 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-q4zkg" podStartSLOduration=3.453287652 podStartE2EDuration="7.690380392s" totalImagesPullingTime="4.23709274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:10.68698623 +0000 UTC m=+7.780115671" watchObservedRunningTime="2026-11-03 19:36:11.690380392 +0000 UTC m=+8.783509836" Nov 03 19:36:12.471347 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:12.471318 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:12.471498 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:12.471421 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wh9tj" podUID="116af81a-d3a0-4558-93c4-8bf061b6815f" Nov 03 19:36:12.678176 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:12.678048 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" event={"ID":"e64f9d9f-c868-470e-bd78-4ce8baabd97a","Type":"ContainerStarted","Data":"62f0f80745fa368e467df64f0bf792c4cd2e1b47d1fb3bd9b3fd60f4c55bd0fd"} Nov 03 19:36:12.979364 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:12.979328 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs\") pod \"network-metrics-daemon-qvxjq\" (UID: \"caf7d6c9-0bb2-4da0-be2d-caea7734bbcb\") " pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:12.979541 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:12.979466 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:36:12.979541 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:12.979524 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs podName:caf7d6c9-0bb2-4da0-be2d-caea7734bbcb nodeName:}" failed. No retries permitted until 2026-11-03 19:36:20.979505836 +0000 UTC m=+18.072635267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs") pod "network-metrics-daemon-qvxjq" (UID: "caf7d6c9-0bb2-4da0-be2d-caea7734bbcb") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:36:13.080256 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:13.080214 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/116af81a-d3a0-4558-93c4-8bf061b6815f-original-pull-secret\") pod \"global-pull-secret-syncer-wh9tj\" (UID: \"116af81a-d3a0-4558-93c4-8bf061b6815f\") " pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:13.080397 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:13.080276 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf5tn\" (UniqueName: \"kubernetes.io/projected/c95249a6-899f-4faf-82dc-6e553453ff80-kube-api-access-lf5tn\") pod \"network-check-target-b4x9q\" (UID: \"c95249a6-899f-4faf-82dc-6e553453ff80\") " pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:13.080506 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:13.080438 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:36:13.080506 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:13.080458 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:36:13.080506 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:13.080470 2580 projected.go:196] Error preparing data for projected volume kube-api-access-lf5tn for pod openshift-network-diagnostics/network-check-target-b4x9q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:36:13.080689 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:13.080523 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c95249a6-899f-4faf-82dc-6e553453ff80-kube-api-access-lf5tn podName:c95249a6-899f-4faf-82dc-6e553453ff80 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:21.080505825 +0000 UTC m=+18.173635258 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lf5tn" (UniqueName: "kubernetes.io/projected/c95249a6-899f-4faf-82dc-6e553453ff80-kube-api-access-lf5tn") pod "network-check-target-b4x9q" (UID: "c95249a6-899f-4faf-82dc-6e553453ff80") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:36:13.080947 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:13.080927 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:36:13.081019 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:13.080985 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/116af81a-d3a0-4558-93c4-8bf061b6815f-original-pull-secret podName:116af81a-d3a0-4558-93c4-8bf061b6815f nodeName:}" failed. No retries permitted until 2026-11-03 19:36:21.080969636 +0000 UTC m=+18.174099068 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/116af81a-d3a0-4558-93c4-8bf061b6815f-original-pull-secret") pod "global-pull-secret-syncer-wh9tj" (UID: "116af81a-d3a0-4558-93c4-8bf061b6815f") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:36:13.473060 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:13.473036 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:13.473204 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:13.473170 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxjq" podUID="caf7d6c9-0bb2-4da0-be2d-caea7734bbcb" Nov 03 19:36:13.473258 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:13.473243 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:13.473348 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:13.473319 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4x9q" podUID="c95249a6-899f-4faf-82dc-6e553453ff80" Nov 03 19:36:13.644303 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:13.642131 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 19:36:13.683735 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:13.683699 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" event={"ID":"e64f9d9f-c868-470e-bd78-4ce8baabd97a","Type":"ContainerStarted","Data":"6e3783f74fe364f15d528fea9fe7e47dcec97bcdb9fd686619dba72571e9d851"} Nov 03 19:36:14.471179 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:14.470717 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:14.471179 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:14.470829 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wh9tj" podUID="116af81a-d3a0-4558-93c4-8bf061b6815f" Nov 03 19:36:14.503072 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:14.502930 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:36:13.642150757Z","UUID":"e16bd93b-18ad-4faf-9e5e-eb404b64961b","Handler":null,"Name":"","Endpoint":""} Nov 03 19:36:14.504803 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:14.504783 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 19:36:14.504897 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:14.504814 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 19:36:14.690572 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:14.689606 2580 generic.go:356] "Generic (PLEG): container finished" podID="b6f0d931-3a92-424f-a5fc-d9ebef985dfd" containerID="b6d4d0233d7933e7962748fb0657527fb9f2f0f264d350c54c1b150e851dc498" exitCode=0 Nov 03 19:36:14.690572 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:14.689722 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxdrc" event={"ID":"b6f0d931-3a92-424f-a5fc-d9ebef985dfd","Type":"ContainerDied","Data":"b6d4d0233d7933e7962748fb0657527fb9f2f0f264d350c54c1b150e851dc498"} Nov 03 19:36:14.693389 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:14.693362 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4d4ll" event={"ID":"94d082d5-1c33-4394-9d93-67645f9b60cd","Type":"ContainerStarted","Data":"d18b8e43dbd50064522fc0e777d7c013a68f097ae690d66dedc8eb0abc9fb923"} Nov 03 19:36:14.695267 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:14.695246 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xlfb8" event={"ID":"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b","Type":"ContainerStarted","Data":"a83805f4560acaf55dc54ed13e3022f999b8d1a1edb4605520ee3dccbf3e32a1"} Nov 03 19:36:14.715494 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:14.715442 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kzr2l" podStartSLOduration=5.443063249 podStartE2EDuration="10.715426542s" totalImagesPullingTime="5.272363293s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:11.695892195 +0000 UTC m=+8.789021637" watchObservedRunningTime="2026-11-03 19:36:14.715426542 +0000 UTC m=+11.808555983" Nov 03 19:36:14.760890 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:14.760812 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-4d4ll" podStartSLOduration=2.444192639 podStartE2EDuration="10.760798031s" totalImagesPullingTime="8.316605392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:14.760792132 +0000 UTC m=+11.853921573" watchObservedRunningTime="2026-11-03 19:36:14.760798031 +0000 UTC m=+11.853927470" Nov 03 19:36:14.761005 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:14.760957 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-xlfb8" podStartSLOduration=2.740231264 podStartE2EDuration="10.760952182s" totalImagesPullingTime="8.020720918s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:14.743606337 +0000 UTC m=+11.836735793" watchObservedRunningTime="2026-11-03 19:36:14.760952182 +0000 UTC m=+11.854081622" Nov 03 19:36:15.471078 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:15.470850 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:15.471223 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:15.471057 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:15.471338 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:15.471270 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxjq" podUID="caf7d6c9-0bb2-4da0-be2d-caea7734bbcb" Nov 03 19:36:15.471410 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:15.471384 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4x9q" podUID="c95249a6-899f-4faf-82dc-6e553453ff80" Nov 03 19:36:15.595504 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:15.595465 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-q4zkg" Nov 03 19:36:15.595942 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:15.595920 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-q4zkg" Nov 03 19:36:15.699841 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:15.699733 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7zx7h" event={"ID":"55f124eb-2ebf-4675-a08a-bad3757061b3","Type":"ContainerStarted","Data":"35b98e30f3ea0019ecbab75b49225b08c138781e9ae6aec64fdeb42f74c72fc5"} Nov 03 19:36:15.702359 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:15.702331 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" event={"ID":"e64f9d9f-c868-470e-bd78-4ce8baabd97a","Type":"ContainerStarted","Data":"08d3bb693f951f36bc6efba78e2222f68815843c3f6226d21fa10ed3fc56ffb0"} Nov 03 19:36:15.705310 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:15.705284 2580 generic.go:356] "Generic (PLEG): container finished" podID="b6f0d931-3a92-424f-a5fc-d9ebef985dfd" containerID="dbaa1409db8655e6bf968118e5ebd222bdf95aacaf439e7ed5c4f593c69f7c20" exitCode=0 Nov 03 19:36:15.705431 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:15.705388 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxdrc" event={"ID":"b6f0d931-3a92-424f-a5fc-d9ebef985dfd","Type":"ContainerDied","Data":"dbaa1409db8655e6bf968118e5ebd222bdf95aacaf439e7ed5c4f593c69f7c20"} Nov 03 19:36:15.718208 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:15.718160 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7zx7h" podStartSLOduration=3.451654983 podStartE2EDuration="11.718143302s" totalImagesPullingTime="8.266488319s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:15.717587136 +0000 UTC m=+12.810716576" watchObservedRunningTime="2026-11-03 19:36:15.718143302 +0000 UTC m=+12.811272744" Nov 03 19:36:15.738541 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:15.738505 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" podStartSLOduration=2.806297272 podStartE2EDuration="11.738488995s" totalImagesPullingTime="8.932191723s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:15.738079397 +0000 UTC m=+12.831208838" watchObservedRunningTime="2026-11-03 19:36:15.738488995 +0000 UTC m=+12.831618434" Nov 03 19:36:16.191131 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:16.191112 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-q4zkg" Nov 03 19:36:16.191795 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:16.191767 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q4zkg" Nov 03 19:36:16.470763 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:16.470696 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:16.470919 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:16.470828 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wh9tj" podUID="116af81a-d3a0-4558-93c4-8bf061b6815f" Nov 03 19:36:16.709346 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:16.709310 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" event={"ID":"53b14f9a-431e-4dbb-a725-4c8470262ca7","Type":"ContainerStarted","Data":"21351f0f37e5a071fb3e95e9001996885abfec52b86208eb3f1a5dde84399ce8"} Nov 03 19:36:16.729897 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:16.729823 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" podStartSLOduration=2.203855427 podStartE2EDuration="12.729810211s" totalImagesPullingTime="10.525954784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:16.729613967 +0000 UTC m=+13.822743430" watchObservedRunningTime="2026-11-03 19:36:16.729810211 +0000 UTC m=+13.822939650" Nov 03 19:36:17.470902 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:17.470866 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:17.471064 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:17.471032 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4x9q" podUID="c95249a6-899f-4faf-82dc-6e553453ff80" Nov 03 19:36:17.471121 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:17.471066 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:17.471189 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:17.471165 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxjq" podUID="caf7d6c9-0bb2-4da0-be2d-caea7734bbcb" Nov 03 19:36:17.714460 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:17.714423 2580 generic.go:356] "Generic (PLEG): container finished" podID="b6f0d931-3a92-424f-a5fc-d9ebef985dfd" containerID="d350c3a6349a6db16ea21d5f7e3613e81cc603fc770eac0d77c7b52ea6723cb8" exitCode=0 Nov 03 19:36:17.718429 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:17.714470 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxdrc" event={"ID":"b6f0d931-3a92-424f-a5fc-d9ebef985dfd","Type":"ContainerDied","Data":"d350c3a6349a6db16ea21d5f7e3613e81cc603fc770eac0d77c7b52ea6723cb8"} Nov 03 19:36:18.471047 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:18.471010 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:18.471205 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:18.471141 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wh9tj" podUID="116af81a-d3a0-4558-93c4-8bf061b6815f" Nov 03 19:36:19.470761 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:19.470730 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:19.471308 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:19.470730 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:19.471308 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:19.470870 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxjq" podUID="caf7d6c9-0bb2-4da0-be2d-caea7734bbcb" Nov 03 19:36:19.471308 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:19.470943 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4x9q" podUID="c95249a6-899f-4faf-82dc-6e553453ff80" Nov 03 19:36:20.471142 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:20.470970 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:20.471500 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:20.471210 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wh9tj" podUID="116af81a-d3a0-4558-93c4-8bf061b6815f" Nov 03 19:36:20.723034 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:20.722967 2580 generic.go:356] "Generic (PLEG): container finished" podID="b6f0d931-3a92-424f-a5fc-d9ebef985dfd" containerID="cde5453097b00d29561b440f383684339fb83484f14f1515f139e905d88315ac" exitCode=0 Nov 03 19:36:20.723034 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:20.723021 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxdrc" event={"ID":"b6f0d931-3a92-424f-a5fc-d9ebef985dfd","Type":"ContainerDied","Data":"cde5453097b00d29561b440f383684339fb83484f14f1515f139e905d88315ac"} Nov 03 19:36:21.043189 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:21.043136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs\") pod \"network-metrics-daemon-qvxjq\" (UID: \"caf7d6c9-0bb2-4da0-be2d-caea7734bbcb\") " pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:21.043302 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:21.043285 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:36:21.043367 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:21.043356 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs podName:caf7d6c9-0bb2-4da0-be2d-caea7734bbcb nodeName:}" failed. No retries permitted until 2026-11-03 19:36:37.043336253 +0000 UTC m=+34.136465684 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs") pod "network-metrics-daemon-qvxjq" (UID: "caf7d6c9-0bb2-4da0-be2d-caea7734bbcb") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:36:21.143528 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:21.143502 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf5tn\" (UniqueName: \"kubernetes.io/projected/c95249a6-899f-4faf-82dc-6e553453ff80-kube-api-access-lf5tn\") pod \"network-check-target-b4x9q\" (UID: \"c95249a6-899f-4faf-82dc-6e553453ff80\") " pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:21.143633 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:21.143584 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/116af81a-d3a0-4558-93c4-8bf061b6815f-original-pull-secret\") pod \"global-pull-secret-syncer-wh9tj\" (UID: \"116af81a-d3a0-4558-93c4-8bf061b6815f\") " pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:21.143729 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:21.143656 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:36:21.143729 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:21.143691 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:36:21.143729 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:21.143696 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:36:21.143862 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:21.143736 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/116af81a-d3a0-4558-93c4-8bf061b6815f-original-pull-secret podName:116af81a-d3a0-4558-93c4-8bf061b6815f nodeName:}" failed. No retries permitted until 2026-11-03 19:36:37.143723956 +0000 UTC m=+34.236853388 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/116af81a-d3a0-4558-93c4-8bf061b6815f-original-pull-secret") pod "global-pull-secret-syncer-wh9tj" (UID: "116af81a-d3a0-4558-93c4-8bf061b6815f") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:36:21.143862 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:21.143700 2580 projected.go:196] Error preparing data for projected volume kube-api-access-lf5tn for pod openshift-network-diagnostics/network-check-target-b4x9q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:36:21.143862 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:21.143779 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c95249a6-899f-4faf-82dc-6e553453ff80-kube-api-access-lf5tn podName:c95249a6-899f-4faf-82dc-6e553453ff80 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:37.143772034 +0000 UTC m=+34.236901452 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lf5tn" (UniqueName: "kubernetes.io/projected/c95249a6-899f-4faf-82dc-6e553453ff80-kube-api-access-lf5tn") pod "network-check-target-b4x9q" (UID: "c95249a6-899f-4faf-82dc-6e553453ff80") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:36:21.471117 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:21.471084 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:21.471239 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:21.471194 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4x9q" podUID="c95249a6-899f-4faf-82dc-6e553453ff80" Nov 03 19:36:21.471525 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:21.471250 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:21.471525 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:21.471353 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxjq" podUID="caf7d6c9-0bb2-4da0-be2d-caea7734bbcb" Nov 03 19:36:21.727452 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:21.727427 2580 generic.go:356] "Generic (PLEG): container finished" podID="b6f0d931-3a92-424f-a5fc-d9ebef985dfd" containerID="62d86fa7bc9b39b63efd2667523c20f75bf119c83a70f7e97185c9173607c514" exitCode=0 Nov 03 19:36:21.727579 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:21.727501 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxdrc" event={"ID":"b6f0d931-3a92-424f-a5fc-d9ebef985dfd","Type":"ContainerDied","Data":"62d86fa7bc9b39b63efd2667523c20f75bf119c83a70f7e97185c9173607c514"} Nov 03 19:36:21.730635 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:21.730610 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" event={"ID":"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42","Type":"ContainerStarted","Data":"a77a444a232d8d2dc8a652fd5810e521b9802eced241368959d604ee8ef49ab2"} Nov 03 19:36:21.730730 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:21.730690 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" event={"ID":"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42","Type":"ContainerStarted","Data":"1c5b1e0c1289f742c2fadaad877f9a1770f694d0f463cf0df396d47820e49e0d"} Nov 03 19:36:21.730809 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:21.730736 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" event={"ID":"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42","Type":"ContainerStarted","Data":"1a880f93e3d2adac6de169936a9f932884920e8bf02e9f19bd4f18156c7c398b"} Nov 03 19:36:21.730809 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:21.730768 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" event={"ID":"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42","Type":"ContainerStarted","Data":"ca6a529e9297b64ca3651073a5c6e7816c2540919e1bcbf811f6605291b35b06"} Nov 03 19:36:21.730809 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:21.730795 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" event={"ID":"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42","Type":"ContainerStarted","Data":"e4440c85e922f8f402806b67bbf322c9f5830b79f6e7caf22a47fe29741af367"} Nov 03 19:36:21.730895 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:21.730821 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" event={"ID":"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42","Type":"ContainerStarted","Data":"612b1a9f2fc2ecedf082b7698641ba56776fbeb52d5b40e7057861a0287708fb"} Nov 03 19:36:22.471384 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:22.471221 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:22.471761 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:22.471456 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wh9tj" podUID="116af81a-d3a0-4558-93c4-8bf061b6815f" Nov 03 19:36:22.735830 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:22.735759 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxdrc" event={"ID":"b6f0d931-3a92-424f-a5fc-d9ebef985dfd","Type":"ContainerStarted","Data":"ed3724bf6ce7956783a2b2c7302d9d985e0bb94d1d88fc170e6c3dd0f6914d0f"} Nov 03 19:36:22.762265 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:22.762215 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cxdrc" podStartSLOduration=7.253177817 podStartE2EDuration="18.762204212s" totalImagesPullingTime="11.509026395s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:22.760738832 +0000 UTC m=+19.853868271" watchObservedRunningTime="2026-11-03 19:36:22.762204212 +0000 UTC m=+19.855333652" Nov 03 19:36:23.472285 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:23.472255 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:23.472641 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:23.472379 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxjq" podUID="caf7d6c9-0bb2-4da0-be2d-caea7734bbcb" Nov 03 19:36:23.472641 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:23.472422 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:23.472641 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:23.472523 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4x9q" podUID="c95249a6-899f-4faf-82dc-6e553453ff80" Nov 03 19:36:23.740884 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:23.740806 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" event={"ID":"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42","Type":"ContainerStarted","Data":"7439b118826ebd6433844803c6dff1facbb835c546f038e9fa7ef576b6a51ea1"} Nov 03 19:36:24.470573 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:24.470547 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:24.470732 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:24.470633 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wh9tj" podUID="116af81a-d3a0-4558-93c4-8bf061b6815f" Nov 03 19:36:25.470789 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:25.470764 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:25.471238 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:25.470800 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:25.471238 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:25.470906 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxjq" podUID="caf7d6c9-0bb2-4da0-be2d-caea7734bbcb" Nov 03 19:36:25.471238 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:25.470973 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4x9q" podUID="c95249a6-899f-4faf-82dc-6e553453ff80" Nov 03 19:36:25.750326 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:25.750246 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" event={"ID":"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42","Type":"ContainerStarted","Data":"df8ae2eeddb6b0121a32b7b240612798a51c02fa64a2245fde58f0b85f3449e5"} Nov 03 19:36:25.750602 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:25.750588 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:25.750700 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:25.750641 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:25.765513 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:25.765491 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:25.785096 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:25.785062 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" podStartSLOduration=6.410369541 podStartE2EDuration="21.785049837s" totalImagesPullingTime="15.374680296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:25.78496909 +0000 UTC m=+22.878098530" watchObservedRunningTime="2026-11-03 19:36:25.785049837 +0000 UTC m=+22.878179267" Nov 03 19:36:26.471240 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:26.471206 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:26.472053 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:26.471324 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wh9tj" podUID="116af81a-d3a0-4558-93c4-8bf061b6815f" Nov 03 19:36:26.753207 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:26.753130 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:26.772880 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:26.772839 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:27.226249 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:27.226216 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b4x9q"] Nov 03 19:36:27.226580 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:27.226377 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:27.226580 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:27.226506 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4x9q" podUID="c95249a6-899f-4faf-82dc-6e553453ff80" Nov 03 19:36:27.227080 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:27.227050 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wh9tj"] Nov 03 19:36:27.227209 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:27.227151 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:27.227267 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:27.227250 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wh9tj" podUID="116af81a-d3a0-4558-93c4-8bf061b6815f" Nov 03 19:36:27.229004 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:27.228973 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qvxjq"] Nov 03 19:36:27.229086 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:27.229076 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:27.229204 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:27.229184 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxjq" podUID="caf7d6c9-0bb2-4da0-be2d-caea7734bbcb" Nov 03 19:36:28.470729 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:28.470698 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:28.470729 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:28.470736 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:28.471129 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:28.470816 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxjq" podUID="caf7d6c9-0bb2-4da0-be2d-caea7734bbcb" Nov 03 19:36:28.471129 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:28.470952 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wh9tj" podUID="116af81a-d3a0-4558-93c4-8bf061b6815f" Nov 03 19:36:29.470512 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:29.470485 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:29.470643 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:29.470605 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4x9q" podUID="c95249a6-899f-4faf-82dc-6e553453ff80" Nov 03 19:36:30.470988 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:30.470743 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:30.470988 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:30.470817 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:30.471474 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:30.470997 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wh9tj" podUID="116af81a-d3a0-4558-93c4-8bf061b6815f" Nov 03 19:36:30.471474 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:30.471055 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxjq" podUID="caf7d6c9-0bb2-4da0-be2d-caea7734bbcb" Nov 03 19:36:31.471228 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:31.471203 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:31.471547 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:31.471312 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b4x9q" podUID="c95249a6-899f-4faf-82dc-6e553453ff80" Nov 03 19:36:32.470744 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:32.470719 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:32.470906 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:32.470719 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:32.470906 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:32.470812 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvxjq" podUID="caf7d6c9-0bb2-4da0-be2d-caea7734bbcb" Nov 03 19:36:32.470906 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:32.470894 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wh9tj" podUID="116af81a-d3a0-4558-93c4-8bf061b6815f" Nov 03 19:36:33.166673 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.166645 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeReady" Nov 03 19:36:33.167013 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.166760 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 19:36:33.199957 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.199928 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bd8f8"] Nov 03 19:36:33.262138 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.262105 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jt54m"] Nov 03 19:36:33.262283 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.262199 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bd8f8" Nov 03 19:36:33.277429 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.277310 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 19:36:33.277834 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.277811 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6hx8t"] Nov 03 19:36:33.278731 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.277902 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jt54m" Nov 03 19:36:33.280530 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.280512 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 19:36:33.285457 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.285391 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b"] Nov 03 19:36:33.285564 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.285466 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6hx8t" Nov 03 19:36:33.298038 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.298017 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wdls7"] Nov 03 19:36:33.298346 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.298328 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:33.303073 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.303044 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 19:36:33.303073 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.303044 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rwt79\"" Nov 03 19:36:33.319521 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.319503 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-f7b87d695-cwlfq"] Nov 03 19:36:33.319622 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.319607 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:33.333777 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.333760 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fsbsp"] Nov 03 19:36:33.333894 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.333880 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:33.347983 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.347965 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x"] Nov 03 19:36:33.348086 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.348070 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fsbsp" Nov 03 19:36:33.368172 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.368150 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx"] Nov 03 19:36:33.368352 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.368337 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" Nov 03 19:36:33.375363 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.375323 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 19:36:33.375363 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.375332 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 19:36:33.375363 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.375353 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 19:36:33.375580 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.375386 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 19:36:33.375580 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.375394 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:36:33.375580 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.375415 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 19:36:33.375580 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.375451 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 19:36:33.375580 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.375459 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 19:36:33.375580 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.375526 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:36:33.375580 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.375536 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 19:36:33.375580 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.375572 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 19:36:33.375906 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.375588 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 19:36:33.375906 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.375603 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 19:36:33.378638 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.378608 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 19:36:33.381944 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.381924 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 19:36:33.383256 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.383239 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-hmc5z"] Nov 03 19:36:33.383365 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.383351 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:33.401736 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.401713 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl"] Nov 03 19:36:33.401831 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.401815 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:33.403587 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.403571 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 19:36:33.403587 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.403579 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 19:36:33.403587 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.403587 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 19:36:33.403587 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.403596 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 19:36:33.403820 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.403618 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 19:36:33.403820 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.403655 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 19:36:33.403820 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.403702 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2r65v\"" Nov 03 19:36:33.403820 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.403767 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-p6m7s\"" Nov 03 19:36:33.403820 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.403791 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xr7jf\"" Nov 03 19:36:33.405036 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.405020 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 19:36:33.405442 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.405428 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fw8qp\"" Nov 03 19:36:33.415913 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.415895 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6dd465cd78-4n8kx"] Nov 03 19:36:33.416008 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.415996 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" Nov 03 19:36:33.426502 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.426485 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-799ffcd7c9-5pdd7"] Nov 03 19:36:33.426595 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.426585 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:33.439341 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.439322 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40834bff-2db0-4690-9cb9-7b4aacdab3ab-tmp\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:33.439422 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.439360 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-h9f7b\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:33.439422 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.439411 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c5zw\" (UniqueName: \"kubernetes.io/projected/f9d935b2-aa51-474c-836d-a74f91efab07-kube-api-access-7c5zw\") pod \"kas-connection-checker-685895c7c9-6hx8t\" (UID: \"f9d935b2-aa51-474c-836d-a74f91efab07\") " pod="kube-system/kas-connection-checker-685895c7c9-6hx8t" Nov 03 19:36:33.439520 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.439463 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jclj5\" (UniqueName: \"kubernetes.io/projected/40834bff-2db0-4690-9cb9-7b4aacdab3ab-kube-api-access-jclj5\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:33.439520 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.439486 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-stats-auth\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:33.439619 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.439545 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40834bff-2db0-4690-9cb9-7b4aacdab3ab-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:33.439619 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.439592 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40834bff-2db0-4690-9cb9-7b4aacdab3ab-service-ca-bundle\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:33.439740 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.439621 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25cd8\" (UniqueName: \"kubernetes.io/projected/0fc518fc-6302-4a1a-9ff9-f1e036a2199b-kube-api-access-25cd8\") pod \"kas-connection-checker-685895c7c9-bd8f8\" (UID: \"0fc518fc-6302-4a1a-9ff9-f1e036a2199b\") " pod="kube-system/kas-connection-checker-685895c7c9-bd8f8" Nov 03 19:36:33.439740 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.439649 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/40834bff-2db0-4690-9cb9-7b4aacdab3ab-serving-cert\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:33.439740 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.439699 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-default-certificate\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:33.439740 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.439728 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjhsn\" (UniqueName: \"kubernetes.io/projected/e2155512-2116-44f6-be5e-77cef72f9e14-kube-api-access-cjhsn\") pod \"kas-connection-checker-685895c7c9-jt54m\" (UID: \"e2155512-2116-44f6-be5e-77cef72f9e14\") " pod="kube-system/kas-connection-checker-685895c7c9-jt54m" Nov 03 19:36:33.439889 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.439789 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8d0db609-b271-4c98-a303-fb3ee71f152c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-h9f7b\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:33.439889 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.439809 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:33.439889 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.439825 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkxhk\" (UniqueName: \"kubernetes.io/projected/8d0db609-b271-4c98-a303-fb3ee71f152c-kube-api-access-mkxhk\") pod \"cluster-monitoring-operator-884576c57-h9f7b\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:33.439889 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.439843 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89g8g\" (UniqueName: \"kubernetes.io/projected/603868b0-8628-4227-8471-1894e2e1de50-kube-api-access-89g8g\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:33.439889 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.439866 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wvnz\" (UniqueName: \"kubernetes.io/projected/1bcd4690-4753-4246-abb4-8668a0a56231-kube-api-access-8wvnz\") pod \"volume-data-source-validator-58c9f5f46d-fsbsp\" (UID: \"1bcd4690-4753-4246-abb4-8668a0a56231\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fsbsp" Nov 03 19:36:33.440053 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.439893 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/40834bff-2db0-4690-9cb9-7b4aacdab3ab-snapshots\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:33.440053 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.439919 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-metrics-certs\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:33.444791 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.444776 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-846mt"] Nov 03 19:36:33.444880 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.444869 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:33.469425 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.469401 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-tvb96"] Nov 03 19:36:33.469538 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.469521 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" Nov 03 19:36:33.475309 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.475288 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:36:33.475377 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.475288 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 19:36:33.475377 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.475289 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 19:36:33.475377 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.475341 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 19:36:33.475517 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.475438 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:36:33.475517 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.475449 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 19:36:33.475517 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.475469 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 19:36:33.475517 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.475505 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 19:36:33.475636 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.475521 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:36:33.475636 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.475545 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 19:36:33.482288 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.482271 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 19:36:33.482368 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.482354 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 19:36:33.491134 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.491121 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bd8f8"] Nov 03 19:36:33.491238 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.491222 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tvb96" Nov 03 19:36:33.491301 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.491238 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:33.494533 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.494515 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jt54m"] Nov 03 19:36:33.494613 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.494547 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b"] Nov 03 19:36:33.494613 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.494565 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fsbsp"] Nov 03 19:36:33.494613 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.494586 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx"] Nov 03 19:36:33.494613 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.494604 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nhf67"] Nov 03 19:36:33.503712 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.503692 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 19:36:33.503797 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.503713 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 19:36:33.503797 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.503721 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bfvcm\"" Nov 03 19:36:33.503797 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.503718 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vf7n8\"" Nov 03 19:36:33.503797 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.503735 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 19:36:33.503797 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.503748 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 19:36:33.503947 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.503807 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5qm8x\"" Nov 03 19:36:33.503947 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.503812 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-dkxw8\"" Nov 03 19:36:33.503947 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.503860 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 19:36:33.512581 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.512565 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-f7b87d695-cwlfq"] Nov 03 19:36:33.512653 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.512589 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl"] Nov 03 19:36:33.512653 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.512605 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fgdh8"] Nov 03 19:36:33.512653 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.512622 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:33.523654 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.523640 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dd465cd78-4n8kx"] Nov 03 19:36:33.523731 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.523683 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6hx8t"] Nov 03 19:36:33.523731 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.523706 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-799ffcd7c9-5pdd7"] Nov 03 19:36:33.523731 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.523724 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x"] Nov 03 19:36:33.523731 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.523728 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fgdh8" Nov 03 19:36:33.523858 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.523743 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-tvb96"] Nov 03 19:36:33.523858 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.523765 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wdls7"] Nov 03 19:36:33.523858 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.523796 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-846mt"] Nov 03 19:36:33.523858 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.523818 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-hmc5z"] Nov 03 19:36:33.523858 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.523838 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fgdh8"] Nov 03 19:36:33.523858 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.523857 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nhf67"] Nov 03 19:36:33.540192 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.540169 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89g8g\" (UniqueName: \"kubernetes.io/projected/603868b0-8628-4227-8471-1894e2e1de50-kube-api-access-89g8g\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:33.540285 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.540198 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c99b0c19-004e-450c-852a-e38338635b61-serving-cert\") pod \"service-ca-operator-544d749c8c-xzpgl\" (UID: \"c99b0c19-004e-450c-852a-e38338635b61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" Nov 03 19:36:33.540285 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.540220 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wvnz\" (UniqueName: \"kubernetes.io/projected/1bcd4690-4753-4246-abb4-8668a0a56231-kube-api-access-8wvnz\") pod \"volume-data-source-validator-58c9f5f46d-fsbsp\" (UID: \"1bcd4690-4753-4246-abb4-8668a0a56231\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fsbsp" Nov 03 19:36:33.540285 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.540239 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-bound-sa-token\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:33.540285 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.540254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-trusted-ca\") pod \"console-operator-7cd56b8dbc-hmc5z\" (UID: \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:33.540477 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.540317 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-h9f7b\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:33.540477 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:33.540405 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 19:36:33.540477 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.540404 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jclj5\" (UniqueName: \"kubernetes.io/projected/40834bff-2db0-4690-9cb9-7b4aacdab3ab-kube-api-access-jclj5\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:33.540477 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.540439 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-stats-auth\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:33.540477 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.540466 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c5zw\" (UniqueName: \"kubernetes.io/projected/f9d935b2-aa51-474c-836d-a74f91efab07-kube-api-access-7c5zw\") pod \"kas-connection-checker-685895c7c9-6hx8t\" (UID: \"f9d935b2-aa51-474c-836d-a74f91efab07\") " pod="kube-system/kas-connection-checker-685895c7c9-6hx8t" Nov 03 19:36:33.540733 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:33.540507 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls podName:8d0db609-b271-4c98-a303-fb3ee71f152c nodeName:}" failed. No retries permitted until 2026-11-03 19:36:34.04048354 +0000 UTC m=+31.133612979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-h9f7b" (UID: "8d0db609-b271-4c98-a303-fb3ee71f152c") : secret "cluster-monitoring-operator-tls" not found Nov 03 19:36:33.540733 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.540534 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-bound-sa-token\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:33.540733 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.540577 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-certificates\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:33.540733 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.540623 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b989ba1e-71f9-424b-90f1-d1450f63e8b1-image-registry-private-configuration\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:33.540733 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.540651 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32065295-c6a2-44ea-88e8-c4fdc5a84083-ca-trust-extracted\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:33.540733 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.540708 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/492c7b4e-84c0-49a7-b12f-153a021f971b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wh95x\" (UID: \"492c7b4e-84c0-49a7-b12f-153a021f971b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" Nov 03 19:36:33.541011 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.540755 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc5qt\" (UniqueName: \"kubernetes.io/projected/40b2621f-854e-44e0-9b01-c877e333f644-kube-api-access-jc5qt\") pod \"cluster-samples-operator-6ddb6c8b94-86qxx\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:33.541011 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.540786 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25cd8\" (UniqueName: \"kubernetes.io/projected/0fc518fc-6302-4a1a-9ff9-f1e036a2199b-kube-api-access-25cd8\") pod \"kas-connection-checker-685895c7c9-bd8f8\" (UID: \"0fc518fc-6302-4a1a-9ff9-f1e036a2199b\") " pod="kube-system/kas-connection-checker-685895c7c9-bd8f8" Nov 03 19:36:33.541011 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.540818 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28xz4\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-kube-api-access-28xz4\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:33.541011 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.540890 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/40834bff-2db0-4690-9cb9-7b4aacdab3ab-serving-cert\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:33.541011 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.540941 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8d0db609-b271-4c98-a303-fb3ee71f152c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-h9f7b\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:33.541011 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.540967 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkxhk\" (UniqueName: \"kubernetes.io/projected/8d0db609-b271-4c98-a303-fb3ee71f152c-kube-api-access-mkxhk\") pod \"cluster-monitoring-operator-884576c57-h9f7b\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:33.541011 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.540995 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b989ba1e-71f9-424b-90f1-d1450f63e8b1-trusted-ca\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:33.541264 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.541023 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv87l\" (UniqueName: \"kubernetes.io/projected/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-kube-api-access-tv87l\") pod \"console-operator-7cd56b8dbc-hmc5z\" (UID: \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:33.541264 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.541048 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c99b0c19-004e-450c-852a-e38338635b61-config\") pod \"service-ca-operator-544d749c8c-xzpgl\" (UID: \"c99b0c19-004e-450c-852a-e38338635b61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" Nov 03 19:36:33.541264 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.541080 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:33.541264 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.541112 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-certificates\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:33.541264 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.541142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/40834bff-2db0-4690-9cb9-7b4aacdab3ab-snapshots\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:33.541264 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.541186 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-metrics-certs\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:33.541264 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.541230 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-serving-cert\") pod \"console-operator-7cd56b8dbc-hmc5z\" (UID: \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:33.541551 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:33.541302 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 19:36:33.541551 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:33.541349 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-metrics-certs podName:603868b0-8628-4227-8471-1894e2e1de50 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:34.041336677 +0000 UTC m=+31.134466099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-metrics-certs") pod "router-default-f7b87d695-cwlfq" (UID: "603868b0-8628-4227-8471-1894e2e1de50") : secret "router-metrics-certs-default" not found Nov 03 19:36:33.541551 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.541378 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-tls\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:33.541551 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.541405 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/492c7b4e-84c0-49a7-b12f-153a021f971b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wh95x\" (UID: \"492c7b4e-84c0-49a7-b12f-153a021f971b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" Nov 03 19:36:33.541551 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.541430 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40834bff-2db0-4690-9cb9-7b4aacdab3ab-tmp\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:33.541551 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.541488 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-86qxx\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:33.541551 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.541520 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32065295-c6a2-44ea-88e8-c4fdc5a84083-trusted-ca\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:33.541551 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.541543 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8d0db609-b271-4c98-a303-fb3ee71f152c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-h9f7b\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:33.541939 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.541546 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b989ba1e-71f9-424b-90f1-d1450f63e8b1-installation-pull-secrets\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:33.541939 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.541678 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbrbm\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-kube-api-access-kbrbm\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:33.541939 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.541702 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk5v5\" (UniqueName: \"kubernetes.io/projected/c99b0c19-004e-450c-852a-e38338635b61-kube-api-access-xk5v5\") pod \"service-ca-operator-544d749c8c-xzpgl\" (UID: \"c99b0c19-004e-450c-852a-e38338635b61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" Nov 03 19:36:33.541939 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.541751 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/40834bff-2db0-4690-9cb9-7b4aacdab3ab-snapshots\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:33.541939 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:33.541776 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle podName:603868b0-8628-4227-8471-1894e2e1de50 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:34.041763311 +0000 UTC m=+31.134892752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle") pod "router-default-f7b87d695-cwlfq" (UID: "603868b0-8628-4227-8471-1894e2e1de50") : configmap references non-existent config key: service-ca.crt Nov 03 19:36:33.541939 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.541804 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40834bff-2db0-4690-9cb9-7b4aacdab3ab-tmp\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:33.541939 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.541838 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40834bff-2db0-4690-9cb9-7b4aacdab3ab-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:33.541939 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.541870 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40834bff-2db0-4690-9cb9-7b4aacdab3ab-service-ca-bundle\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:33.542301 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.542080 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-86qxx\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:33.542301 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.542134 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32065295-c6a2-44ea-88e8-c4fdc5a84083-image-registry-private-configuration\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:33.542301 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.542164 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b989ba1e-71f9-424b-90f1-d1450f63e8b1-ca-trust-extracted\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:33.542301 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.542194 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-tls\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:33.542301 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.542264 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-default-certificate\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:33.542530 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.542308 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjhsn\" (UniqueName: \"kubernetes.io/projected/e2155512-2116-44f6-be5e-77cef72f9e14-kube-api-access-cjhsn\") pod \"kas-connection-checker-685895c7c9-jt54m\" (UID: \"e2155512-2116-44f6-be5e-77cef72f9e14\") " pod="kube-system/kas-connection-checker-685895c7c9-jt54m" Nov 03 19:36:33.542530 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.542351 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32065295-c6a2-44ea-88e8-c4fdc5a84083-installation-pull-secrets\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:33.542530 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.542385 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40834bff-2db0-4690-9cb9-7b4aacdab3ab-service-ca-bundle\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:33.542530 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.542388 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgtxr\" (UniqueName: \"kubernetes.io/projected/492c7b4e-84c0-49a7-b12f-153a021f971b-kube-api-access-hgtxr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wh95x\" (UID: \"492c7b4e-84c0-49a7-b12f-153a021f971b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" Nov 03 19:36:33.542530 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.542493 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40834bff-2db0-4690-9cb9-7b4aacdab3ab-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:33.542530 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.542512 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-config\") pod \"console-operator-7cd56b8dbc-hmc5z\" (UID: \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:33.544398 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.544369 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/40834bff-2db0-4690-9cb9-7b4aacdab3ab-serving-cert\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:33.544471 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.544374 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-stats-auth\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:33.544642 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.544624 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-default-certificate\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:33.554032 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.553602 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c5zw\" (UniqueName: \"kubernetes.io/projected/f9d935b2-aa51-474c-836d-a74f91efab07-kube-api-access-7c5zw\") pod \"kas-connection-checker-685895c7c9-6hx8t\" (UID: \"f9d935b2-aa51-474c-836d-a74f91efab07\") " pod="kube-system/kas-connection-checker-685895c7c9-6hx8t" Nov 03 19:36:33.554867 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.554841 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wvnz\" (UniqueName: \"kubernetes.io/projected/1bcd4690-4753-4246-abb4-8668a0a56231-kube-api-access-8wvnz\") pod \"volume-data-source-validator-58c9f5f46d-fsbsp\" (UID: \"1bcd4690-4753-4246-abb4-8668a0a56231\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fsbsp" Nov 03 19:36:33.555143 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.555118 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jclj5\" (UniqueName: \"kubernetes.io/projected/40834bff-2db0-4690-9cb9-7b4aacdab3ab-kube-api-access-jclj5\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:33.556543 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.556496 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkxhk\" (UniqueName: \"kubernetes.io/projected/8d0db609-b271-4c98-a303-fb3ee71f152c-kube-api-access-mkxhk\") pod \"cluster-monitoring-operator-884576c57-h9f7b\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:33.556650 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.556624 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25cd8\" (UniqueName: \"kubernetes.io/projected/0fc518fc-6302-4a1a-9ff9-f1e036a2199b-kube-api-access-25cd8\") pod \"kas-connection-checker-685895c7c9-bd8f8\" (UID: \"0fc518fc-6302-4a1a-9ff9-f1e036a2199b\") " pod="kube-system/kas-connection-checker-685895c7c9-bd8f8" Nov 03 19:36:33.556776 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.556755 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjhsn\" (UniqueName: \"kubernetes.io/projected/e2155512-2116-44f6-be5e-77cef72f9e14-kube-api-access-cjhsn\") pod \"kas-connection-checker-685895c7c9-jt54m\" (UID: \"e2155512-2116-44f6-be5e-77cef72f9e14\") " pod="kube-system/kas-connection-checker-685895c7c9-jt54m" Nov 03 19:36:33.557543 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.557524 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89g8g\" (UniqueName: \"kubernetes.io/projected/603868b0-8628-4227-8471-1894e2e1de50-kube-api-access-89g8g\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:33.575411 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.575394 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 19:36:33.575411 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.575401 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 19:36:33.575536 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.575401 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 19:36:33.575536 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.575442 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 19:36:33.575536 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.575497 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 19:36:33.585112 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.585093 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bd8f8" Nov 03 19:36:33.604176 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.604145 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 19:36:33.604315 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.604300 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qg9jj\"" Nov 03 19:36:33.604394 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.604337 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 19:36:33.604394 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.604363 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dgjzc\"" Nov 03 19:36:33.610495 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.610479 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jt54m" Nov 03 19:36:33.645231 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.643860 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-serving-cert\") pod \"console-operator-7cd56b8dbc-hmc5z\" (UID: \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:33.645231 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.643889 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-tls\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:33.645231 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.643912 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/492c7b4e-84c0-49a7-b12f-153a021f971b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wh95x\" (UID: \"492c7b4e-84c0-49a7-b12f-153a021f971b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" Nov 03 19:36:33.645231 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:33.644015 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:36:33.645231 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:33.644031 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd465cd78-4n8kx: secret "image-registry-tls" not found Nov 03 19:36:33.645231 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:33.644085 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-tls podName:b989ba1e-71f9-424b-90f1-d1450f63e8b1 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:34.14406451 +0000 UTC m=+31.237193950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-tls") pod "image-registry-6dd465cd78-4n8kx" (UID: "b989ba1e-71f9-424b-90f1-d1450f63e8b1") : secret "image-registry-tls" not found Nov 03 19:36:33.645231 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.644295 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-86qxx\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:33.645231 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.644360 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32065295-c6a2-44ea-88e8-c4fdc5a84083-trusted-ca\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:33.645231 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.644414 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b989ba1e-71f9-424b-90f1-d1450f63e8b1-installation-pull-secrets\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:33.645231 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.644462 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbrbm\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-kube-api-access-kbrbm\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:33.645231 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.644497 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk5v5\" (UniqueName: \"kubernetes.io/projected/c99b0c19-004e-450c-852a-e38338635b61-kube-api-access-xk5v5\") pod \"service-ca-operator-544d749c8c-xzpgl\" (UID: \"c99b0c19-004e-450c-852a-e38338635b61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" Nov 03 19:36:33.645231 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.644552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-86qxx\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:33.645231 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.644577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32065295-c6a2-44ea-88e8-c4fdc5a84083-image-registry-private-configuration\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:33.645231 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.644633 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/04a982f9-143c-406c-99d5-9ef73d5c571a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-846mt\" (UID: \"04a982f9-143c-406c-99d5-9ef73d5c571a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" Nov 03 19:36:33.645231 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.644685 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b989ba1e-71f9-424b-90f1-d1450f63e8b1-ca-trust-extracted\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:33.645231 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.644713 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-tls\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:33.646066 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.644764 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32065295-c6a2-44ea-88e8-c4fdc5a84083-installation-pull-secrets\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:33.646066 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.644791 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgtxr\" (UniqueName: \"kubernetes.io/projected/492c7b4e-84c0-49a7-b12f-153a021f971b-kube-api-access-hgtxr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wh95x\" (UID: \"492c7b4e-84c0-49a7-b12f-153a021f971b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" Nov 03 19:36:33.646066 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.644819 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-config\") pod \"console-operator-7cd56b8dbc-hmc5z\" (UID: \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:33.646066 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.644871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c99b0c19-004e-450c-852a-e38338635b61-serving-cert\") pod \"service-ca-operator-544d749c8c-xzpgl\" (UID: \"c99b0c19-004e-450c-852a-e38338635b61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" Nov 03 19:36:33.646066 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.644929 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drmpl\" (UniqueName: \"kubernetes.io/projected/18ffe451-b8eb-41ad-a19d-0390f3e1f8e2-kube-api-access-drmpl\") pod \"ingress-canary-fgdh8\" (UID: \"18ffe451-b8eb-41ad-a19d-0390f3e1f8e2\") " pod="openshift-ingress-canary/ingress-canary-fgdh8" Nov 03 19:36:33.646066 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.644967 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-bound-sa-token\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:33.646066 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.645019 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-trusted-ca\") pod \"console-operator-7cd56b8dbc-hmc5z\" (UID: \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:33.646066 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.645017 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-86qxx\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:33.646066 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.645059 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:33.646066 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.645114 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-846mt\" (UID: \"04a982f9-143c-406c-99d5-9ef73d5c571a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" Nov 03 19:36:33.646066 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.645163 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-bound-sa-token\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:33.646066 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.645204 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-certificates\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:33.646066 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.645251 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b989ba1e-71f9-424b-90f1-d1450f63e8b1-image-registry-private-configuration\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:33.646066 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.645276 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32065295-c6a2-44ea-88e8-c4fdc5a84083-ca-trust-extracted\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:33.646066 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.645309 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b989ba1e-71f9-424b-90f1-d1450f63e8b1-ca-trust-extracted\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:33.646769 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.645323 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/492c7b4e-84c0-49a7-b12f-153a021f971b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wh95x\" (UID: \"492c7b4e-84c0-49a7-b12f-153a021f971b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" Nov 03 19:36:33.646769 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.644416 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/492c7b4e-84c0-49a7-b12f-153a021f971b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wh95x\" (UID: \"492c7b4e-84c0-49a7-b12f-153a021f971b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" Nov 03 19:36:33.646769 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.645356 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc5qt\" (UniqueName: \"kubernetes.io/projected/40b2621f-854e-44e0-9b01-c877e333f644-kube-api-access-jc5qt\") pod \"cluster-samples-operator-6ddb6c8b94-86qxx\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:33.646769 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.645400 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tckv\" (UniqueName: \"kubernetes.io/projected/419841be-7a31-4cdb-b0b7-9d90a3c376c7-kube-api-access-9tckv\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:33.646769 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.645433 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28xz4\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-kube-api-access-28xz4\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:33.646769 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.645500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b989ba1e-71f9-424b-90f1-d1450f63e8b1-trusted-ca\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:33.646769 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.645530 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv87l\" (UniqueName: \"kubernetes.io/projected/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-kube-api-access-tv87l\") pod \"console-operator-7cd56b8dbc-hmc5z\" (UID: \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:33.646769 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.645584 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c99b0c19-004e-450c-852a-e38338635b61-config\") pod \"service-ca-operator-544d749c8c-xzpgl\" (UID: \"c99b0c19-004e-450c-852a-e38338635b61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" Nov 03 19:36:33.646769 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:33.645631 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 19:36:33.646769 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.645647 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18ffe451-b8eb-41ad-a19d-0390f3e1f8e2-cert\") pod \"ingress-canary-fgdh8\" (UID: \"18ffe451-b8eb-41ad-a19d-0390f3e1f8e2\") " pod="openshift-ingress-canary/ingress-canary-fgdh8" Nov 03 19:36:33.646769 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:33.645694 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls podName:40b2621f-854e-44e0-9b01-c877e333f644 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:34.145679406 +0000 UTC m=+31.238808838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-86qxx" (UID: "40b2621f-854e-44e0-9b01-c877e333f644") : secret "samples-operator-tls" not found Nov 03 19:36:33.646769 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.645768 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-certificates\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:33.646769 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.645802 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/419841be-7a31-4cdb-b0b7-9d90a3c376c7-config-volume\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:33.646769 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.645850 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/419841be-7a31-4cdb-b0b7-9d90a3c376c7-tmp-dir\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:33.646769 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.645895 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tvmx\" (UniqueName: \"kubernetes.io/projected/0a985b50-c4c0-4a39-abed-29f53e2be747-kube-api-access-8tvmx\") pod \"network-check-source-6f78d59f99-tvb96\" (UID: \"0a985b50-c4c0-4a39-abed-29f53e2be747\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tvb96" Nov 03 19:36:33.646769 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.645903 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-serving-cert\") pod \"console-operator-7cd56b8dbc-hmc5z\" (UID: \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:33.647501 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:33.645973 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:36:33.647501 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:33.645984 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799ffcd7c9-5pdd7: secret "image-registry-tls" not found Nov 03 19:36:33.647501 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.646940 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32065295-c6a2-44ea-88e8-c4fdc5a84083-ca-trust-extracted\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:33.647501 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:33.647012 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-tls podName:32065295-c6a2-44ea-88e8-c4fdc5a84083 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:34.146995759 +0000 UTC m=+31.240125191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-tls") pod "image-registry-799ffcd7c9-5pdd7" (UID: "32065295-c6a2-44ea-88e8-c4fdc5a84083") : secret "image-registry-tls" not found Nov 03 19:36:33.647501 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.647292 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c99b0c19-004e-450c-852a-e38338635b61-config\") pod \"service-ca-operator-544d749c8c-xzpgl\" (UID: \"c99b0c19-004e-450c-852a-e38338635b61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" Nov 03 19:36:33.647707 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.647618 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-trusted-ca\") pod \"console-operator-7cd56b8dbc-hmc5z\" (UID: \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:33.650476 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.648297 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-certificates\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:33.650476 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.648593 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-config\") pod \"console-operator-7cd56b8dbc-hmc5z\" (UID: \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:33.650476 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.648687 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-certificates\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:33.650476 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.648808 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b989ba1e-71f9-424b-90f1-d1450f63e8b1-installation-pull-secrets\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:33.650476 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.649027 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6hx8t" Nov 03 19:36:33.650476 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.649547 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b989ba1e-71f9-424b-90f1-d1450f63e8b1-trusted-ca\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:33.650476 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.649945 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b989ba1e-71f9-424b-90f1-d1450f63e8b1-image-registry-private-configuration\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:33.650476 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.650170 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32065295-c6a2-44ea-88e8-c4fdc5a84083-trusted-ca\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:33.650476 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.650447 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32065295-c6a2-44ea-88e8-c4fdc5a84083-image-registry-private-configuration\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:33.650972 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.650928 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c99b0c19-004e-450c-852a-e38338635b61-serving-cert\") pod \"service-ca-operator-544d749c8c-xzpgl\" (UID: \"c99b0c19-004e-450c-852a-e38338635b61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" Nov 03 19:36:33.650972 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.650939 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32065295-c6a2-44ea-88e8-c4fdc5a84083-installation-pull-secrets\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:33.651259 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.651237 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/492c7b4e-84c0-49a7-b12f-153a021f971b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wh95x\" (UID: \"492c7b4e-84c0-49a7-b12f-153a021f971b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" Nov 03 19:36:33.660978 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.660955 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-bound-sa-token\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:33.661529 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.661443 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc5qt\" (UniqueName: \"kubernetes.io/projected/40b2621f-854e-44e0-9b01-c877e333f644-kube-api-access-jc5qt\") pod \"cluster-samples-operator-6ddb6c8b94-86qxx\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:33.662938 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.662812 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv87l\" (UniqueName: \"kubernetes.io/projected/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-kube-api-access-tv87l\") pod \"console-operator-7cd56b8dbc-hmc5z\" (UID: \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:33.664809 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.663455 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk5v5\" (UniqueName: \"kubernetes.io/projected/c99b0c19-004e-450c-852a-e38338635b61-kube-api-access-xk5v5\") pod \"service-ca-operator-544d749c8c-xzpgl\" (UID: \"c99b0c19-004e-450c-852a-e38338635b61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" Nov 03 19:36:33.664809 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.663836 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgtxr\" (UniqueName: \"kubernetes.io/projected/492c7b4e-84c0-49a7-b12f-153a021f971b-kube-api-access-hgtxr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wh95x\" (UID: \"492c7b4e-84c0-49a7-b12f-153a021f971b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" Nov 03 19:36:33.664809 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.664209 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-bound-sa-token\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:33.664809 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.664769 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28xz4\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-kube-api-access-28xz4\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:33.666521 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.666480 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbrbm\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-kube-api-access-kbrbm\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:33.684102 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.684057 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:33.698313 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.698121 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fsbsp" Nov 03 19:36:33.730687 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.726992 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" Nov 03 19:36:33.740381 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.739889 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" Nov 03 19:36:33.747375 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.747348 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tckv\" (UniqueName: \"kubernetes.io/projected/419841be-7a31-4cdb-b0b7-9d90a3c376c7-kube-api-access-9tckv\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:33.749627 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.747593 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18ffe451-b8eb-41ad-a19d-0390f3e1f8e2-cert\") pod \"ingress-canary-fgdh8\" (UID: \"18ffe451-b8eb-41ad-a19d-0390f3e1f8e2\") " pod="openshift-ingress-canary/ingress-canary-fgdh8" Nov 03 19:36:33.749627 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.747634 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/419841be-7a31-4cdb-b0b7-9d90a3c376c7-config-volume\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:33.749627 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.747699 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/419841be-7a31-4cdb-b0b7-9d90a3c376c7-tmp-dir\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:33.749627 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.747738 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tvmx\" (UniqueName: \"kubernetes.io/projected/0a985b50-c4c0-4a39-abed-29f53e2be747-kube-api-access-8tvmx\") pod \"network-check-source-6f78d59f99-tvb96\" (UID: \"0a985b50-c4c0-4a39-abed-29f53e2be747\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tvb96" Nov 03 19:36:33.749627 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.747826 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/04a982f9-143c-406c-99d5-9ef73d5c571a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-846mt\" (UID: \"04a982f9-143c-406c-99d5-9ef73d5c571a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" Nov 03 19:36:33.749627 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.747881 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drmpl\" (UniqueName: \"kubernetes.io/projected/18ffe451-b8eb-41ad-a19d-0390f3e1f8e2-kube-api-access-drmpl\") pod \"ingress-canary-fgdh8\" (UID: \"18ffe451-b8eb-41ad-a19d-0390f3e1f8e2\") " pod="openshift-ingress-canary/ingress-canary-fgdh8" Nov 03 19:36:33.749627 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.747923 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:33.749627 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.747951 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-846mt\" (UID: \"04a982f9-143c-406c-99d5-9ef73d5c571a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" Nov 03 19:36:33.749627 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:33.748081 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 19:36:33.749627 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:33.748157 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert podName:04a982f9-143c-406c-99d5-9ef73d5c571a nodeName:}" failed. No retries permitted until 2026-11-03 19:36:34.248137139 +0000 UTC m=+31.341266579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-846mt" (UID: "04a982f9-143c-406c-99d5-9ef73d5c571a") : secret "networking-console-plugin-cert" not found Nov 03 19:36:33.749627 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.748306 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/419841be-7a31-4cdb-b0b7-9d90a3c376c7-config-volume\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:33.749627 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.748612 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/419841be-7a31-4cdb-b0b7-9d90a3c376c7-tmp-dir\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:33.749627 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:33.748793 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:36:33.749627 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:33.748837 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls podName:419841be-7a31-4cdb-b0b7-9d90a3c376c7 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:34.248823131 +0000 UTC m=+31.341952550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls") pod "dns-default-nhf67" (UID: "419841be-7a31-4cdb-b0b7-9d90a3c376c7") : secret "dns-default-metrics-tls" not found Nov 03 19:36:33.749627 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.749446 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/04a982f9-143c-406c-99d5-9ef73d5c571a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-846mt\" (UID: \"04a982f9-143c-406c-99d5-9ef73d5c571a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" Nov 03 19:36:33.753567 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.752399 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18ffe451-b8eb-41ad-a19d-0390f3e1f8e2-cert\") pod \"ingress-canary-fgdh8\" (UID: \"18ffe451-b8eb-41ad-a19d-0390f3e1f8e2\") " pod="openshift-ingress-canary/ingress-canary-fgdh8" Nov 03 19:36:33.763150 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.761956 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tckv\" (UniqueName: \"kubernetes.io/projected/419841be-7a31-4cdb-b0b7-9d90a3c376c7-kube-api-access-9tckv\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:33.763150 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.762304 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drmpl\" (UniqueName: \"kubernetes.io/projected/18ffe451-b8eb-41ad-a19d-0390f3e1f8e2-kube-api-access-drmpl\") pod \"ingress-canary-fgdh8\" (UID: \"18ffe451-b8eb-41ad-a19d-0390f3e1f8e2\") " pod="openshift-ingress-canary/ingress-canary-fgdh8" Nov 03 19:36:33.763150 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.762349 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tvmx\" (UniqueName: \"kubernetes.io/projected/0a985b50-c4c0-4a39-abed-29f53e2be747-kube-api-access-8tvmx\") pod \"network-check-source-6f78d59f99-tvb96\" (UID: \"0a985b50-c4c0-4a39-abed-29f53e2be747\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tvb96" Nov 03 19:36:33.763150 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.762412 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:33.835909 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.835693 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tvb96" Nov 03 19:36:33.880927 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.880900 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fgdh8" Nov 03 19:36:33.897338 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.897303 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bd8f8"] Nov 03 19:36:33.900534 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:33.900356 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc518fc_6302_4a1a_9ff9_f1e036a2199b.slice/crio-78e9081e9ae7d7fd90676dd54dbde8f3e82156ec5f28724f4b1e82150cb9f4ff WatchSource:0}: Error finding container 78e9081e9ae7d7fd90676dd54dbde8f3e82156ec5f28724f4b1e82150cb9f4ff: Status 404 returned error can't find the container with id 78e9081e9ae7d7fd90676dd54dbde8f3e82156ec5f28724f4b1e82150cb9f4ff Nov 03 19:36:33.919728 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.915554 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jt54m"] Nov 03 19:36:33.923338 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:33.923306 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2155512_2116_44f6_be5e_77cef72f9e14.slice/crio-e781524536bca1b272b961d24e61957bd19573364a9d4afb8efb7d5734c4b324 WatchSource:0}: Error finding container e781524536bca1b272b961d24e61957bd19573364a9d4afb8efb7d5734c4b324: Status 404 returned error can't find the container with id e781524536bca1b272b961d24e61957bd19573364a9d4afb8efb7d5734c4b324 Nov 03 19:36:33.955876 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.947806 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6hx8t"] Nov 03 19:36:33.981772 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:33.981738 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d935b2_aa51_474c_836d_a74f91efab07.slice/crio-5bc830e6d84d3cd29bae7a5df0255f8f86285729e706443279b9042876a37e1c WatchSource:0}: Error finding container 5bc830e6d84d3cd29bae7a5df0255f8f86285729e706443279b9042876a37e1c: Status 404 returned error can't find the container with id 5bc830e6d84d3cd29bae7a5df0255f8f86285729e706443279b9042876a37e1c Nov 03 19:36:33.994908 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.993564 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wdls7"] Nov 03 19:36:33.996753 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:33.996589 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fsbsp"] Nov 03 19:36:33.999614 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:33.999190 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40834bff_2db0_4690_9cb9_7b4aacdab3ab.slice/crio-32222fdc67b4f6469d917ab85ca77d14f3f5514ef515ff284628d7056c596a70 WatchSource:0}: Error finding container 32222fdc67b4f6469d917ab85ca77d14f3f5514ef515ff284628d7056c596a70: Status 404 returned error can't find the container with id 32222fdc67b4f6469d917ab85ca77d14f3f5514ef515ff284628d7056c596a70 Nov 03 19:36:34.002757 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:34.001984 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bcd4690_4753_4246_abb4_8668a0a56231.slice/crio-3c720fc599e07edb9f53c3258393229ef983d842fab015bf43f5f59b3342d40f WatchSource:0}: Error finding container 3c720fc599e07edb9f53c3258393229ef983d842fab015bf43f5f59b3342d40f: Status 404 returned error can't find the container with id 3c720fc599e07edb9f53c3258393229ef983d842fab015bf43f5f59b3342d40f Nov 03 19:36:34.041301 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.041242 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x"] Nov 03 19:36:34.050048 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.048540 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl"] Nov 03 19:36:34.069746 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.069725 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-h9f7b\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:34.069828 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.069796 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:34.069828 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.069821 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-metrics-certs\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:34.069937 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:34.069876 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 19:36:34.069937 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:34.069906 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 19:36:34.070028 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:34.069940 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle podName:603868b0-8628-4227-8471-1894e2e1de50 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:35.069921461 +0000 UTC m=+32.163050879 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle") pod "router-default-f7b87d695-cwlfq" (UID: "603868b0-8628-4227-8471-1894e2e1de50") : configmap references non-existent config key: service-ca.crt Nov 03 19:36:34.070028 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:34.069979 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-metrics-certs podName:603868b0-8628-4227-8471-1894e2e1de50 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:35.069954264 +0000 UTC m=+32.163083682 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-metrics-certs") pod "router-default-f7b87d695-cwlfq" (UID: "603868b0-8628-4227-8471-1894e2e1de50") : secret "router-metrics-certs-default" not found Nov 03 19:36:34.070028 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:34.069999 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls podName:8d0db609-b271-4c98-a303-fb3ee71f152c nodeName:}" failed. No retries permitted until 2026-11-03 19:36:35.06998957 +0000 UTC m=+32.163118988 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-h9f7b" (UID: "8d0db609-b271-4c98-a303-fb3ee71f152c") : secret "cluster-monitoring-operator-tls" not found Nov 03 19:36:34.078999 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.078953 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-hmc5z"] Nov 03 19:36:34.083849 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:34.082140 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b140c9_9e09_4cbd_847b_82ee1e3be6ac.slice/crio-38325d12703d972e1ae36cba1ea5a54367988c1d6b25ab7c5727d1b58e368726 WatchSource:0}: Error finding container 38325d12703d972e1ae36cba1ea5a54367988c1d6b25ab7c5727d1b58e368726: Status 404 returned error can't find the container with id 38325d12703d972e1ae36cba1ea5a54367988c1d6b25ab7c5727d1b58e368726 Nov 03 19:36:34.148731 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.148709 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-tvb96"] Nov 03 19:36:34.150547 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:34.150525 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a985b50_c4c0_4a39_abed_29f53e2be747.slice/crio-574c6906b17ca7382d033e3c1490a3c3ffef5f8642f16e2bde9dbd6349e080a2 WatchSource:0}: Error finding container 574c6906b17ca7382d033e3c1490a3c3ffef5f8642f16e2bde9dbd6349e080a2: Status 404 returned error can't find the container with id 574c6906b17ca7382d033e3c1490a3c3ffef5f8642f16e2bde9dbd6349e080a2 Nov 03 19:36:34.170582 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.170556 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-tls\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:34.170875 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.170609 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-86qxx\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:34.170875 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.170641 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-tls\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:34.170875 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:34.170712 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:36:34.170875 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:34.170728 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd465cd78-4n8kx: secret "image-registry-tls" not found Nov 03 19:36:34.170875 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:34.170748 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 19:36:34.170875 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:34.170776 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-tls podName:b989ba1e-71f9-424b-90f1-d1450f63e8b1 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:35.170762728 +0000 UTC m=+32.263892153 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-tls") pod "image-registry-6dd465cd78-4n8kx" (UID: "b989ba1e-71f9-424b-90f1-d1450f63e8b1") : secret "image-registry-tls" not found Nov 03 19:36:34.170875 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:34.170753 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:36:34.170875 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:34.170789 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799ffcd7c9-5pdd7: secret "image-registry-tls" not found Nov 03 19:36:34.170875 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:34.170790 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls podName:40b2621f-854e-44e0-9b01-c877e333f644 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:35.170783824 +0000 UTC m=+32.263913241 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-86qxx" (UID: "40b2621f-854e-44e0-9b01-c877e333f644") : secret "samples-operator-tls" not found Nov 03 19:36:34.170875 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:34.170826 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-tls podName:32065295-c6a2-44ea-88e8-c4fdc5a84083 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:35.170812675 +0000 UTC m=+32.263942109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-tls") pod "image-registry-799ffcd7c9-5pdd7" (UID: "32065295-c6a2-44ea-88e8-c4fdc5a84083") : secret "image-registry-tls" not found Nov 03 19:36:34.177513 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.177493 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fgdh8"] Nov 03 19:36:34.180062 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:34.180045 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ffe451_b8eb_41ad_a19d_0390f3e1f8e2.slice/crio-6b14c8771941b7fdef847ecbb077cf12b273fee2ba5086c2edd527956ab372a4 WatchSource:0}: Error finding container 6b14c8771941b7fdef847ecbb077cf12b273fee2ba5086c2edd527956ab372a4: Status 404 returned error can't find the container with id 6b14c8771941b7fdef847ecbb077cf12b273fee2ba5086c2edd527956ab372a4 Nov 03 19:36:34.271647 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.271625 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:34.271748 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.271655 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-846mt\" (UID: \"04a982f9-143c-406c-99d5-9ef73d5c571a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" Nov 03 19:36:34.271791 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:34.271766 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:36:34.271823 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:34.271795 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 19:36:34.271823 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:34.271817 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls podName:419841be-7a31-4cdb-b0b7-9d90a3c376c7 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:35.271802513 +0000 UTC m=+32.364931933 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls") pod "dns-default-nhf67" (UID: "419841be-7a31-4cdb-b0b7-9d90a3c376c7") : secret "dns-default-metrics-tls" not found Nov 03 19:36:34.271884 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:34.271833 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert podName:04a982f9-143c-406c-99d5-9ef73d5c571a nodeName:}" failed. No retries permitted until 2026-11-03 19:36:35.271825825 +0000 UTC m=+32.364955243 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-846mt" (UID: "04a982f9-143c-406c-99d5-9ef73d5c571a") : secret "networking-console-plugin-cert" not found Nov 03 19:36:34.470716 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.470692 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:34.470842 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.470825 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:34.474725 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.474704 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 19:36:34.474725 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.474711 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vlmgx\"" Nov 03 19:36:34.474871 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.474815 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tx8mk\"" Nov 03 19:36:34.475199 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.475183 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 19:36:34.784171 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.784095 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wdls7" event={"ID":"40834bff-2db0-4690-9cb9-7b4aacdab3ab","Type":"ContainerStarted","Data":"32222fdc67b4f6469d917ab85ca77d14f3f5514ef515ff284628d7056c596a70"} Nov 03 19:36:34.786144 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.786106 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jt54m" event={"ID":"e2155512-2116-44f6-be5e-77cef72f9e14","Type":"ContainerStarted","Data":"27b2092e2b1ae773885e5c47b4223f33ad58ad3523d44d0fdfca69cbbee17e22"} Nov 03 19:36:34.786276 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.786162 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jt54m" event={"ID":"e2155512-2116-44f6-be5e-77cef72f9e14","Type":"ContainerStarted","Data":"e781524536bca1b272b961d24e61957bd19573364a9d4afb8efb7d5734c4b324"} Nov 03 19:36:34.788091 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.788065 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fgdh8" event={"ID":"18ffe451-b8eb-41ad-a19d-0390f3e1f8e2","Type":"ContainerStarted","Data":"6b14c8771941b7fdef847ecbb077cf12b273fee2ba5086c2edd527956ab372a4"} Nov 03 19:36:34.789823 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.789795 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tvb96" event={"ID":"0a985b50-c4c0-4a39-abed-29f53e2be747","Type":"ContainerStarted","Data":"574c6906b17ca7382d033e3c1490a3c3ffef5f8642f16e2bde9dbd6349e080a2"} Nov 03 19:36:34.792827 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.792804 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" event={"ID":"c99b0c19-004e-450c-852a-e38338635b61","Type":"ContainerStarted","Data":"22cc4b81bd4c7dc1a411b6935fbe5de145ecdb4a6cb45704d4810326ae0a896e"} Nov 03 19:36:34.794472 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.794451 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" event={"ID":"492c7b4e-84c0-49a7-b12f-153a021f971b","Type":"ContainerStarted","Data":"1af68c1e1deeba4a098d6b6a340842f1096a672a7c9f458990ee87e7fd746ec3"} Nov 03 19:36:34.797367 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.797297 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6hx8t" event={"ID":"f9d935b2-aa51-474c-836d-a74f91efab07","Type":"ContainerStarted","Data":"e7fcbceef75077181869de0f40c073966ec17006f2685b45f17586d4fcf3eeb3"} Nov 03 19:36:34.797367 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.797336 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6hx8t" event={"ID":"f9d935b2-aa51-474c-836d-a74f91efab07","Type":"ContainerStarted","Data":"5bc830e6d84d3cd29bae7a5df0255f8f86285729e706443279b9042876a37e1c"} Nov 03 19:36:34.799306 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.799285 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fsbsp" event={"ID":"1bcd4690-4753-4246-abb4-8668a0a56231","Type":"ContainerStarted","Data":"3c720fc599e07edb9f53c3258393229ef983d842fab015bf43f5f59b3342d40f"} Nov 03 19:36:34.800975 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.800919 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" event={"ID":"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac","Type":"ContainerStarted","Data":"38325d12703d972e1ae36cba1ea5a54367988c1d6b25ab7c5727d1b58e368726"} Nov 03 19:36:34.805886 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.802881 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bd8f8" event={"ID":"0fc518fc-6302-4a1a-9ff9-f1e036a2199b","Type":"ContainerStarted","Data":"13a83c66d1721d1a4304250d96ba6c4a613cfb0ba59048a8e63f219d8a6f74c8"} Nov 03 19:36:34.805886 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.802927 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bd8f8" event={"ID":"0fc518fc-6302-4a1a-9ff9-f1e036a2199b","Type":"ContainerStarted","Data":"78e9081e9ae7d7fd90676dd54dbde8f3e82156ec5f28724f4b1e82150cb9f4ff"} Nov 03 19:36:34.825458 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.825404 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-jt54m" podStartSLOduration=390.825389542 podStartE2EDuration="6m30.825389542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:34.805645714 +0000 UTC m=+31.898775157" watchObservedRunningTime="2026-11-03 19:36:34.825389542 +0000 UTC m=+31.918518983" Nov 03 19:36:34.845457 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.845407 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-bd8f8" podStartSLOduration=390.845393489 podStartE2EDuration="6m30.845393489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:34.82562558 +0000 UTC m=+31.918755026" watchObservedRunningTime="2026-11-03 19:36:34.845393489 +0000 UTC m=+31.938522929" Nov 03 19:36:34.845936 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:34.845793 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6hx8t" podStartSLOduration=390.845783283 podStartE2EDuration="6m30.845783283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:34.844169071 +0000 UTC m=+31.937298528" watchObservedRunningTime="2026-11-03 19:36:34.845783283 +0000 UTC m=+31.938912738" Nov 03 19:36:35.082086 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:35.081178 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-h9f7b\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:35.082086 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:35.081280 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:35.082086 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:35.081317 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-metrics-certs\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:35.082086 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:35.081459 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 19:36:35.082086 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:35.081520 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-metrics-certs podName:603868b0-8628-4227-8471-1894e2e1de50 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:37.081501145 +0000 UTC m=+34.174630577 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-metrics-certs") pod "router-default-f7b87d695-cwlfq" (UID: "603868b0-8628-4227-8471-1894e2e1de50") : secret "router-metrics-certs-default" not found Nov 03 19:36:35.082086 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:35.081941 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 19:36:35.082086 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:35.081993 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls podName:8d0db609-b271-4c98-a303-fb3ee71f152c nodeName:}" failed. No retries permitted until 2026-11-03 19:36:37.081977577 +0000 UTC m=+34.175107010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-h9f7b" (UID: "8d0db609-b271-4c98-a303-fb3ee71f152c") : secret "cluster-monitoring-operator-tls" not found Nov 03 19:36:35.082086 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:35.082061 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle podName:603868b0-8628-4227-8471-1894e2e1de50 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:37.082050522 +0000 UTC m=+34.175179954 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle") pod "router-default-f7b87d695-cwlfq" (UID: "603868b0-8628-4227-8471-1894e2e1de50") : configmap references non-existent config key: service-ca.crt Nov 03 19:36:35.182849 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:35.181885 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-tls\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:35.182849 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:35.181953 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-86qxx\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:35.182849 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:35.181980 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-tls\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:35.182849 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:35.182215 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:36:35.182849 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:35.182231 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799ffcd7c9-5pdd7: secret "image-registry-tls" not found Nov 03 19:36:35.182849 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:35.182285 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-tls podName:32065295-c6a2-44ea-88e8-c4fdc5a84083 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:37.182267944 +0000 UTC m=+34.275397378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-tls") pod "image-registry-799ffcd7c9-5pdd7" (UID: "32065295-c6a2-44ea-88e8-c4fdc5a84083") : secret "image-registry-tls" not found Nov 03 19:36:35.182849 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:35.182649 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:36:35.182849 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:35.182677 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd465cd78-4n8kx: secret "image-registry-tls" not found Nov 03 19:36:35.182849 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:35.182719 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-tls podName:b989ba1e-71f9-424b-90f1-d1450f63e8b1 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:37.182705289 +0000 UTC m=+34.275834722 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-tls") pod "image-registry-6dd465cd78-4n8kx" (UID: "b989ba1e-71f9-424b-90f1-d1450f63e8b1") : secret "image-registry-tls" not found Nov 03 19:36:35.182849 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:35.182782 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 19:36:35.182849 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:35.182813 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls podName:40b2621f-854e-44e0-9b01-c877e333f644 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:37.182803026 +0000 UTC m=+34.275932458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-86qxx" (UID: "40b2621f-854e-44e0-9b01-c877e333f644") : secret "samples-operator-tls" not found Nov 03 19:36:35.284410 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:35.283499 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:35.284410 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:35.283547 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-846mt\" (UID: \"04a982f9-143c-406c-99d5-9ef73d5c571a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" Nov 03 19:36:35.284410 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:35.283796 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 19:36:35.284410 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:35.283862 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert podName:04a982f9-143c-406c-99d5-9ef73d5c571a nodeName:}" failed. No retries permitted until 2026-11-03 19:36:37.283844206 +0000 UTC m=+34.376973639 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-846mt" (UID: "04a982f9-143c-406c-99d5-9ef73d5c571a") : secret "networking-console-plugin-cert" not found Nov 03 19:36:35.284410 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:35.284250 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:36:35.284410 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:35.284299 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls podName:419841be-7a31-4cdb-b0b7-9d90a3c376c7 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:37.284286233 +0000 UTC m=+34.377415665 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls") pod "dns-default-nhf67" (UID: "419841be-7a31-4cdb-b0b7-9d90a3c376c7") : secret "dns-default-metrics-tls" not found Nov 03 19:36:36.815898 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:36.815788 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fsbsp" event={"ID":"1bcd4690-4753-4246-abb4-8668a0a56231","Type":"ContainerStarted","Data":"56aa53c1d031f7ab6c282efd0400e4a7abf4371a2d9df0ad33beb6d199334684"} Nov 03 19:36:37.105646 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:37.105605 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:37.105810 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:37.105683 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-metrics-certs\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:37.105810 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:37.105712 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs\") pod \"network-metrics-daemon-qvxjq\" (UID: \"caf7d6c9-0bb2-4da0-be2d-caea7734bbcb\") " pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:37.105927 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:37.105833 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-h9f7b\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:37.106025 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:37.106004 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 19:36:37.106105 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:37.106078 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls podName:8d0db609-b271-4c98-a303-fb3ee71f152c nodeName:}" failed. No retries permitted until 2026-11-03 19:36:41.10606022 +0000 UTC m=+38.199189660 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-h9f7b" (UID: "8d0db609-b271-4c98-a303-fb3ee71f152c") : secret "cluster-monitoring-operator-tls" not found Nov 03 19:36:37.106600 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:37.106579 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 19:36:37.106759 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:37.106624 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 19:36:37.106759 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:37.106637 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs podName:caf7d6c9-0bb2-4da0-be2d-caea7734bbcb nodeName:}" failed. No retries permitted until 2026-11-03 19:37:09.106621521 +0000 UTC m=+66.199750941 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs") pod "network-metrics-daemon-qvxjq" (UID: "caf7d6c9-0bb2-4da0-be2d-caea7734bbcb") : secret "metrics-daemon-secret" not found Nov 03 19:36:37.106759 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:37.106720 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-metrics-certs podName:603868b0-8628-4227-8471-1894e2e1de50 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:41.106702544 +0000 UTC m=+38.199831963 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-metrics-certs") pod "router-default-f7b87d695-cwlfq" (UID: "603868b0-8628-4227-8471-1894e2e1de50") : secret "router-metrics-certs-default" not found Nov 03 19:36:37.106759 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:37.106737 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle podName:603868b0-8628-4227-8471-1894e2e1de50 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:41.10672755 +0000 UTC m=+38.199856973 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle") pod "router-default-f7b87d695-cwlfq" (UID: "603868b0-8628-4227-8471-1894e2e1de50") : configmap references non-existent config key: service-ca.crt Nov 03 19:36:37.207218 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:37.207152 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-86qxx\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:37.207218 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:37.207188 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-tls\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:37.207431 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:37.207291 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf5tn\" (UniqueName: \"kubernetes.io/projected/c95249a6-899f-4faf-82dc-6e553453ff80-kube-api-access-lf5tn\") pod \"network-check-target-b4x9q\" (UID: \"c95249a6-899f-4faf-82dc-6e553453ff80\") " pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:37.207431 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:37.207306 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 19:36:37.207431 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:37.207369 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls podName:40b2621f-854e-44e0-9b01-c877e333f644 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:41.207352499 +0000 UTC m=+38.300481925 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-86qxx" (UID: "40b2621f-854e-44e0-9b01-c877e333f644") : secret "samples-operator-tls" not found Nov 03 19:36:37.207431 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:37.207402 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-tls\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:37.207565 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:37.207427 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:36:37.207565 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:37.207452 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799ffcd7c9-5pdd7: secret "image-registry-tls" not found Nov 03 19:36:37.207565 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:37.207511 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-tls podName:32065295-c6a2-44ea-88e8-c4fdc5a84083 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:41.20749184 +0000 UTC m=+38.300621274 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-tls") pod "image-registry-799ffcd7c9-5pdd7" (UID: "32065295-c6a2-44ea-88e8-c4fdc5a84083") : secret "image-registry-tls" not found Nov 03 19:36:37.207565 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:37.207434 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/116af81a-d3a0-4558-93c4-8bf061b6815f-original-pull-secret\") pod \"global-pull-secret-syncer-wh9tj\" (UID: \"116af81a-d3a0-4558-93c4-8bf061b6815f\") " pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:37.207982 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:37.207962 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:36:37.207982 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:37.207982 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd465cd78-4n8kx: secret "image-registry-tls" not found Nov 03 19:36:37.208150 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:37.208022 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-tls podName:b989ba1e-71f9-424b-90f1-d1450f63e8b1 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:41.208008924 +0000 UTC m=+38.301138365 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-tls") pod "image-registry-6dd465cd78-4n8kx" (UID: "b989ba1e-71f9-424b-90f1-d1450f63e8b1") : secret "image-registry-tls" not found Nov 03 19:36:37.217155 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:37.217108 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/116af81a-d3a0-4558-93c4-8bf061b6815f-original-pull-secret\") pod \"global-pull-secret-syncer-wh9tj\" (UID: \"116af81a-d3a0-4558-93c4-8bf061b6815f\") " pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:37.219358 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:37.219333 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf5tn\" (UniqueName: \"kubernetes.io/projected/c95249a6-899f-4faf-82dc-6e553453ff80-kube-api-access-lf5tn\") pod \"network-check-target-b4x9q\" (UID: \"c95249a6-899f-4faf-82dc-6e553453ff80\") " pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:37.308336 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:37.308294 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:37.308496 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:37.308345 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-846mt\" (UID: \"04a982f9-143c-406c-99d5-9ef73d5c571a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" Nov 03 19:36:37.308561 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:37.308552 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 19:36:37.308622 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:37.308610 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert podName:04a982f9-143c-406c-99d5-9ef73d5c571a nodeName:}" failed. No retries permitted until 2026-11-03 19:36:41.308594252 +0000 UTC m=+38.401723683 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-846mt" (UID: "04a982f9-143c-406c-99d5-9ef73d5c571a") : secret "networking-console-plugin-cert" not found Nov 03 19:36:37.308784 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:37.308769 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:36:37.308837 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:37.308821 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls podName:419841be-7a31-4cdb-b0b7-9d90a3c376c7 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:41.308809339 +0000 UTC m=+38.401938770 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls") pod "dns-default-nhf67" (UID: "419841be-7a31-4cdb-b0b7-9d90a3c376c7") : secret "dns-default-metrics-tls" not found Nov 03 19:36:37.454059 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:37.453829 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:37.516981 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:37.516793 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:37.820575 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:37.820540 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b4x9q"] Nov 03 19:36:37.832345 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:37.832295 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc95249a6_899f_4faf_82dc_6e553453ff80.slice/crio-7f2c2611f3929ca717acc0cd7b339637d4192b9b0f500398744cedf2c5f27953 WatchSource:0}: Error finding container 7f2c2611f3929ca717acc0cd7b339637d4192b9b0f500398744cedf2c5f27953: Status 404 returned error can't find the container with id 7f2c2611f3929ca717acc0cd7b339637d4192b9b0f500398744cedf2c5f27953 Nov 03 19:36:37.845638 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:37.844944 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fsbsp" podStartSLOduration=364.216067854 podStartE2EDuration="6m6.844927797s" totalImagesPullingTime="2.628859943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:37.841946901 +0000 UTC m=+34.935076342" watchObservedRunningTime="2026-11-03 19:36:37.844927797 +0000 UTC m=+34.938057239" Nov 03 19:36:37.845638 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:37.845554 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wh9tj"] Nov 03 19:36:37.854976 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:37.854942 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116af81a_d3a0_4558_93c4_8bf061b6815f.slice/crio-e15414be96b0fe9f7d586967b509dcccb51d5baec5da68fd4c509d22b5bda8f1 WatchSource:0}: Error finding container e15414be96b0fe9f7d586967b509dcccb51d5baec5da68fd4c509d22b5bda8f1: Status 404 returned error can't find the container with id e15414be96b0fe9f7d586967b509dcccb51d5baec5da68fd4c509d22b5bda8f1 Nov 03 19:36:38.826986 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:38.826957 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b4x9q" event={"ID":"c95249a6-899f-4faf-82dc-6e553453ff80","Type":"ContainerStarted","Data":"7f2c2611f3929ca717acc0cd7b339637d4192b9b0f500398744cedf2c5f27953"} Nov 03 19:36:38.828284 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:38.828258 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wh9tj" event={"ID":"116af81a-d3a0-4558-93c4-8bf061b6815f","Type":"ContainerStarted","Data":"e15414be96b0fe9f7d586967b509dcccb51d5baec5da68fd4c509d22b5bda8f1"} Nov 03 19:36:39.837161 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:39.836240 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/0.log" Nov 03 19:36:39.837161 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:39.836284 2580 generic.go:356] "Generic (PLEG): container finished" podID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" containerID="21c0839a7c5782de9455b7b089f47980a110a80bfe80ebf17197f047f1eefe0d" exitCode=255 Nov 03 19:36:39.837161 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:39.836361 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" event={"ID":"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac","Type":"ContainerDied","Data":"21c0839a7c5782de9455b7b089f47980a110a80bfe80ebf17197f047f1eefe0d"} Nov 03 19:36:39.837161 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:39.836625 2580 scope.go:122] "RemoveContainer" containerID="21c0839a7c5782de9455b7b089f47980a110a80bfe80ebf17197f047f1eefe0d" Nov 03 19:36:39.840251 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:39.839774 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wdls7" event={"ID":"40834bff-2db0-4690-9cb9-7b4aacdab3ab","Type":"ContainerStarted","Data":"974d23ae8d30a4fce6a921ba2dc0d90ab07c28ac162cd3380ddc95968846b3c1"} Nov 03 19:36:39.842519 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:39.842199 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" event={"ID":"c99b0c19-004e-450c-852a-e38338635b61","Type":"ContainerStarted","Data":"481852318efef15a99c90fc33e8f7d88cfa6b35f952bd0e4e6c4993664dcc592"} Nov 03 19:36:39.844443 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:39.844139 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" event={"ID":"492c7b4e-84c0-49a7-b12f-153a021f971b","Type":"ContainerStarted","Data":"418374b5472981e5cdbb3a5be895780bfb3ad1fc0b28a5e01266b5d997136ed3"} Nov 03 19:36:39.876762 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:39.876707 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" podStartSLOduration=364.085865038 podStartE2EDuration="6m8.876688247s" totalImagesPullingTime="4.790823209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:39.874829598 +0000 UTC m=+36.967959051" watchObservedRunningTime="2026-11-03 19:36:39.876688247 +0000 UTC m=+36.969817684" Nov 03 19:36:39.931298 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:39.931237 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-wdls7" podStartSLOduration=364.094514688 podStartE2EDuration="6m8.931217892s" totalImagesPullingTime="4.836703204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:39.901677039 +0000 UTC m=+36.994806476" watchObservedRunningTime="2026-11-03 19:36:39.931217892 +0000 UTC m=+37.024347333" Nov 03 19:36:39.932402 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:39.931652 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" podStartSLOduration=364.128523149 podStartE2EDuration="6m8.931641873s" totalImagesPullingTime="4.803118724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:39.930552061 +0000 UTC m=+37.023681504" watchObservedRunningTime="2026-11-03 19:36:39.931641873 +0000 UTC m=+37.024771313" Nov 03 19:36:39.976133 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:39.975914 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-pbgwk"] Nov 03 19:36:39.998343 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:39.998316 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6k9n9"] Nov 03 19:36:39.998472 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:39.998442 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pbgwk" Nov 03 19:36:40.017858 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.017830 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 19:36:40.018029 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.017920 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 19:36:40.036895 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.036867 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdss4\" (UniqueName: \"kubernetes.io/projected/815b15b6-2c37-4ca3-9b38-e7e1328d0aa8-kube-api-access-mdss4\") pod \"migrator-66df5d496b-pbgwk\" (UID: \"815b15b6-2c37-4ca3-9b38-e7e1328d0aa8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pbgwk" Nov 03 19:36:40.049701 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.049413 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-pbgwk"] Nov 03 19:36:40.049701 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.049462 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6k9n9"] Nov 03 19:36:40.049701 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.049562 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6k9n9" Nov 03 19:36:40.070791 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.070748 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4p6h4\"" Nov 03 19:36:40.093768 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.093702 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-bhdxh-d8v5q"] Nov 03 19:36:40.111874 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.111846 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hmnx4"] Nov 03 19:36:40.112253 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.112207 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" Nov 03 19:36:40.136801 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.136775 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bhdxh-d8v5q"] Nov 03 19:36:40.136981 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.136820 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hmnx4"] Nov 03 19:36:40.136981 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.136935 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:40.137427 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.137402 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdss4\" (UniqueName: \"kubernetes.io/projected/815b15b6-2c37-4ca3-9b38-e7e1328d0aa8-kube-api-access-mdss4\") pod \"migrator-66df5d496b-pbgwk\" (UID: \"815b15b6-2c37-4ca3-9b38-e7e1328d0aa8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pbgwk" Nov 03 19:36:40.137518 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.137462 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b79231d-e954-41b2-924d-67808a734ecf-service-ca-bundle\") pod \"periodic-gathering-bhdxh-d8v5q\" (UID: \"4b79231d-e954-41b2-924d-67808a734ecf\") " pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" Nov 03 19:36:40.137574 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.137517 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86bkx\" (UniqueName: \"kubernetes.io/projected/f485893c-6f14-449b-b837-8ac181315aeb-kube-api-access-86bkx\") pod \"migrator-66df5d496b-6k9n9\" (UID: \"f485893c-6f14-449b-b837-8ac181315aeb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6k9n9" Nov 03 19:36:40.137574 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.137543 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4b79231d-e954-41b2-924d-67808a734ecf-archives-path\") pod \"periodic-gathering-bhdxh-d8v5q\" (UID: \"4b79231d-e954-41b2-924d-67808a734ecf\") " pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" Nov 03 19:36:40.137706 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.137580 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b79231d-e954-41b2-924d-67808a734ecf-trusted-ca-bundle\") pod \"periodic-gathering-bhdxh-d8v5q\" (UID: \"4b79231d-e954-41b2-924d-67808a734ecf\") " pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" Nov 03 19:36:40.137706 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.137612 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhrbz\" (UniqueName: \"kubernetes.io/projected/4b79231d-e954-41b2-924d-67808a734ecf-kube-api-access-nhrbz\") pod \"periodic-gathering-bhdxh-d8v5q\" (UID: \"4b79231d-e954-41b2-924d-67808a734ecf\") " pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" Nov 03 19:36:40.147919 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.147874 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 19:36:40.154994 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.154944 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdss4\" (UniqueName: \"kubernetes.io/projected/815b15b6-2c37-4ca3-9b38-e7e1328d0aa8-kube-api-access-mdss4\") pod \"migrator-66df5d496b-pbgwk\" (UID: \"815b15b6-2c37-4ca3-9b38-e7e1328d0aa8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pbgwk" Nov 03 19:36:40.170517 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.170478 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 19:36:40.170614 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.170555 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b2f85\"" Nov 03 19:36:40.241468 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.238598 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/377ffcf5-b097-495d-b986-e61c58ae8ec1-data-volume\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:40.241468 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.238731 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b79231d-e954-41b2-924d-67808a734ecf-service-ca-bundle\") pod \"periodic-gathering-bhdxh-d8v5q\" (UID: \"4b79231d-e954-41b2-924d-67808a734ecf\") " pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" Nov 03 19:36:40.241468 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.238815 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86bkx\" (UniqueName: \"kubernetes.io/projected/f485893c-6f14-449b-b837-8ac181315aeb-kube-api-access-86bkx\") pod \"migrator-66df5d496b-6k9n9\" (UID: \"f485893c-6f14-449b-b837-8ac181315aeb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6k9n9" Nov 03 19:36:40.241468 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.238845 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:40.241468 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.238926 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4b79231d-e954-41b2-924d-67808a734ecf-archives-path\") pod \"periodic-gathering-bhdxh-d8v5q\" (UID: \"4b79231d-e954-41b2-924d-67808a734ecf\") " pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" Nov 03 19:36:40.241468 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.238982 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b79231d-e954-41b2-924d-67808a734ecf-trusted-ca-bundle\") pod \"periodic-gathering-bhdxh-d8v5q\" (UID: \"4b79231d-e954-41b2-924d-67808a734ecf\") " pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" Nov 03 19:36:40.241468 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.239012 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzh28\" (UniqueName: \"kubernetes.io/projected/377ffcf5-b097-495d-b986-e61c58ae8ec1-kube-api-access-hzh28\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:40.241468 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.239033 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/377ffcf5-b097-495d-b986-e61c58ae8ec1-crio-socket\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:40.241468 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.239048 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/377ffcf5-b097-495d-b986-e61c58ae8ec1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:40.241468 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.239065 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhrbz\" (UniqueName: \"kubernetes.io/projected/4b79231d-e954-41b2-924d-67808a734ecf-kube-api-access-nhrbz\") pod \"periodic-gathering-bhdxh-d8v5q\" (UID: \"4b79231d-e954-41b2-924d-67808a734ecf\") " pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" Nov 03 19:36:40.241468 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.239958 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b79231d-e954-41b2-924d-67808a734ecf-service-ca-bundle\") pod \"periodic-gathering-bhdxh-d8v5q\" (UID: \"4b79231d-e954-41b2-924d-67808a734ecf\") " pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" Nov 03 19:36:40.241468 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.240466 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4b79231d-e954-41b2-924d-67808a734ecf-archives-path\") pod \"periodic-gathering-bhdxh-d8v5q\" (UID: \"4b79231d-e954-41b2-924d-67808a734ecf\") " pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" Nov 03 19:36:40.241468 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.241371 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b79231d-e954-41b2-924d-67808a734ecf-trusted-ca-bundle\") pod \"periodic-gathering-bhdxh-d8v5q\" (UID: \"4b79231d-e954-41b2-924d-67808a734ecf\") " pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" Nov 03 19:36:40.252992 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.252967 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhrbz\" (UniqueName: \"kubernetes.io/projected/4b79231d-e954-41b2-924d-67808a734ecf-kube-api-access-nhrbz\") pod \"periodic-gathering-bhdxh-d8v5q\" (UID: \"4b79231d-e954-41b2-924d-67808a734ecf\") " pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" Nov 03 19:36:40.253141 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.253119 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86bkx\" (UniqueName: \"kubernetes.io/projected/f485893c-6f14-449b-b837-8ac181315aeb-kube-api-access-86bkx\") pod \"migrator-66df5d496b-6k9n9\" (UID: \"f485893c-6f14-449b-b837-8ac181315aeb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6k9n9" Nov 03 19:36:40.339712 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.339684 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pbgwk" Nov 03 19:36:40.340069 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.340049 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzh28\" (UniqueName: \"kubernetes.io/projected/377ffcf5-b097-495d-b986-e61c58ae8ec1-kube-api-access-hzh28\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:40.340154 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.340097 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/377ffcf5-b097-495d-b986-e61c58ae8ec1-crio-socket\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:40.340154 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.340114 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/377ffcf5-b097-495d-b986-e61c58ae8ec1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:40.340253 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.340153 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/377ffcf5-b097-495d-b986-e61c58ae8ec1-data-volume\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:40.340319 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.340268 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:40.340512 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:40.340433 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:36:40.340512 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.340489 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/377ffcf5-b097-495d-b986-e61c58ae8ec1-crio-socket\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:40.340512 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:40.340504 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls podName:377ffcf5-b097-495d-b986-e61c58ae8ec1 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:40.840486622 +0000 UTC m=+37.933616054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hmnx4" (UID: "377ffcf5-b097-495d-b986-e61c58ae8ec1") : secret "insights-runtime-extractor-tls" not found Nov 03 19:36:40.340703 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:40.340437 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 19:36:40.340703 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:40.340567 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/377ffcf5-b097-495d-b986-e61c58ae8ec1-kube-rbac-proxy-cm podName:377ffcf5-b097-495d-b986-e61c58ae8ec1 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:40.840550147 +0000 UTC m=+37.933679586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/377ffcf5-b097-495d-b986-e61c58ae8ec1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hmnx4" (UID: "377ffcf5-b097-495d-b986-e61c58ae8ec1") : configmap "kube-rbac-proxy" not found Nov 03 19:36:40.340851 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.340824 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/377ffcf5-b097-495d-b986-e61c58ae8ec1-data-volume\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:40.356282 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.356259 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzh28\" (UniqueName: \"kubernetes.io/projected/377ffcf5-b097-495d-b986-e61c58ae8ec1-kube-api-access-hzh28\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:40.377673 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.377643 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6k9n9" Nov 03 19:36:40.439687 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.438619 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-brvgz"] Nov 03 19:36:40.461443 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.461409 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-brvgz"] Nov 03 19:36:40.461572 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.461527 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" Nov 03 19:36:40.462321 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.462294 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" Nov 03 19:36:40.467418 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.467374 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 19:36:40.469452 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.469316 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-b22jw\"" Nov 03 19:36:40.469452 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.469377 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 19:36:40.469786 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.469679 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 19:36:40.469786 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.469756 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 19:36:40.549340 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.546803 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rnvd\" (UniqueName: \"kubernetes.io/projected/7b8992fc-fba6-4135-90f8-fb259e765dd5-kube-api-access-7rnvd\") pod \"service-ca-5fffc97bdf-brvgz\" (UID: \"7b8992fc-fba6-4135-90f8-fb259e765dd5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" Nov 03 19:36:40.549340 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.546877 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7b8992fc-fba6-4135-90f8-fb259e765dd5-signing-key\") pod \"service-ca-5fffc97bdf-brvgz\" (UID: \"7b8992fc-fba6-4135-90f8-fb259e765dd5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" Nov 03 19:36:40.549340 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.546923 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7b8992fc-fba6-4135-90f8-fb259e765dd5-signing-cabundle\") pod \"service-ca-5fffc97bdf-brvgz\" (UID: \"7b8992fc-fba6-4135-90f8-fb259e765dd5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" Nov 03 19:36:40.645980 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.645909 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-pbgwk"] Nov 03 19:36:40.648424 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.648398 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rnvd\" (UniqueName: \"kubernetes.io/projected/7b8992fc-fba6-4135-90f8-fb259e765dd5-kube-api-access-7rnvd\") pod \"service-ca-5fffc97bdf-brvgz\" (UID: \"7b8992fc-fba6-4135-90f8-fb259e765dd5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" Nov 03 19:36:40.648524 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.648465 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7b8992fc-fba6-4135-90f8-fb259e765dd5-signing-key\") pod \"service-ca-5fffc97bdf-brvgz\" (UID: \"7b8992fc-fba6-4135-90f8-fb259e765dd5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" Nov 03 19:36:40.648580 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.648505 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7b8992fc-fba6-4135-90f8-fb259e765dd5-signing-cabundle\") pod \"service-ca-5fffc97bdf-brvgz\" (UID: \"7b8992fc-fba6-4135-90f8-fb259e765dd5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" Nov 03 19:36:40.649624 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.649602 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7b8992fc-fba6-4135-90f8-fb259e765dd5-signing-cabundle\") pod \"service-ca-5fffc97bdf-brvgz\" (UID: \"7b8992fc-fba6-4135-90f8-fb259e765dd5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" Nov 03 19:36:40.653156 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.653130 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7b8992fc-fba6-4135-90f8-fb259e765dd5-signing-key\") pod \"service-ca-5fffc97bdf-brvgz\" (UID: \"7b8992fc-fba6-4135-90f8-fb259e765dd5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" Nov 03 19:36:40.662903 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.662861 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rnvd\" (UniqueName: \"kubernetes.io/projected/7b8992fc-fba6-4135-90f8-fb259e765dd5-kube-api-access-7rnvd\") pod \"service-ca-5fffc97bdf-brvgz\" (UID: \"7b8992fc-fba6-4135-90f8-fb259e765dd5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" Nov 03 19:36:40.696783 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.696759 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6k9n9"] Nov 03 19:36:40.711632 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:40.711602 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod815b15b6_2c37_4ca3_9b38_e7e1328d0aa8.slice/crio-6c41b8db3ecc464cfdadc0046d8144c354dc482d49d376e427a16f417211c4bc WatchSource:0}: Error finding container 6c41b8db3ecc464cfdadc0046d8144c354dc482d49d376e427a16f417211c4bc: Status 404 returned error can't find the container with id 6c41b8db3ecc464cfdadc0046d8144c354dc482d49d376e427a16f417211c4bc Nov 03 19:36:40.712944 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:40.712908 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf485893c_6f14_449b_b837_8ac181315aeb.slice/crio-57dc8ba289eb00c068e01a84058a62327d180d9194361eb4011c73b22e8d29f2 WatchSource:0}: Error finding container 57dc8ba289eb00c068e01a84058a62327d180d9194361eb4011c73b22e8d29f2: Status 404 returned error can't find the container with id 57dc8ba289eb00c068e01a84058a62327d180d9194361eb4011c73b22e8d29f2 Nov 03 19:36:40.768291 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.768226 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bhdxh-d8v5q"] Nov 03 19:36:40.772290 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:40.772258 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b79231d_e954_41b2_924d_67808a734ecf.slice/crio-84dcb8f62c1c8e86cbc9805e06cd0f30068afc5169cf3efd95ae1cfd8d7d1404 WatchSource:0}: Error finding container 84dcb8f62c1c8e86cbc9805e06cd0f30068afc5169cf3efd95ae1cfd8d7d1404: Status 404 returned error can't find the container with id 84dcb8f62c1c8e86cbc9805e06cd0f30068afc5169cf3efd95ae1cfd8d7d1404 Nov 03 19:36:40.799516 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.799472 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" Nov 03 19:36:40.851217 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.851188 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" event={"ID":"4b79231d-e954-41b2-924d-67808a734ecf","Type":"ContainerStarted","Data":"84dcb8f62c1c8e86cbc9805e06cd0f30068afc5169cf3efd95ae1cfd8d7d1404"} Nov 03 19:36:40.851675 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.851644 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/377ffcf5-b097-495d-b986-e61c58ae8ec1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:40.851795 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.851777 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:40.851927 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:40.851910 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:36:40.851986 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:40.851956 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls podName:377ffcf5-b097-495d-b986-e61c58ae8ec1 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:41.851943841 +0000 UTC m=+38.945073272 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hmnx4" (UID: "377ffcf5-b097-495d-b986-e61c58ae8ec1") : secret "insights-runtime-extractor-tls" not found Nov 03 19:36:40.852060 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:40.851998 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 19:36:40.852060 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:40.852020 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/377ffcf5-b097-495d-b986-e61c58ae8ec1-kube-rbac-proxy-cm podName:377ffcf5-b097-495d-b986-e61c58ae8ec1 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:41.852013399 +0000 UTC m=+38.945142830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/377ffcf5-b097-495d-b986-e61c58ae8ec1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hmnx4" (UID: "377ffcf5-b097-495d-b986-e61c58ae8ec1") : configmap "kube-rbac-proxy" not found Nov 03 19:36:40.853240 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.853215 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6k9n9" event={"ID":"f485893c-6f14-449b-b837-8ac181315aeb","Type":"ContainerStarted","Data":"57dc8ba289eb00c068e01a84058a62327d180d9194361eb4011c73b22e8d29f2"} Nov 03 19:36:40.856369 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.856335 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fgdh8" event={"ID":"18ffe451-b8eb-41ad-a19d-0390f3e1f8e2","Type":"ContainerStarted","Data":"a97001bc3b35b58c9e4c2fa65912856ee3382100b3b217b811203f4b9f51e4fd"} Nov 03 19:36:40.861595 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.861571 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tvb96" event={"ID":"0a985b50-c4c0-4a39-abed-29f53e2be747","Type":"ContainerStarted","Data":"b3dbf9416fa12ac2df38a62843937fa243f692c05fb2159a754be69427937d5d"} Nov 03 19:36:40.864974 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.864954 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 19:36:40.865786 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.865768 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/0.log" Nov 03 19:36:40.865898 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.865802 2580 generic.go:356] "Generic (PLEG): container finished" podID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" containerID="7bbcf23cd573d5c9e806dd6604b68ae67663631e74f89f722cb793dfee52aa69" exitCode=255 Nov 03 19:36:40.866249 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.866091 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" event={"ID":"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac","Type":"ContainerDied","Data":"7bbcf23cd573d5c9e806dd6604b68ae67663631e74f89f722cb793dfee52aa69"} Nov 03 19:36:40.866249 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.866144 2580 scope.go:122] "RemoveContainer" containerID="21c0839a7c5782de9455b7b089f47980a110a80bfe80ebf17197f047f1eefe0d" Nov 03 19:36:40.866502 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.866478 2580 scope.go:122] "RemoveContainer" containerID="7bbcf23cd573d5c9e806dd6604b68ae67663631e74f89f722cb793dfee52aa69" Nov 03 19:36:40.866740 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:40.866672 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-hmc5z_openshift-console-operator(f8b140c9-9e09-4cbd-847b-82ee1e3be6ac)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" podUID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" Nov 03 19:36:40.869526 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.869273 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pbgwk" event={"ID":"815b15b6-2c37-4ca3-9b38-e7e1328d0aa8","Type":"ContainerStarted","Data":"6c41b8db3ecc464cfdadc0046d8144c354dc482d49d376e427a16f417211c4bc"} Nov 03 19:36:40.909844 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:40.907335 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tvb96" podStartSLOduration=359.726330921 podStartE2EDuration="6m5.907312826s" totalImagesPullingTime="6.180981905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:40.88567553 +0000 UTC m=+37.978804962" watchObservedRunningTime="2026-11-03 19:36:40.907312826 +0000 UTC m=+38.000442267" Nov 03 19:36:41.123922 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:41.123839 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-brvgz"] Nov 03 19:36:41.126857 ip-10-0-132-83 kubenswrapper[2580]: W1103 19:36:41.126824 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8992fc_fba6_4135_90f8_fb259e765dd5.slice/crio-7f52997d73f0c127088919f5e5f96fae8499e63d1b444d7b1b7b4b0d1a3ad067 WatchSource:0}: Error finding container 7f52997d73f0c127088919f5e5f96fae8499e63d1b444d7b1b7b4b0d1a3ad067: Status 404 returned error can't find the container with id 7f52997d73f0c127088919f5e5f96fae8499e63d1b444d7b1b7b4b0d1a3ad067 Nov 03 19:36:41.157093 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:41.156901 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:41.157093 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:41.156968 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-metrics-certs\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:41.157276 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:41.157093 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle podName:603868b0-8628-4227-8471-1894e2e1de50 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:49.157071016 +0000 UTC m=+46.250200437 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle") pod "router-default-f7b87d695-cwlfq" (UID: "603868b0-8628-4227-8471-1894e2e1de50") : configmap references non-existent config key: service-ca.crt Nov 03 19:36:41.157276 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:41.157131 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-h9f7b\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:41.157388 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:41.157281 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 19:36:41.157388 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:41.157325 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls podName:8d0db609-b271-4c98-a303-fb3ee71f152c nodeName:}" failed. No retries permitted until 2026-11-03 19:36:49.157312748 +0000 UTC m=+46.250442166 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-h9f7b" (UID: "8d0db609-b271-4c98-a303-fb3ee71f152c") : secret "cluster-monitoring-operator-tls" not found Nov 03 19:36:41.157509 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:41.157393 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 19:36:41.157509 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:41.157429 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-metrics-certs podName:603868b0-8628-4227-8471-1894e2e1de50 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:49.15741807 +0000 UTC m=+46.250547515 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-metrics-certs") pod "router-default-f7b87d695-cwlfq" (UID: "603868b0-8628-4227-8471-1894e2e1de50") : secret "router-metrics-certs-default" not found Nov 03 19:36:41.258168 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:41.258141 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-tls\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:41.258298 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:41.258198 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-86qxx\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:41.258298 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:41.258226 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-tls\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:41.258404 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:41.258375 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:36:41.258404 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:41.258390 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799ffcd7c9-5pdd7: secret "image-registry-tls" not found Nov 03 19:36:41.258496 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:41.258441 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-tls podName:32065295-c6a2-44ea-88e8-c4fdc5a84083 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:49.258423062 +0000 UTC m=+46.351552498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-tls") pod "image-registry-799ffcd7c9-5pdd7" (UID: "32065295-c6a2-44ea-88e8-c4fdc5a84083") : secret "image-registry-tls" not found Nov 03 19:36:41.258870 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:41.258849 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 19:36:41.258870 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:41.258871 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd465cd78-4n8kx: secret "image-registry-tls" not found Nov 03 19:36:41.259021 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:41.258918 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-tls podName:b989ba1e-71f9-424b-90f1-d1450f63e8b1 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:49.25890293 +0000 UTC m=+46.352032362 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-tls") pod "image-registry-6dd465cd78-4n8kx" (UID: "b989ba1e-71f9-424b-90f1-d1450f63e8b1") : secret "image-registry-tls" not found Nov 03 19:36:41.259021 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:41.258981 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 19:36:41.259021 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:41.259017 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls podName:40b2621f-854e-44e0-9b01-c877e333f644 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:49.259006511 +0000 UTC m=+46.352135944 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-86qxx" (UID: "40b2621f-854e-44e0-9b01-c877e333f644") : secret "samples-operator-tls" not found Nov 03 19:36:41.359578 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:41.359548 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:41.359578 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:41.359585 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-846mt\" (UID: \"04a982f9-143c-406c-99d5-9ef73d5c571a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" Nov 03 19:36:41.359843 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:41.359718 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:36:41.359843 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:41.359778 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls podName:419841be-7a31-4cdb-b0b7-9d90a3c376c7 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:49.359763329 +0000 UTC m=+46.452892767 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls") pod "dns-default-nhf67" (UID: "419841be-7a31-4cdb-b0b7-9d90a3c376c7") : secret "dns-default-metrics-tls" not found Nov 03 19:36:41.359987 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:41.359832 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 19:36:41.359987 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:41.359944 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert podName:04a982f9-143c-406c-99d5-9ef73d5c571a nodeName:}" failed. No retries permitted until 2026-11-03 19:36:49.359909282 +0000 UTC m=+46.453038705 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-846mt" (UID: "04a982f9-143c-406c-99d5-9ef73d5c571a") : secret "networking-console-plugin-cert" not found Nov 03 19:36:41.866365 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:41.866075 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:41.866365 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:41.866200 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/377ffcf5-b097-495d-b986-e61c58ae8ec1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:41.866365 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:41.866318 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:36:41.866365 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:41.866349 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 19:36:41.867202 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:41.866411 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls podName:377ffcf5-b097-495d-b986-e61c58ae8ec1 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:43.866389134 +0000 UTC m=+40.959518567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hmnx4" (UID: "377ffcf5-b097-495d-b986-e61c58ae8ec1") : secret "insights-runtime-extractor-tls" not found Nov 03 19:36:41.867202 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:41.866432 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/377ffcf5-b097-495d-b986-e61c58ae8ec1-kube-rbac-proxy-cm podName:377ffcf5-b097-495d-b986-e61c58ae8ec1 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:43.866422069 +0000 UTC m=+40.959551487 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/377ffcf5-b097-495d-b986-e61c58ae8ec1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hmnx4" (UID: "377ffcf5-b097-495d-b986-e61c58ae8ec1") : configmap "kube-rbac-proxy" not found Nov 03 19:36:41.876933 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:41.876814 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 19:36:41.877603 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:41.877378 2580 scope.go:122] "RemoveContainer" containerID="7bbcf23cd573d5c9e806dd6604b68ae67663631e74f89f722cb793dfee52aa69" Nov 03 19:36:41.878088 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:41.878034 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-hmc5z_openshift-console-operator(f8b140c9-9e09-4cbd-847b-82ee1e3be6ac)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" podUID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" Nov 03 19:36:41.879897 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:41.879365 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" event={"ID":"7b8992fc-fba6-4135-90f8-fb259e765dd5","Type":"ContainerStarted","Data":"a9bfcc2a7fbcc7ee6fea838b485c6e2a9dbb15881176ff436fbfec03c0df0eaf"} Nov 03 19:36:41.879897 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:41.879437 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" event={"ID":"7b8992fc-fba6-4135-90f8-fb259e765dd5","Type":"ContainerStarted","Data":"7f52997d73f0c127088919f5e5f96fae8499e63d1b444d7b1b7b4b0d1a3ad067"} Nov 03 19:36:41.882083 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:41.882059 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" event={"ID":"4b79231d-e954-41b2-924d-67808a734ecf","Type":"ContainerStarted","Data":"41d3b07ff68a6e28275ab819a514afbe80d479b939d817bbf8e5094b3a82d99c"} Nov 03 19:36:41.916283 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:41.916233 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fgdh8" podStartSLOduration=2.355197997 podStartE2EDuration="8.916217274s" totalImagesPullingTime="6.561019277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:41.915800198 +0000 UTC m=+39.008929640" watchObservedRunningTime="2026-11-03 19:36:41.916217274 +0000 UTC m=+39.009346715" Nov 03 19:36:41.933994 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:41.933953 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" podStartSLOduration=1.933944142 podStartE2EDuration="1.933944142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:41.933612677 +0000 UTC m=+39.026742116" watchObservedRunningTime="2026-11-03 19:36:41.933944142 +0000 UTC m=+39.027073581" Nov 03 19:36:41.952150 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:41.952114 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" podStartSLOduration=1.952099947 podStartE2EDuration="1.952099947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:41.951598384 +0000 UTC m=+39.044727824" watchObservedRunningTime="2026-11-03 19:36:41.952099947 +0000 UTC m=+39.045229388" Nov 03 19:36:42.888279 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:42.888199 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b4x9q" event={"ID":"c95249a6-899f-4faf-82dc-6e553453ff80","Type":"ContainerStarted","Data":"fa5cb5a843991b3495175e8ea4acb78312827714ddb7e10fc3fc15b84c49b22f"} Nov 03 19:36:42.906440 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:42.906393 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b4x9q" podStartSLOduration=34.617670867 podStartE2EDuration="38.906378502s" totalImagesPullingTime="4.288707635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:42.905496385 +0000 UTC m=+39.998625822" watchObservedRunningTime="2026-11-03 19:36:42.906378502 +0000 UTC m=+39.999507944" Nov 03 19:36:43.209091 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:43.209024 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4d4ll_94d082d5-1c33-4394-9d93-67645f9b60cd/dns-node-resolver/0.log" Nov 03 19:36:43.763328 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:43.763284 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:43.763494 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:43.763391 2580 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:43.763807 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:43.763785 2580 scope.go:122] "RemoveContainer" containerID="7bbcf23cd573d5c9e806dd6604b68ae67663631e74f89f722cb793dfee52aa69" Nov 03 19:36:43.764037 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:43.764014 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-hmc5z_openshift-console-operator(f8b140c9-9e09-4cbd-847b-82ee1e3be6ac)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" podUID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" Nov 03 19:36:43.887297 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:43.887266 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:43.887443 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:43.887374 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:36:43.887443 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:43.887397 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/377ffcf5-b097-495d-b986-e61c58ae8ec1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:43.887551 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:43.887446 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls podName:377ffcf5-b097-495d-b986-e61c58ae8ec1 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:47.887429137 +0000 UTC m=+44.980558570 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hmnx4" (UID: "377ffcf5-b097-495d-b986-e61c58ae8ec1") : secret "insights-runtime-extractor-tls" not found Nov 03 19:36:43.887803 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:43.887783 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 19:36:43.887863 ip-10-0-132-83 kubenswrapper[2580]: E1103 19:36:43.887850 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/377ffcf5-b097-495d-b986-e61c58ae8ec1-kube-rbac-proxy-cm podName:377ffcf5-b097-495d-b986-e61c58ae8ec1 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:47.887834416 +0000 UTC m=+44.980963834 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/377ffcf5-b097-495d-b986-e61c58ae8ec1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hmnx4" (UID: "377ffcf5-b097-495d-b986-e61c58ae8ec1") : configmap "kube-rbac-proxy" not found Nov 03 19:36:43.893578 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:43.893546 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6k9n9" event={"ID":"f485893c-6f14-449b-b837-8ac181315aeb","Type":"ContainerStarted","Data":"5d874b39931c2408a3ab4748545ce2ab475106ee717ad14248ee3ca358ef09d0"} Nov 03 19:36:43.895303 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:43.895279 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pbgwk" event={"ID":"815b15b6-2c37-4ca3-9b38-e7e1328d0aa8","Type":"ContainerStarted","Data":"c3bd71ffe2f7af1614a32a2b8d3f234a09fc878537716084e68bd018236a5528"} Nov 03 19:36:43.895592 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:43.895571 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:43.912415 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:43.912332 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6k9n9" podStartSLOduration=2.5976628330000002 podStartE2EDuration="4.912320257s" totalImagesPullingTime="2.314657424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:43.910932345 +0000 UTC m=+41.004061818" watchObservedRunningTime="2026-11-03 19:36:43.912320257 +0000 UTC m=+41.005449724" Nov 03 19:36:43.929175 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:43.929140 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pbgwk" podStartSLOduration=2.610284617 podStartE2EDuration="4.929130764s" totalImagesPullingTime="2.318846147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:43.928830715 +0000 UTC m=+41.021960155" watchObservedRunningTime="2026-11-03 19:36:43.929130764 +0000 UTC m=+41.022260204" Nov 03 19:36:44.208262 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:44.208232 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kzr2l_2398ba99-35d3-4461-b66e-f44ac013b2d2/node-ca/0.log" Nov 03 19:36:45.008026 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:45.007975 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fgdh8_18ffe451-b8eb-41ad-a19d-0390f3e1f8e2/serve-healthcheck-canary/0.log" Nov 03 19:36:45.208957 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:45.208932 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-6k9n9_f485893c-6f14-449b-b837-8ac181315aeb/migrator/0.log" Nov 03 19:36:45.408836 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:45.408813 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-pbgwk_815b15b6-2c37-4ca3-9b38-e7e1328d0aa8/migrator/0.log" Nov 03 19:36:45.609597 ip-10-0-132-83 kubenswrapper[2580]: I1103 19:36:45.609572 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-wh95x_492c7b4e-84c0-49a7-b12f-153a021f971b/kube-storage-version-migrator-operator/0.log" Nov 03 19:36:45.659873 ip-10-0-132-83 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 19:36:45.676029 ip-10-0-132-83 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 19:36:45.676193 ip-10-0-132-83 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 19:36:45.676239 ip-10-0-132-83 systemd[1]: kubelet.service: Consumed 3.091s CPU time. Nov 03 19:36:45.694134 ip-10-0-132-83 systemd[1]: Starting Kubernetes Kubelet... Nov 03 19:36:45.771104 ip-10-0-132-83 kubenswrapper[5059]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:36:45.771104 ip-10-0-132-83 kubenswrapper[5059]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 19:36:45.771104 ip-10-0-132-83 kubenswrapper[5059]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:36:45.771104 ip-10-0-132-83 kubenswrapper[5059]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:36:45.773296 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773209 5059 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:36:45.773296 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773291 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:36:45.773296 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773297 5059 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:36:45.773296 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773300 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:36:45.773448 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773304 5059 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:36:45.773448 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773307 5059 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:36:45.773448 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773310 5059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:36:45.773448 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773312 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:36:45.773448 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773315 5059 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:36:45.773448 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773318 5059 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:36:45.773448 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773321 5059 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:36:45.773448 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773323 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:36:45.773448 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773326 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:36:45.773448 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773329 5059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:36:45.773448 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773331 5059 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:36:45.773448 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773334 5059 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:36:45.773448 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773337 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:36:45.773448 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773339 5059 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:36:45.773448 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773342 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:36:45.773448 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773345 5059 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:36:45.773448 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773348 5059 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:36:45.773448 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773350 5059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:36:45.773448 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773353 5059 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:36:45.773930 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773356 5059 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:36:45.773930 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773359 5059 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:36:45.773930 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773362 5059 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:36:45.773930 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773367 5059 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:36:45.773930 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773377 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:36:45.773930 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773380 5059 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:36:45.773930 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773383 5059 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:36:45.773930 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773386 5059 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:36:45.773930 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773389 5059 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:36:45.773930 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773392 5059 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:36:45.773930 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773395 5059 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:36:45.773930 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773398 5059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:36:45.773930 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773400 5059 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:36:45.773930 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773403 5059 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:36:45.773930 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773406 5059 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:36:45.773930 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773408 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:36:45.773930 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773411 5059 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:36:45.773930 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773415 5059 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:36:45.773930 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773419 5059 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:36:45.774396 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773422 5059 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:36:45.774396 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773425 5059 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:36:45.774396 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773428 5059 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:36:45.774396 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773431 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:36:45.774396 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773434 5059 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:36:45.774396 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773437 5059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:36:45.774396 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773440 5059 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:36:45.774396 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773442 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:36:45.774396 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773446 5059 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:36:45.774396 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773449 5059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:36:45.774396 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773451 5059 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:36:45.774396 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773454 5059 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:36:45.774396 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773457 5059 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:36:45.774396 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773460 5059 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:36:45.774396 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773462 5059 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:36:45.774396 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773471 5059 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:36:45.774396 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773474 5059 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:36:45.774396 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773478 5059 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:36:45.774396 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773481 5059 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:36:45.774396 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773483 5059 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:36:45.774928 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773486 5059 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:36:45.774928 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773489 5059 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:36:45.774928 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773492 5059 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:36:45.774928 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773494 5059 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:36:45.774928 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773497 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:36:45.774928 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773500 5059 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:36:45.774928 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773503 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:36:45.774928 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773506 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:36:45.774928 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773508 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:36:45.774928 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773511 5059 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:36:45.774928 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773514 5059 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:36:45.774928 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773516 5059 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:36:45.774928 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773519 5059 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:36:45.774928 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773521 5059 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:36:45.774928 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773524 5059 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:36:45.774928 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773526 5059 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:36:45.774928 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773529 5059 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:36:45.774928 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773532 5059 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:36:45.774928 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773535 5059 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:36:45.774928 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773538 5059 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:36:45.775431 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773540 5059 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:36:45.775431 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773543 5059 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:36:45.775431 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773545 5059 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:36:45.775431 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773548 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:36:45.775431 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773551 5059 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:36:45.775431 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773553 5059 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:36:45.775431 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773556 5059 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:36:45.775431 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773559 5059 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:36:45.775431 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773561 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:36:45.775431 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773564 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:36:45.775431 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773567 5059 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:36:45.775431 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773569 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:36:45.775431 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773572 5059 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:36:45.775431 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773574 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:36:45.775431 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773577 5059 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:36:45.775431 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773580 5059 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:36:45.775431 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773582 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:36:45.775431 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773586 5059 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:36:45.775431 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773589 5059 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:36:45.775948 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773591 5059 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:36:45.775948 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773594 5059 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:36:45.775948 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773597 5059 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:36:45.775948 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773600 5059 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:36:45.775948 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.773603 5059 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:36:45.775948 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774393 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:36:45.775948 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774402 5059 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:36:45.775948 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774405 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:36:45.775948 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774408 5059 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:36:45.775948 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774411 5059 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:36:45.775948 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774414 5059 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:36:45.775948 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774417 5059 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:36:45.775948 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774420 5059 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:36:45.775948 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774423 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:36:45.775948 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774426 5059 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:36:45.775948 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774428 5059 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:36:45.775948 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774431 5059 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:36:45.775948 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774434 5059 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:36:45.775948 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774437 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:36:45.775948 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774440 5059 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:36:45.776452 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774443 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:36:45.776452 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774445 5059 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:36:45.776452 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774448 5059 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:36:45.776452 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774451 5059 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:36:45.776452 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774453 5059 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:36:45.776452 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774456 5059 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:36:45.776452 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774458 5059 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:36:45.776452 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774461 5059 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:36:45.776452 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774464 5059 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:36:45.776452 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774466 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:36:45.776452 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774470 5059 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:36:45.776452 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774473 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:36:45.776452 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774475 5059 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:36:45.776452 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774478 5059 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:36:45.776452 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774481 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:36:45.776452 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774483 5059 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:36:45.776452 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774486 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:36:45.776452 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774489 5059 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:36:45.776452 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774491 5059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:36:45.777007 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774494 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:36:45.777007 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774496 5059 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:36:45.777007 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774499 5059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:36:45.777007 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774501 5059 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:36:45.777007 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774504 5059 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:36:45.777007 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774507 5059 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:36:45.777007 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774510 5059 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:36:45.777007 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774512 5059 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:36:45.777007 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774515 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:36:45.777007 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774518 5059 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:36:45.777007 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774521 5059 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:36:45.777007 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774525 5059 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:36:45.777007 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774530 5059 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:36:45.777007 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774533 5059 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:36:45.777007 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774536 5059 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:36:45.777007 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774539 5059 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:36:45.777007 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774541 5059 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:36:45.777007 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774544 5059 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:36:45.777007 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774546 5059 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:36:45.777007 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774549 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:36:45.777500 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774552 5059 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:36:45.777500 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774554 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:36:45.777500 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774557 5059 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:36:45.777500 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774559 5059 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:36:45.777500 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774563 5059 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:36:45.777500 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774566 5059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:36:45.777500 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774570 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:36:45.777500 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774573 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:36:45.777500 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774575 5059 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:36:45.777500 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774578 5059 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:36:45.777500 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774581 5059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:36:45.777500 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774583 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:36:45.777500 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774586 5059 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:36:45.777500 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774588 5059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:36:45.777500 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774591 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:36:45.777500 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774594 5059 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:36:45.777500 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774596 5059 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:36:45.777500 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774599 5059 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:36:45.777500 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774602 5059 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:36:45.777986 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774605 5059 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:36:45.777986 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774610 5059 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:36:45.777986 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774613 5059 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:36:45.777986 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774616 5059 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:36:45.777986 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774618 5059 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:36:45.777986 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774621 5059 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:36:45.777986 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774624 5059 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:36:45.777986 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774626 5059 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:36:45.777986 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774629 5059 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:36:45.777986 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774632 5059 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:36:45.777986 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774634 5059 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:36:45.777986 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774637 5059 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:36:45.777986 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774640 5059 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:36:45.777986 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774642 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:36:45.777986 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774645 5059 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:36:45.777986 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774647 5059 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:36:45.777986 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774651 5059 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:36:45.777986 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774653 5059 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:36:45.777986 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774690 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:36:45.777986 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774693 5059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:36:45.778522 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774696 5059 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:36:45.778522 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774698 5059 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:36:45.778522 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774701 5059 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:36:45.778522 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774705 5059 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:36:45.778522 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774708 5059 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:36:45.778522 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774710 5059 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:36:45.778522 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774713 5059 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:36:45.778522 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774716 5059 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:36:45.778522 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774718 5059 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:36:45.778522 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774721 5059 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:36:45.778522 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774723 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:36:45.778522 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774726 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:36:45.778522 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.774734 5059 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:36:45.778522 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775242 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:36:45.778522 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775248 5059 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:36:45.778522 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775251 5059 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:36:45.778522 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775254 5059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:36:45.778522 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775257 5059 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:36:45.778522 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775260 5059 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:36:45.778522 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775263 5059 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:36:45.779073 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775266 5059 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:36:45.779073 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775268 5059 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:36:45.779073 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775271 5059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:36:45.779073 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775274 5059 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:36:45.779073 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775277 5059 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:36:45.779073 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775281 5059 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:36:45.779073 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775285 5059 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:36:45.779073 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775288 5059 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:36:45.779073 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775291 5059 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:36:45.779073 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775294 5059 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:36:45.779073 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775297 5059 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:36:45.779073 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775299 5059 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:36:45.779073 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775302 5059 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:36:45.779073 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775304 5059 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:36:45.779073 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775307 5059 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:36:45.779073 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775310 5059 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:36:45.779073 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775312 5059 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:36:45.779073 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775315 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:36:45.779073 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775318 5059 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:36:45.779537 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775321 5059 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:36:45.779537 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775324 5059 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:36:45.779537 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775326 5059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:36:45.779537 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775329 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:36:45.779537 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775332 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:36:45.779537 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775335 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:36:45.779537 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775340 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:36:45.779537 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775342 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:36:45.779537 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775345 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:36:45.779537 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775348 5059 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:36:45.779537 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775350 5059 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:36:45.779537 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775353 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:36:45.779537 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775355 5059 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:36:45.779537 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775358 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:36:45.779537 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775360 5059 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:36:45.779537 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775363 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:36:45.779537 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775366 5059 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:36:45.779537 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775368 5059 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:36:45.779537 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775371 5059 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:36:45.779537 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775373 5059 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:36:45.780052 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775376 5059 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:36:45.780052 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775379 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:36:45.780052 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775381 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:36:45.780052 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775384 5059 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:36:45.780052 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775389 5059 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:36:45.780052 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775392 5059 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:36:45.780052 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775395 5059 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:36:45.780052 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775398 5059 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:36:45.780052 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775400 5059 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:36:45.780052 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775403 5059 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:36:45.780052 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775405 5059 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:36:45.780052 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775408 5059 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:36:45.780052 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775411 5059 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:36:45.780052 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775413 5059 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:36:45.780052 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775416 5059 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:36:45.780052 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775419 5059 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:36:45.780052 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775421 5059 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:36:45.780052 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775424 5059 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:36:45.780052 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775429 5059 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:36:45.780052 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775432 5059 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:36:45.780548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775435 5059 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:36:45.780548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775438 5059 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:36:45.780548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775441 5059 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:36:45.780548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775443 5059 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:36:45.780548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775446 5059 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:36:45.780548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775449 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:36:45.780548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775451 5059 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:36:45.780548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775454 5059 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:36:45.780548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775456 5059 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:36:45.780548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775459 5059 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:36:45.780548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775462 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:36:45.780548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775465 5059 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:36:45.780548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775467 5059 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:36:45.780548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775470 5059 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:36:45.780548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775473 5059 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:36:45.780548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775475 5059 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:36:45.780548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775479 5059 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:36:45.780548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775482 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:36:45.780548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775484 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:36:45.780548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775487 5059 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:36:45.781043 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775489 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:36:45.781043 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775492 5059 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:36:45.781043 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775495 5059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:36:45.781043 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775497 5059 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:36:45.781043 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775500 5059 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:36:45.781043 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775502 5059 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:36:45.781043 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775505 5059 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:36:45.781043 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775507 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:36:45.781043 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775510 5059 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:36:45.781043 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775512 5059 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:36:45.781043 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775516 5059 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:36:45.781043 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775519 5059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:36:45.781043 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775521 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:36:45.781043 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775525 5059 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:36:45.781043 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775529 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:36:45.781043 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775532 5059 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:36:45.781043 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775535 5059 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:36:45.781043 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775538 5059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:36:45.781043 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775542 5059 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:36:45.781525 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.775544 5059 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:36:45.781525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775640 5059 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 19:36:45.781525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775650 5059 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 19:36:45.781525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775671 5059 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 19:36:45.781525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775677 5059 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 19:36:45.781525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775682 5059 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 19:36:45.781525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775685 5059 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 19:36:45.781525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775689 5059 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 19:36:45.781525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775693 5059 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 19:36:45.781525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775697 5059 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 19:36:45.781525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775700 5059 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 19:36:45.781525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775704 5059 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 19:36:45.781525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775707 5059 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 19:36:45.781525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775711 5059 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 19:36:45.781525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775714 5059 flags.go:64] FLAG: --cgroup-root="" Nov 03 19:36:45.781525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775717 5059 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 19:36:45.781525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775720 5059 flags.go:64] FLAG: --client-ca-file="" Nov 03 19:36:45.781525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775723 5059 flags.go:64] FLAG: --cloud-provider="external" Nov 03 19:36:45.781525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775726 5059 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 19:36:45.781525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775731 5059 flags.go:64] FLAG: --cluster-domain="" Nov 03 19:36:45.781525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775734 5059 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 19:36:45.781525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775738 5059 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 19:36:45.781525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775741 5059 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 19:36:45.781525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775745 5059 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775749 5059 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775753 5059 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775757 5059 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775760 5059 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775763 5059 flags.go:64] FLAG: --contention-profiling="false" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775767 5059 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775770 5059 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775773 5059 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775776 5059 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775781 5059 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775784 5059 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775787 5059 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775790 5059 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775793 5059 flags.go:64] FLAG: --enable-server="true" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775796 5059 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775800 5059 flags.go:64] FLAG: --event-burst="100" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775803 5059 flags.go:64] FLAG: --event-qps="50" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775807 5059 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775810 5059 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775813 5059 flags.go:64] FLAG: --eviction-hard="" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775817 5059 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775820 5059 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775824 5059 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775827 5059 flags.go:64] FLAG: --eviction-soft="" Nov 03 19:36:45.782132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775830 5059 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 19:36:45.782761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775834 5059 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 19:36:45.782761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775837 5059 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 19:36:45.782761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775840 5059 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 19:36:45.782761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775843 5059 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 19:36:45.782761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775846 5059 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 19:36:45.782761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775849 5059 flags.go:64] FLAG: --feature-gates="" Nov 03 19:36:45.782761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775853 5059 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 19:36:45.782761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775856 5059 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 19:36:45.782761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775859 5059 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 19:36:45.782761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775863 5059 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 19:36:45.782761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775867 5059 flags.go:64] FLAG: --healthz-port="10248" Nov 03 19:36:45.782761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775870 5059 flags.go:64] FLAG: --help="false" Nov 03 19:36:45.782761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775874 5059 flags.go:64] FLAG: --hostname-override="ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:45.782761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775877 5059 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 19:36:45.782761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775880 5059 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 19:36:45.782761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775884 5059 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 19:36:45.782761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775887 5059 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 19:36:45.782761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775907 5059 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 19:36:45.782761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775911 5059 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 19:36:45.782761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775915 5059 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 19:36:45.782761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775918 5059 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 19:36:45.782761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775922 5059 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 19:36:45.782761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775927 5059 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775931 5059 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775934 5059 flags.go:64] FLAG: --kube-reserved="" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775937 5059 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775940 5059 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775943 5059 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775946 5059 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775949 5059 flags.go:64] FLAG: --lock-file="" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775953 5059 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775956 5059 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775959 5059 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775965 5059 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775968 5059 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775971 5059 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775974 5059 flags.go:64] FLAG: --logging-format="text" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775977 5059 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775981 5059 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775984 5059 flags.go:64] FLAG: --manifest-url="" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775987 5059 flags.go:64] FLAG: --manifest-url-header="" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775992 5059 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.775995 5059 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776000 5059 flags.go:64] FLAG: --max-pods="110" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776003 5059 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776006 5059 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776009 5059 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 19:36:45.783381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776012 5059 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776015 5059 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776018 5059 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776022 5059 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776029 5059 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776032 5059 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776035 5059 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776040 5059 flags.go:64] FLAG: --pod-cidr="" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776043 5059 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776046 5059 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776049 5059 flags.go:64] FLAG: --pods-per-core="0" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776052 5059 flags.go:64] FLAG: --port="10250" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776057 5059 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776060 5059 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-094694846601f51d3" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776063 5059 flags.go:64] FLAG: --qos-reserved="" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776066 5059 flags.go:64] FLAG: --read-only-port="10255" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776069 5059 flags.go:64] FLAG: --register-node="true" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776072 5059 flags.go:64] FLAG: --register-with-taints="" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776076 5059 flags.go:64] FLAG: --registry-burst="10" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776079 5059 flags.go:64] FLAG: --registry-qps="5" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776082 5059 flags.go:64] FLAG: --reserved-cpus="" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776085 5059 flags.go:64] FLAG: --reserved-memory="" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776089 5059 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776092 5059 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776095 5059 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776098 5059 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 19:36:45.784391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776101 5059 flags.go:64] FLAG: --runonce="false" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776104 5059 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776108 5059 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776111 5059 flags.go:64] FLAG: --seccomp-default="false" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776114 5059 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776117 5059 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776120 5059 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776123 5059 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776127 5059 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776130 5059 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776133 5059 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776136 5059 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776138 5059 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776143 5059 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776146 5059 flags.go:64] FLAG: --system-cgroups="" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776149 5059 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776154 5059 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776157 5059 flags.go:64] FLAG: --tls-cert-file="" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776162 5059 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776166 5059 flags.go:64] FLAG: --tls-min-version="" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776169 5059 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776172 5059 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776175 5059 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776178 5059 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776181 5059 flags.go:64] FLAG: --v="2" Nov 03 19:36:45.785056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776186 5059 flags.go:64] FLAG: --version="false" Nov 03 19:36:45.785704 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776190 5059 flags.go:64] FLAG: --vmodule="" Nov 03 19:36:45.785704 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776194 5059 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 19:36:45.785704 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776197 5059 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 19:36:45.785704 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776333 5059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:36:45.785704 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776338 5059 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:36:45.785704 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776341 5059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:36:45.785704 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776344 5059 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:36:45.785704 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776347 5059 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:36:45.785704 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776349 5059 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:36:45.785704 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776353 5059 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:36:45.785704 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776356 5059 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:36:45.785704 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776358 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:36:45.785704 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776361 5059 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:36:45.785704 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776364 5059 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:36:45.785704 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776366 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:36:45.785704 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776369 5059 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:36:45.785704 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776371 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:36:45.785704 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776374 5059 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:36:45.785704 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776377 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:36:45.785704 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776381 5059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:36:45.786213 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776383 5059 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:36:45.786213 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776386 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:36:45.786213 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776388 5059 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:36:45.786213 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776391 5059 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:36:45.786213 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776396 5059 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:36:45.786213 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776399 5059 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:36:45.786213 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776401 5059 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:36:45.786213 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776404 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:36:45.786213 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776407 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:36:45.786213 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776409 5059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:36:45.786213 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776412 5059 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:36:45.786213 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776415 5059 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:36:45.786213 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776418 5059 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:36:45.786213 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776420 5059 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:36:45.786213 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776423 5059 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:36:45.786213 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776426 5059 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:36:45.786213 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776428 5059 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:36:45.786213 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776431 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:36:45.786213 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776434 5059 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:36:45.786213 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776436 5059 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:36:45.787078 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776439 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:36:45.787078 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776442 5059 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:36:45.787078 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776445 5059 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:36:45.787078 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776447 5059 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:36:45.787078 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776450 5059 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:36:45.787078 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776453 5059 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:36:45.787078 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776456 5059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:36:45.787078 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776459 5059 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:36:45.787078 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776461 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:36:45.787078 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776464 5059 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:36:45.787078 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776467 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:36:45.787078 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776470 5059 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:36:45.787078 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776473 5059 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:36:45.787078 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776476 5059 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:36:45.787078 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776479 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:36:45.787078 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776481 5059 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:36:45.787078 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776485 5059 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:36:45.787078 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776488 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:36:45.787078 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776491 5059 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:36:45.787712 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776494 5059 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:36:45.787712 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776497 5059 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:36:45.787712 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776500 5059 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:36:45.787712 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776502 5059 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:36:45.787712 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776505 5059 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:36:45.787712 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776508 5059 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:36:45.787712 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776512 5059 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:36:45.787712 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776515 5059 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:36:45.787712 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776519 5059 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:36:45.787712 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776522 5059 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:36:45.787712 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776524 5059 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:36:45.787712 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776528 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:36:45.787712 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776531 5059 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:36:45.787712 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776533 5059 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:36:45.787712 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776536 5059 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:36:45.787712 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776539 5059 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:36:45.787712 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776542 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:36:45.787712 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776545 5059 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:36:45.787712 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776548 5059 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:36:45.787712 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776550 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:36:45.788309 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776553 5059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:36:45.788309 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776555 5059 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:36:45.788309 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776558 5059 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:36:45.788309 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776560 5059 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:36:45.788309 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776564 5059 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:36:45.788309 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776567 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:36:45.788309 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776570 5059 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:36:45.788309 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776572 5059 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:36:45.788309 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776575 5059 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:36:45.788309 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776580 5059 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:36:45.788309 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776583 5059 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:36:45.788309 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776586 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:36:45.788309 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776588 5059 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:36:45.788309 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776591 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:36:45.788309 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776594 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:36:45.788309 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776596 5059 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:36:45.788309 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776599 5059 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:36:45.788309 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776601 5059 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:36:45.788309 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776604 5059 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:36:45.788798 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776607 5059 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:36:45.788798 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776609 5059 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:36:45.788798 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776612 5059 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:36:45.788798 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776616 5059 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:36:45.788798 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776619 5059 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:36:45.788798 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776622 5059 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:36:45.788798 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776625 5059 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:36:45.788798 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776628 5059 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:36:45.788798 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776630 5059 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:36:45.788798 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776633 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:36:45.788798 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776636 5059 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:36:45.788798 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.776676 5059 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:36:45.788798 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776713 5059 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:36:45.788798 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776717 5059 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:36:45.788798 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776721 5059 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:36:45.789173 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776724 5059 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:36:45.789173 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776726 5059 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:36:45.789173 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776729 5059 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:36:45.789173 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776734 5059 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:36:45.789173 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776738 5059 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:36:45.789173 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776742 5059 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:36:45.789173 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776744 5059 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:36:45.789173 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776749 5059 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:36:45.789173 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776752 5059 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:36:45.789173 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776754 5059 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:36:45.789173 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776757 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:36:45.789173 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776760 5059 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:36:45.789173 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776762 5059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:36:45.789173 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776765 5059 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:36:45.789173 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776767 5059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:36:45.789173 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776770 5059 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:36:45.789173 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776773 5059 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:36:45.789173 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776776 5059 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:36:45.789173 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776778 5059 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:36:45.789736 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776781 5059 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:36:45.789736 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776784 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:36:45.789736 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776786 5059 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:36:45.789736 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776789 5059 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:36:45.789736 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776791 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:36:45.789736 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776794 5059 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:36:45.789736 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776797 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:36:45.789736 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776799 5059 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:36:45.789736 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776802 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:36:45.789736 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776805 5059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:36:45.789736 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776807 5059 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:36:45.789736 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776810 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:36:45.789736 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776813 5059 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:36:45.789736 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776815 5059 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:36:45.789736 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776818 5059 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:36:45.789736 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776821 5059 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:36:45.789736 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776825 5059 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:36:45.789736 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776828 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:36:45.789736 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776830 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:36:45.789736 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776833 5059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:36:45.790378 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776837 5059 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:36:45.790378 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776839 5059 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:36:45.790378 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776842 5059 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:36:45.790378 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776846 5059 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:36:45.790378 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776848 5059 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:36:45.790378 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776851 5059 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:36:45.790378 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776854 5059 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:36:45.790378 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776856 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:36:45.790378 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776859 5059 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:36:45.790378 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776862 5059 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:36:45.790378 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776864 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:36:45.790378 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776867 5059 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:36:45.790378 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776869 5059 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:36:45.790378 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776872 5059 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:36:45.790378 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776875 5059 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:36:45.790378 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776877 5059 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:36:45.790378 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776880 5059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:36:45.790378 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776882 5059 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:36:45.790378 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776885 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:36:45.790378 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776887 5059 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:36:45.791002 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776890 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:36:45.791002 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776892 5059 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:36:45.791002 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776895 5059 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:36:45.791002 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776898 5059 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:36:45.791002 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776900 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:36:45.791002 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776903 5059 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:36:45.791002 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776906 5059 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:36:45.791002 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776908 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:36:45.791002 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776913 5059 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:36:45.791002 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776915 5059 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:36:45.791002 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776918 5059 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:36:45.791002 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776921 5059 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:36:45.791002 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776925 5059 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:36:45.791002 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776928 5059 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:36:45.791002 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776931 5059 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:36:45.791002 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776935 5059 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:36:45.791002 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776938 5059 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:36:45.791002 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776940 5059 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:36:45.791002 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776943 5059 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:36:45.791465 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776946 5059 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:36:45.791465 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776948 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:36:45.791465 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776951 5059 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:36:45.791465 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776954 5059 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:36:45.791465 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776957 5059 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:36:45.791465 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776959 5059 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:36:45.791465 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776962 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:36:45.791465 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776965 5059 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:36:45.791465 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776968 5059 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:36:45.791465 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776971 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:36:45.791465 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776973 5059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:36:45.791465 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776976 5059 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:36:45.791465 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776978 5059 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:36:45.791465 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776981 5059 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:36:45.791465 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776983 5059 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:36:45.791465 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776986 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:36:45.791465 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776989 5059 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:36:45.791465 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776991 5059 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:36:45.791465 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776994 5059 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:36:45.791940 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776996 5059 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:36:45.791940 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.776999 5059 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:36:45.791940 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.777003 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:36:45.791940 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.777005 5059 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:36:45.791940 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.777008 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:36:45.791940 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.777011 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:36:45.791940 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.781199 5059 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 19:36:45.791940 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.781214 5059 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 19:36:45.791940 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.781231 5059 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 19:36:45.791940 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.781236 5059 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 19:36:45.791940 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781308 5059 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:36:45.791940 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781312 5059 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:36:45.791940 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781316 5059 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:36:45.791940 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781320 5059 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:36:45.791940 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781323 5059 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:36:45.791940 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781327 5059 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:36:45.791940 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781329 5059 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:36:45.791940 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781332 5059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:36:45.791940 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781335 5059 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:36:45.792420 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781338 5059 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:36:45.792420 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781340 5059 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:36:45.792420 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781343 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:36:45.792420 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781346 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:36:45.792420 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781349 5059 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:36:45.792420 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781351 5059 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:36:45.792420 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781354 5059 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:36:45.792420 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781356 5059 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:36:45.792420 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781359 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:36:45.792420 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781362 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:36:45.792420 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781364 5059 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:36:45.792420 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781367 5059 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:36:45.792420 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781370 5059 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:36:45.792420 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781378 5059 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:36:45.792420 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781381 5059 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:36:45.792420 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781385 5059 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:36:45.792420 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781387 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:36:45.792420 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781390 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:36:45.792420 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781393 5059 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:36:45.792420 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781396 5059 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:36:45.792944 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781399 5059 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:36:45.792944 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781401 5059 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:36:45.792944 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781404 5059 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:36:45.792944 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781406 5059 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:36:45.792944 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781409 5059 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:36:45.792944 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781413 5059 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:36:45.792944 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781417 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:36:45.792944 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781420 5059 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:36:45.792944 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781422 5059 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:36:45.792944 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781425 5059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:36:45.792944 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781428 5059 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:36:45.792944 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781431 5059 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:36:45.792944 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781433 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:36:45.792944 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781442 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:36:45.792944 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781445 5059 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:36:45.792944 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781448 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:36:45.792944 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781450 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:36:45.792944 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781453 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:36:45.792944 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781456 5059 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:36:45.793395 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781458 5059 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:36:45.793395 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781460 5059 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:36:45.793395 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781463 5059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:36:45.793395 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781466 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:36:45.793395 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781468 5059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:36:45.793395 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781471 5059 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:36:45.793395 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781473 5059 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:36:45.793395 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781482 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:36:45.793395 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781485 5059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:36:45.793395 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781487 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:36:45.793395 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781490 5059 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:36:45.793395 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781492 5059 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:36:45.793395 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781495 5059 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:36:45.793395 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781498 5059 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:36:45.793395 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781500 5059 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:36:45.793395 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781503 5059 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:36:45.793395 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781505 5059 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:36:45.793395 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781508 5059 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:36:45.793395 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781512 5059 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:36:45.793856 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781516 5059 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:36:45.793856 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781519 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:36:45.793856 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781522 5059 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:36:45.793856 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781524 5059 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:36:45.793856 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781527 5059 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:36:45.793856 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781530 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:36:45.793856 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781533 5059 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:36:45.793856 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781536 5059 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:36:45.793856 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781539 5059 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:36:45.793856 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781541 5059 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:36:45.793856 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781544 5059 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:36:45.793856 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781547 5059 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:36:45.793856 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781550 5059 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:36:45.793856 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781552 5059 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:36:45.793856 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781555 5059 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:36:45.793856 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781557 5059 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:36:45.793856 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781560 5059 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:36:45.793856 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781563 5059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:36:45.793856 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781568 5059 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:36:45.793856 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781572 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:36:45.794333 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781576 5059 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:36:45.794333 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781588 5059 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:36:45.794333 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781592 5059 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:36:45.794333 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781597 5059 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:36:45.794333 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781601 5059 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:36:45.794333 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781630 5059 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:36:45.794333 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781633 5059 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:36:45.794333 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781636 5059 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:36:45.794333 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781638 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:36:45.794333 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781641 5059 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:36:45.794333 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781644 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:36:45.794333 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781646 5059 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:36:45.794333 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781649 5059 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:36:45.794333 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781651 5059 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:36:45.794333 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781654 5059 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:36:45.794333 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781657 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:36:45.794333 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781675 5059 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:36:45.794333 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781678 5059 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:36:45.794333 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781681 5059 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:36:45.794842 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.781712 5059 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:36:45.794842 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781905 5059 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:36:45.794842 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781910 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:36:45.794842 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781913 5059 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:36:45.794842 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781916 5059 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:36:45.794842 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781919 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:36:45.794842 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781921 5059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:36:45.794842 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781924 5059 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:36:45.794842 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781927 5059 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:36:45.794842 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781929 5059 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:36:45.794842 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781932 5059 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:36:45.794842 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781935 5059 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:36:45.794842 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781938 5059 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:36:45.794842 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781940 5059 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:36:45.794842 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781943 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:36:45.794842 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781952 5059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:36:45.795292 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781955 5059 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:36:45.795292 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781958 5059 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:36:45.795292 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781960 5059 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:36:45.795292 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781963 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:36:45.795292 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781966 5059 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:36:45.795292 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781968 5059 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:36:45.795292 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781971 5059 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:36:45.795292 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781974 5059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:36:45.795292 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781976 5059 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:36:45.795292 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781979 5059 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:36:45.795292 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781981 5059 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:36:45.795292 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781984 5059 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:36:45.795292 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781986 5059 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:36:45.795292 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781989 5059 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:36:45.795292 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781992 5059 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:36:45.795292 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781995 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:36:45.795292 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.781997 5059 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:36:45.795292 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782000 5059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:36:45.795292 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782002 5059 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:36:45.795292 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782005 5059 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:36:45.795853 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782007 5059 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:36:45.795853 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782010 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:36:45.795853 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782013 5059 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:36:45.795853 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782015 5059 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:36:45.795853 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782018 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:36:45.795853 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782020 5059 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:36:45.795853 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782023 5059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:36:45.795853 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782025 5059 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:36:45.795853 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782028 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:36:45.795853 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782031 5059 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:36:45.795853 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782034 5059 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:36:45.795853 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782036 5059 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:36:45.795853 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782045 5059 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:36:45.795853 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782048 5059 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:36:45.795853 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782050 5059 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:36:45.795853 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782053 5059 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:36:45.795853 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782055 5059 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:36:45.795853 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782058 5059 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:36:45.795853 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782061 5059 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:36:45.796548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782063 5059 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:36:45.796548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782067 5059 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:36:45.796548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782071 5059 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:36:45.796548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782074 5059 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:36:45.796548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782077 5059 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:36:45.796548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782079 5059 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:36:45.796548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782082 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:36:45.796548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782085 5059 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:36:45.796548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782088 5059 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:36:45.796548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782090 5059 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:36:45.796548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782093 5059 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:36:45.796548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782095 5059 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:36:45.796548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782098 5059 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:36:45.796548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782100 5059 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:36:45.796548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782103 5059 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:36:45.796548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782113 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:36:45.796548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782116 5059 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:36:45.796548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782119 5059 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:36:45.796548 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782121 5059 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:36:45.797216 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782124 5059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:36:45.797216 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782127 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:36:45.797216 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782129 5059 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:36:45.797216 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782133 5059 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:36:45.797216 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782137 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:36:45.797216 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782140 5059 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:36:45.797216 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782143 5059 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:36:45.797216 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782151 5059 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:36:45.797216 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782154 5059 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:36:45.797216 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782157 5059 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:36:45.797216 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782159 5059 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:36:45.797216 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782162 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:36:45.797216 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782164 5059 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:36:45.797216 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782167 5059 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:36:45.797216 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782170 5059 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:36:45.797216 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782172 5059 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:36:45.797216 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782175 5059 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:36:45.797216 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782177 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:36:45.797216 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782180 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:36:45.797807 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782182 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:36:45.797807 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782185 5059 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:36:45.797807 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782187 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:36:45.797807 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782190 5059 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:36:45.797807 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782193 5059 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:36:45.797807 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782195 5059 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:36:45.797807 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782198 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:36:45.797807 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782200 5059 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:36:45.797807 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782203 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:36:45.797807 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782205 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:36:45.797807 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782208 5059 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:36:45.797807 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782210 5059 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:36:45.797807 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782213 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:36:45.797807 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782215 5059 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:36:45.797807 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.782238 5059 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:36:45.798374 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782280 5059 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:36:45.798374 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782285 5059 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:36:45.798374 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782288 5059 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:36:45.798374 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782290 5059 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:36:45.798374 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782293 5059 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:36:45.798374 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782295 5059 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:36:45.798374 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782307 5059 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:36:45.798374 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782310 5059 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:36:45.798374 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782313 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:36:45.798374 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782316 5059 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:36:45.798374 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782319 5059 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:36:45.798374 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782322 5059 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:36:45.798374 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782324 5059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:36:45.798374 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782327 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:36:45.798374 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782329 5059 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:36:45.798374 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782333 5059 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:36:45.798374 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782336 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:36:45.798374 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782339 5059 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:36:45.798374 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782342 5059 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:36:45.799132 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782345 5059 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:36:45.799132 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782348 5059 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:36:45.799132 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782350 5059 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:36:45.799132 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782353 5059 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:36:45.799132 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782355 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:36:45.799132 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782358 5059 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:36:45.799132 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782360 5059 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:36:45.799132 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782363 5059 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:36:45.799132 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782365 5059 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:36:45.799132 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782368 5059 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:36:45.799132 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782371 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:36:45.799132 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782373 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:36:45.799132 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782375 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:36:45.799132 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782378 5059 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:36:45.799132 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782381 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:36:45.799132 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782384 5059 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:36:45.799132 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782386 5059 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:36:45.799132 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782389 5059 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:36:45.799132 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782391 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:36:45.799965 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782394 5059 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:36:45.799965 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782404 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:36:45.799965 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782407 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:36:45.799965 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782410 5059 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:36:45.799965 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782412 5059 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:36:45.799965 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782415 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:36:45.799965 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782418 5059 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:36:45.799965 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782420 5059 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:36:45.799965 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782423 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:36:45.799965 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782426 5059 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:36:45.799965 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782428 5059 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:36:45.799965 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782431 5059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:36:45.799965 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782433 5059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:36:45.799965 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782436 5059 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:36:45.799965 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782439 5059 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:36:45.799965 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782448 5059 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:36:45.799965 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782452 5059 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:36:45.799965 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782455 5059 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:36:45.799965 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782457 5059 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:36:45.799965 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782460 5059 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:36:45.800554 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782463 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:36:45.800554 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782465 5059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:36:45.800554 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782468 5059 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:36:45.800554 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782470 5059 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:36:45.800554 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782473 5059 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:36:45.800554 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782475 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:36:45.800554 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782480 5059 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:36:45.800554 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782483 5059 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:36:45.800554 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782485 5059 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:36:45.800554 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782488 5059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:36:45.800554 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782491 5059 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:36:45.800554 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782493 5059 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:36:45.800554 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782496 5059 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:36:45.800554 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782500 5059 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:36:45.800554 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782508 5059 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:36:45.800554 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782511 5059 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:36:45.800554 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782513 5059 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:36:45.800554 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782516 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:36:45.800554 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782519 5059 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:36:45.800554 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782521 5059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:36:45.801063 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782524 5059 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:36:45.801063 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782526 5059 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:36:45.801063 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782529 5059 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:36:45.801063 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782532 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:36:45.801063 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782534 5059 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:36:45.801063 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782537 5059 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:36:45.801063 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782540 5059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:36:45.801063 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782543 5059 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:36:45.801063 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782545 5059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:36:45.801063 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782548 5059 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:36:45.801063 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782550 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:36:45.801063 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782553 5059 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:36:45.801063 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782556 5059 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:36:45.801063 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782558 5059 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:36:45.801063 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782561 5059 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:36:45.801063 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782563 5059 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:36:45.801063 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782566 5059 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:36:45.801063 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782569 5059 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:36:45.801063 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782573 5059 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:36:45.801063 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782575 5059 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:36:45.801593 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782578 5059 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:36:45.801593 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782580 5059 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:36:45.801593 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782585 5059 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:36:45.801593 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782588 5059 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:36:45.801593 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782591 5059 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:36:45.801593 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782596 5059 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:36:45.801593 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782599 5059 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:36:45.801593 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:45.782607 5059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:36:45.801593 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.782683 5059 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 19:36:45.801593 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.784267 5059 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 19:36:45.801593 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.784329 5059 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 19:36:45.801593 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.784805 5059 server.go:1014] "Starting client certificate rotation" Nov 03 19:36:45.801593 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.784927 5059 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 19:36:45.801593 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.784973 5059 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:31:03 +0000 UTC" deadline="2028-06-03 03:37:42.279503456 +0000 UTC" Nov 03 19:36:45.801593 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.785007 5059 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13856h0m56.494498202s" Nov 03 19:36:45.801995 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.785291 5059 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:36:45.801995 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.786400 5059 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:36:45.801995 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.787700 5059 log.go:25] "Validated CRI v1 runtime API" Nov 03 19:36:45.801995 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.789340 5059 log.go:25] "Validated CRI v1 image API" Nov 03 19:36:45.801995 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.790150 5059 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 19:36:45.801995 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.792100 5059 fs.go:135] Filesystem UUIDs: map[160a260a-1b31-4b52-8c5a-0749e0241f24:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 98c29d79-7c2a-4cc8-bcbf-d3d2f24eb05b:/dev/nvme0n1p4] Nov 03 19:36:45.802240 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.792121 5059 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03461b119eb96fc911cff538b5a6be0cb57a44cba669682e00c64daa7dd146fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03461b119eb96fc911cff538b5a6be0cb57a44cba669682e00c64daa7dd146fd/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c4f7b61fcd4ba2db6accad32dd25c0948b92f9ceb71408c97376cd46a8690c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c4f7b61fcd4ba2db6accad32dd25c0948b92f9ceb71408c97376cd46a8690c4/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ff53ee8da3120ef2b5adedd44d8db06f9192eab24e2f05017e50093b4b8bb2d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ff53ee8da3120ef2b5adedd44d8db06f9192eab24e2f05017e50093b4b8bb2d/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1af68c1e1deeba4a098d6b6a340842f1096a672a7c9f458990ee87e7fd746ec3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1af68c1e1deeba4a098d6b6a340842f1096a672a7c9f458990ee87e7fd746ec3/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/22cc4b81bd4c7dc1a411b6935fbe5de145ecdb4a6cb45704d4810326ae0a896e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/22cc4b81bd4c7dc1a411b6935fbe5de145ecdb4a6cb45704d4810326ae0a896e/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32222fdc67b4f6469d917ab85ca77d14f3f5514ef515ff284628d7056c596a70/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32222fdc67b4f6469d917ab85ca77d14f3f5514ef515ff284628d7056c596a70/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/378e0842a4e27216e6788143c711761cf336eabefb9faface56ff15da16e8218/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/378e0842a4e27216e6788143c711761cf336eabefb9faface56ff15da16e8218/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/38325d12703d972e1ae36cba1ea5a54367988c1d6b25ab7c5727d1b58e368726/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/38325d12703d972e1ae36cba1ea5a54367988c1d6b25ab7c5727d1b58e368726/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c720fc599e07edb9f53c3258393229ef983d842fab015bf43f5f59b3342d40f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c720fc599e07edb9f53c3258393229ef983d842fab015bf43f5f59b3342d40f/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/574c6906b17ca7382d033e3c1490a3c3ffef5f8642f16e2bde9dbd6349e080a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/574c6906b17ca7382d033e3c1490a3c3ffef5f8642f16e2bde9dbd6349e080a2/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57dc8ba289eb00c068e01a84058a62327d180d9194361eb4011c73b22e8d29f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57dc8ba289eb00c068e01a84058a62327d180d9194361eb4011c73b22e8d29f2/userdata/shm major:0 minor:333 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5bc830e6d84d3cd29bae7a5df0255f8f86285729e706443279b9042876a37e1c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5bc830e6d84d3cd29bae7a5df0255f8f86285729e706443279b9042876a37e1c/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/68f02c50c3562aee1180e793d24de99a71dde738bf562bb7e9f0dc8649d1eab4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/68f02c50c3562aee1180e793d24de99a71dde738bf562bb7e9f0dc8649d1eab4/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b14c8771941b7fdef847ecbb077cf12b273fee2ba5086c2edd527956ab372a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b14c8771941b7fdef847ecbb077cf12b273fee2ba5086c2edd527956ab372a4/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c41b8db3ecc464cfdadc0046d8144c354dc482d49d376e427a16f417211c4bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c41b8db3ecc464cfdadc0046d8144c354dc482d49d376e427a16f417211c4bc/userdata/shm major:0 minor:314 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/78e9081e9ae7d7fd90676dd54dbde8f3e82156ec5f28724f4b1e82150cb9f4ff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/78e9081e9ae7d7fd90676dd54dbde8f3e82156ec5f28724f4b1e82150cb9f4ff/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f2c2611f3929ca717acc0cd7b339637d4192b9b0f500398744cedf2c5f27953/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f2c2611f3929ca717acc0cd7b339637d4192b9b0f500398744cedf2c5f27953/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f52997d73f0c127088919f5e5f96fae8499e63d1b444d7b1b7b4b0d1a3ad067/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f52997d73f0c127088919f5e5f96fae8499e63d1b444d7b1b7b4b0d1a3ad067/userdata/shm major:0 minor:354 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84dcb8f62c1c8e86cbc9805e06cd0f30068afc5169cf3efd95ae1cfd8d7d1404/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84dcb8f62c1c8e86cbc9805e06cd0f30068afc5169cf3efd95ae1cfd8d7d1404/userdata/shm major:0 minor:315 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a115502e7ad24bf1026d336ac422c9fc0e0473373c5d1488bcfc2fc861f4aec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a115502e7ad24bf1026d336ac422c9fc0e0473373c5d1488bcfc2fc861f4aec/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ba62026c6d8111a62ecf49684a219609734418cf6b76e7f93021ff6e6d26ddc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ba62026c6d8111a62ecf49684a219609734418cf6b76e7f93021ff6e6d26ddc/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b164ebe179b4dc809db2208bdd9376de1000c8be52a466df2cf5cc5c8cdf45e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b164ebe179b4dc809db2208bdd9376de1000c8be52a466df2cf5cc5c8cdf45e9/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c2b962f17b61966cbbfd2bd9acd8c1d47071750979c7e64374959e75a9332477/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c2b962f17b61966cbbfd2bd9acd8c1d47071750979c7e64374959e75a9332477/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d74276b7e45093444f92ecc5311a048e8ade116d042b902e48a8c8dd526a8f79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d74276b7e45093444f92ecc5311a048e8ade116d042b902e48a8c8dd526a8f79/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e15414be96b0fe9f7d586967b509dcccb51d5baec5da68fd4c509d22b5bda8f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e15414be96b0fe9f7d586967b509dcccb51d5baec5da68fd4c509d22b5bda8f1/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e781524536bca1b272b961d24e61957bd19573364a9d4afb8efb7d5734c4b324/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e781524536bca1b272b961d24e61957bd19573364a9d4afb8efb7d5734c4b324/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a985b50-c4c0-4a39-abed-29f53e2be747/volumes/kubernetes.io~projected/kube-api-access-8tvmx:{mountpoint:/var/lib/kubelet/pods/0a985b50-c4c0-4a39-abed-29f53e2be747/volumes/kubernetes.io~projected/kube-api-access-8tvmx major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fc518fc-6302-4a1a-9ff9-f1e036a2199b/volumes/kubernetes.io~projected/kube-api-access-25cd8:{mountpoint:/var/lib/kubelet/pods/0fc518fc-6302-4a1a-9ff9-f1e036a2199b/volumes/kubernetes.io~projected/kube-api-access-25cd8 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/116af81a-d3a0-4558-93c4-8bf061b6815f/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/116af81a-d3a0-4558-93c4-8bf061b6815f/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18ffe451-b8eb-41ad-a19d-0390f3e1f8e2/volumes/kubernetes.io~projected/kube-api-access-drmpl:{mountpoint:/var/lib/kubelet/pods/18ffe451-b8eb-41ad-a19d-0390f3e1f8e2/volumes/kubernetes.io~projected/kube-api-access-drmpl major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18ffe451-b8eb-41ad-a19d-0390f3e1f8e2/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/18ffe451-b8eb-41ad-a19d-0390f3e1f8e2/volumes/kubernetes.io~secret/cert major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bcd4690-4753-4246-abb4-8668a0a56231/volumes/kubernetes.io~projected/kube-api-access-8wvnz:{mountpoint:/var/lib/kubelet/pods/1bcd4690-4753-4246-abb4-8668a0a56231/volumes/kubernetes.io~projected/kube-api-access-8wvnz major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2398ba99-35d3-4461-b66e-f44ac013b2d2/volumes/kubernetes.io~projected/kube-api-access-jx7kl:{mountpoint:/var/lib/kubelet/pods/2398ba99-35d3-4461-b66e-f44ac013b2d2/volumes/kubernetes.io~projected/kube-api-access-jx7kl major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32065295-c6a2-44ea-88e8-c4fdc5a84083/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/32065295-c6a2-44ea-88e8-c4fdc5a84083/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32065295-c6a2-44ea-88e8-c4fdc5a84083/volumes/kubernetes.io~projected/kube-api-access-kbrbm:{mountpoint:/var/lib/kubelet/pods/32065295-c6a2-44ea-88e8-c4fdc5a84083/volumes/kubernetes.io~projected/kube-api-access-kbrbm major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32065295-c6a2-44ea-88e8-c4fdc5a84083/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/32065295-c6a2-44ea-88e8-c4fdc5a84083/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32065295-c6a2-44ea-88e8-c4fdc5a84083/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/32065295-c6a2-44ea-88e8-c4fdc5a84083/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/377ffcf5-b097-495d-b986-e61c58ae8ec1/volumes/kubernetes.io~projected/kube-api-access-hzh28:{mountpoint:/var/lib/kubelet/pods/377ffcf5-b097-495d-b986-e61c58ae8ec1/volumes/kubernetes.io~projected/kube-api-access-hzh28 major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40834bff-2db0-4690-9cb9-7b4aacdab3ab/volumes/kubernetes.io~projected/kube-api-access-jclj5:{mountpoint:/var/lib/kubelet/pods/40834bff-2db0-4690-9cb9-7b4aacdab3ab/volumes/kubernetes.io~projected/kube-api-access-jclj5 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40834bff-2db0-4690-9cb9-7b4aacdab3ab/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/40834bff-2db0-4690-9cb9-7b4aacdab3ab/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40b2621f-854e-44e0-9b01-c877e333f644/volumes/kubernetes.io~projected/kube-api-access-jc5qt:{mountpoint:/var/lib/kubelet/pods/40b2621f-854e-44e0-9b01-c877e333f644/volumes/kubernetes.io~projected/kube-api-access-jc5qt major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/419841be-7a31-4cdb-b0b7-9d90a3c376c7/volumes/kubernetes.io~projected/kube-api-access-9tckv:{mountpoint:/var/lib/kubelet/pods/419841be-7a31-4cdb-b0b7-9d90a3c376c7/volumes/kubernetes.io~projected/kube-api-access-9tckv major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/492c7b4e-84c0-49a7-b12f-153a021f971b/volumes/kubernetes.io~projected/kube-api-access-hgtxr:{mountpoint:/var/lib/kubelet/pods/492c7b4e-84c0-49a7-b12f-153a021f971b/volumes/kubernetes.io~projected/kube-api-access-hgtxr major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/492c7b4e-84c0-49a7-b12f-153a021f971b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/492c7b4e-84c0-49a7-b12f-153a021f971b/volumes/kubernetes.io~secret/serving-cert major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b79231d-e954-41b2-924d-67808a734ecf/volumes/kubernetes.io~projected/kube-api-access-nhrbz:{mountpoint:/var/lib/kubelet/pods/4b79231d-e954-41b2-924d-67808a734ecf/volumes/kubernetes.io~projected/kube-api-access-nhrbz major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fdbd17a-1e15-4442-b715-aaec1087da11/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/4fdbd17a-1e15-4442-b715-aaec1087da11/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53b14f9a-431e-4dbb-a725-4c8470262ca7/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/53b14f9a-431e-4dbb-a725-4c8470262ca7/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53b14f9a-431e-4dbb-a725-4c8470262ca7/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/53b14f9a-431e-4dbb-a725-4c8470262ca7/volumes/kubernetes.io~empty-dir/tmp major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53b14f9a-431e-4dbb-a725-4c8470262ca7/volumes/kubernetes.io~projected/kube-api-access-m5xvg:{mountpoint:/var/lib/kubelet/pods/53b14f9a-431e-4dbb-a725-4c8470262ca7/volumes/kubernetes.io~projected/kube-api-access-m5xvg major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55f124eb-2ebf-4675-a08a-bad3757061b3/volumes/kubernetes.io~projected/kube-api-access-lx2l9:{mountpoint:/var/lib/kubelet/pods/55f124eb-2ebf-4675-a08a-bad3757061b3/volumes/kubernetes.io~projected/kube-api-access-lx2l9 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/603868b0-8628-4227-8471-1894e2e1de50/volumes/kubernetes.io~projected/kube-api-access-89g8g:{mountpoint:/var/lib/kubelet/pods/603868b0-8628-4227-8471-1894e2e1de50/volumes/kubernetes.io~projected/kube-api-access-89g8g major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/603868b0-8628-4227-8471-1894e2e1de50/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/603868b0-8628-4227-8471-1894e2e1de50/volumes/kubernetes.io~secret/default-certificate major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/603868b0-8628-4227-8471-1894e2e1de50/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/603868b0-8628-4227-8471-1894e2e1de50/volumes/kubernetes.io~secret/stats-auth major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b8992fc-fba6-4135-90f8-fb259e765dd5/volumes/kubernetes.io~projected/kube-api-access-7rnvd:{mountpoint:/var/lib/kubelet/pods/7b8992fc-fba6-4135-90f8-fb259e765dd5/volumes/kubernetes.io~projected/kube-api-access-7rnvd major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b8992fc-fba6-4135-90f8-fb259e765dd5/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/7b8992fc-fba6-4135-90f8-fb259e765dd5/volumes/kubernetes.io~secret/signing-key major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/815b15b6-2c37-4ca3-9b38-e7e1328d0aa8/volumes/kubernetes.io~projected/kube-api-access-mdss4:{mountpoint:/var/lib/kubelet/pods/815b15b6-2c37-4ca3-9b38-e7e1328d0aa8/volumes/kubernetes.io~projected/kube-api-access-mdss4 major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d0db609-b271-4c98-a303-fb3ee71f152c/volumes/kubernetes.io~projected/kube-api-access-mkxhk:{mountpoint:/var/lib/kubelet/pods/8d0db609-b271-4c98-a303-fb3ee71f152c/volumes/kubernetes.io~projected/kube-api-access-mkxhk major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94d082d5-1c33-4394-9d93-67645f9b60cd/volumes/kubernetes.io~projected/kube-api-access-rsppf:{mountpoint:/var/lib/kubelet/pods/94d082d5-1c33-4394-9d93-67645f9b60cd/volumes/kubernetes.io~projected/kube-api-access-rsppf major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6f0d931-3a92-424f-a5fc-d9ebef985dfd/volumes/kubernetes.io~projected/kube-api-access-lnslt:{mountpoint:/var/lib/kubelet/pods/b6f0d931-3a92-424f-a5fc-d9ebef985dfd/volumes/kubernetes.io~projected/kube-api-access-lnslt major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b989ba1e-71f9-424b-90f1-d1450f63e8b1/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/b989ba1e-71f9-424b-90f1-d1450f63e8b1/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b989ba1e-71f9-424b-90f1-d1450f63e8b1/volumes/kubernetes.io~projected/kube-api-access-28xz4:{mountpoint:/var/lib/kubelet/pods/b989ba1e-71f9-424b-90f1-d1450f63e8b1/volumes/kubernetes.io~projected/kube-api-access-28xz4 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b989ba1e-71f9-424b-90f1-d1450f63e8b1/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/b989ba1e-71f9-424b-90f1-d1450f63e8b1/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b989ba1e-71f9-424b-90f1-d1450f63e8b1/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/b989ba1e-71f9-424b-90f1-d1450f63e8b1/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b/volumes/kubernetes.io~projected/kube-api-access-7p75v:{mountpoint:/var/lib/kubelet/pods/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b/volumes/kubernetes.io~projected/kube-api-access-7p75v major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/volumes/kubernetes.io~projected/kube-api-access-9hszb:{mountpoint:/var/lib/kubelet/pods/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/volumes/kubernetes.io~projected/kube-api-access-9hszb major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c95249a6-899f-4faf-82dc-6e553453ff80/volumes/kubernetes.io~projected/kube-api-access-lf5tn:{mountpoint:/var/lib/kubelet/pods/c95249a6-899f-4faf-82dc-6e553453ff80/volumes/kubernetes.io~projected/kube-api-access-lf5tn major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c99b0c19-004e-450c-852a-e38338635b61/volumes/kubernetes.io~projected/kube-api-access-xk5v5:{mountpoint:/var/lib/kubelet/pods/c99b0c19-004e-450c-852a-e38338635b61/volumes/kubernetes.io~projected/kube-api-access-xk5v5 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c99b0c19-004e-450c-852a-e38338635b61/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c99b0c19-004e-450c-852a-e38338635b61/volumes/kubernetes.io~secret/serving-cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb/volumes/kubernetes.io~projected/kube-api-access-2tpvj:{mountpoint:/var/lib/kubelet/pods/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb/volumes/kubernetes.io~projected/kube-api-access-2tpvj major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2155512-2116-44f6-be5e-77cef72f9e14/volumes/kubernetes.io~projected/kube-api-access-cjhsn:{mountpoint:/var/lib/kubelet/pods/e2155512-2116-44f6-be5e-77cef72f9e14/volumes/kubernetes.io~projected/kube-api-access-cjhsn major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e64f9d9f-c868-470e-bd78-4ce8baabd97a/volumes/kubernetes.io~projected/kube-api-access-mn9fc:{mountpoint:/var/lib/kubelet/pods/e64f9d9f-c868-470e-bd78-4ce8baabd97a/volumes/kubernetes.io~projected/kube-api-access-mn9fc major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f485893c-6f14-449b-b837-8ac181315aeb/volumes/kubernetes.io~projected/kube-api-access-86bkx:{mountpoint:/var/lib/kubelet/pods/f485893c-6f14-449b-b837-8ac181315aeb/volumes/kubernetes.io~projected/kube-api-access-86bkx major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/volumes/kubernetes.io~projected/kube-api-access-tv87l:{mountpoint:/var/lib/kubelet/pods/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/volumes/kubernetes.io~projected/kube-api-access-tv87l major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9d935b2-aa51-474c-836d-a74f91efab07/volumes/kubernetes.io~projected/kube-api-access-7c5zw:{mountpoint:/var/lib/kubelet/pods/f9d935b2-aa51-474c-836d-a74f91efab07/volumes/kubernetes.io~projected/kube-api-access-7c5zw major:0 minor:202 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/807b5d91ec044b6e3690fd687ada9bd4e01ef9aefd7c5be109d86f8020447460/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/4b8288f2e07855b92c0673979bbc56f16b95f104d7f3d331523a2b249fd005be/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5630a22ba1b338988a20699748117abbe435074b2a4d9dcff245e43eb9767391/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/af2813e3cc345e41fada3e12c6cc0bafde54c03ea986315db35b5d8e8606a959/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/d68139bb7d74a6179bfbf30966bab9231b936b5fed801b743293927edf9117a4/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/18bc1e0ac5f089f3489772fb39b4cc72a12ceea81b05d60570e379b8672727f2/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/15de5c78c58cb7031b136a1c35a90045dd6aebec927308d49a43d91a20ef351d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/3590b0870ebbb7484f75a61269deee6b5456d7e78326def3787ba771f3783e4d/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/4a77bbd76b8d6f139795f66a89d4209b4d85e3189b0d33316e94acef23c21ee8/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/db08eff9b22a4500cec0d1680cd6484dadd3fdd3de35cc7f22a12a0590ccdc68/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/7d118d46ddd9118020120bf4abeda717a9745f884905c6c04f3bec41139d79e2/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/de6ca20b9377c28cabe4038e64abbe6a95d1dd277c5ca5ddfa3af5b8c5e76c54/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/b85378590d3a8b51c3720c14d24215dbda1e35f20453ad77550072c510cf3714/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/395fbf4d5d4ec8a134ec8da50084515d06500fbb092d7ed16790f32d60204633/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/7fcc2c309088f71f15e3de261c00b834cb912549820428227aa73c7f19b19398/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/43201a78b713ec15496bb789b9294c6a0013d5c89dba58b4ae0edd1a0971be62/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/f26445c31409eb167bf27cc2cbab2017408dbd457f03729e67cd2f30f1ea53f0/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/9edd96ae054cc1dc8e9dfc2a28e2169a5c4bcc43353e82ed8bf524e2c8467dbc/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/ccf806ac6ec23ad74bcc74db02ca5f266c08af5a5088b87059866bfd5ad3a508/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/29312326d1c5d67a009d3f2a0ead3df745c17fd18ca61679cf5a8d648501ee1c/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/d3141a050734ea30444cd519454c37c2cee0e793a05fad7726c15fd6b8e5ce74/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/3e151306ed961ad6bfc0a65d6d7858de0210f8149234a8750218b7bce9f28c39/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/3fe862ed3e308dc9fd551945d6f248a08da522305664f95336d9bb32796ea201/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/1ab252080a6da0c26d9aed23639cf17237b7e42ba6e207939b96152dca047393/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/67643f6bfc43ab27acce59d3f46ca4439902175e226d9e26fa2b0544a356d3a7/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/d91a2eef7704086128be09e4b79681dd76ac562bcbb98517cf669c2b35c77693/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/cbb879871a44605c610147751343b85dedca5ecbe37123acd39e2fd41eedaa35/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/56a1a4fe0beea2934f0067c34f07d9b9de48a79ddeaebc10dcc1918e249e4fe9/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/e508a4fba182975f628a2965bdd2bc06db2f762a03cb80182d6faab7219e0ccd/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/974994766dd4ff7caa644bd6347d79d35dd36116080142add2f681ab8815f5ab/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/3c1933e8657ed24a888e3078150a15152408aef3de6d06715661d3b955bdeb03/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/e3a08dd38e0673fee7ee93b1f7ca301b5aa65f8f1089002352132e3cd010d249/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/f76791e3b40bd77e4177e755af9593b8264a6f9584df82df5f3ad48f8536a824/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/2acad7ba6294ceb7ed21eec32f82e3dd0f18687ff5816a03411e1545b134088c/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/285095d558a08566a705a3940ab4af0b45e7f143d3fb6d23370fc566c03184ba/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/3664b495f860b8b12a14f83f6f7268d45d1d6788d533b83f182667924c5c86e2/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/364e21f717075de023707ce0eadcab5ff3e172710140545d125c62e36740887e/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/67b680590f5eefdd8ad88492c01e8616609706fb8ff8b7571351b1b139c2be6d/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/facdfe40bf1d495f6de5e16730be25692c4e02b216527bd53f3820b40d730669/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/e85f90ce9c7dee5bcd4a959cf6355747a084ba8d57e2182068ae8933f45dad10/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/403fb18d4100ac35b0fcff3a33b393fca67fc0623169d31e16b4889c870be753/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/d1b34dee7bb386b4b08b6201e76895f6f53c61c71d5047b48d1b40a81fdc836c/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/0bcd3ad108cb6e59ca34fbe5d0bdabe1fa2c72489d6b64f202cd98113a84a542/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/b6cd9e4f4ab5200d0481e1c96d9431f1e12949004fd0870f5eab0a08be25c4ee/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/ed0eafee322a8d2b52fa0003bbc19044e534837ac93210edf91724599788f996/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/0edfa7bdd1c0bb6e171a437f6bce6c93a31b9489e465414ee1c1d9d1a8585a8a/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/a5b97e3cd348c806e09cd4029d62a599c5ce3ce7e0f239aa65465e6412865061/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/c37ca7a1d4a292f83c6190b9174a510fc5046ebdb1f5f1453a9ab145c3e6857a/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/ee7f02d9be1fdeb93909fdbcfaf9d4038d7c16d9e26b087b8d5ace145804b85a/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/5b3629ccac63294c67978800bc554ad8685d0a4172b11f35016df65821206978/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/8d5a8bfb065b77d3516c9b12dc7440a9b213e9e14b732e71883de279b65cd3ce/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/421c143b7c80827d2cc5a062879acfb554cf4fb285709dba5f301636b77ba0e8/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/6108362e423e122429ec08608da59a2f40f00273b2584dca92bd8a7570448b0c/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/7f28e0f7f0452864f37340b84f720939062a84b92404c9b09cbac54186e86f29/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/036ad7274ad1232ed107d37abc90a18d84e2c58366eba450be383b1c7539919b/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/2ab4d09781f5bb38cf39dad12e66699114c6e92eb3b29dd65c026a0af1b404cf/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/cc33b7715971fc9b11e0f8ee7fff767603d70576b6ba7c1fde7231b29d5e89b6/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/0c52062b38c59af46beb68c242b3a38f795b07b0c8b1ad76126162c8c3c3e1d5/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/0a728ec770e6ffa3c0d8df9535746dbcb124a23ff2fea94ab2fc314ec32ae537/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/05b98974003a4181cc525d06dd6951339b1182ba0e8da0ec7cc1521a072638d0/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/fff3e4757f69e495c651e4c9e0afd850cf031c4c2b428196eb46a31c2ff022ea/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/2b818e04a987bed97dc81c53c55626df56032b4ea5b65abebfe974b29ef334b6/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 19:36:45.802601 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.800747 5059 manager.go:217] Machine: {Timestamp:2026-11-03 19:36:45.80026208 +0000 UTC m=+0.058059334 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3087087 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26ff604744fdec5b889a367c7545e9 SystemUUID:ec26ff60-4744-fdec-5b88-9a367c7545e9 BootID:2904ca26-dd83-4680-b3d7-33f1ad2aa3e3 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32222fdc67b4f6469d917ab85ca77d14f3f5514ef515ff284628d7056c596a70/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c41b8db3ecc464cfdadc0046d8144c354dc482d49d376e427a16f417211c4bc/userdata/shm DeviceMajor:0 DeviceMinor:314 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4fdbd17a-1e15-4442-b715-aaec1087da11/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/53b14f9a-431e-4dbb-a725-4c8470262ca7/volumes/kubernetes.io~projected/kube-api-access-m5xvg DeviceMajor:0 DeviceMinor:65 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c95249a6-899f-4faf-82dc-6e553453ff80/volumes/kubernetes.io~projected/kube-api-access-lf5tn DeviceMajor:0 DeviceMinor:293 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94d082d5-1c33-4394-9d93-67645f9b60cd/volumes/kubernetes.io~projected/kube-api-access-rsppf DeviceMajor:0 DeviceMinor:66 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b6f0d931-3a92-424f-a5fc-d9ebef985dfd/volumes/kubernetes.io~projected/kube-api-access-lnslt DeviceMajor:0 DeviceMinor:67 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/603868b0-8628-4227-8471-1894e2e1de50/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:201 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e781524536bca1b272b961d24e61957bd19573364a9d4afb8efb7d5734c4b324/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/e64f9d9f-c868-470e-bd78-4ce8baabd97a/volumes/kubernetes.io~projected/kube-api-access-mn9fc DeviceMajor:0 DeviceMinor:60 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40834bff-2db0-4690-9cb9-7b4aacdab3ab/volumes/kubernetes.io~projected/kube-api-access-jclj5 DeviceMajor:0 DeviceMinor:204 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e15414be96b0fe9f7d586967b509dcccb51d5baec5da68fd4c509d22b5bda8f1/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40b2621f-854e-44e0-9b01-c877e333f644/volumes/kubernetes.io~projected/kube-api-access-jc5qt DeviceMajor:0 DeviceMinor:224 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/18ffe451-b8eb-41ad-a19d-0390f3e1f8e2/volumes/kubernetes.io~projected/kube-api-access-drmpl DeviceMajor:0 DeviceMinor:236 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c720fc599e07edb9f53c3258393229ef983d842fab015bf43f5f59b3342d40f/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/419841be-7a31-4cdb-b0b7-9d90a3c376c7/volumes/kubernetes.io~projected/kube-api-access-9tckv DeviceMajor:0 DeviceMinor:237 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f485893c-6f14-449b-b837-8ac181315aeb/volumes/kubernetes.io~projected/kube-api-access-86bkx DeviceMajor:0 DeviceMinor:312 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7b8992fc-fba6-4135-90f8-fb259e765dd5/volumes/kubernetes.io~projected/kube-api-access-7rnvd DeviceMajor:0 DeviceMinor:337 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2398ba99-35d3-4461-b66e-f44ac013b2d2/volumes/kubernetes.io~projected/kube-api-access-jx7kl DeviceMajor:0 DeviceMinor:62 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c99b0c19-004e-450c-852a-e38338635b61/volumes/kubernetes.io~projected/kube-api-access-xk5v5 DeviceMajor:0 DeviceMinor:221 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/32065295-c6a2-44ea-88e8-c4fdc5a84083/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:222 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a115502e7ad24bf1026d336ac422c9fc0e0473373c5d1488bcfc2fc861f4aec/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/32065295-c6a2-44ea-88e8-c4fdc5a84083/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:217 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/32065295-c6a2-44ea-88e8-c4fdc5a84083/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:216 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d0db609-b271-4c98-a303-fb3ee71f152c/volumes/kubernetes.io~projected/kube-api-access-mkxhk DeviceMajor:0 DeviceMinor:206 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03461b119eb96fc911cff538b5a6be0cb57a44cba669682e00c64daa7dd146fd/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b/volumes/kubernetes.io~projected/kube-api-access-7p75v DeviceMajor:0 DeviceMinor:61 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/68f02c50c3562aee1180e793d24de99a71dde738bf562bb7e9f0dc8649d1eab4/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/38325d12703d972e1ae36cba1ea5a54367988c1d6b25ab7c5727d1b58e368726/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/492c7b4e-84c0-49a7-b12f-153a021f971b/volumes/kubernetes.io~projected/kube-api-access-hgtxr DeviceMajor:0 DeviceMinor:223 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c4f7b61fcd4ba2db6accad32dd25c0948b92f9ceb71408c97376cd46a8690c4/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/53b14f9a-431e-4dbb-a725-4c8470262ca7/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:64 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ff53ee8da3120ef2b5adedd44d8db06f9192eab24e2f05017e50093b4b8bb2d/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b989ba1e-71f9-424b-90f1-d1450f63e8b1/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:225 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b989ba1e-71f9-424b-90f1-d1450f63e8b1/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:214 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b989ba1e-71f9-424b-90f1-d1450f63e8b1/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:215 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/492c7b4e-84c0-49a7-b12f-153a021f971b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:219 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/78e9081e9ae7d7fd90676dd54dbde8f3e82156ec5f28724f4b1e82150cb9f4ff/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0a985b50-c4c0-4a39-abed-29f53e2be747/volumes/kubernetes.io~projected/kube-api-access-8tvmx DeviceMajor:0 DeviceMinor:238 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/574c6906b17ca7382d033e3c1490a3c3ffef5f8642f16e2bde9dbd6349e080a2/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84dcb8f62c1c8e86cbc9805e06cd0f30068afc5169cf3efd95ae1cfd8d7d1404/userdata/shm DeviceMajor:0 DeviceMinor:315 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1bcd4690-4753-4246-abb4-8668a0a56231/volumes/kubernetes.io~projected/kube-api-access-8wvnz DeviceMajor:0 DeviceMinor:203 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b8992fc-fba6-4135-90f8-fb259e765dd5/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:336 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/53b14f9a-431e-4dbb-a725-4c8470262ca7/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:63 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b164ebe179b4dc809db2208bdd9376de1000c8be52a466df2cf5cc5c8cdf45e9/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e2155512-2116-44f6-be5e-77cef72f9e14/volumes/kubernetes.io~projected/kube-api-access-cjhsn DeviceMajor:0 DeviceMinor:205 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/32065295-c6a2-44ea-88e8-c4fdc5a84083/volumes/kubernetes.io~projected/kube-api-access-kbrbm DeviceMajor:0 DeviceMinor:228 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb/volumes/kubernetes.io~projected/kube-api-access-2tpvj DeviceMajor:0 DeviceMinor:59 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5bc830e6d84d3cd29bae7a5df0255f8f86285729e706443279b9042876a37e1c/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/volumes/kubernetes.io~projected/kube-api-access-tv87l DeviceMajor:0 DeviceMinor:226 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4b79231d-e954-41b2-924d-67808a734ecf/volumes/kubernetes.io~projected/kube-api-access-nhrbz DeviceMajor:0 DeviceMinor:311 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d74276b7e45093444f92ecc5311a048e8ade116d042b902e48a8c8dd526a8f79/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57dc8ba289eb00c068e01a84058a62327d180d9194361eb4011c73b22e8d29f2/userdata/shm DeviceMajor:0 DeviceMinor:333 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ba62026c6d8111a62ecf49684a219609734418cf6b76e7f93021ff6e6d26ddc/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1af68c1e1deeba4a098d6b6a340842f1096a672a7c9f458990ee87e7fd746ec3/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f2c2611f3929ca717acc0cd7b339637d4192b9b0f500398744cedf2c5f27953/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/603868b0-8628-4227-8471-1894e2e1de50/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:45 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/603868b0-8628-4227-8471-1894e2e1de50/volumes/kubernetes.io~projected/kube-api-access-89g8g DeviceMajor:0 DeviceMinor:208 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/22cc4b81bd4c7dc1a411b6935fbe5de145ecdb4a6cb45704d4810326ae0a896e/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/116af81a-d3a0-4558-93c4-8bf061b6815f/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/volumes/kubernetes.io~projected/kube-api-access-9hszb DeviceMajor:0 DeviceMinor:58 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c2b962f17b61966cbbfd2bd9acd8c1d47071750979c7e64374959e75a9332477/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/40834bff-2db0-4690-9cb9-7b4aacdab3ab/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f9d935b2-aa51-474c-836d-a74f91efab07/volumes/kubernetes.io~projected/kube-api-access-7c5zw DeviceMajor:0 DeviceMinor:202 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0fc518fc-6302-4a1a-9ff9-f1e036a2199b/volumes/kubernetes.io~projected/kube-api-access-25cd8 DeviceMajor:0 DeviceMinor:207 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c99b0c19-004e-450c-852a-e38338635b61/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:218 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/55f124eb-2ebf-4675-a08a-bad3757061b3/volumes/kubernetes.io~projected/kube-api-access-lx2l9 DeviceMajor:0 DeviceMinor:57 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b14c8771941b7fdef847ecbb077cf12b273fee2ba5086c2edd527956ab372a4/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f52997d73f0c127088919f5e5f96fae8499e63d1b444d7b1b7b4b0d1a3ad067/userdata/shm DeviceMajor:0 DeviceMinor:354 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/378e0842a4e27216e6788143c711761cf336eabefb9faface56ff15da16e8218/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18ffe451-b8eb-41ad-a19d-0390f3e1f8e2/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:234 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/815b15b6-2c37-4ca3-9b38-e7e1328d0aa8/volumes/kubernetes.io~projected/kube-api-access-mdss4 DeviceMajor:0 DeviceMinor:310 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/377ffcf5-b097-495d-b986-e61c58ae8ec1/volumes/kubernetes.io~projected/kube-api-access-hzh28 DeviceMajor:0 DeviceMinor:313 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b989ba1e-71f9-424b-90f1-d1450f63e8b1/volumes/kubernetes.io~projected/kube-api-access-28xz4 DeviceMajor:0 DeviceMinor:227 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1af68c1e1deeba4 MacAddress:26:6c:62:b3:ed:22 Speed:10000 Mtu:8901} {Name:22cc4b81bd4c7dc MacAddress:66:8c:b2:3e:20:10 Speed:10000 Mtu:8901} {Name:32222fdc67b4f64 MacAddress:42:40:70:61:65:fd Speed:10000 Mtu:8901} {Name:38325d12703d972 MacAddress:22:16:ae:94:85:51 Speed:10000 Mtu:8901} {Name:3c720fc599e07ed MacAddress:c2:61:30:25:26:d0 Speed:10000 Mtu:8901} {Name:574c6906b17ca73 MacAddress:9e:50:17:02:e1:8f Speed:10000 Mtu:8901} {Name:57dc8ba289eb00c MacAddress:a2:90:bf:05:ce:ec Speed:10000 Mtu:8901} {Name:5bc830e6d84d3cd MacAddress:b2:c6:11:03:6b:72 Speed:10000 Mtu:8901} {Name:6b14c8771941b7f MacAddress:9e:8f:22:54:16:05 Speed:10000 Mtu:8901} {Name:6c41b8db3ecc464 MacAddress:9a:b7:68:5d:ec:8c Speed:10000 Mtu:8901} {Name:78e9081e9ae7d7f MacAddress:72:76:47:99:a5:c1 Speed:10000 Mtu:8901} {Name:7f2c2611f3929ca MacAddress:f6:b1:41:8f:e4:2d Speed:10000 Mtu:8901} {Name:7f52997d73f0c12 MacAddress:56:33:5d:c2:07:29 Speed:10000 Mtu:8901} {Name:84dcb8f62c1c8e8 MacAddress:7e:2c:ff:1c:b4:50 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c4:77:56:bb Speed:0 Mtu:9001} {Name:br-int MacAddress:4e:1b:c5:67:7b:fa Speed:0 Mtu:8901} {Name:e15414be96b0fe9 MacAddress:46:26:ea:ed:13:54 Speed:10000 Mtu:8901} {Name:e781524536bca1b MacAddress:9a:76:a4:a6:4d:2b Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c4:77:56:bb Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:ae:3f:bf:14:55:31 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:0a:dd:de:9f:65:4c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 19:36:45.802601 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.801079 5059 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 19:36:45.802601 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.801120 5059 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 19:36:45.802601 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.801431 5059 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 19:36:45.802601 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.801448 5059 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-83.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 19:36:45.802601 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.801588 5059 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 19:36:45.802601 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.801595 5059 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 19:36:45.802601 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.801608 5059 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:36:45.802601 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.801624 5059 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:36:45.802601 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.801642 5059 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 19:36:45.802601 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.801817 5059 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 19:36:45.802601 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.801944 5059 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 19:36:45.802601 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.802201 5059 kubelet.go:493] "Attempting to sync node with API server" Nov 03 19:36:45.802601 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.802242 5059 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 19:36:45.802601 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.802259 5059 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 19:36:45.802601 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.802276 5059 kubelet.go:405] "Adding apiserver pod source" Nov 03 19:36:45.802601 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.802290 5059 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 19:36:45.804719 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.804500 5059 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 19:36:45.804785 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.804725 5059 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:36:45.805187 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.805169 5059 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 19:36:45.807440 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.807426 5059 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 19:36:45.807440 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.807439 5059 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:36:45.807532 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.807523 5059 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 19:36:45.807724 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.807712 5059 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 19:36:45.807755 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.807742 5059 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 19:36:45.807834 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.807825 5059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 19:36:45.807879 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.807867 5059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 19:36:45.807908 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.807886 5059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 19:36:45.807908 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.807892 5059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 19:36:45.807908 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.807898 5059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 19:36:45.807908 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.807904 5059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 19:36:45.808026 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.807911 5059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 19:36:45.808026 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.807917 5059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 19:36:45.808026 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.807923 5059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 19:36:45.808026 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.807934 5059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 19:36:45.808026 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.807946 5059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 19:36:45.808026 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.807956 5059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 19:36:45.808026 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.807980 5059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 19:36:45.808026 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.807985 5059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 19:36:45.811630 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.811614 5059 server.go:1263] "Started kubelet" Nov 03 19:36:45.811846 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.811790 5059 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 19:36:45.812294 ip-10-0-132-83 systemd[1]: Started Kubernetes Kubelet. Nov 03 19:36:45.813336 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.813269 5059 server.go:317] "Adding debug handlers to kubelet server" Nov 03 19:36:45.814493 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.814354 5059 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 19:36:45.814596 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.814528 5059 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 19:36:45.814841 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.814803 5059 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.818025 5059 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.818370 5059 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.818385 5059 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.818402 5059 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.818459 5059 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.818870 5059 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:31:05 +0000 UTC" deadline="2028-04-05 15:32:36.697528376 +0000 UTC" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.818891 5059 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12451h55m50.8786407s" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:45.819188 5059 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-83.us-east-2.compute.internal\" not found" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821100 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94d082d5-1c33-4394-9d93-67645f9b60cd" volumeName="kubernetes.io/projected/94d082d5-1c33-4394-9d93-67645f9b60cd-kube-api-access-rsppf" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821122 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6f0d931-3a92-424f-a5fc-d9ebef985dfd" volumeName="kubernetes.io/configmap/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821134 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e64f9d9f-c868-470e-bd78-4ce8baabd97a" volumeName="kubernetes.io/projected/e64f9d9f-c868-470e-bd78-4ce8baabd97a-kube-api-access-mn9fc" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821142 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18ffe451-b8eb-41ad-a19d-0390f3e1f8e2" volumeName="kubernetes.io/secret/18ffe451-b8eb-41ad-a19d-0390f3e1f8e2-cert" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821151 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bcd4690-4753-4246-abb4-8668a0a56231" volumeName="kubernetes.io/projected/1bcd4690-4753-4246-abb4-8668a0a56231-kube-api-access-8wvnz" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821158 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="419841be-7a31-4cdb-b0b7-9d90a3c376c7" volumeName="kubernetes.io/configmap/419841be-7a31-4cdb-b0b7-9d90a3c376c7-config-volume" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821166 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53b14f9a-431e-4dbb-a725-4c8470262ca7" volumeName="kubernetes.io/projected/53b14f9a-431e-4dbb-a725-4c8470262ca7-kube-api-access-m5xvg" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821173 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6f0d931-3a92-424f-a5fc-d9ebef985dfd" volumeName="kubernetes.io/configmap/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821181 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c95249a6-899f-4faf-82dc-6e553453ff80" volumeName="kubernetes.io/projected/c95249a6-899f-4faf-82dc-6e553453ff80-kube-api-access-lf5tn" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821189 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" volumeName="kubernetes.io/secret/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-serving-cert" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821197 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04a982f9-143c-406c-99d5-9ef73d5c571a" volumeName="kubernetes.io/configmap/04a982f9-143c-406c-99d5-9ef73d5c571a-nginx-conf" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821204 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18ffe451-b8eb-41ad-a19d-0390f3e1f8e2" volumeName="kubernetes.io/projected/18ffe451-b8eb-41ad-a19d-0390f3e1f8e2-kube-api-access-drmpl" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821211 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40834bff-2db0-4690-9cb9-7b4aacdab3ab" volumeName="kubernetes.io/secret/40834bff-2db0-4690-9cb9-7b4aacdab3ab-serving-cert" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821218 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b79231d-e954-41b2-924d-67808a734ecf" volumeName="kubernetes.io/configmap/4b79231d-e954-41b2-924d-67808a734ecf-trusted-ca-bundle" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821228 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b8992fc-fba6-4135-90f8-fb259e765dd5" volumeName="kubernetes.io/configmap/7b8992fc-fba6-4135-90f8-fb259e765dd5-signing-cabundle" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821240 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42" volumeName="kubernetes.io/projected/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-kube-api-access-9hszb" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821247 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b79231d-e954-41b2-924d-67808a734ecf" volumeName="kubernetes.io/empty-dir/4b79231d-e954-41b2-924d-67808a734ecf-archives-path" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821255 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b989ba1e-71f9-424b-90f1-d1450f63e8b1" volumeName="kubernetes.io/configmap/b989ba1e-71f9-424b-90f1-d1450f63e8b1-trusted-ca" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821262 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c99b0c19-004e-450c-852a-e38338635b61" volumeName="kubernetes.io/projected/c99b0c19-004e-450c-852a-e38338635b61-kube-api-access-xk5v5" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821268 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c99b0c19-004e-450c-852a-e38338635b61" volumeName="kubernetes.io/secret/c99b0c19-004e-450c-852a-e38338635b61-serving-cert" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821275 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="377ffcf5-b097-495d-b986-e61c58ae8ec1" volumeName="kubernetes.io/projected/377ffcf5-b097-495d-b986-e61c58ae8ec1-kube-api-access-hzh28" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821282 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="492c7b4e-84c0-49a7-b12f-153a021f971b" volumeName="kubernetes.io/projected/492c7b4e-84c0-49a7-b12f-153a021f971b-kube-api-access-hgtxr" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821291 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53b14f9a-431e-4dbb-a725-4c8470262ca7" volumeName="kubernetes.io/empty-dir/53b14f9a-431e-4dbb-a725-4c8470262ca7-tmp" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821298 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" volumeName="kubernetes.io/configmap/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-config" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821305 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40b2621f-854e-44e0-9b01-c877e333f644" volumeName="kubernetes.io/projected/40b2621f-854e-44e0-9b01-c877e333f644-kube-api-access-jc5qt" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821326 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55f124eb-2ebf-4675-a08a-bad3757061b3" volumeName="kubernetes.io/configmap/55f124eb-2ebf-4675-a08a-bad3757061b3-iptables-alerter-script" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821337 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b989ba1e-71f9-424b-90f1-d1450f63e8b1" volumeName="kubernetes.io/secret/b989ba1e-71f9-424b-90f1-d1450f63e8b1-installation-pull-secrets" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821346 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" volumeName="kubernetes.io/configmap/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-trusted-ca" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821356 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a985b50-c4c0-4a39-abed-29f53e2be747" volumeName="kubernetes.io/projected/0a985b50-c4c0-4a39-abed-29f53e2be747-kube-api-access-8tvmx" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821364 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32065295-c6a2-44ea-88e8-c4fdc5a84083" volumeName="kubernetes.io/configmap/32065295-c6a2-44ea-88e8-c4fdc5a84083-trusted-ca" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821371 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fdbd17a-1e15-4442-b715-aaec1087da11" volumeName="kubernetes.io/configmap/4fdbd17a-1e15-4442-b715-aaec1087da11-konnectivity-ca" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821378 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="815b15b6-2c37-4ca3-9b38-e7e1328d0aa8" volumeName="kubernetes.io/projected/815b15b6-2c37-4ca3-9b38-e7e1328d0aa8-kube-api-access-mdss4" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821384 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b989ba1e-71f9-424b-90f1-d1450f63e8b1" volumeName="kubernetes.io/empty-dir/b989ba1e-71f9-424b-90f1-d1450f63e8b1-ca-trust-extracted" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821391 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b989ba1e-71f9-424b-90f1-d1450f63e8b1" volumeName="kubernetes.io/secret/b989ba1e-71f9-424b-90f1-d1450f63e8b1-image-registry-private-configuration" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821399 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32065295-c6a2-44ea-88e8-c4fdc5a84083" volumeName="kubernetes.io/empty-dir/32065295-c6a2-44ea-88e8-c4fdc5a84083-ca-trust-extracted" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821408 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32065295-c6a2-44ea-88e8-c4fdc5a84083" volumeName="kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-bound-sa-token" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821415 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40834bff-2db0-4690-9cb9-7b4aacdab3ab" volumeName="kubernetes.io/empty-dir/40834bff-2db0-4690-9cb9-7b4aacdab3ab-tmp" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821422 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d0db609-b271-4c98-a303-fb3ee71f152c" volumeName="kubernetes.io/projected/8d0db609-b271-4c98-a303-fb3ee71f152c-kube-api-access-mkxhk" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821428 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6f0d931-3a92-424f-a5fc-d9ebef985dfd" volumeName="kubernetes.io/configmap/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-cni-binary-copy" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821435 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b989ba1e-71f9-424b-90f1-d1450f63e8b1" volumeName="kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-bound-sa-token" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821443 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42" volumeName="kubernetes.io/secret/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821451 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40834bff-2db0-4690-9cb9-7b4aacdab3ab" volumeName="kubernetes.io/projected/40834bff-2db0-4690-9cb9-7b4aacdab3ab-kube-api-access-jclj5" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821457 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="492c7b4e-84c0-49a7-b12f-153a021f971b" volumeName="kubernetes.io/configmap/492c7b4e-84c0-49a7-b12f-153a021f971b-config" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821464 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="603868b0-8628-4227-8471-1894e2e1de50" volumeName="kubernetes.io/projected/603868b0-8628-4227-8471-1894e2e1de50-kube-api-access-89g8g" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821471 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b8992fc-fba6-4135-90f8-fb259e765dd5" volumeName="kubernetes.io/projected/7b8992fc-fba6-4135-90f8-fb259e765dd5-kube-api-access-7rnvd" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821478 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b989ba1e-71f9-424b-90f1-d1450f63e8b1" volumeName="kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-kube-api-access-28xz4" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821493 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caf7d6c9-0bb2-4da0-be2d-caea7734bbcb" volumeName="kubernetes.io/projected/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-kube-api-access-2tpvj" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821501 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2398ba99-35d3-4461-b66e-f44ac013b2d2" volumeName="kubernetes.io/configmap/2398ba99-35d3-4461-b66e-f44ac013b2d2-serviceca" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821508 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2398ba99-35d3-4461-b66e-f44ac013b2d2" volumeName="kubernetes.io/projected/2398ba99-35d3-4461-b66e-f44ac013b2d2-kube-api-access-jx7kl" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821515 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40834bff-2db0-4690-9cb9-7b4aacdab3ab" volumeName="kubernetes.io/configmap/40834bff-2db0-4690-9cb9-7b4aacdab3ab-service-ca-bundle" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821523 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53b14f9a-431e-4dbb-a725-4c8470262ca7" volumeName="kubernetes.io/empty-dir/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-tuned" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821530 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b8992fc-fba6-4135-90f8-fb259e765dd5" volumeName="kubernetes.io/secret/7b8992fc-fba6-4135-90f8-fb259e765dd5-signing-key" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821539 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d0db609-b271-4c98-a303-fb3ee71f152c" volumeName="kubernetes.io/configmap/8d0db609-b271-4c98-a303-fb3ee71f152c-telemetry-config" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821546 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32065295-c6a2-44ea-88e8-c4fdc5a84083" volumeName="kubernetes.io/configmap/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-certificates" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821554 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32065295-c6a2-44ea-88e8-c4fdc5a84083" volumeName="kubernetes.io/secret/32065295-c6a2-44ea-88e8-c4fdc5a84083-image-registry-private-configuration" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821561 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="419841be-7a31-4cdb-b0b7-9d90a3c376c7" volumeName="kubernetes.io/empty-dir/419841be-7a31-4cdb-b0b7-9d90a3c376c7-tmp-dir" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821573 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b79231d-e954-41b2-924d-67808a734ecf" volumeName="kubernetes.io/configmap/4b79231d-e954-41b2-924d-67808a734ecf-service-ca-bundle" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821580 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94d082d5-1c33-4394-9d93-67645f9b60cd" volumeName="kubernetes.io/empty-dir/94d082d5-1c33-4394-9d93-67645f9b60cd-tmp-dir" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821588 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b989ba1e-71f9-424b-90f1-d1450f63e8b1" volumeName="kubernetes.io/configmap/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-certificates" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821596 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2155512-2116-44f6-be5e-77cef72f9e14" volumeName="kubernetes.io/projected/e2155512-2116-44f6-be5e-77cef72f9e14-kube-api-access-cjhsn" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821603 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55f124eb-2ebf-4675-a08a-bad3757061b3" volumeName="kubernetes.io/projected/55f124eb-2ebf-4675-a08a-bad3757061b3-kube-api-access-lx2l9" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821610 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="603868b0-8628-4227-8471-1894e2e1de50" volumeName="kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-stats-auth" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821618 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f485893c-6f14-449b-b837-8ac181315aeb" volumeName="kubernetes.io/projected/f485893c-6f14-449b-b837-8ac181315aeb-kube-api-access-86bkx" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821625 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="377ffcf5-b097-495d-b986-e61c58ae8ec1" volumeName="kubernetes.io/empty-dir/377ffcf5-b097-495d-b986-e61c58ae8ec1-data-volume" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821632 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="419841be-7a31-4cdb-b0b7-9d90a3c376c7" volumeName="kubernetes.io/projected/419841be-7a31-4cdb-b0b7-9d90a3c376c7-kube-api-access-9tckv" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821639 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6f0d931-3a92-424f-a5fc-d9ebef985dfd" volumeName="kubernetes.io/projected/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-kube-api-access-lnslt" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821646 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6025fec-2ce4-4638-9a74-1f58c7ba7e8b" volumeName="kubernetes.io/projected/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-kube-api-access-7p75v" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821654 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" volumeName="kubernetes.io/projected/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-kube-api-access-tv87l" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821676 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40834bff-2db0-4690-9cb9-7b4aacdab3ab" volumeName="kubernetes.io/configmap/40834bff-2db0-4690-9cb9-7b4aacdab3ab-trusted-ca-bundle" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821686 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42" volumeName="kubernetes.io/configmap/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-ovnkube-config" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821694 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9d935b2-aa51-474c-836d-a74f91efab07" volumeName="kubernetes.io/projected/f9d935b2-aa51-474c-836d-a74f91efab07-kube-api-access-7c5zw" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821702 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fc518fc-6302-4a1a-9ff9-f1e036a2199b" volumeName="kubernetes.io/projected/0fc518fc-6302-4a1a-9ff9-f1e036a2199b-kube-api-access-25cd8" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821709 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32065295-c6a2-44ea-88e8-c4fdc5a84083" volumeName="kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-kube-api-access-kbrbm" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821718 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40b2621f-854e-44e0-9b01-c877e333f644" volumeName="kubernetes.io/configmap/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-config" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821725 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b79231d-e954-41b2-924d-67808a734ecf" volumeName="kubernetes.io/projected/4b79231d-e954-41b2-924d-67808a734ecf-kube-api-access-nhrbz" seLinuxMountContext="" Nov 03 19:36:45.821683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821732 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fdbd17a-1e15-4442-b715-aaec1087da11" volumeName="kubernetes.io/secret/4fdbd17a-1e15-4442-b715-aaec1087da11-agent-certs" seLinuxMountContext="" Nov 03 19:36:45.823964 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821740 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6025fec-2ce4-4638-9a74-1f58c7ba7e8b" volumeName="kubernetes.io/configmap/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-cni-binary-copy" seLinuxMountContext="" Nov 03 19:36:45.823964 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821747 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6025fec-2ce4-4638-9a74-1f58c7ba7e8b" volumeName="kubernetes.io/configmap/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-multus-daemon-config" seLinuxMountContext="" Nov 03 19:36:45.823964 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821755 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42" volumeName="kubernetes.io/configmap/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-ovnkube-script-lib" seLinuxMountContext="" Nov 03 19:36:45.823964 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821762 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="116af81a-d3a0-4558-93c4-8bf061b6815f" volumeName="kubernetes.io/secret/116af81a-d3a0-4558-93c4-8bf061b6815f-original-pull-secret" seLinuxMountContext="" Nov 03 19:36:45.823964 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821769 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40834bff-2db0-4690-9cb9-7b4aacdab3ab" volumeName="kubernetes.io/empty-dir/40834bff-2db0-4690-9cb9-7b4aacdab3ab-snapshots" seLinuxMountContext="" Nov 03 19:36:45.823964 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821776 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="492c7b4e-84c0-49a7-b12f-153a021f971b" volumeName="kubernetes.io/secret/492c7b4e-84c0-49a7-b12f-153a021f971b-serving-cert" seLinuxMountContext="" Nov 03 19:36:45.823964 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821783 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42" volumeName="kubernetes.io/configmap/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-env-overrides" seLinuxMountContext="" Nov 03 19:36:45.823964 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821790 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c99b0c19-004e-450c-852a-e38338635b61" volumeName="kubernetes.io/configmap/c99b0c19-004e-450c-852a-e38338635b61-config" seLinuxMountContext="" Nov 03 19:36:45.823964 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821797 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32065295-c6a2-44ea-88e8-c4fdc5a84083" volumeName="kubernetes.io/secret/32065295-c6a2-44ea-88e8-c4fdc5a84083-installation-pull-secrets" seLinuxMountContext="" Nov 03 19:36:45.823964 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821805 5059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="603868b0-8628-4227-8471-1894e2e1de50" volumeName="kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-default-certificate" seLinuxMountContext="" Nov 03 19:36:45.823964 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821812 5059 reconstruct.go:97] "Volume reconstruction finished" Nov 03 19:36:45.823964 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.821818 5059 reconciler.go:29] "Reconciler: start to sync state" Nov 03 19:36:45.823964 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.823776 5059 factory.go:55] Registering systemd factory Nov 03 19:36:45.823964 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.823790 5059 factory.go:223] Registration of the systemd container factory successfully Nov 03 19:36:45.824475 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.823985 5059 factory.go:153] Registering CRI-O factory Nov 03 19:36:45.824475 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.823995 5059 factory.go:223] Registration of the crio container factory successfully Nov 03 19:36:45.824475 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.824043 5059 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 19:36:45.824475 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.824064 5059 factory.go:103] Registering Raw factory Nov 03 19:36:45.824475 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.824075 5059 manager.go:1199] Started watching for new ooms in manager Nov 03 19:36:45.824475 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.824414 5059 manager.go:319] Starting recovery of all containers Nov 03 19:36:45.826501 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.826479 5059 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:36:45.826989 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.826972 5059 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:36:45.827109 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.827096 5059 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 19:36:45.829062 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.829040 5059 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:36:45.837483 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.837463 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 19:36:45.837571 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.837550 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 19:36:45.847359 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.847332 5059 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 19:36:45.848932 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.848914 5059 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 19:36:45.848932 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.848932 5059 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 19:36:45.849074 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.848949 5059 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 19:36:45.849074 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:45.848990 5059 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 19:36:45.852682 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.852643 5059 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:36:45.857721 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.857701 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 19:36:45.857817 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.857733 5059 generic.go:356] "Generic (PLEG): container finished" podID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" containerID="7bbcf23cd573d5c9e806dd6604b68ae67663631e74f89f722cb793dfee52aa69" exitCode=255 Nov 03 19:36:45.893034 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.893002 5059 generic.go:356] "Generic (PLEG): container finished" podID="3e0047dbb7ee8d4e56f23e3bd51214c3" containerID="c6978819fa7c37d1667e66b210c3e508059c55dfaaee2d872abd178cd61f6c5d" exitCode=0 Nov 03 19:36:45.900918 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.900891 5059 generic.go:356] "Generic (PLEG): container finished" podID="b6f0d931-3a92-424f-a5fc-d9ebef985dfd" containerID="62d86fa7bc9b39b63efd2667523c20f75bf119c83a70f7e97185c9173607c514" exitCode=0 Nov 03 19:36:45.900918 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.900918 5059 generic.go:356] "Generic (PLEG): container finished" podID="b6f0d931-3a92-424f-a5fc-d9ebef985dfd" containerID="cde5453097b00d29561b440f383684339fb83484f14f1515f139e905d88315ac" exitCode=0 Nov 03 19:36:45.901061 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.900928 5059 generic.go:356] "Generic (PLEG): container finished" podID="b6f0d931-3a92-424f-a5fc-d9ebef985dfd" containerID="d350c3a6349a6db16ea21d5f7e3613e81cc603fc770eac0d77c7b52ea6723cb8" exitCode=0 Nov 03 19:36:45.901061 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.900937 5059 generic.go:356] "Generic (PLEG): container finished" podID="b6f0d931-3a92-424f-a5fc-d9ebef985dfd" containerID="dbaa1409db8655e6bf968118e5ebd222bdf95aacaf439e7ed5c4f593c69f7c20" exitCode=0 Nov 03 19:36:45.901061 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.900946 5059 generic.go:356] "Generic (PLEG): container finished" podID="b6f0d931-3a92-424f-a5fc-d9ebef985dfd" containerID="b6d4d0233d7933e7962748fb0657527fb9f2f0f264d350c54c1b150e851dc498" exitCode=0 Nov 03 19:36:45.901061 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.900954 5059 generic.go:356] "Generic (PLEG): container finished" podID="b6f0d931-3a92-424f-a5fc-d9ebef985dfd" containerID="8375618a0150cf854feba7f5b17ca368bd2306889a76ded11e3299f2f6eba684" exitCode=0 Nov 03 19:36:45.921429 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.921385 5059 manager.go:324] Recovery completed Nov 03 19:36:45.941433 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.941415 5059 cpu_manager.go:226] "Starting" policy="none" Nov 03 19:36:45.941433 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.941429 5059 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 19:36:45.941616 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.941445 5059 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 19:36:45.941616 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.941572 5059 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 19:36:45.941616 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.941583 5059 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 19:36:45.941616 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.941599 5059 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 19:36:45.941616 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.941605 5059 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 19:36:45.941616 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.941610 5059 policy_none.go:50] "Start" Nov 03 19:36:45.941814 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.941627 5059 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 19:36:45.941814 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.941637 5059 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 19:36:45.941814 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.941788 5059 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 19:36:45.941814 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.941796 5059 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 19:36:45.941814 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.941801 5059 policy_none.go:44] "Start" Nov 03 19:36:45.945944 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.945917 5059 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 19:36:45.948412 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.948395 5059 manager.go:341] "Starting Device Plugin manager" Nov 03 19:36:45.948941 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:45.948420 5059 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 19:36:45.948941 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.948940 5059 server.go:87] "Starting device plugin registration server" Nov 03 19:36:45.949195 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.949181 5059 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 19:36:45.949273 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.949193 5059 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 19:36:45.949351 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.949284 5059 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 19:36:45.949351 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:45.949200 5059 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 19:36:45.949442 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.949372 5059 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 19:36:45.949442 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.949414 5059 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 19:36:45.949442 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.949421 5059 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 19:36:45.949592 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.949470 5059 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:36:45.949402762Z","UUID":"93d14099-072e-4c14-96fd-b59bbbb46b3e","Handler":null,"Name":"","Endpoint":""} Nov 03 19:36:45.950878 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.950863 5059 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 19:36:45.950961 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:45.950886 5059 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 19:36:45.951056 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:45.951039 5059 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 19:36:46.050147 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.050118 5059 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:36:46.051300 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.051276 5059 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:36:46.051404 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.051313 5059 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:36:46.051404 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.051324 5059 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:36:46.051404 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.051361 5059 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:46.063738 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.063718 5059 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:46.063794 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.063773 5059 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:46.150015 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.149964 5059 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal"] Nov 03 19:36:46.150656 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.150624 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal" event={"ID":"ea3063cb5f7f736e0060bab60ff36a5e","Type":"ContainerStarted","Data":"c9c1860be7dfaad8a21ff56677a08bb380d30d2c7b48c834c50e89b4e2eb7555"} Nov 03 19:36:46.150721 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.150691 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal" event={"ID":"ea3063cb5f7f736e0060bab60ff36a5e","Type":"ContainerStarted","Data":"d74276b7e45093444f92ecc5311a048e8ade116d042b902e48a8c8dd526a8f79"} Nov 03 19:36:46.150721 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.150712 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" event={"ID":"3e0047dbb7ee8d4e56f23e3bd51214c3","Type":"ContainerStarted","Data":"88ccefea5dcfebefffd8248d43bf2bd987ea1b0002b05b75afaa845ea72bd60e"} Nov 03 19:36:46.150777 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.150732 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" event={"ID":"3e0047dbb7ee8d4e56f23e3bd51214c3","Type":"ContainerDied","Data":"c6978819fa7c37d1667e66b210c3e508059c55dfaaee2d872abd178cd61f6c5d"} Nov 03 19:36:46.150777 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.150735 5059 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:46.150777 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.150745 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" event={"ID":"3e0047dbb7ee8d4e56f23e3bd51214c3","Type":"ContainerStarted","Data":"0c4f7b61fcd4ba2db6accad32dd25c0948b92f9ceb71408c97376cd46a8690c4"} Nov 03 19:36:46.150908 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.150778 5059 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:46.165039 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.165022 5059 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:36:46.165082 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:46.165061 5059 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:46.165467 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.165455 5059 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:36:46.165497 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:46.165487 5059 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:46.229077 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.229013 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e0047dbb7ee8d4e56f23e3bd51214c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal\" (UID: \"3e0047dbb7ee8d4e56f23e3bd51214c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:46.229077 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.229041 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e0047dbb7ee8d4e56f23e3bd51214c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal\" (UID: \"3e0047dbb7ee8d4e56f23e3bd51214c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:46.229077 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.229062 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ea3063cb5f7f736e0060bab60ff36a5e-config\") pod \"kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal\" (UID: \"ea3063cb5f7f736e0060bab60ff36a5e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:46.329679 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.329634 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e0047dbb7ee8d4e56f23e3bd51214c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal\" (UID: \"3e0047dbb7ee8d4e56f23e3bd51214c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:46.329679 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.329585 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e0047dbb7ee8d4e56f23e3bd51214c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal\" (UID: \"3e0047dbb7ee8d4e56f23e3bd51214c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:46.329874 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.329728 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e0047dbb7ee8d4e56f23e3bd51214c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal\" (UID: \"3e0047dbb7ee8d4e56f23e3bd51214c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:46.329874 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.329751 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ea3063cb5f7f736e0060bab60ff36a5e-config\") pod \"kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal\" (UID: \"ea3063cb5f7f736e0060bab60ff36a5e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:46.329874 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.329793 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ea3063cb5f7f736e0060bab60ff36a5e-config\") pod \"kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal\" (UID: \"ea3063cb5f7f736e0060bab60ff36a5e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:46.329874 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.329823 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e0047dbb7ee8d4e56f23e3bd51214c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal\" (UID: \"3e0047dbb7ee8d4e56f23e3bd51214c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:46.804031 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.804004 5059 apiserver.go:52] "Watching apiserver" Nov 03 19:36:46.814588 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.814558 5059 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 19:36:46.815272 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.815245 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fgdh8","openshift-multus/multus-xlfb8","kube-system/kas-connection-checker-685895c7c9-jt54m","openshift-image-registry/node-ca-kzr2l","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x","kube-system/kas-connection-checker-685895c7c9-bd8f8","openshift-image-registry/image-registry-799ffcd7c9-5pdd7","openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl","openshift-multus/network-metrics-daemon-qvxjq","openshift-network-console/networking-console-plugin-86c478bb88-846mt","openshift-dns/node-resolver-4d4ll","openshift-image-registry/image-registry-6dd465cd78-4n8kx","openshift-kube-storage-version-migrator/migrator-66df5d496b-6k9n9","kube-system/kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal","openshift-dns/dns-default-nhf67","openshift-service-ca/service-ca-5fffc97bdf-brvgz","openshift-ingress/router-default-f7b87d695-cwlfq","openshift-ovn-kubernetes/ovnkube-node-2m4mq","openshift-multus/multus-additional-cni-plugins-cxdrc","openshift-kube-storage-version-migrator/migrator-66df5d496b-pbgwk","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fsbsp","openshift-insights/periodic-gathering-bhdxh-d8v5q","openshift-insights/insights-runtime-extractor-hmnx4","kube-system/global-pull-secret-syncer-wh9tj","openshift-network-diagnostics/network-check-target-b4x9q","openshift-network-diagnostics/network-check-source-6f78d59f99-tvb96","kube-system/kas-connection-checker-685895c7c9-6hx8t","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx","openshift-insights/insights-operator-7c4986d449-wdls7","openshift-console-operator/console-operator-7cd56b8dbc-hmc5z","openshift-cluster-node-tuning-operator/tuned-mkm7n","kube-system/konnectivity-agent-q4zkg","openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal","openshift-network-operator/iptables-alerter-7zx7h"] Nov 03 19:36:46.817048 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.817026 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:46.818274 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.818255 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:46.818726 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.818710 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:46.819201 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.819182 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:46.820323 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.820305 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" Nov 03 19:36:46.826802 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.826777 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 19:36:46.827192 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.827173 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 19:36:46.827371 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.827181 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 19:36:46.827457 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.827390 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 19:36:46.828347 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.828327 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 19:36:46.828544 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.828433 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 19:36:46.828544 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.828470 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 19:36:46.828544 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.828450 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 19:36:46.828747 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.828625 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fw8qp\"" Nov 03 19:36:46.828889 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.828874 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 19:36:46.828966 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.828944 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:36:46.829008 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.828992 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:36:46.829046 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.829010 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-p6m7s\"" Nov 03 19:36:46.829046 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.829037 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 19:36:46.829113 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.829085 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 19:36:46.829167 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.829117 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 19:36:46.829167 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.829158 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xr7jf\"" Nov 03 19:36:46.829260 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.829166 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 19:36:46.829260 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.829192 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2r65v\"" Nov 03 19:36:46.829355 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.829258 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bfvcm\"" Nov 03 19:36:46.829355 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.829267 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 19:36:46.829355 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.829311 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 19:36:46.829355 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.829342 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 19:36:46.829355 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.829313 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 19:36:46.829583 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.829365 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 19:36:46.829583 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.829405 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 19:36:46.829583 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.829431 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rwt79\"" Nov 03 19:36:46.829762 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.829620 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 19:36:46.830390 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.830367 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:36:46.830390 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.830386 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 19:36:46.830607 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.830572 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 19:36:46.830710 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.830689 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 19:36:46.830768 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.830713 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 19:36:46.830833 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.830792 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:36:46.830833 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.830817 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 19:36:46.830996 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.830834 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 19:36:46.830996 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.830904 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:36:46.830996 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.830797 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 19:36:46.830996 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.830909 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 19:36:46.830996 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.830941 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 19:36:46.831210 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.831125 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 19:36:46.831274 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.831252 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 19:36:46.831465 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.831440 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 19:36:46.831532 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.831485 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:36:46.831532 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.831513 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 19:36:46.831646 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.831596 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 19:36:46.831646 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.831607 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 19:36:46.831847 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.831650 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 19:36:46.831847 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.831654 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 19:36:46.831847 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.831832 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 19:36:46.832048 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.831952 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 19:36:46.832048 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.831985 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 19:36:46.832048 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.832026 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:46.832296 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.832274 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-metrics-certs\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:46.832378 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.832314 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c99b0c19-004e-450c-852a-e38338635b61-config\") pod \"service-ca-operator-544d749c8c-xzpgl\" (UID: \"c99b0c19-004e-450c-852a-e38338635b61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" Nov 03 19:36:46.832543 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.832527 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 19:36:46.832594 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.832565 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40834bff-2db0-4690-9cb9-7b4aacdab3ab-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:46.832644 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.832615 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40834bff-2db0-4690-9cb9-7b4aacdab3ab-service-ca-bundle\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:46.832722 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.832650 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkxhk\" (UniqueName: \"kubernetes.io/projected/8d0db609-b271-4c98-a303-fb3ee71f152c-kube-api-access-mkxhk\") pod \"cluster-monitoring-operator-884576c57-h9f7b\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:46.832722 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.832700 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c99b0c19-004e-450c-852a-e38338635b61-config\") pod \"service-ca-operator-544d749c8c-xzpgl\" (UID: \"c99b0c19-004e-450c-852a-e38338635b61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" Nov 03 19:36:46.832815 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.832704 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/492c7b4e-84c0-49a7-b12f-153a021f971b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wh95x\" (UID: \"492c7b4e-84c0-49a7-b12f-153a021f971b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" Nov 03 19:36:46.832815 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.832777 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40834bff-2db0-4690-9cb9-7b4aacdab3ab-tmp\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:46.832922 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.832816 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8d0db609-b271-4c98-a303-fb3ee71f152c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-h9f7b\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:46.832922 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.832849 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c5zw\" (UniqueName: \"kubernetes.io/projected/f9d935b2-aa51-474c-836d-a74f91efab07-kube-api-access-7c5zw\") pod \"kas-connection-checker-685895c7c9-6hx8t\" (UID: \"f9d935b2-aa51-474c-836d-a74f91efab07\") " pod="kube-system/kas-connection-checker-685895c7c9-6hx8t" Nov 03 19:36:46.832922 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.832877 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/492c7b4e-84c0-49a7-b12f-153a021f971b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wh95x\" (UID: \"492c7b4e-84c0-49a7-b12f-153a021f971b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" Nov 03 19:36:46.832922 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.832910 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgtxr\" (UniqueName: \"kubernetes.io/projected/492c7b4e-84c0-49a7-b12f-153a021f971b-kube-api-access-hgtxr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wh95x\" (UID: \"492c7b4e-84c0-49a7-b12f-153a021f971b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" Nov 03 19:36:46.833163 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.832964 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/492c7b4e-84c0-49a7-b12f-153a021f971b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wh95x\" (UID: \"492c7b4e-84c0-49a7-b12f-153a021f971b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" Nov 03 19:36:46.833163 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.832971 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40834bff-2db0-4690-9cb9-7b4aacdab3ab-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:46.833163 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.832992 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-stats-auth\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:46.833163 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833024 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv87l\" (UniqueName: \"kubernetes.io/projected/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-kube-api-access-tv87l\") pod \"console-operator-7cd56b8dbc-hmc5z\" (UID: \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:46.833163 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833069 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-86qxx\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:46.833163 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833093 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-default-certificate\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:46.833163 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833117 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk5v5\" (UniqueName: \"kubernetes.io/projected/c99b0c19-004e-450c-852a-e38338635b61-kube-api-access-xk5v5\") pod \"service-ca-operator-544d749c8c-xzpgl\" (UID: \"c99b0c19-004e-450c-852a-e38338635b61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" Nov 03 19:36:46.833163 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833124 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40834bff-2db0-4690-9cb9-7b4aacdab3ab-service-ca-bundle\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:46.833163 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833144 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jclj5\" (UniqueName: \"kubernetes.io/projected/40834bff-2db0-4690-9cb9-7b4aacdab3ab-kube-api-access-jclj5\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:46.833588 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833174 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-config\") pod \"console-operator-7cd56b8dbc-hmc5z\" (UID: \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:46.833588 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833366 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40834bff-2db0-4690-9cb9-7b4aacdab3ab-tmp\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:46.833588 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833396 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/40834bff-2db0-4690-9cb9-7b4aacdab3ab-snapshots\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:46.833588 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833403 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-stats-auth\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:46.833588 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833430 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-serving-cert\") pod \"console-operator-7cd56b8dbc-hmc5z\" (UID: \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:46.833588 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833429 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-config\") pod \"console-operator-7cd56b8dbc-hmc5z\" (UID: \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:46.833588 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833461 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/40834bff-2db0-4690-9cb9-7b4aacdab3ab-serving-cert\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:46.833588 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833485 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c99b0c19-004e-450c-852a-e38338635b61-serving-cert\") pod \"service-ca-operator-544d749c8c-xzpgl\" (UID: \"c99b0c19-004e-450c-852a-e38338635b61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" Nov 03 19:36:46.833588 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833520 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc5qt\" (UniqueName: \"kubernetes.io/projected/40b2621f-854e-44e0-9b01-c877e333f644-kube-api-access-jc5qt\") pod \"cluster-samples-operator-6ddb6c8b94-86qxx\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:46.833588 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833516 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/492c7b4e-84c0-49a7-b12f-153a021f971b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wh95x\" (UID: \"492c7b4e-84c0-49a7-b12f-153a021f971b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" Nov 03 19:36:46.833588 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833562 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-h9f7b\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:46.834152 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833599 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-86qxx\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:46.834152 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833614 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8d0db609-b271-4c98-a303-fb3ee71f152c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-h9f7b\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:46.834152 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833643 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25cd8\" (UniqueName: \"kubernetes.io/projected/0fc518fc-6302-4a1a-9ff9-f1e036a2199b-kube-api-access-25cd8\") pod \"kas-connection-checker-685895c7c9-bd8f8\" (UID: \"0fc518fc-6302-4a1a-9ff9-f1e036a2199b\") " pod="kube-system/kas-connection-checker-685895c7c9-bd8f8" Nov 03 19:36:46.834152 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833683 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:46.834152 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833673 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-86qxx\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:46.834152 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833725 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89g8g\" (UniqueName: \"kubernetes.io/projected/603868b0-8628-4227-8471-1894e2e1de50-kube-api-access-89g8g\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:46.834152 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833754 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjhsn\" (UniqueName: \"kubernetes.io/projected/e2155512-2116-44f6-be5e-77cef72f9e14-kube-api-access-cjhsn\") pod \"kas-connection-checker-685895c7c9-jt54m\" (UID: \"e2155512-2116-44f6-be5e-77cef72f9e14\") " pod="kube-system/kas-connection-checker-685895c7c9-jt54m" Nov 03 19:36:46.834152 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833778 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-trusted-ca\") pod \"console-operator-7cd56b8dbc-hmc5z\" (UID: \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:46.834152 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833793 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-default-certificate\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:46.834152 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833801 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c99b0c19-004e-450c-852a-e38338635b61-serving-cert\") pod \"service-ca-operator-544d749c8c-xzpgl\" (UID: \"c99b0c19-004e-450c-852a-e38338635b61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" Nov 03 19:36:46.834152 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833866 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/40834bff-2db0-4690-9cb9-7b4aacdab3ab-snapshots\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:46.834152 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833926 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/40834bff-2db0-4690-9cb9-7b4aacdab3ab-serving-cert\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:46.834152 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.833985 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-serving-cert\") pod \"console-operator-7cd56b8dbc-hmc5z\" (UID: \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:46.834152 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.834057 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-trusted-ca\") pod \"console-operator-7cd56b8dbc-hmc5z\" (UID: \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:46.834914 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.834329 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:46.834914 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.834540 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 19:36:46.834914 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.834586 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 19:36:46.834914 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.834606 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:36:46.834914 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.834702 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 19:36:46.834914 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.834707 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 19:36:46.834914 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.834751 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 19:36:46.835587 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.835561 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 19:36:46.839474 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.839447 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:46.839891 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.839867 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 19:36:46.840290 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.840190 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 19:36:46.840418 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.840307 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 19:36:46.840418 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.840334 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 19:36:46.840560 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.840484 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 19:36:46.840560 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.840523 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 19:36:46.840655 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.840553 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 19:36:46.840655 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.840568 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 19:36:46.840803 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.840678 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 19:36:46.840803 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.840722 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 19:36:46.840972 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.840952 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 19:36:46.841141 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.841101 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 19:36:46.841217 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.841202 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 19:36:46.842775 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.842745 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:46.843730 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.843710 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 19:36:46.843730 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.843715 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 19:36:46.843844 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.843764 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 19:36:46.843987 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.843962 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c5zw\" (UniqueName: \"kubernetes.io/projected/f9d935b2-aa51-474c-836d-a74f91efab07-kube-api-access-7c5zw\") pod \"kas-connection-checker-685895c7c9-6hx8t\" (UID: \"f9d935b2-aa51-474c-836d-a74f91efab07\") " pod="kube-system/kas-connection-checker-685895c7c9-6hx8t" Nov 03 19:36:46.844066 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.844048 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgtxr\" (UniqueName: \"kubernetes.io/projected/492c7b4e-84c0-49a7-b12f-153a021f971b-kube-api-access-hgtxr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wh95x\" (UID: \"492c7b4e-84c0-49a7-b12f-153a021f971b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" Nov 03 19:36:46.844148 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.844131 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 19:36:46.844203 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.844155 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc5qt\" (UniqueName: \"kubernetes.io/projected/40b2621f-854e-44e0-9b01-c877e333f644-kube-api-access-jc5qt\") pod \"cluster-samples-operator-6ddb6c8b94-86qxx\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:46.844283 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.844265 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkxhk\" (UniqueName: \"kubernetes.io/projected/8d0db609-b271-4c98-a303-fb3ee71f152c-kube-api-access-mkxhk\") pod \"cluster-monitoring-operator-884576c57-h9f7b\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:46.845647 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.845630 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 19:36:46.846394 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.846376 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 19:36:46.846470 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.846420 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 19:36:46.846470 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.846431 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 19:36:46.846555 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.846472 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 19:36:46.849007 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.848984 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk5v5\" (UniqueName: \"kubernetes.io/projected/c99b0c19-004e-450c-852a-e38338635b61-kube-api-access-xk5v5\") pod \"service-ca-operator-544d749c8c-xzpgl\" (UID: \"c99b0c19-004e-450c-852a-e38338635b61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" Nov 03 19:36:46.849253 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.849235 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25cd8\" (UniqueName: \"kubernetes.io/projected/0fc518fc-6302-4a1a-9ff9-f1e036a2199b-kube-api-access-25cd8\") pod \"kas-connection-checker-685895c7c9-bd8f8\" (UID: \"0fc518fc-6302-4a1a-9ff9-f1e036a2199b\") " pod="kube-system/kas-connection-checker-685895c7c9-bd8f8" Nov 03 19:36:46.849385 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.849366 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv87l\" (UniqueName: \"kubernetes.io/projected/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-kube-api-access-tv87l\") pod \"console-operator-7cd56b8dbc-hmc5z\" (UID: \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:46.849429 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.849376 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jclj5\" (UniqueName: \"kubernetes.io/projected/40834bff-2db0-4690-9cb9-7b4aacdab3ab-kube-api-access-jclj5\") pod \"insights-operator-7c4986d449-wdls7\" (UID: \"40834bff-2db0-4690-9cb9-7b4aacdab3ab\") " pod="openshift-insights/insights-operator-7c4986d449-wdls7" Nov 03 19:36:46.849429 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.849418 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89g8g\" (UniqueName: \"kubernetes.io/projected/603868b0-8628-4227-8471-1894e2e1de50-kube-api-access-89g8g\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:46.849723 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.849705 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjhsn\" (UniqueName: \"kubernetes.io/projected/e2155512-2116-44f6-be5e-77cef72f9e14-kube-api-access-cjhsn\") pod \"kas-connection-checker-685895c7c9-jt54m\" (UID: \"e2155512-2116-44f6-be5e-77cef72f9e14\") " pod="kube-system/kas-connection-checker-685895c7c9-jt54m" Nov 03 19:36:46.905657 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.905637 5059 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:46.905813 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.905797 5059 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:46.915863 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.915842 5059 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:36:46.915943 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:46.915884 5059 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:46.920156 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.920141 5059 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 19:36:46.921230 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.921216 5059 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:36:46.921293 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:46.921247 5059 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal" Nov 03 19:36:46.921801 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.921787 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 19:36:46.922084 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.922069 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 19:36:46.922348 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.922322 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tx8mk\"" Nov 03 19:36:46.922460 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.922364 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 19:36:46.922460 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.922421 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 19:36:46.922549 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.922456 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dgjzc\"" Nov 03 19:36:46.922549 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.922477 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 19:36:46.922549 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.922544 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qhpjg\"" Nov 03 19:36:46.922709 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.922691 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5qm8x\"" Nov 03 19:36:46.922811 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.922796 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jzp4r\"" Nov 03 19:36:46.922854 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.922801 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 19:36:46.922854 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.922817 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vlmgx\"" Nov 03 19:36:46.922854 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.922800 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 19:36:46.923537 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.923523 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 19:36:46.923766 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.923683 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b2f85\"" Nov 03 19:36:46.923766 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.923728 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vf7n8\"" Nov 03 19:36:46.923766 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.923740 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lrbp4\"" Nov 03 19:36:46.923766 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.923747 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bkd9f\"" Nov 03 19:36:46.924024 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.923833 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xbkg2\"" Nov 03 19:36:46.924024 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.923854 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 19:36:46.924024 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.923862 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ldllm\"" Nov 03 19:36:46.924024 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.923878 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-dkxw8\"" Nov 03 19:36:46.924024 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.923878 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-b22jw\"" Nov 03 19:36:46.924024 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.923929 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xv4gr\"" Nov 03 19:36:46.924024 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.923932 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4p6h4\"" Nov 03 19:36:46.924024 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.923942 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ndfpg\"" Nov 03 19:36:46.924024 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.923943 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qg9jj\"" Nov 03 19:36:46.924024 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.923940 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 19:36:46.924024 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.923953 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rs44g\"" Nov 03 19:36:46.934276 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.934257 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/377ffcf5-b097-495d-b986-e61c58ae8ec1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:46.934359 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.934291 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsppf\" (UniqueName: \"kubernetes.io/projected/94d082d5-1c33-4394-9d93-67645f9b60cd-kube-api-access-rsppf\") pod \"node-resolver-4d4ll\" (UID: \"94d082d5-1c33-4394-9d93-67645f9b60cd\") " pod="openshift-dns/node-resolver-4d4ll" Nov 03 19:36:46.934359 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.934316 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28xz4\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-kube-api-access-28xz4\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:46.934460 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.934423 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-systemd-units\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.934460 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.934448 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55f124eb-2ebf-4675-a08a-bad3757061b3-host-slash\") pod \"iptables-alerter-7zx7h\" (UID: \"55f124eb-2ebf-4675-a08a-bad3757061b3\") " pod="openshift-network-operator/iptables-alerter-7zx7h" Nov 03 19:36:46.934569 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.934463 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-etc-kubernetes\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:46.934569 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.934510 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tpvj\" (UniqueName: \"kubernetes.io/projected/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-kube-api-access-2tpvj\") pod \"network-metrics-daemon-qvxjq\" (UID: \"caf7d6c9-0bb2-4da0-be2d-caea7734bbcb\") " pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:46.934569 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.934542 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-systemd\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:46.934692 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.934593 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-ovnkube-config\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.934692 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.934648 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/377ffcf5-b097-495d-b986-e61c58ae8ec1-crio-socket\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:46.934763 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.934699 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-run-k8s-cni-cncf-io\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:46.934763 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.934738 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-var-lib-kubelet\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:46.934825 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.934767 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-run-openvswitch\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.934825 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.934799 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:46.934914 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.934895 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-ovnkube-config\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.934967 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:46.934926 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle podName:603868b0-8628-4227-8471-1894e2e1de50 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:47.434907373 +0000 UTC m=+1.692704634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle") pod "router-default-f7b87d695-cwlfq" (UID: "603868b0-8628-4227-8471-1894e2e1de50") : configmap references non-existent config key: service-ca.crt Nov 03 19:36:46.934967 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.934942 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnslt\" (UniqueName: \"kubernetes.io/projected/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-kube-api-access-lnslt\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:46.935080 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.934975 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4fdbd17a-1e15-4442-b715-aaec1087da11-agent-certs\") pod \"konnectivity-agent-q4zkg\" (UID: \"4fdbd17a-1e15-4442-b715-aaec1087da11\") " pod="kube-system/konnectivity-agent-q4zkg" Nov 03 19:36:46.935080 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935011 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32065295-c6a2-44ea-88e8-c4fdc5a84083-trusted-ca\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:46.935080 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935035 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b989ba1e-71f9-424b-90f1-d1450f63e8b1-trusted-ca\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:46.935080 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935059 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzh28\" (UniqueName: \"kubernetes.io/projected/377ffcf5-b097-495d-b986-e61c58ae8ec1-kube-api-access-hzh28\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:46.935265 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935171 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-metrics-certs\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:46.935265 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935203 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-etc-selinux\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:46.935265 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935233 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-run-multus-certs\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:46.935265 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935253 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/116af81a-d3a0-4558-93c4-8bf061b6815f-kubelet-config\") pod \"global-pull-secret-syncer-wh9tj\" (UID: \"116af81a-d3a0-4558-93c4-8bf061b6815f\") " pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:46.935413 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935282 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-bound-sa-token\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:46.935413 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935334 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4fdbd17a-1e15-4442-b715-aaec1087da11-agent-certs\") pod \"konnectivity-agent-q4zkg\" (UID: \"4fdbd17a-1e15-4442-b715-aaec1087da11\") " pod="kube-system/konnectivity-agent-q4zkg" Nov 03 19:36:46.935506 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935424 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wvnz\" (UniqueName: \"kubernetes.io/projected/1bcd4690-4753-4246-abb4-8668a0a56231-kube-api-access-8wvnz\") pod \"volume-data-source-validator-58c9f5f46d-fsbsp\" (UID: \"1bcd4690-4753-4246-abb4-8668a0a56231\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fsbsp" Nov 03 19:36:46.935506 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935432 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32065295-c6a2-44ea-88e8-c4fdc5a84083-trusted-ca\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:46.935506 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935465 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4b79231d-e954-41b2-924d-67808a734ecf-archives-path\") pod \"periodic-gathering-bhdxh-d8v5q\" (UID: \"4b79231d-e954-41b2-924d-67808a734ecf\") " pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" Nov 03 19:36:46.935643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935502 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/04a982f9-143c-406c-99d5-9ef73d5c571a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-846mt\" (UID: \"04a982f9-143c-406c-99d5-9ef73d5c571a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" Nov 03 19:36:46.935643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935534 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-run\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:46.935643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935519 5059 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 19:36:46.935643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935576 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:46.935643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935584 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4b79231d-e954-41b2-924d-67808a734ecf-archives-path\") pod \"periodic-gathering-bhdxh-d8v5q\" (UID: \"4b79231d-e954-41b2-924d-67808a734ecf\") " pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" Nov 03 19:36:46.935643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935606 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-multus-conf-dir\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:46.935863 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935650 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4fdbd17a-1e15-4442-b715-aaec1087da11-konnectivity-ca\") pod \"konnectivity-agent-q4zkg\" (UID: \"4fdbd17a-1e15-4442-b715-aaec1087da11\") " pod="kube-system/konnectivity-agent-q4zkg" Nov 03 19:36:46.935863 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935781 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/04a982f9-143c-406c-99d5-9ef73d5c571a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-846mt\" (UID: \"04a982f9-143c-406c-99d5-9ef73d5c571a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" Nov 03 19:36:46.935863 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935790 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-tls\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:46.935863 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935838 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/377ffcf5-b097-495d-b986-e61c58ae8ec1-data-volume\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:46.935999 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935864 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-system-cni-dir\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:46.935999 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935879 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4fdbd17a-1e15-4442-b715-aaec1087da11-konnectivity-ca\") pod \"konnectivity-agent-q4zkg\" (UID: \"4fdbd17a-1e15-4442-b715-aaec1087da11\") " pod="kube-system/konnectivity-agent-q4zkg" Nov 03 19:36:46.935999 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935884 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-cnibin\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:46.935999 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935918 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/377ffcf5-b097-495d-b986-e61c58ae8ec1-data-volume\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:46.935999 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935921 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rnvd\" (UniqueName: \"kubernetes.io/projected/7b8992fc-fba6-4135-90f8-fb259e765dd5-kube-api-access-7rnvd\") pod \"service-ca-5fffc97bdf-brvgz\" (UID: \"7b8992fc-fba6-4135-90f8-fb259e765dd5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" Nov 03 19:36:46.935999 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935972 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-run-netns\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.935999 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935980 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b989ba1e-71f9-424b-90f1-d1450f63e8b1-trusted-ca\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:46.936273 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.935999 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-run-ovn\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.936273 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.936024 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-var-lib-kubelet\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:46.936273 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.936050 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-cni-netd\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.936273 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.936095 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-86qxx\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:46.936273 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.936124 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-multus-socket-dir-parent\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:46.936273 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.936141 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-var-lib-cni-bin\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:46.936273 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.936162 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32065295-c6a2-44ea-88e8-c4fdc5a84083-image-registry-private-configuration\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:46.936273 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.936178 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-lib-modules\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:46.936273 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:46.936173 5059 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 19:36:46.936273 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:46.936237 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls podName:40b2621f-854e-44e0-9b01-c877e333f644 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:47.436222583 +0000 UTC m=+1.694019843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-86qxx" (UID: "40b2621f-854e-44e0-9b01-c877e333f644") : secret "samples-operator-tls" not found Nov 03 19:36:46.936273 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.936257 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5xvg\" (UniqueName: \"kubernetes.io/projected/53b14f9a-431e-4dbb-a725-4c8470262ca7-kube-api-access-m5xvg\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:46.936789 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.936319 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-slash\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.936789 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.936354 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7b8992fc-fba6-4135-90f8-fb259e765dd5-signing-key\") pod \"service-ca-5fffc97bdf-brvgz\" (UID: \"7b8992fc-fba6-4135-90f8-fb259e765dd5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" Nov 03 19:36:46.936789 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.936371 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32065295-c6a2-44ea-88e8-c4fdc5a84083-image-registry-private-configuration\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:46.936789 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.936387 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32065295-c6a2-44ea-88e8-c4fdc5a84083-installation-pull-secrets\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:46.936789 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.936418 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-log-socket\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.936789 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.936474 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b79231d-e954-41b2-924d-67808a734ecf-trusted-ca-bundle\") pod \"periodic-gathering-bhdxh-d8v5q\" (UID: \"4b79231d-e954-41b2-924d-67808a734ecf\") " pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" Nov 03 19:36:46.936789 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.936552 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbrbm\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-kube-api-access-kbrbm\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:46.936789 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.936590 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7b8992fc-fba6-4135-90f8-fb259e765dd5-signing-key\") pod \"service-ca-5fffc97bdf-brvgz\" (UID: \"7b8992fc-fba6-4135-90f8-fb259e765dd5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" Nov 03 19:36:46.936789 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.936595 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32065295-c6a2-44ea-88e8-c4fdc5a84083-installation-pull-secrets\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:46.936789 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.936592 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-tuned\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:46.936789 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.936640 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-etc-openvswitch\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.936789 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.936692 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-tuned\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:46.936789 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.936739 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-cni-bin\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.936789 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.936773 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:46.937434 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.936883 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:46.937434 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.936962 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:46.937434 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.937095 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-socket-dir\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:46.937434 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.937144 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn9fc\" (UniqueName: \"kubernetes.io/projected/e64f9d9f-c868-470e-bd78-4ce8baabd97a-kube-api-access-mn9fc\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:46.937434 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.937201 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-ovnkube-script-lib\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.937434 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.937262 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-multus-daemon-config\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:46.937434 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.937287 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:46.937434 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.937403 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p75v\" (UniqueName: \"kubernetes.io/projected/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-kube-api-access-7p75v\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:46.937824 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.937470 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-sysconfig\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:46.937824 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.937540 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-ovnkube-script-lib\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.937824 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.937587 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-sys\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:46.937824 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.937630 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-var-lib-openvswitch\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.937824 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.937655 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b79231d-e954-41b2-924d-67808a734ecf-trusted-ca-bundle\") pod \"periodic-gathering-bhdxh-d8v5q\" (UID: \"4b79231d-e954-41b2-924d-67808a734ecf\") " pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" Nov 03 19:36:46.937824 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.937684 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-multus-daemon-config\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:46.937824 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.937683 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/419841be-7a31-4cdb-b0b7-9d90a3c376c7-tmp-dir\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:46.937824 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.937759 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/419841be-7a31-4cdb-b0b7-9d90a3c376c7-tmp-dir\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:46.937824 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.937782 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/94d082d5-1c33-4394-9d93-67645f9b60cd-hosts-file\") pod \"node-resolver-4d4ll\" (UID: \"94d082d5-1c33-4394-9d93-67645f9b60cd\") " pod="openshift-dns/node-resolver-4d4ll" Nov 03 19:36:46.937824 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.937806 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs\") pod \"network-metrics-daemon-qvxjq\" (UID: \"caf7d6c9-0bb2-4da0-be2d-caea7734bbcb\") " pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:46.938319 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.937845 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx7kl\" (UniqueName: \"kubernetes.io/projected/2398ba99-35d3-4461-b66e-f44ac013b2d2-kube-api-access-jx7kl\") pod \"node-ca-kzr2l\" (UID: \"2398ba99-35d3-4461-b66e-f44ac013b2d2\") " pod="openshift-image-registry/node-ca-kzr2l" Nov 03 19:36:46.938319 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.937970 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86bkx\" (UniqueName: \"kubernetes.io/projected/f485893c-6f14-449b-b837-8ac181315aeb-kube-api-access-86bkx\") pod \"migrator-66df5d496b-6k9n9\" (UID: \"f485893c-6f14-449b-b837-8ac181315aeb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6k9n9" Nov 03 19:36:46.938319 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.938029 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-h9f7b\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:46.938319 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.938050 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-run-netns\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:46.938319 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.938071 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-node-log\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.938319 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.938141 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hszb\" (UniqueName: \"kubernetes.io/projected/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-kube-api-access-9hszb\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.938319 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:46.938184 5059 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 19:36:46.938319 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.938177 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b79231d-e954-41b2-924d-67808a734ecf-service-ca-bundle\") pod \"periodic-gathering-bhdxh-d8v5q\" (UID: \"4b79231d-e954-41b2-924d-67808a734ecf\") " pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" Nov 03 19:36:46.938319 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:46.938246 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls podName:8d0db609-b271-4c98-a303-fb3ee71f152c nodeName:}" failed. No retries permitted until 2026-11-03 19:36:47.438218222 +0000 UTC m=+1.696015476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-h9f7b" (UID: "8d0db609-b271-4c98-a303-fb3ee71f152c") : secret "cluster-monitoring-operator-tls" not found Nov 03 19:36:46.938319 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.938312 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:46.938820 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.938350 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-kubelet\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.938820 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.938367 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b79231d-e954-41b2-924d-67808a734ecf-service-ca-bundle\") pod \"periodic-gathering-bhdxh-d8v5q\" (UID: \"4b79231d-e954-41b2-924d-67808a734ecf\") " pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" Nov 03 19:36:46.938820 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.938380 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tvmx\" (UniqueName: \"kubernetes.io/projected/0a985b50-c4c0-4a39-abed-29f53e2be747-kube-api-access-8tvmx\") pod \"network-check-source-6f78d59f99-tvb96\" (UID: \"0a985b50-c4c0-4a39-abed-29f53e2be747\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tvb96" Nov 03 19:36:46.938820 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.938414 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94d082d5-1c33-4394-9d93-67645f9b60cd-tmp-dir\") pod \"node-resolver-4d4ll\" (UID: \"94d082d5-1c33-4394-9d93-67645f9b60cd\") " pod="openshift-dns/node-resolver-4d4ll" Nov 03 19:36:46.938820 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.938432 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-kubernetes\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:46.938820 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.938447 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drmpl\" (UniqueName: \"kubernetes.io/projected/18ffe451-b8eb-41ad-a19d-0390f3e1f8e2-kube-api-access-drmpl\") pod \"ingress-canary-fgdh8\" (UID: \"18ffe451-b8eb-41ad-a19d-0390f3e1f8e2\") " pod="openshift-ingress-canary/ingress-canary-fgdh8" Nov 03 19:36:46.938820 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.938463 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-run-ovn-kubernetes\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.938820 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.938482 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-env-overrides\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.938820 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.938520 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94d082d5-1c33-4394-9d93-67645f9b60cd-tmp-dir\") pod \"node-resolver-4d4ll\" (UID: \"94d082d5-1c33-4394-9d93-67645f9b60cd\") " pod="openshift-dns/node-resolver-4d4ll" Nov 03 19:36:46.938820 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.938517 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhrbz\" (UniqueName: \"kubernetes.io/projected/4b79231d-e954-41b2-924d-67808a734ecf-kube-api-access-nhrbz\") pod \"periodic-gathering-bhdxh-d8v5q\" (UID: \"4b79231d-e954-41b2-924d-67808a734ecf\") " pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" Nov 03 19:36:46.938820 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.938559 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b989ba1e-71f9-424b-90f1-d1450f63e8b1-ca-trust-extracted\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:46.938820 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.938697 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-sys-fs\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:46.938820 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.938715 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:46.938820 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.938768 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55f124eb-2ebf-4675-a08a-bad3757061b3-iptables-alerter-script\") pod \"iptables-alerter-7zx7h\" (UID: \"55f124eb-2ebf-4675-a08a-bad3757061b3\") " pod="openshift-network-operator/iptables-alerter-7zx7h" Nov 03 19:36:46.938820 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.938778 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-env-overrides\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.938820 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.938811 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b989ba1e-71f9-424b-90f1-d1450f63e8b1-ca-trust-extracted\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:46.939486 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.938811 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b989ba1e-71f9-424b-90f1-d1450f63e8b1-image-registry-private-configuration\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:46.939486 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.938913 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:46.939486 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939028 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7b8992fc-fba6-4135-90f8-fb259e765dd5-signing-cabundle\") pod \"service-ca-5fffc97bdf-brvgz\" (UID: \"7b8992fc-fba6-4135-90f8-fb259e765dd5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" Nov 03 19:36:46.939486 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939051 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-cni-binary-copy\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:46.939486 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939066 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx2l9\" (UniqueName: \"kubernetes.io/projected/55f124eb-2ebf-4675-a08a-bad3757061b3-kube-api-access-lx2l9\") pod \"iptables-alerter-7zx7h\" (UID: \"55f124eb-2ebf-4675-a08a-bad3757061b3\") " pod="openshift-network-operator/iptables-alerter-7zx7h" Nov 03 19:36:46.939486 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939091 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55f124eb-2ebf-4675-a08a-bad3757061b3-iptables-alerter-script\") pod \"iptables-alerter-7zx7h\" (UID: \"55f124eb-2ebf-4675-a08a-bad3757061b3\") " pod="openshift-network-operator/iptables-alerter-7zx7h" Nov 03 19:36:46.939486 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939097 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-hostroot\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:46.939486 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939124 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-metrics-certs\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:46.939486 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939139 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-ovn-node-metrics-cert\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.939486 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939149 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b989ba1e-71f9-424b-90f1-d1450f63e8b1-image-registry-private-configuration\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:46.939486 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939243 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-registration-dir\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:46.939486 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939271 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-cni-binary-copy\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:46.939486 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939298 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-846mt\" (UID: \"04a982f9-143c-406c-99d5-9ef73d5c571a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" Nov 03 19:36:46.939486 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939320 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-multus-cni-dir\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:46.939486 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939384 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7b8992fc-fba6-4135-90f8-fb259e765dd5-signing-cabundle\") pod \"service-ca-5fffc97bdf-brvgz\" (UID: \"7b8992fc-fba6-4135-90f8-fb259e765dd5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" Nov 03 19:36:46.939486 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939399 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-ovn-node-metrics-cert\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.939486 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939398 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-sysctl-conf\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:46.939486 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939454 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.939486 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939472 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-var-lib-cni-multus\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:46.939486 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939487 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/116af81a-d3a0-4558-93c4-8bf061b6815f-dbus\") pod \"global-pull-secret-syncer-wh9tj\" (UID: \"116af81a-d3a0-4558-93c4-8bf061b6815f\") " pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939527 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-certificates\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939591 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-certificates\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939694 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-modprobe-d\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939726 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-sysctl-d\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939751 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-host\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939817 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tckv\" (UniqueName: \"kubernetes.io/projected/419841be-7a31-4cdb-b0b7-9d90a3c376c7-kube-api-access-9tckv\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939852 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-device-dir\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939882 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/116af81a-d3a0-4558-93c4-8bf061b6815f-original-pull-secret\") pod \"global-pull-secret-syncer-wh9tj\" (UID: \"116af81a-d3a0-4558-93c4-8bf061b6815f\") " pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.939907 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2398ba99-35d3-4461-b66e-f44ac013b2d2-serviceca\") pod \"node-ca-kzr2l\" (UID: \"2398ba99-35d3-4461-b66e-f44ac013b2d2\") " pod="openshift-image-registry/node-ca-kzr2l" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.940070 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf5tn\" (UniqueName: \"kubernetes.io/projected/c95249a6-899f-4faf-82dc-6e553453ff80-kube-api-access-lf5tn\") pod \"network-check-target-b4x9q\" (UID: \"c95249a6-899f-4faf-82dc-6e553453ff80\") " pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.940102 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-cni-binary-copy\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.940130 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-tls\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.940156 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32065295-c6a2-44ea-88e8-c4fdc5a84083-ca-trust-extracted\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.940182 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53b14f9a-431e-4dbb-a725-4c8470262ca7-tmp\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.940208 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-system-cni-dir\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.940229 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18ffe451-b8eb-41ad-a19d-0390f3e1f8e2-cert\") pod \"ingress-canary-fgdh8\" (UID: \"18ffe451-b8eb-41ad-a19d-0390f3e1f8e2\") " pod="openshift-ingress-canary/ingress-canary-fgdh8" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.940249 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/419841be-7a31-4cdb-b0b7-9d90a3c376c7-config-volume\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.940271 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdss4\" (UniqueName: \"kubernetes.io/projected/815b15b6-2c37-4ca3-9b38-e7e1328d0aa8-kube-api-access-mdss4\") pod \"migrator-66df5d496b-pbgwk\" (UID: \"815b15b6-2c37-4ca3-9b38-e7e1328d0aa8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pbgwk" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.940274 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/116af81a-d3a0-4558-93c4-8bf061b6815f-original-pull-secret\") pod \"global-pull-secret-syncer-wh9tj\" (UID: \"116af81a-d3a0-4558-93c4-8bf061b6815f\") " pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.940288 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-certificates\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.940299 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-run-systemd\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.940346 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-certificates\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.940328 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-cni-binary-copy\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.940378 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32065295-c6a2-44ea-88e8-c4fdc5a84083-ca-trust-extracted\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:46.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.940415 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-cnibin\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:46.941245 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.940451 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-bound-sa-token\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:46.941245 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.940494 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b989ba1e-71f9-424b-90f1-d1450f63e8b1-installation-pull-secrets\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:46.941245 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.940552 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53b14f9a-431e-4dbb-a725-4c8470262ca7-tmp\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:46.941245 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.940563 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18ffe451-b8eb-41ad-a19d-0390f3e1f8e2-cert\") pod \"ingress-canary-fgdh8\" (UID: \"18ffe451-b8eb-41ad-a19d-0390f3e1f8e2\") " pod="openshift-ingress-canary/ingress-canary-fgdh8" Nov 03 19:36:46.941245 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.940587 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/419841be-7a31-4cdb-b0b7-9d90a3c376c7-config-volume\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:46.941245 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.940587 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2398ba99-35d3-4461-b66e-f44ac013b2d2-host\") pod \"node-ca-kzr2l\" (UID: \"2398ba99-35d3-4461-b66e-f44ac013b2d2\") " pod="openshift-image-registry/node-ca-kzr2l" Nov 03 19:36:46.941245 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.940611 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2398ba99-35d3-4461-b66e-f44ac013b2d2-serviceca\") pod \"node-ca-kzr2l\" (UID: \"2398ba99-35d3-4461-b66e-f44ac013b2d2\") " pod="openshift-image-registry/node-ca-kzr2l" Nov 03 19:36:46.941245 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.940726 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b989ba1e-71f9-424b-90f1-d1450f63e8b1-installation-pull-secrets\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:46.945768 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.945745 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tpvj\" (UniqueName: \"kubernetes.io/projected/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-kube-api-access-2tpvj\") pod \"network-metrics-daemon-qvxjq\" (UID: \"caf7d6c9-0bb2-4da0-be2d-caea7734bbcb\") " pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:46.945850 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.945829 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzh28\" (UniqueName: \"kubernetes.io/projected/377ffcf5-b097-495d-b986-e61c58ae8ec1-kube-api-access-hzh28\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:46.945894 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.945878 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsppf\" (UniqueName: \"kubernetes.io/projected/94d082d5-1c33-4394-9d93-67645f9b60cd-kube-api-access-rsppf\") pod \"node-resolver-4d4ll\" (UID: \"94d082d5-1c33-4394-9d93-67645f9b60cd\") " pod="openshift-dns/node-resolver-4d4ll" Nov 03 19:36:46.945934 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.945919 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnslt\" (UniqueName: \"kubernetes.io/projected/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-kube-api-access-lnslt\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:46.947187 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.947166 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-bound-sa-token\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:46.947491 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.947474 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5xvg\" (UniqueName: \"kubernetes.io/projected/53b14f9a-431e-4dbb-a725-4c8470262ca7-kube-api-access-m5xvg\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:46.947572 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.947549 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbrbm\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-kube-api-access-kbrbm\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:46.947777 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.947751 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wvnz\" (UniqueName: \"kubernetes.io/projected/1bcd4690-4753-4246-abb4-8668a0a56231-kube-api-access-8wvnz\") pod \"volume-data-source-validator-58c9f5f46d-fsbsp\" (UID: \"1bcd4690-4753-4246-abb4-8668a0a56231\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fsbsp" Nov 03 19:36:46.947838 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.947755 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rnvd\" (UniqueName: \"kubernetes.io/projected/7b8992fc-fba6-4135-90f8-fb259e765dd5-kube-api-access-7rnvd\") pod \"service-ca-5fffc97bdf-brvgz\" (UID: \"7b8992fc-fba6-4135-90f8-fb259e765dd5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" Nov 03 19:36:46.947838 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.947812 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28xz4\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-kube-api-access-28xz4\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:46.949898 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.949880 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn9fc\" (UniqueName: \"kubernetes.io/projected/e64f9d9f-c868-470e-bd78-4ce8baabd97a-kube-api-access-mn9fc\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:46.950074 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.950051 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p75v\" (UniqueName: \"kubernetes.io/projected/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-kube-api-access-7p75v\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:46.950779 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.950738 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx7kl\" (UniqueName: \"kubernetes.io/projected/2398ba99-35d3-4461-b66e-f44ac013b2d2-kube-api-access-jx7kl\") pod \"node-ca-kzr2l\" (UID: \"2398ba99-35d3-4461-b66e-f44ac013b2d2\") " pod="openshift-image-registry/node-ca-kzr2l" Nov 03 19:36:46.950856 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.950810 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hszb\" (UniqueName: \"kubernetes.io/projected/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-kube-api-access-9hszb\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:46.954323 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.951158 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tvmx\" (UniqueName: \"kubernetes.io/projected/0a985b50-c4c0-4a39-abed-29f53e2be747-kube-api-access-8tvmx\") pod \"network-check-source-6f78d59f99-tvb96\" (UID: \"0a985b50-c4c0-4a39-abed-29f53e2be747\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tvb96" Nov 03 19:36:46.954323 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.951181 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx2l9\" (UniqueName: \"kubernetes.io/projected/55f124eb-2ebf-4675-a08a-bad3757061b3-kube-api-access-lx2l9\") pod \"iptables-alerter-7zx7h\" (UID: \"55f124eb-2ebf-4675-a08a-bad3757061b3\") " pod="openshift-network-operator/iptables-alerter-7zx7h" Nov 03 19:36:46.955435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.955414 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhrbz\" (UniqueName: \"kubernetes.io/projected/4b79231d-e954-41b2-924d-67808a734ecf-kube-api-access-nhrbz\") pod \"periodic-gathering-bhdxh-d8v5q\" (UID: \"4b79231d-e954-41b2-924d-67808a734ecf\") " pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" Nov 03 19:36:46.955580 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.955563 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drmpl\" (UniqueName: \"kubernetes.io/projected/18ffe451-b8eb-41ad-a19d-0390f3e1f8e2-kube-api-access-drmpl\") pod \"ingress-canary-fgdh8\" (UID: \"18ffe451-b8eb-41ad-a19d-0390f3e1f8e2\") " pod="openshift-ingress-canary/ingress-canary-fgdh8" Nov 03 19:36:46.955995 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.955978 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf5tn\" (UniqueName: \"kubernetes.io/projected/c95249a6-899f-4faf-82dc-6e553453ff80-kube-api-access-lf5tn\") pod \"network-check-target-b4x9q\" (UID: \"c95249a6-899f-4faf-82dc-6e553453ff80\") " pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:46.958859 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.958842 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tckv\" (UniqueName: \"kubernetes.io/projected/419841be-7a31-4cdb-b0b7-9d90a3c376c7-kube-api-access-9tckv\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:46.962015 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.961996 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-bound-sa-token\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:46.962077 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.962045 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86bkx\" (UniqueName: \"kubernetes.io/projected/f485893c-6f14-449b-b837-8ac181315aeb-kube-api-access-86bkx\") pod \"migrator-66df5d496b-6k9n9\" (UID: \"f485893c-6f14-449b-b837-8ac181315aeb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6k9n9" Nov 03 19:36:46.962162 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:46.962144 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdss4\" (UniqueName: \"kubernetes.io/projected/815b15b6-2c37-4ca3-9b38-e7e1328d0aa8-kube-api-access-mdss4\") pod \"migrator-66df5d496b-pbgwk\" (UID: \"815b15b6-2c37-4ca3-9b38-e7e1328d0aa8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pbgwk" Nov 03 19:36:47.041585 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.041561 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:47.041816 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.041595 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-multus-conf-dir\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.041816 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.041646 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-multus-conf-dir\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.041816 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.041657 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:47.041816 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.041653 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-tls\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:47.041816 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.041815 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-system-cni-dir\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:47.042072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.041837 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-cnibin\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:47.042072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.041862 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-run-netns\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.042072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.041885 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-run-ovn\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.042072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.041891 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-system-cni-dir\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:47.042072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.041908 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-var-lib-kubelet\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:47.042072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.041925 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-run-netns\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.042072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.041933 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-cni-netd\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.042072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.041888 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-cnibin\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:47.042072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.041944 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-run-ovn\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.042072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.041979 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-multus-socket-dir-parent\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.042072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.041990 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-var-lib-kubelet\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:47.042072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042024 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-cni-netd\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.042072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042069 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-var-lib-cni-bin\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042114 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-multus-socket-dir-parent\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042188 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-var-lib-cni-bin\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042228 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-lib-modules\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042255 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-slash\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042278 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-log-socket\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042302 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-etc-openvswitch\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042315 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-slash\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042322 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-cni-bin\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042344 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-log-socket\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042353 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-etc-openvswitch\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042345 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-lib-modules\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042355 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042359 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-cni-bin\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:47.042411 5059 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:47.042462 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls podName:377ffcf5-b097-495d-b986-e61c58ae8ec1 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:47.542446316 +0000 UTC m=+1.800243566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hmnx4" (UID: "377ffcf5-b097-495d-b986-e61c58ae8ec1") : secret "insights-runtime-extractor-tls" not found Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042479 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042506 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-socket-dir\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042533 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-sysconfig\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042555 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-sys\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042533 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6f0d931-3a92-424f-a5fc-d9ebef985dfd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cxdrc\" (UID: \"b6f0d931-3a92-424f-a5fc-d9ebef985dfd\") " pod="openshift-multus/multus-additional-cni-plugins-cxdrc" Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042571 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-socket-dir\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042588 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-var-lib-openvswitch\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042609 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/94d082d5-1c33-4394-9d93-67645f9b60cd-hosts-file\") pod \"node-resolver-4d4ll\" (UID: \"94d082d5-1c33-4394-9d93-67645f9b60cd\") " pod="openshift-dns/node-resolver-4d4ll" Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042623 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs\") pod \"network-metrics-daemon-qvxjq\" (UID: \"caf7d6c9-0bb2-4da0-be2d-caea7734bbcb\") " pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:47.042643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042623 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-sysconfig\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042657 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-run-netns\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042692 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/94d082d5-1c33-4394-9d93-67645f9b60cd-hosts-file\") pod \"node-resolver-4d4ll\" (UID: \"94d082d5-1c33-4394-9d93-67645f9b60cd\") " pod="openshift-dns/node-resolver-4d4ll" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042697 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-node-log\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042702 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-sys\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042741 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-var-lib-openvswitch\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042742 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-run-netns\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042743 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-kubelet\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042773 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-kubelet\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042783 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-kubernetes\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042808 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-run-ovn-kubernetes\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042821 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-kubernetes\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042772 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-node-log\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042835 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-sys-fs\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042859 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042866 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-run-ovn-kubernetes\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042883 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-hostroot\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042907 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-registration-dir\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:47.042933 5059 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042933 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-sys-fs\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042958 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-hostroot\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.042929 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-846mt\" (UID: \"04a982f9-143c-406c-99d5-9ef73d5c571a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:47.042973 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls podName:419841be-7a31-4cdb-b0b7-9d90a3c376c7 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:47.542959252 +0000 UTC m=+1.800756509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls") pod "dns-default-nhf67" (UID: "419841be-7a31-4cdb-b0b7-9d90a3c376c7") : secret "dns-default-metrics-tls" not found Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:47.042981 5059 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043013 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-registration-dir\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043023 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-multus-cni-dir\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043048 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-sysctl-conf\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043054 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-multus-cni-dir\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:47.043084 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert podName:04a982f9-143c-406c-99d5-9ef73d5c571a nodeName:}" failed. No retries permitted until 2026-11-03 19:36:47.543073066 +0000 UTC m=+1.800870310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-846mt" (UID: "04a982f9-143c-406c-99d5-9ef73d5c571a") : secret "networking-console-plugin-cert" not found Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043071 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043113 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-var-lib-cni-multus\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043126 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043135 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/116af81a-d3a0-4558-93c4-8bf061b6815f-dbus\") pod \"global-pull-secret-syncer-wh9tj\" (UID: \"116af81a-d3a0-4558-93c4-8bf061b6815f\") " pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043147 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-sysctl-conf\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043171 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-modprobe-d\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043181 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-var-lib-cni-multus\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043195 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-sysctl-d\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043216 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-host\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043223 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/116af81a-d3a0-4558-93c4-8bf061b6815f-dbus\") pod \"global-pull-secret-syncer-wh9tj\" (UID: \"116af81a-d3a0-4558-93c4-8bf061b6815f\") " pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043240 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-device-dir\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043255 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-modprobe-d\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043272 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-tls\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043298 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-host\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043301 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-system-cni-dir\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043330 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-device-dir\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043336 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-sysctl-d\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043330 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-run-systemd\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043375 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-cnibin\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043386 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-system-cni-dir\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043397 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2398ba99-35d3-4461-b66e-f44ac013b2d2-host\") pod \"node-ca-kzr2l\" (UID: \"2398ba99-35d3-4461-b66e-f44ac013b2d2\") " pod="openshift-image-registry/node-ca-kzr2l" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043435 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-cnibin\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043440 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2398ba99-35d3-4461-b66e-f44ac013b2d2-host\") pod \"node-ca-kzr2l\" (UID: \"2398ba99-35d3-4461-b66e-f44ac013b2d2\") " pod="openshift-image-registry/node-ca-kzr2l" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043459 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/377ffcf5-b097-495d-b986-e61c58ae8ec1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043479 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-systemd-units\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043483 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-run-systemd\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043495 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55f124eb-2ebf-4675-a08a-bad3757061b3-host-slash\") pod \"iptables-alerter-7zx7h\" (UID: \"55f124eb-2ebf-4675-a08a-bad3757061b3\") " pod="openshift-network-operator/iptables-alerter-7zx7h" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043521 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-etc-kubernetes\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043540 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-systemd\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043561 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/377ffcf5-b097-495d-b986-e61c58ae8ec1-crio-socket\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043575 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-run-k8s-cni-cncf-io\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043590 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-var-lib-kubelet\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043604 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-run-openvswitch\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043641 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-etc-selinux\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043682 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-run-multus-certs\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043702 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-etc-systemd\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043710 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/116af81a-d3a0-4558-93c4-8bf061b6815f-kubelet-config\") pod \"global-pull-secret-syncer-wh9tj\" (UID: \"116af81a-d3a0-4558-93c4-8bf061b6815f\") " pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043737 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-run\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043742 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-systemd-units\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.043753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043792 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55f124eb-2ebf-4675-a08a-bad3757061b3-host-slash\") pod \"iptables-alerter-7zx7h\" (UID: \"55f124eb-2ebf-4675-a08a-bad3757061b3\") " pod="openshift-network-operator/iptables-alerter-7zx7h" Nov 03 19:36:47.046768 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043814 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/53b14f9a-431e-4dbb-a725-4c8470262ca7-run\") pod \"tuned-mkm7n\" (UID: \"53b14f9a-431e-4dbb-a725-4c8470262ca7\") " pod="openshift-cluster-node-tuning-operator/tuned-mkm7n" Nov 03 19:36:47.046768 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043841 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-etc-kubernetes\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.046768 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043861 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/377ffcf5-b097-495d-b986-e61c58ae8ec1-crio-socket\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:47.046768 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043880 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42-run-openvswitch\") pod \"ovnkube-node-2m4mq\" (UID: \"c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42\") " pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.046768 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043926 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-run-k8s-cni-cncf-io\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.046768 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043960 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-var-lib-kubelet\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.046768 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.043996 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/377ffcf5-b097-495d-b986-e61c58ae8ec1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:47.046768 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.044012 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c6025fec-2ce4-4638-9a74-1f58c7ba7e8b-host-run-multus-certs\") pod \"multus-xlfb8\" (UID: \"c6025fec-2ce4-4638-9a74-1f58c7ba7e8b\") " pod="openshift-multus/multus-xlfb8" Nov 03 19:36:47.046768 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.044097 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/116af81a-d3a0-4558-93c4-8bf061b6815f-kubelet-config\") pod \"global-pull-secret-syncer-wh9tj\" (UID: \"116af81a-d3a0-4558-93c4-8bf061b6815f\") " pod="kube-system/global-pull-secret-syncer-wh9tj" Nov 03 19:36:47.046768 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.044107 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e64f9d9f-c868-470e-bd78-4ce8baabd97a-etc-selinux\") pod \"aws-ebs-csi-driver-node-gb8r8\" (UID: \"e64f9d9f-c868-470e-bd78-4ce8baabd97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" Nov 03 19:36:47.046768 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.044404 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-tls\") pod \"image-registry-6dd465cd78-4n8kx\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:47.046768 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.045104 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf7d6c9-0bb2-4da0-be2d-caea7734bbcb-metrics-certs\") pod \"network-metrics-daemon-qvxjq\" (UID: \"caf7d6c9-0bb2-4da0-be2d-caea7734bbcb\") " pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:47.046768 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.045704 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-tls\") pod \"image-registry-799ffcd7c9-5pdd7\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:47.117261 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.117238 5059 scope.go:122] "RemoveContainer" containerID="7bbcf23cd573d5c9e806dd6604b68ae67663631e74f89f722cb793dfee52aa69" Nov 03 19:36:47.119644 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.119625 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:47.133269 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.133241 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvxjq" Nov 03 19:36:47.135525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.135504 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:47.403994 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.403927 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dd465cd78-4n8kx"] Nov 03 19:36:47.407166 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:47.407133 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb989ba1e_71f9_424b_90f1_d1450f63e8b1.slice/crio-8e109501f0db9a519b2048e1b15d4d7add9f81e67d9123165db0a963b15b1b2d WatchSource:0}: Error finding container 8e109501f0db9a519b2048e1b15d4d7add9f81e67d9123165db0a963b15b1b2d: Status 404 returned error can't find the container with id 8e109501f0db9a519b2048e1b15d4d7add9f81e67d9123165db0a963b15b1b2d Nov 03 19:36:47.432556 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.432531 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qvxjq"] Nov 03 19:36:47.440632 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.440611 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-799ffcd7c9-5pdd7"] Nov 03 19:36:47.442453 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.442434 5059 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:36:47.443876 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:47.443850 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32065295_c6a2_44ea_88e8_c4fdc5a84083.slice/crio-b6de17f35c0d2cd0522a618a81d7c9aa8c0584954008f35de924edb4305cf227 WatchSource:0}: Error finding container b6de17f35c0d2cd0522a618a81d7c9aa8c0584954008f35de924edb4305cf227: Status 404 returned error can't find the container with id b6de17f35c0d2cd0522a618a81d7c9aa8c0584954008f35de924edb4305cf227 Nov 03 19:36:47.446329 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.446309 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-h9f7b\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:47.446425 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.446389 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:47.446479 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.446433 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-86qxx\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:47.446642 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:47.446622 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle podName:603868b0-8628-4227-8471-1894e2e1de50 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:48.446604034 +0000 UTC m=+2.704401283 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle") pod "router-default-f7b87d695-cwlfq" (UID: "603868b0-8628-4227-8471-1894e2e1de50") : configmap references non-existent config key: service-ca.crt Nov 03 19:36:47.446642 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:47.446626 5059 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 19:36:47.446803 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:47.446706 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls podName:8d0db609-b271-4c98-a303-fb3ee71f152c nodeName:}" failed. No retries permitted until 2026-11-03 19:36:48.446692452 +0000 UTC m=+2.704489707 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-h9f7b" (UID: "8d0db609-b271-4c98-a303-fb3ee71f152c") : secret "cluster-monitoring-operator-tls" not found Nov 03 19:36:47.446803 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:47.446712 5059 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 19:36:47.446803 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:47.446765 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls podName:40b2621f-854e-44e0-9b01-c877e333f644 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:48.44674842 +0000 UTC m=+2.704545669 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-86qxx" (UID: "40b2621f-854e-44e0-9b01-c877e333f644") : secret "samples-operator-tls" not found Nov 03 19:36:47.547569 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.547520 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:47.547751 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.547614 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:47.547751 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.547649 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-846mt\" (UID: \"04a982f9-143c-406c-99d5-9ef73d5c571a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" Nov 03 19:36:47.547879 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:47.547783 5059 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:36:47.547879 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:47.547849 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls podName:419841be-7a31-4cdb-b0b7-9d90a3c376c7 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:48.547834732 +0000 UTC m=+2.805631972 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls") pod "dns-default-nhf67" (UID: "419841be-7a31-4cdb-b0b7-9d90a3c376c7") : secret "dns-default-metrics-tls" not found Nov 03 19:36:47.547879 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:47.547783 5059 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:36:47.548041 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:47.547894 5059 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 19:36:47.548041 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:47.547902 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls podName:377ffcf5-b097-495d-b986-e61c58ae8ec1 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:48.54789117 +0000 UTC m=+2.805688425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hmnx4" (UID: "377ffcf5-b097-495d-b986-e61c58ae8ec1") : secret "insights-runtime-extractor-tls" not found Nov 03 19:36:47.548041 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:47.547952 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert podName:04a982f9-143c-406c-99d5-9ef73d5c571a nodeName:}" failed. No retries permitted until 2026-11-03 19:36:48.547936334 +0000 UTC m=+2.805733576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-846mt" (UID: "04a982f9-143c-406c-99d5-9ef73d5c571a") : secret "networking-console-plugin-cert" not found Nov 03 19:36:47.828673 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.828640 5059 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:47.881365 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.881337 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:47.912313 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.912291 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 19:36:47.912420 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.912382 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" event={"ID":"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac","Type":"ContainerStarted","Data":"b7138eb33c5d6fa9eeb22093ad7d51deb0e28bca02ed9a1262e5df9e2a221d24"} Nov 03 19:36:47.913904 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.913876 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" event={"ID":"32065295-c6a2-44ea-88e8-c4fdc5a84083","Type":"ContainerStarted","Data":"5609de2d9f0e6127c11efd5b0a9f67fa54e1525ea6353e6a712e0ed74644d2b7"} Nov 03 19:36:47.914027 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.913925 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" event={"ID":"32065295-c6a2-44ea-88e8-c4fdc5a84083","Type":"ContainerStarted","Data":"b6de17f35c0d2cd0522a618a81d7c9aa8c0584954008f35de924edb4305cf227"} Nov 03 19:36:47.915078 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.915053 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvxjq" event={"ID":"caf7d6c9-0bb2-4da0-be2d-caea7734bbcb","Type":"ContainerStarted","Data":"6efb3eb7ab0e442436b0a63ac2d1b66cb1d8a1fabbdb3536579fa102028be4e6"} Nov 03 19:36:47.916672 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.916628 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" event={"ID":"b989ba1e-71f9-424b-90f1-d1450f63e8b1","Type":"ContainerStarted","Data":"7041265b830d65324d40a06784f0a77e789f5ea1c1de24e08a751fd193048631"} Nov 03 19:36:47.916746 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:47.916691 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" event={"ID":"b989ba1e-71f9-424b-90f1-d1450f63e8b1","Type":"ContainerStarted","Data":"8e109501f0db9a519b2048e1b15d4d7add9f81e67d9123165db0a963b15b1b2d"} Nov 03 19:36:48.458614 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:48.458569 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-86qxx\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:48.458804 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:48.458674 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-h9f7b\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:48.458804 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:48.458745 5059 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 19:36:48.458804 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:48.458773 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:48.458949 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:48.458832 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls podName:40b2621f-854e-44e0-9b01-c877e333f644 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:50.458799534 +0000 UTC m=+4.716596786 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-86qxx" (UID: "40b2621f-854e-44e0-9b01-c877e333f644") : secret "samples-operator-tls" not found Nov 03 19:36:48.458949 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:48.458836 5059 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 19:36:48.458949 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:48.458882 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle podName:603868b0-8628-4227-8471-1894e2e1de50 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:50.458866811 +0000 UTC m=+4.716664048 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle") pod "router-default-f7b87d695-cwlfq" (UID: "603868b0-8628-4227-8471-1894e2e1de50") : configmap references non-existent config key: service-ca.crt Nov 03 19:36:48.458949 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:48.458902 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls podName:8d0db609-b271-4c98-a303-fb3ee71f152c nodeName:}" failed. No retries permitted until 2026-11-03 19:36:50.458892187 +0000 UTC m=+4.716689424 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-h9f7b" (UID: "8d0db609-b271-4c98-a303-fb3ee71f152c") : secret "cluster-monitoring-operator-tls" not found Nov 03 19:36:48.559952 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:48.559914 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:48.560133 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:48.559970 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-846mt\" (UID: \"04a982f9-143c-406c-99d5-9ef73d5c571a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" Nov 03 19:36:48.560133 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:48.560086 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:48.560245 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:48.560227 5059 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:36:48.560312 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:48.560290 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls podName:377ffcf5-b097-495d-b986-e61c58ae8ec1 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:50.560271755 +0000 UTC m=+4.818069003 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hmnx4" (UID: "377ffcf5-b097-495d-b986-e61c58ae8ec1") : secret "insights-runtime-extractor-tls" not found Nov 03 19:36:48.560968 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:48.560866 5059 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 19:36:48.560968 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:48.560938 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert podName:04a982f9-143c-406c-99d5-9ef73d5c571a nodeName:}" failed. No retries permitted until 2026-11-03 19:36:50.56092048 +0000 UTC m=+4.818717728 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-846mt" (UID: "04a982f9-143c-406c-99d5-9ef73d5c571a") : secret "networking-console-plugin-cert" not found Nov 03 19:36:48.563472 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:48.563435 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/419841be-7a31-4cdb-b0b7-9d90a3c376c7-metrics-tls\") pod \"dns-default-nhf67\" (UID: \"419841be-7a31-4cdb-b0b7-9d90a3c376c7\") " pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:48.640511 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:48.640488 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:48.770397 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:48.770375 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:36:48.809355 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:48.809318 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:36:48.915570 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:48.915528 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nhf67"] Nov 03 19:36:48.918281 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:48.918244 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419841be_7a31_4cdb_b0b7_9d90a3c376c7.slice/crio-59e4266ded340023fd9ea49b7c3ffeff030e3eba619228a5b775a20a30c7fc38 WatchSource:0}: Error finding container 59e4266ded340023fd9ea49b7c3ffeff030e3eba619228a5b775a20a30c7fc38: Status 404 returned error can't find the container with id 59e4266ded340023fd9ea49b7c3ffeff030e3eba619228a5b775a20a30c7fc38 Nov 03 19:36:48.921698 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:48.921636 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvxjq" event={"ID":"caf7d6c9-0bb2-4da0-be2d-caea7734bbcb","Type":"ContainerStarted","Data":"07b024c1fb12d50be2fa203dbe709734aaed7522f408b57ceafe5661b506daf7"} Nov 03 19:36:48.921698 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:48.921684 5059 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:36:48.921866 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:48.921706 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvxjq" event={"ID":"caf7d6c9-0bb2-4da0-be2d-caea7734bbcb","Type":"ContainerStarted","Data":"c2ed614fa0b2fb59ea94ca40bf0c73974856b12427fa51cbfe641e5708b3a568"} Nov 03 19:36:48.921866 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:48.921721 5059 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:36:48.922299 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:48.922283 5059 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:36:48.922582 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:48.922566 5059 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:36:49.927426 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:49.927367 5059 generic.go:356] "Generic (PLEG): container finished" podID="492c7b4e-84c0-49a7-b12f-153a021f971b" containerID="418374b5472981e5cdbb3a5be895780bfb3ad1fc0b28a5e01266b5d997136ed3" exitCode=0 Nov 03 19:36:49.927999 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:49.927496 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" event={"ID":"492c7b4e-84c0-49a7-b12f-153a021f971b","Type":"ContainerDied","Data":"418374b5472981e5cdbb3a5be895780bfb3ad1fc0b28a5e01266b5d997136ed3"} Nov 03 19:36:49.927999 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:49.927872 5059 scope.go:122] "RemoveContainer" containerID="418374b5472981e5cdbb3a5be895780bfb3ad1fc0b28a5e01266b5d997136ed3" Nov 03 19:36:49.929307 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:49.929282 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nhf67" event={"ID":"419841be-7a31-4cdb-b0b7-9d90a3c376c7","Type":"ContainerStarted","Data":"59e4266ded340023fd9ea49b7c3ffeff030e3eba619228a5b775a20a30c7fc38"} Nov 03 19:36:50.479183 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:50.479145 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:50.479363 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:50.479208 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-86qxx\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:50.479363 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:50.479260 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-h9f7b\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:50.479363 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:50.479299 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle podName:603868b0-8628-4227-8471-1894e2e1de50 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:54.479275556 +0000 UTC m=+8.737072805 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle") pod "router-default-f7b87d695-cwlfq" (UID: "603868b0-8628-4227-8471-1894e2e1de50") : configmap references non-existent config key: service-ca.crt Nov 03 19:36:50.479496 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:50.479360 5059 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 19:36:50.479496 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:50.479414 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls podName:40b2621f-854e-44e0-9b01-c877e333f644 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:54.479399953 +0000 UTC m=+8.737197207 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-86qxx" (UID: "40b2621f-854e-44e0-9b01-c877e333f644") : secret "samples-operator-tls" not found Nov 03 19:36:50.481915 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:50.481894 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-h9f7b\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:50.579815 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:50.579774 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:50.579989 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:50.579835 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-846mt\" (UID: \"04a982f9-143c-406c-99d5-9ef73d5c571a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" Nov 03 19:36:50.579989 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:50.579978 5059 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:36:50.580090 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:36:50.580026 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls podName:377ffcf5-b097-495d-b986-e61c58ae8ec1 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:54.580013652 +0000 UTC m=+8.837810888 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hmnx4" (UID: "377ffcf5-b097-495d-b986-e61c58ae8ec1") : secret "insights-runtime-extractor-tls" not found Nov 03 19:36:50.582810 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:50.582786 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-846mt\" (UID: \"04a982f9-143c-406c-99d5-9ef73d5c571a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" Nov 03 19:36:50.637587 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:50.637560 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:50.637730 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:50.637694 5059 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:36:50.643183 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:50.643155 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 19:36:50.719867 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:50.719834 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 19:36:50.720940 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:50.720908 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" Nov 03 19:36:50.935236 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:50.935203 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" event={"ID":"492c7b4e-84c0-49a7-b12f-153a021f971b","Type":"ContainerStarted","Data":"8b68d8a54774f3a932e8c394d04b88681ee2fd3757c1b50ef500cf35cb4119f9"} Nov 03 19:36:50.937065 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:50.937040 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nhf67" event={"ID":"419841be-7a31-4cdb-b0b7-9d90a3c376c7","Type":"ContainerStarted","Data":"35a73b47bde0ee9cb02a4cf13d0e462ceee9fc440a10634260b7b9986ed2273c"} Nov 03 19:36:51.007342 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:51.007279 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-846mt"] Nov 03 19:36:51.009951 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:51.009922 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a982f9_143c_406c_99d5_9ef73d5c571a.slice/crio-887eb13a4950ee22ec72790db18fd5bfedfc6957e0af69fc8c65d2c77f870323 WatchSource:0}: Error finding container 887eb13a4950ee22ec72790db18fd5bfedfc6957e0af69fc8c65d2c77f870323: Status 404 returned error can't find the container with id 887eb13a4950ee22ec72790db18fd5bfedfc6957e0af69fc8c65d2c77f870323 Nov 03 19:36:51.012899 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:51.012857 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b"] Nov 03 19:36:51.014503 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:51.014461 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d0db609_b271_4c98_a303_fb3ee71f152c.slice/crio-bc5f2d96ab72bf6de58c0862a759ae4375b7675d1de97b000932e8d79181cd70 WatchSource:0}: Error finding container bc5f2d96ab72bf6de58c0862a759ae4375b7675d1de97b000932e8d79181cd70: Status 404 returned error can't find the container with id bc5f2d96ab72bf6de58c0862a759ae4375b7675d1de97b000932e8d79181cd70 Nov 03 19:36:51.943161 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:51.943117 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" event={"ID":"04a982f9-143c-406c-99d5-9ef73d5c571a","Type":"ContainerStarted","Data":"887eb13a4950ee22ec72790db18fd5bfedfc6957e0af69fc8c65d2c77f870323"} Nov 03 19:36:51.945811 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:51.945778 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nhf67" event={"ID":"419841be-7a31-4cdb-b0b7-9d90a3c376c7","Type":"ContainerStarted","Data":"946f3a457e0132915ab570431fa1a49aead5110073e632e38c09438360249b47"} Nov 03 19:36:51.945961 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:51.945896 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nhf67" Nov 03 19:36:51.947881 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:51.947854 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" event={"ID":"8d0db609-b271-4c98-a303-fb3ee71f152c","Type":"ContainerStarted","Data":"bc5f2d96ab72bf6de58c0862a759ae4375b7675d1de97b000932e8d79181cd70"} Nov 03 19:36:52.476886 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:52.476795 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:36:52.954018 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:52.953980 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" event={"ID":"8d0db609-b271-4c98-a303-fb3ee71f152c","Type":"ContainerStarted","Data":"6a679e526b0df571226e2d9d4986921d285477efff7809df408a7dcd93178490"} Nov 03 19:36:52.955779 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:52.955749 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" event={"ID":"04a982f9-143c-406c-99d5-9ef73d5c571a","Type":"ContainerStarted","Data":"147857ae5dcfb3795e8ee333ec3863a15882c306754cc03d2571ba058fc49ce3"} Nov 03 19:36:54.518334 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:54.518288 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:54.518747 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:54.518349 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-86qxx\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:54.519008 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:54.518982 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle\") pod \"router-default-f7b87d695-cwlfq\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:54.520649 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:54.520622 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-86qxx\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:54.617648 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:54.617618 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:54.618714 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:54.618695 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 19:36:54.618860 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:54.618840 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:54.620799 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:54.620777 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/377ffcf5-b097-495d-b986-e61c58ae8ec1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hmnx4\" (UID: \"377ffcf5-b097-495d-b986-e61c58ae8ec1\") " pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:54.643909 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:54.643880 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hmnx4" Nov 03 19:36:54.665059 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:54.665033 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:36:54.901031 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:54.901006 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-f7b87d695-cwlfq"] Nov 03 19:36:54.903158 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:54.903133 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx"] Nov 03 19:36:54.904082 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:54.903903 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603868b0_8628_4227_8471_1894e2e1de50.slice/crio-56ca0571aedc921d703ca9a0d179a99f24a724e4d9f206adc2364d70071a3b71 WatchSource:0}: Error finding container 56ca0571aedc921d703ca9a0d179a99f24a724e4d9f206adc2364d70071a3b71: Status 404 returned error can't find the container with id 56ca0571aedc921d703ca9a0d179a99f24a724e4d9f206adc2364d70071a3b71 Nov 03 19:36:54.929141 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:54.929116 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hmnx4"] Nov 03 19:36:54.931404 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:36:54.931378 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377ffcf5_b097_495d_b986_e61c58ae8ec1.slice/crio-ccf3db467e696ebdb91feb01394f40a0698f92a9a9cf197df8f77c1886fa7608 WatchSource:0}: Error finding container ccf3db467e696ebdb91feb01394f40a0698f92a9a9cf197df8f77c1886fa7608: Status 404 returned error can't find the container with id ccf3db467e696ebdb91feb01394f40a0698f92a9a9cf197df8f77c1886fa7608 Nov 03 19:36:54.965231 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:54.965199 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-f7b87d695-cwlfq" event={"ID":"603868b0-8628-4227-8471-1894e2e1de50","Type":"ContainerStarted","Data":"a75a1e78b8fa43ee7161dbf1263337919ca861621fb782933da9fe9d87a6c4b1"} Nov 03 19:36:54.965312 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:54.965254 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-f7b87d695-cwlfq" event={"ID":"603868b0-8628-4227-8471-1894e2e1de50","Type":"ContainerStarted","Data":"56ca0571aedc921d703ca9a0d179a99f24a724e4d9f206adc2364d70071a3b71"} Nov 03 19:36:54.966895 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:54.966862 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hmnx4" event={"ID":"377ffcf5-b097-495d-b986-e61c58ae8ec1","Type":"ContainerStarted","Data":"ccf3db467e696ebdb91feb01394f40a0698f92a9a9cf197df8f77c1886fa7608"} Nov 03 19:36:54.968071 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:54.968050 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" event={"ID":"40b2621f-854e-44e0-9b01-c877e333f644","Type":"ContainerStarted","Data":"d8b3f14031a53eecbdb2db39f15186180f1b7059e2868a65f8b7a4fe42d2380a"} Nov 03 19:36:55.618479 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:55.618437 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:55.621837 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:55.621595 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:36:55.973145 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:55.973059 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hmnx4" event={"ID":"377ffcf5-b097-495d-b986-e61c58ae8ec1","Type":"ContainerStarted","Data":"da80fa8a0498111540442aa3510825ea09afb373c9c7d2792ddb7b00125fbdaa"} Nov 03 19:36:55.973145 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:55.973132 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hmnx4" event={"ID":"377ffcf5-b097-495d-b986-e61c58ae8ec1","Type":"ContainerStarted","Data":"043cb06041d851f5b08e43de4ddd8bcec258fcfefabc7c17851b3d2b458b7695"} Nov 03 19:36:56.979472 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:56.979389 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" event={"ID":"40b2621f-854e-44e0-9b01-c877e333f644","Type":"ContainerStarted","Data":"1a4ab42465871189364c0b8d0033b852cbd3488b01bc6dd2ae20a8268416653f"} Nov 03 19:36:56.979896 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:56.979479 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" event={"ID":"40b2621f-854e-44e0-9b01-c877e333f644","Type":"ContainerStarted","Data":"b2569884782bd34d772a62814be138ac35a8f3a6ac0255af71fcef484fe98a3d"} Nov 03 19:36:56.979896 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:56.979402 5059 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:36:57.986468 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:36:57.986434 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hmnx4" event={"ID":"377ffcf5-b097-495d-b986-e61c58ae8ec1","Type":"ContainerStarted","Data":"a5ba72a6af7c7b24a279be39f35131a53c4a12a30bb49194fe5f104c93f38027"} Nov 03 19:37:01.127773 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.127739 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:37:01.128153 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.127862 5059 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:37:01.129028 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.129005 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 19:37:01.585852 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.585810 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qchj6"] Nov 03 19:37:01.593611 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.593582 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-c88f8"] Nov 03 19:37:01.593796 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.593741 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qchj6" Nov 03 19:37:01.597234 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.597211 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6dd465cd78-4n8kx"] Nov 03 19:37:01.597366 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.597333 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-c88f8" Nov 03 19:37:01.597857 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.597838 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qchj6"] Nov 03 19:37:01.604273 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.604253 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-c88f8"] Nov 03 19:37:01.610166 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.610140 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 19:37:01.610361 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.610343 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 19:37:01.610856 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.610830 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 19:37:01.610964 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.610897 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-wftlv\"" Nov 03 19:37:01.611021 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.610966 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-h4l58\"" Nov 03 19:37:01.676339 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.676301 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7rw8\" (UniqueName: \"kubernetes.io/projected/6a7937ca-c705-4091-967e-2e640c622a26-kube-api-access-q7rw8\") pod \"downloads-6dd789c8b7-c88f8\" (UID: \"6a7937ca-c705-4091-967e-2e640c622a26\") " pod="openshift-console/downloads-6dd789c8b7-c88f8" Nov 03 19:37:01.676485 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.676439 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a7937ca-c705-4091-967e-2e640c622a26-tmp\") pod \"downloads-6dd789c8b7-c88f8\" (UID: \"6a7937ca-c705-4091-967e-2e640c622a26\") " pod="openshift-console/downloads-6dd789c8b7-c88f8" Nov 03 19:37:01.676542 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.676508 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ad491ce3-2f92-45a7-9b4f-278309989e64-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qchj6\" (UID: \"ad491ce3-2f92-45a7-9b4f-278309989e64\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qchj6" Nov 03 19:37:01.680405 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.680381 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c57db574-fmm6v"] Nov 03 19:37:01.683673 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.683648 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.696812 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.696788 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c57db574-fmm6v"] Nov 03 19:37:01.777760 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.777733 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n55dd\" (UniqueName: \"kubernetes.io/projected/3265bd58-d654-467c-9b2a-14cedc27b754-kube-api-access-n55dd\") pod \"image-registry-6c57db574-fmm6v\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.777864 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.777765 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3265bd58-d654-467c-9b2a-14cedc27b754-trusted-ca\") pod \"image-registry-6c57db574-fmm6v\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.777864 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.777790 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a7937ca-c705-4091-967e-2e640c622a26-tmp\") pod \"downloads-6dd789c8b7-c88f8\" (UID: \"6a7937ca-c705-4091-967e-2e640c622a26\") " pod="openshift-console/downloads-6dd789c8b7-c88f8" Nov 03 19:37:01.777984 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.777961 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3265bd58-d654-467c-9b2a-14cedc27b754-installation-pull-secrets\") pod \"image-registry-6c57db574-fmm6v\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.778063 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.778024 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3265bd58-d654-467c-9b2a-14cedc27b754-registry-tls\") pod \"image-registry-6c57db574-fmm6v\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.778129 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.778055 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3265bd58-d654-467c-9b2a-14cedc27b754-registry-certificates\") pod \"image-registry-6c57db574-fmm6v\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.778179 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.778149 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a7937ca-c705-4091-967e-2e640c622a26-tmp\") pod \"downloads-6dd789c8b7-c88f8\" (UID: \"6a7937ca-c705-4091-967e-2e640c622a26\") " pod="openshift-console/downloads-6dd789c8b7-c88f8" Nov 03 19:37:01.778179 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.778164 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ad491ce3-2f92-45a7-9b4f-278309989e64-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qchj6\" (UID: \"ad491ce3-2f92-45a7-9b4f-278309989e64\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qchj6" Nov 03 19:37:01.778266 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.778195 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7rw8\" (UniqueName: \"kubernetes.io/projected/6a7937ca-c705-4091-967e-2e640c622a26-kube-api-access-q7rw8\") pod \"downloads-6dd789c8b7-c88f8\" (UID: \"6a7937ca-c705-4091-967e-2e640c622a26\") " pod="openshift-console/downloads-6dd789c8b7-c88f8" Nov 03 19:37:01.778266 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.778227 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3265bd58-d654-467c-9b2a-14cedc27b754-ca-trust-extracted\") pod \"image-registry-6c57db574-fmm6v\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.778368 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.778284 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3265bd58-d654-467c-9b2a-14cedc27b754-image-registry-private-configuration\") pod \"image-registry-6c57db574-fmm6v\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.778368 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.778305 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3265bd58-d654-467c-9b2a-14cedc27b754-bound-sa-token\") pod \"image-registry-6c57db574-fmm6v\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.780564 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.780540 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ad491ce3-2f92-45a7-9b4f-278309989e64-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qchj6\" (UID: \"ad491ce3-2f92-45a7-9b4f-278309989e64\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qchj6" Nov 03 19:37:01.792580 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.792558 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7rw8\" (UniqueName: \"kubernetes.io/projected/6a7937ca-c705-4091-967e-2e640c622a26-kube-api-access-q7rw8\") pod \"downloads-6dd789c8b7-c88f8\" (UID: \"6a7937ca-c705-4091-967e-2e640c622a26\") " pod="openshift-console/downloads-6dd789c8b7-c88f8" Nov 03 19:37:01.879353 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.879294 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3265bd58-d654-467c-9b2a-14cedc27b754-ca-trust-extracted\") pod \"image-registry-6c57db574-fmm6v\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.879353 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.879331 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3265bd58-d654-467c-9b2a-14cedc27b754-image-registry-private-configuration\") pod \"image-registry-6c57db574-fmm6v\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.879353 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.879350 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3265bd58-d654-467c-9b2a-14cedc27b754-bound-sa-token\") pod \"image-registry-6c57db574-fmm6v\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.879705 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.879377 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n55dd\" (UniqueName: \"kubernetes.io/projected/3265bd58-d654-467c-9b2a-14cedc27b754-kube-api-access-n55dd\") pod \"image-registry-6c57db574-fmm6v\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.879705 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.879404 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3265bd58-d654-467c-9b2a-14cedc27b754-trusted-ca\") pod \"image-registry-6c57db574-fmm6v\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.879705 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.879448 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3265bd58-d654-467c-9b2a-14cedc27b754-installation-pull-secrets\") pod \"image-registry-6c57db574-fmm6v\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.879705 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.879489 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3265bd58-d654-467c-9b2a-14cedc27b754-registry-tls\") pod \"image-registry-6c57db574-fmm6v\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.879705 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.879527 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3265bd58-d654-467c-9b2a-14cedc27b754-registry-certificates\") pod \"image-registry-6c57db574-fmm6v\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.879972 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.879714 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3265bd58-d654-467c-9b2a-14cedc27b754-ca-trust-extracted\") pod \"image-registry-6c57db574-fmm6v\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.880432 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.880405 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3265bd58-d654-467c-9b2a-14cedc27b754-registry-certificates\") pod \"image-registry-6c57db574-fmm6v\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.880514 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.880454 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3265bd58-d654-467c-9b2a-14cedc27b754-trusted-ca\") pod \"image-registry-6c57db574-fmm6v\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.881723 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.881703 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3265bd58-d654-467c-9b2a-14cedc27b754-image-registry-private-configuration\") pod \"image-registry-6c57db574-fmm6v\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.881860 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.881835 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3265bd58-d654-467c-9b2a-14cedc27b754-installation-pull-secrets\") pod \"image-registry-6c57db574-fmm6v\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.881952 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.881935 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3265bd58-d654-467c-9b2a-14cedc27b754-registry-tls\") pod \"image-registry-6c57db574-fmm6v\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.892337 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.892310 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3265bd58-d654-467c-9b2a-14cedc27b754-bound-sa-token\") pod \"image-registry-6c57db574-fmm6v\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.892337 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.892327 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n55dd\" (UniqueName: \"kubernetes.io/projected/3265bd58-d654-467c-9b2a-14cedc27b754-kube-api-access-n55dd\") pod \"image-registry-6c57db574-fmm6v\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:01.922353 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.922333 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qchj6" Nov 03 19:37:01.927595 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.927567 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-c88f8" Nov 03 19:37:01.958848 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:01.958815 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nhf67" Nov 03 19:37:02.010694 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:02.009866 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:02.202767 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:02.198460 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qchj6"] Nov 03 19:37:02.205296 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:02.205271 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-c88f8"] Nov 03 19:37:02.207067 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:37:02.207045 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7937ca_c705_4091_967e_2e640c622a26.slice/crio-18aa1249881644f7f3beb3555c25607ffb06ef4df7000b71ce35b76f7c4cf22a WatchSource:0}: Error finding container 18aa1249881644f7f3beb3555c25607ffb06ef4df7000b71ce35b76f7c4cf22a: Status 404 returned error can't find the container with id 18aa1249881644f7f3beb3555c25607ffb06ef4df7000b71ce35b76f7c4cf22a Nov 03 19:37:02.296167 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:02.296102 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c57db574-fmm6v"] Nov 03 19:37:02.298580 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:37:02.298553 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3265bd58_d654_467c_9b2a_14cedc27b754.slice/crio-dbf317add738c05f7cddd2b4c4bf1f3d7defd27f850f8568bdb62aa2afa1b980 WatchSource:0}: Error finding container dbf317add738c05f7cddd2b4c4bf1f3d7defd27f850f8568bdb62aa2afa1b980: Status 404 returned error can't find the container with id dbf317add738c05f7cddd2b4c4bf1f3d7defd27f850f8568bdb62aa2afa1b980 Nov 03 19:37:02.482503 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:02.482421 5059 patch_prober.go:29] interesting pod/image-registry-799ffcd7c9-5pdd7 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 19:37:02.482651 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:02.482480 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" podUID="32065295-c6a2-44ea-88e8-c4fdc5a84083" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:37:03.022643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:03.022562 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-c88f8" event={"ID":"6a7937ca-c705-4091-967e-2e640c622a26","Type":"ContainerStarted","Data":"18aa1249881644f7f3beb3555c25607ffb06ef4df7000b71ce35b76f7c4cf22a"} Nov 03 19:37:03.025653 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:03.025625 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c57db574-fmm6v" event={"ID":"3265bd58-d654-467c-9b2a-14cedc27b754","Type":"ContainerStarted","Data":"c00733af03a11c26991be300ebced8bc887b057aa45430c49b4b704481238514"} Nov 03 19:37:03.025800 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:03.025687 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c57db574-fmm6v" event={"ID":"3265bd58-d654-467c-9b2a-14cedc27b754","Type":"ContainerStarted","Data":"dbf317add738c05f7cddd2b4c4bf1f3d7defd27f850f8568bdb62aa2afa1b980"} Nov 03 19:37:03.026379 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:03.026357 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:03.028072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:03.028052 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qchj6" event={"ID":"ad491ce3-2f92-45a7-9b4f-278309989e64","Type":"ContainerStarted","Data":"55e4a9cfd3117a07fe7fd35c39dc1e744a4e090608b00a10f19636b940937dce"} Nov 03 19:37:03.049160 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:03.049100 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c57db574-fmm6v" podStartSLOduration=2.049087853 podStartE2EDuration="2.049087853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:03.048135741 +0000 UTC m=+17.305933009" watchObservedRunningTime="2026-11-03 19:37:03.049087853 +0000 UTC m=+17.306885130" Nov 03 19:37:04.035319 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:04.035230 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qchj6" event={"ID":"ad491ce3-2f92-45a7-9b4f-278309989e64","Type":"ContainerStarted","Data":"d84ec766302d997e0dd2646c72025bbffe88f5776ae0c76a2bedd7ae24b833ab"} Nov 03 19:37:04.035806 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:04.035487 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qchj6" Nov 03 19:37:04.041138 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:04.041114 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qchj6" Nov 03 19:37:04.054562 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:04.054512 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qchj6" podStartSLOduration=1.9247365090000002 podStartE2EDuration="3.054495815s" totalImagesPullingTime="1.129759306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:04.052868532 +0000 UTC m=+18.310665790" watchObservedRunningTime="2026-11-03 19:37:04.054495815 +0000 UTC m=+18.312293069" Nov 03 19:37:04.753933 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:04.753900 5059 patch_prober.go:29] interesting pod/image-registry-799ffcd7c9-5pdd7 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 19:37:04.754109 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:04.753948 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" podUID="32065295-c6a2-44ea-88e8-c4fdc5a84083" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:37:04.808526 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:04.808498 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2m4mq" Nov 03 19:37:08.873435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:08.873396 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8s8ss"] Nov 03 19:37:08.904279 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:08.904249 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:08.949932 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:08.949896 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 19:37:08.950113 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:08.950067 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lhxcg\"" Nov 03 19:37:08.950252 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:08.950116 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 19:37:08.950252 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:08.950123 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 19:37:08.950252 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:08.950173 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 19:37:09.055611 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.055576 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cc202826-dc4c-4f05-b4bc-63fc108188c2-node-exporter-accelerators-collector-config\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.055798 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.055637 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc202826-dc4c-4f05-b4bc-63fc108188c2-metrics-client-ca\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.055844 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.055799 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cc202826-dc4c-4f05-b4bc-63fc108188c2-node-exporter-wtmp\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.055844 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.055832 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc202826-dc4c-4f05-b4bc-63fc108188c2-sys\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.055936 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.055861 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cc202826-dc4c-4f05-b4bc-63fc108188c2-root\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.055936 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.055884 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc202826-dc4c-4f05-b4bc-63fc108188c2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.056020 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.055969 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cc202826-dc4c-4f05-b4bc-63fc108188c2-node-exporter-tls\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.056054 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.056014 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cc202826-dc4c-4f05-b4bc-63fc108188c2-node-exporter-textfile\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.056054 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.056039 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9gl7\" (UniqueName: \"kubernetes.io/projected/cc202826-dc4c-4f05-b4bc-63fc108188c2-kube-api-access-v9gl7\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.156684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.156587 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cc202826-dc4c-4f05-b4bc-63fc108188c2-node-exporter-tls\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.156684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.156626 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cc202826-dc4c-4f05-b4bc-63fc108188c2-node-exporter-textfile\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.156684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.156643 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9gl7\" (UniqueName: \"kubernetes.io/projected/cc202826-dc4c-4f05-b4bc-63fc108188c2-kube-api-access-v9gl7\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.156925 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.156691 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cc202826-dc4c-4f05-b4bc-63fc108188c2-node-exporter-accelerators-collector-config\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.156925 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.156729 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc202826-dc4c-4f05-b4bc-63fc108188c2-metrics-client-ca\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.156925 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.156766 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cc202826-dc4c-4f05-b4bc-63fc108188c2-node-exporter-wtmp\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.156925 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.156795 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc202826-dc4c-4f05-b4bc-63fc108188c2-sys\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.156925 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.156836 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cc202826-dc4c-4f05-b4bc-63fc108188c2-root\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.156925 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.156860 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc202826-dc4c-4f05-b4bc-63fc108188c2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.157156 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.156999 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cc202826-dc4c-4f05-b4bc-63fc108188c2-node-exporter-wtmp\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.159983 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.159953 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc202826-dc4c-4f05-b4bc-63fc108188c2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.160124 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.160003 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cc202826-dc4c-4f05-b4bc-63fc108188c2-node-exporter-textfile\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.162501 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.160421 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc202826-dc4c-4f05-b4bc-63fc108188c2-metrics-client-ca\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.162501 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.160454 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc202826-dc4c-4f05-b4bc-63fc108188c2-sys\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.162501 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.157035 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cc202826-dc4c-4f05-b4bc-63fc108188c2-root\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.162501 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.160753 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cc202826-dc4c-4f05-b4bc-63fc108188c2-node-exporter-tls\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.162501 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.160997 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cc202826-dc4c-4f05-b4bc-63fc108188c2-node-exporter-accelerators-collector-config\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.169960 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.169913 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9gl7\" (UniqueName: \"kubernetes.io/projected/cc202826-dc4c-4f05-b4bc-63fc108188c2-kube-api-access-v9gl7\") pod \"node-exporter-8s8ss\" (UID: \"cc202826-dc4c-4f05-b4bc-63fc108188c2\") " pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.275052 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:09.275012 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8s8ss" Nov 03 19:37:09.312780 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:37:09.312749 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc202826_dc4c_4f05_b4bc_63fc108188c2.slice/crio-e371af3226022be9064b93d6b7c69c55442c3983d2ac65e998cfed766d17575a WatchSource:0}: Error finding container e371af3226022be9064b93d6b7c69c55442c3983d2ac65e998cfed766d17575a: Status 404 returned error can't find the container with id e371af3226022be9064b93d6b7c69c55442c3983d2ac65e998cfed766d17575a Nov 03 19:37:10.070325 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:10.070283 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8s8ss" event={"ID":"cc202826-dc4c-4f05-b4bc-63fc108188c2","Type":"ContainerStarted","Data":"e371af3226022be9064b93d6b7c69c55442c3983d2ac65e998cfed766d17575a"} Nov 03 19:37:10.607728 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:37:10.607687 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:37:11.078593 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:11.078097 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8s8ss" event={"ID":"cc202826-dc4c-4f05-b4bc-63fc108188c2","Type":"ContainerStarted","Data":"f8b64e16ca2be56f9552a1a7e5885f47937164ea8f047db62d5c5e163dbeb974"} Nov 03 19:37:11.603424 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:11.603391 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:37:11.897553 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:11.897472 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q4zkg" Nov 03 19:37:12.087309 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:12.087275 5059 generic.go:356] "Generic (PLEG): container finished" podID="cc202826-dc4c-4f05-b4bc-63fc108188c2" containerID="f8b64e16ca2be56f9552a1a7e5885f47937164ea8f047db62d5c5e163dbeb974" exitCode=0 Nov 03 19:37:12.087687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:12.087392 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8s8ss" event={"ID":"cc202826-dc4c-4f05-b4bc-63fc108188c2","Type":"ContainerDied","Data":"f8b64e16ca2be56f9552a1a7e5885f47937164ea8f047db62d5c5e163dbeb974"} Nov 03 19:37:12.482870 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:12.482793 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:37:13.095193 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:13.095156 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8s8ss" event={"ID":"cc202826-dc4c-4f05-b4bc-63fc108188c2","Type":"ContainerStarted","Data":"69b679dae6f77778d7779b96a0904943f882e687cc63de120ae9c4e705742e26"} Nov 03 19:37:13.095570 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:13.095211 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8s8ss" event={"ID":"cc202826-dc4c-4f05-b4bc-63fc108188c2","Type":"ContainerStarted","Data":"dcdc404e30b6ffc425aba1f190db3f5d6405b5118aae4c7e5531b3b04f36a29f"} Nov 03 19:37:13.119930 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:13.119882 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8s8ss" podStartSLOduration=3.48354492 podStartE2EDuration="5.119867285s" totalImagesPullingTime="1.5579618s" totalInitContainerRuntime="78.360565ms" isStatefulPod=true podCreationTimestamp="2026-11-03 19:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:13.118133262 +0000 UTC m=+27.375930524" watchObservedRunningTime="2026-11-03 19:37:13.119867285 +0000 UTC m=+27.377664541" Nov 03 19:37:18.774000 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:18.773920 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b4x9q" Nov 03 19:37:19.124100 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:19.124061 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-c88f8" event={"ID":"6a7937ca-c705-4091-967e-2e640c622a26","Type":"ContainerStarted","Data":"cecbb61d72ca19f659a4e15f9281290fd0e1acefbff34b44775daf81cb4ed42c"} Nov 03 19:37:19.124329 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:19.124303 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-c88f8" Nov 03 19:37:19.136981 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:19.136955 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-c88f8" Nov 03 19:37:19.169778 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:19.169728 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-c88f8" podStartSLOduration=2.042536502 podStartE2EDuration="18.169714369s" totalImagesPullingTime="16.127177867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:19.147227123 +0000 UTC m=+33.405024381" watchObservedRunningTime="2026-11-03 19:37:19.169714369 +0000 UTC m=+33.427511623" Nov 03 19:37:25.044912 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:25.044879 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 19:37:25.108162 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:25.108129 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-799ffcd7c9-5pdd7"] Nov 03 19:37:26.629995 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:26.629895 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" podUID="b989ba1e-71f9-424b-90f1-d1450f63e8b1" containerName="registry" containerID="cri-o://7041265b830d65324d40a06784f0a77e789f5ea1c1de24e08a751fd193048631" gracePeriod=30 Nov 03 19:37:26.936041 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:26.936010 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:37:27.018608 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.018574 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-certificates\") pod \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " Nov 03 19:37:27.018792 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.018617 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-tls\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-tls\") pod \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " Nov 03 19:37:27.018792 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.018754 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b989ba1e-71f9-424b-90f1-d1450f63e8b1-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b989ba1e-71f9-424b-90f1-d1450f63e8b1-ca-trust-extracted\") pod \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " Nov 03 19:37:27.018922 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.018806 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-bound-sa-token\") pod \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " Nov 03 19:37:27.018922 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.018850 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b989ba1e-71f9-424b-90f1-d1450f63e8b1-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b989ba1e-71f9-424b-90f1-d1450f63e8b1-image-registry-private-configuration\") pod \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " Nov 03 19:37:27.018922 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.018895 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b989ba1e-71f9-424b-90f1-d1450f63e8b1-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b989ba1e-71f9-424b-90f1-d1450f63e8b1-trusted-ca\") pod \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " Nov 03 19:37:27.019079 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.018926 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b989ba1e-71f9-424b-90f1-d1450f63e8b1-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b989ba1e-71f9-424b-90f1-d1450f63e8b1-installation-pull-secrets\") pod \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " Nov 03 19:37:27.019079 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.018969 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-kube-api-access-28xz4\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-kube-api-access-28xz4\") pod \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\" (UID: \"b989ba1e-71f9-424b-90f1-d1450f63e8b1\") " Nov 03 19:37:27.019186 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.019106 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-certificates" pod "b989ba1e-71f9-424b-90f1-d1450f63e8b1" (UID: "b989ba1e-71f9-424b-90f1-d1450f63e8b1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:37:27.019311 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.019251 5059 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-certificates\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:27.019744 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.019655 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b989ba1e-71f9-424b-90f1-d1450f63e8b1-trusted-ca" pod "b989ba1e-71f9-424b-90f1-d1450f63e8b1" (UID: "b989ba1e-71f9-424b-90f1-d1450f63e8b1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:37:27.021642 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.021605 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b989ba1e-71f9-424b-90f1-d1450f63e8b1-image-registry-private-configuration" pod "b989ba1e-71f9-424b-90f1-d1450f63e8b1" (UID: "b989ba1e-71f9-424b-90f1-d1450f63e8b1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:37:27.021796 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.021653 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-bound-sa-token" pod "b989ba1e-71f9-424b-90f1-d1450f63e8b1" (UID: "b989ba1e-71f9-424b-90f1-d1450f63e8b1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:37:27.022039 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.022007 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-tls" pod "b989ba1e-71f9-424b-90f1-d1450f63e8b1" (UID: "b989ba1e-71f9-424b-90f1-d1450f63e8b1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:37:27.028205 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.028177 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b989ba1e-71f9-424b-90f1-d1450f63e8b1-ca-trust-extracted" pod "b989ba1e-71f9-424b-90f1-d1450f63e8b1" (UID: "b989ba1e-71f9-424b-90f1-d1450f63e8b1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:37:27.028833 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.028805 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-kube-api-access-28xz4" pod "b989ba1e-71f9-424b-90f1-d1450f63e8b1" (UID: "b989ba1e-71f9-424b-90f1-d1450f63e8b1"). InnerVolumeSpecName "kube-api-access-28xz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:37:27.030608 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.030579 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b989ba1e-71f9-424b-90f1-d1450f63e8b1-installation-pull-secrets" pod "b989ba1e-71f9-424b-90f1-d1450f63e8b1" (UID: "b989ba1e-71f9-424b-90f1-d1450f63e8b1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:37:27.119786 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.119758 5059 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-registry-tls\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:27.119786 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.119786 5059 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b989ba1e-71f9-424b-90f1-d1450f63e8b1-ca-trust-extracted\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:27.120011 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.119800 5059 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-bound-sa-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:27.120011 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.119815 5059 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b989ba1e-71f9-424b-90f1-d1450f63e8b1-image-registry-private-configuration\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:27.120011 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.119830 5059 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b989ba1e-71f9-424b-90f1-d1450f63e8b1-trusted-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:27.120011 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.119843 5059 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b989ba1e-71f9-424b-90f1-d1450f63e8b1-installation-pull-secrets\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:27.120011 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.119857 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28xz4\" (UniqueName: \"kubernetes.io/projected/b989ba1e-71f9-424b-90f1-d1450f63e8b1-kube-api-access-28xz4\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:27.168006 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.167947 5059 generic.go:356] "Generic (PLEG): container finished" podID="b989ba1e-71f9-424b-90f1-d1450f63e8b1" containerID="7041265b830d65324d40a06784f0a77e789f5ea1c1de24e08a751fd193048631" exitCode=0 Nov 03 19:37:27.168006 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.167996 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" Nov 03 19:37:27.168119 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.168022 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" event={"ID":"b989ba1e-71f9-424b-90f1-d1450f63e8b1","Type":"ContainerDied","Data":"7041265b830d65324d40a06784f0a77e789f5ea1c1de24e08a751fd193048631"} Nov 03 19:37:27.168119 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.168067 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd465cd78-4n8kx" event={"ID":"b989ba1e-71f9-424b-90f1-d1450f63e8b1","Type":"ContainerDied","Data":"8e109501f0db9a519b2048e1b15d4d7add9f81e67d9123165db0a963b15b1b2d"} Nov 03 19:37:27.168119 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.168091 5059 scope.go:122] "RemoveContainer" containerID="7041265b830d65324d40a06784f0a77e789f5ea1c1de24e08a751fd193048631" Nov 03 19:37:27.176579 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.176556 5059 scope.go:122] "RemoveContainer" containerID="7041265b830d65324d40a06784f0a77e789f5ea1c1de24e08a751fd193048631" Nov 03 19:37:27.176866 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:37:27.176847 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7041265b830d65324d40a06784f0a77e789f5ea1c1de24e08a751fd193048631\": container with ID starting with 7041265b830d65324d40a06784f0a77e789f5ea1c1de24e08a751fd193048631 not found: ID does not exist" containerID="7041265b830d65324d40a06784f0a77e789f5ea1c1de24e08a751fd193048631" Nov 03 19:37:27.176925 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.176874 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7041265b830d65324d40a06784f0a77e789f5ea1c1de24e08a751fd193048631"} err="failed to get container status \"7041265b830d65324d40a06784f0a77e789f5ea1c1de24e08a751fd193048631\": rpc error: code = NotFound desc = could not find container \"7041265b830d65324d40a06784f0a77e789f5ea1c1de24e08a751fd193048631\": container with ID starting with 7041265b830d65324d40a06784f0a77e789f5ea1c1de24e08a751fd193048631 not found: ID does not exist" Nov 03 19:37:27.192399 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.192367 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6dd465cd78-4n8kx"] Nov 03 19:37:27.196913 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.196892 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6dd465cd78-4n8kx"] Nov 03 19:37:27.855800 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:27.855769 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b989ba1e-71f9-424b-90f1-d1450f63e8b1" path="/var/lib/kubelet/pods/b989ba1e-71f9-424b-90f1-d1450f63e8b1/volumes" Nov 03 19:37:40.603169 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:37:40.603126 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:37:50.134951 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.134901 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" podUID="32065295-c6a2-44ea-88e8-c4fdc5a84083" containerName="registry" containerID="cri-o://5609de2d9f0e6127c11efd5b0a9f67fa54e1525ea6353e6a712e0ed74644d2b7" gracePeriod=30 Nov 03 19:37:50.258418 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.258386 5059 generic.go:356] "Generic (PLEG): container finished" podID="32065295-c6a2-44ea-88e8-c4fdc5a84083" containerID="5609de2d9f0e6127c11efd5b0a9f67fa54e1525ea6353e6a712e0ed74644d2b7" exitCode=0 Nov 03 19:37:50.258545 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.258461 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" event={"ID":"32065295-c6a2-44ea-88e8-c4fdc5a84083","Type":"ContainerDied","Data":"5609de2d9f0e6127c11efd5b0a9f67fa54e1525ea6353e6a712e0ed74644d2b7"} Nov 03 19:37:50.406941 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.406887 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:37:50.502710 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.502680 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-tls\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-tls\") pod \"32065295-c6a2-44ea-88e8-c4fdc5a84083\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " Nov 03 19:37:50.502866 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.502718 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-bound-sa-token\") pod \"32065295-c6a2-44ea-88e8-c4fdc5a84083\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " Nov 03 19:37:50.502866 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.502777 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-certificates\") pod \"32065295-c6a2-44ea-88e8-c4fdc5a84083\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " Nov 03 19:37:50.502866 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.502833 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32065295-c6a2-44ea-88e8-c4fdc5a84083-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32065295-c6a2-44ea-88e8-c4fdc5a84083-image-registry-private-configuration\") pod \"32065295-c6a2-44ea-88e8-c4fdc5a84083\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " Nov 03 19:37:50.503030 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.502864 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32065295-c6a2-44ea-88e8-c4fdc5a84083-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32065295-c6a2-44ea-88e8-c4fdc5a84083-installation-pull-secrets\") pod \"32065295-c6a2-44ea-88e8-c4fdc5a84083\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " Nov 03 19:37:50.503030 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.502907 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32065295-c6a2-44ea-88e8-c4fdc5a84083-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32065295-c6a2-44ea-88e8-c4fdc5a84083-ca-trust-extracted\") pod \"32065295-c6a2-44ea-88e8-c4fdc5a84083\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " Nov 03 19:37:50.503030 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.502949 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32065295-c6a2-44ea-88e8-c4fdc5a84083-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32065295-c6a2-44ea-88e8-c4fdc5a84083-trusted-ca\") pod \"32065295-c6a2-44ea-88e8-c4fdc5a84083\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " Nov 03 19:37:50.503030 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.502986 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-kube-api-access-kbrbm\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-kube-api-access-kbrbm\") pod \"32065295-c6a2-44ea-88e8-c4fdc5a84083\" (UID: \"32065295-c6a2-44ea-88e8-c4fdc5a84083\") " Nov 03 19:37:50.503326 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.503291 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-certificates" pod "32065295-c6a2-44ea-88e8-c4fdc5a84083" (UID: "32065295-c6a2-44ea-88e8-c4fdc5a84083"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:37:50.503560 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.503518 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32065295-c6a2-44ea-88e8-c4fdc5a84083-trusted-ca" pod "32065295-c6a2-44ea-88e8-c4fdc5a84083" (UID: "32065295-c6a2-44ea-88e8-c4fdc5a84083"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:37:50.505192 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.505163 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-bound-sa-token" pod "32065295-c6a2-44ea-88e8-c4fdc5a84083" (UID: "32065295-c6a2-44ea-88e8-c4fdc5a84083"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:37:50.505290 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.505192 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32065295-c6a2-44ea-88e8-c4fdc5a84083-installation-pull-secrets" pod "32065295-c6a2-44ea-88e8-c4fdc5a84083" (UID: "32065295-c6a2-44ea-88e8-c4fdc5a84083"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:37:50.505347 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.505312 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-tls" pod "32065295-c6a2-44ea-88e8-c4fdc5a84083" (UID: "32065295-c6a2-44ea-88e8-c4fdc5a84083"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:37:50.505464 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.505440 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32065295-c6a2-44ea-88e8-c4fdc5a84083-image-registry-private-configuration" pod "32065295-c6a2-44ea-88e8-c4fdc5a84083" (UID: "32065295-c6a2-44ea-88e8-c4fdc5a84083"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:37:50.505625 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.505602 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-kube-api-access-kbrbm" pod "32065295-c6a2-44ea-88e8-c4fdc5a84083" (UID: "32065295-c6a2-44ea-88e8-c4fdc5a84083"). InnerVolumeSpecName "kube-api-access-kbrbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:37:50.510382 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.510356 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32065295-c6a2-44ea-88e8-c4fdc5a84083-ca-trust-extracted" pod "32065295-c6a2-44ea-88e8-c4fdc5a84083" (UID: "32065295-c6a2-44ea-88e8-c4fdc5a84083"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:37:50.604038 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.604012 5059 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32065295-c6a2-44ea-88e8-c4fdc5a84083-ca-trust-extracted\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:50.604038 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.604036 5059 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32065295-c6a2-44ea-88e8-c4fdc5a84083-trusted-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:50.604168 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.604046 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbrbm\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-kube-api-access-kbrbm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:50.604168 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.604056 5059 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-tls\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:50.604168 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.604066 5059 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32065295-c6a2-44ea-88e8-c4fdc5a84083-bound-sa-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:50.604168 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.604075 5059 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32065295-c6a2-44ea-88e8-c4fdc5a84083-registry-certificates\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:50.604168 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.604085 5059 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32065295-c6a2-44ea-88e8-c4fdc5a84083-image-registry-private-configuration\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:50.604168 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:50.604094 5059 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32065295-c6a2-44ea-88e8-c4fdc5a84083-installation-pull-secrets\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:51.263956 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:51.263928 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" Nov 03 19:37:51.263956 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:51.263944 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-799ffcd7c9-5pdd7" event={"ID":"32065295-c6a2-44ea-88e8-c4fdc5a84083","Type":"ContainerDied","Data":"b6de17f35c0d2cd0522a618a81d7c9aa8c0584954008f35de924edb4305cf227"} Nov 03 19:37:51.264386 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:51.263994 5059 scope.go:122] "RemoveContainer" containerID="5609de2d9f0e6127c11efd5b0a9f67fa54e1525ea6353e6a712e0ed74644d2b7" Nov 03 19:37:51.293230 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:51.293205 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-799ffcd7c9-5pdd7"] Nov 03 19:37:51.297830 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:51.297803 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-799ffcd7c9-5pdd7"] Nov 03 19:37:51.854960 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:51.854931 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32065295-c6a2-44ea-88e8-c4fdc5a84083" path="/var/lib/kubelet/pods/32065295-c6a2-44ea-88e8-c4fdc5a84083/volumes" Nov 03 19:37:55.281872 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:55.281835 5059 generic.go:356] "Generic (PLEG): container finished" podID="40834bff-2db0-4690-9cb9-7b4aacdab3ab" containerID="974d23ae8d30a4fce6a921ba2dc0d90ab07c28ac162cd3380ddc95968846b3c1" exitCode=0 Nov 03 19:37:55.282311 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:55.281909 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wdls7" event={"ID":"40834bff-2db0-4690-9cb9-7b4aacdab3ab","Type":"ContainerDied","Data":"974d23ae8d30a4fce6a921ba2dc0d90ab07c28ac162cd3380ddc95968846b3c1"} Nov 03 19:37:55.282311 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:55.282225 5059 scope.go:122] "RemoveContainer" containerID="974d23ae8d30a4fce6a921ba2dc0d90ab07c28ac162cd3380ddc95968846b3c1" Nov 03 19:37:55.283636 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:55.283614 5059 generic.go:356] "Generic (PLEG): container finished" podID="c99b0c19-004e-450c-852a-e38338635b61" containerID="481852318efef15a99c90fc33e8f7d88cfa6b35f952bd0e4e6c4993664dcc592" exitCode=0 Nov 03 19:37:55.283771 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:55.283652 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" event={"ID":"c99b0c19-004e-450c-852a-e38338635b61","Type":"ContainerDied","Data":"481852318efef15a99c90fc33e8f7d88cfa6b35f952bd0e4e6c4993664dcc592"} Nov 03 19:37:55.283928 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:55.283915 5059 scope.go:122] "RemoveContainer" containerID="481852318efef15a99c90fc33e8f7d88cfa6b35f952bd0e4e6c4993664dcc592" Nov 03 19:37:55.746450 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:55.746410 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-bhdxh-d8v5q"] Nov 03 19:37:55.746714 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:55.746642 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" podUID="4b79231d-e954-41b2-924d-67808a734ecf" containerName="insights-gathering" containerID="cri-o://41d3b07ff68a6e28275ab819a514afbe80d479b939d817bbf8e5094b3a82d99c" gracePeriod=30 Nov 03 19:37:55.848256 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:55.848223 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-ftqrx-rt56g"] Nov 03 19:37:55.848582 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:55.848568 5059 cpu_manager.go:414] "containerMap: removing container" podUID="32065295-c6a2-44ea-88e8-c4fdc5a84083" containerName="registry" Nov 03 19:37:55.848628 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:55.848583 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32065295-c6a2-44ea-88e8-c4fdc5a84083" containerName="registry" Nov 03 19:37:55.848628 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:55.848598 5059 cpu_manager.go:414] "containerMap: removing container" podUID="b989ba1e-71f9-424b-90f1-d1450f63e8b1" containerName="registry" Nov 03 19:37:55.848628 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:55.848603 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b989ba1e-71f9-424b-90f1-d1450f63e8b1" containerName="registry" Nov 03 19:37:55.848752 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:55.848678 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b989ba1e-71f9-424b-90f1-d1450f63e8b1" containerName="registry" Nov 03 19:37:55.848752 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:55.848691 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="32065295-c6a2-44ea-88e8-c4fdc5a84083" containerName="registry" Nov 03 19:37:55.871599 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:55.871563 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ftqrx-rt56g" Nov 03 19:37:55.878525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:55.878497 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-ftqrx-rt56g"] Nov 03 19:37:56.036163 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.036135 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" Nov 03 19:37:56.047418 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.047392 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t6c7\" (UniqueName: \"kubernetes.io/projected/a60d7aa8-156a-4c7e-964c-3c20241c02ec-kube-api-access-7t6c7\") pod \"periodic-gathering-ftqrx-rt56g\" (UID: \"a60d7aa8-156a-4c7e-964c-3c20241c02ec\") " pod="openshift-insights/periodic-gathering-ftqrx-rt56g" Nov 03 19:37:56.047539 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.047437 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a60d7aa8-156a-4c7e-964c-3c20241c02ec-archives-path\") pod \"periodic-gathering-ftqrx-rt56g\" (UID: \"a60d7aa8-156a-4c7e-964c-3c20241c02ec\") " pod="openshift-insights/periodic-gathering-ftqrx-rt56g" Nov 03 19:37:56.047646 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.047625 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a60d7aa8-156a-4c7e-964c-3c20241c02ec-service-ca-bundle\") pod \"periodic-gathering-ftqrx-rt56g\" (UID: \"a60d7aa8-156a-4c7e-964c-3c20241c02ec\") " pod="openshift-insights/periodic-gathering-ftqrx-rt56g" Nov 03 19:37:56.047840 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.047820 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a60d7aa8-156a-4c7e-964c-3c20241c02ec-trusted-ca-bundle\") pod \"periodic-gathering-ftqrx-rt56g\" (UID: \"a60d7aa8-156a-4c7e-964c-3c20241c02ec\") " pod="openshift-insights/periodic-gathering-ftqrx-rt56g" Nov 03 19:37:56.148626 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.148586 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b79231d-e954-41b2-924d-67808a734ecf-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b79231d-e954-41b2-924d-67808a734ecf-trusted-ca-bundle\") pod \"4b79231d-e954-41b2-924d-67808a734ecf\" (UID: \"4b79231d-e954-41b2-924d-67808a734ecf\") " Nov 03 19:37:56.148791 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.148728 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b79231d-e954-41b2-924d-67808a734ecf-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4b79231d-e954-41b2-924d-67808a734ecf-archives-path\") pod \"4b79231d-e954-41b2-924d-67808a734ecf\" (UID: \"4b79231d-e954-41b2-924d-67808a734ecf\") " Nov 03 19:37:56.148852 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.148810 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b79231d-e954-41b2-924d-67808a734ecf-kube-api-access-nhrbz\" (UniqueName: \"kubernetes.io/projected/4b79231d-e954-41b2-924d-67808a734ecf-kube-api-access-nhrbz\") pod \"4b79231d-e954-41b2-924d-67808a734ecf\" (UID: \"4b79231d-e954-41b2-924d-67808a734ecf\") " Nov 03 19:37:56.148903 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.148853 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b79231d-e954-41b2-924d-67808a734ecf-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b79231d-e954-41b2-924d-67808a734ecf-service-ca-bundle\") pod \"4b79231d-e954-41b2-924d-67808a734ecf\" (UID: \"4b79231d-e954-41b2-924d-67808a734ecf\") " Nov 03 19:37:56.148973 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.148951 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a60d7aa8-156a-4c7e-964c-3c20241c02ec-archives-path\") pod \"periodic-gathering-ftqrx-rt56g\" (UID: \"a60d7aa8-156a-4c7e-964c-3c20241c02ec\") " pod="openshift-insights/periodic-gathering-ftqrx-rt56g" Nov 03 19:37:56.149026 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.149017 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a60d7aa8-156a-4c7e-964c-3c20241c02ec-service-ca-bundle\") pod \"periodic-gathering-ftqrx-rt56g\" (UID: \"a60d7aa8-156a-4c7e-964c-3c20241c02ec\") " pod="openshift-insights/periodic-gathering-ftqrx-rt56g" Nov 03 19:37:56.149077 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.149054 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b79231d-e954-41b2-924d-67808a734ecf-trusted-ca-bundle" pod "4b79231d-e954-41b2-924d-67808a734ecf" (UID: "4b79231d-e954-41b2-924d-67808a734ecf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:37:56.149177 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.149068 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b79231d-e954-41b2-924d-67808a734ecf-archives-path" pod "4b79231d-e954-41b2-924d-67808a734ecf" (UID: "4b79231d-e954-41b2-924d-67808a734ecf"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:37:56.149258 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.149174 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a60d7aa8-156a-4c7e-964c-3c20241c02ec-trusted-ca-bundle\") pod \"periodic-gathering-ftqrx-rt56g\" (UID: \"a60d7aa8-156a-4c7e-964c-3c20241c02ec\") " pod="openshift-insights/periodic-gathering-ftqrx-rt56g" Nov 03 19:37:56.149258 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.149233 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t6c7\" (UniqueName: \"kubernetes.io/projected/a60d7aa8-156a-4c7e-964c-3c20241c02ec-kube-api-access-7t6c7\") pod \"periodic-gathering-ftqrx-rt56g\" (UID: \"a60d7aa8-156a-4c7e-964c-3c20241c02ec\") " pod="openshift-insights/periodic-gathering-ftqrx-rt56g" Nov 03 19:37:56.149258 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.149235 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b79231d-e954-41b2-924d-67808a734ecf-service-ca-bundle" pod "4b79231d-e954-41b2-924d-67808a734ecf" (UID: "4b79231d-e954-41b2-924d-67808a734ecf"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:37:56.149400 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.149296 5059 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b79231d-e954-41b2-924d-67808a734ecf-trusted-ca-bundle\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:56.149400 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.149313 5059 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4b79231d-e954-41b2-924d-67808a734ecf-archives-path\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:56.149503 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.149399 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a60d7aa8-156a-4c7e-964c-3c20241c02ec-archives-path\") pod \"periodic-gathering-ftqrx-rt56g\" (UID: \"a60d7aa8-156a-4c7e-964c-3c20241c02ec\") " pod="openshift-insights/periodic-gathering-ftqrx-rt56g" Nov 03 19:37:56.149717 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.149699 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a60d7aa8-156a-4c7e-964c-3c20241c02ec-service-ca-bundle\") pod \"periodic-gathering-ftqrx-rt56g\" (UID: \"a60d7aa8-156a-4c7e-964c-3c20241c02ec\") " pod="openshift-insights/periodic-gathering-ftqrx-rt56g" Nov 03 19:37:56.149967 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.149946 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a60d7aa8-156a-4c7e-964c-3c20241c02ec-trusted-ca-bundle\") pod \"periodic-gathering-ftqrx-rt56g\" (UID: \"a60d7aa8-156a-4c7e-964c-3c20241c02ec\") " pod="openshift-insights/periodic-gathering-ftqrx-rt56g" Nov 03 19:37:56.151261 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.151233 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b79231d-e954-41b2-924d-67808a734ecf-kube-api-access-nhrbz" pod "4b79231d-e954-41b2-924d-67808a734ecf" (UID: "4b79231d-e954-41b2-924d-67808a734ecf"). InnerVolumeSpecName "kube-api-access-nhrbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:37:56.166733 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.166710 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t6c7\" (UniqueName: \"kubernetes.io/projected/a60d7aa8-156a-4c7e-964c-3c20241c02ec-kube-api-access-7t6c7\") pod \"periodic-gathering-ftqrx-rt56g\" (UID: \"a60d7aa8-156a-4c7e-964c-3c20241c02ec\") " pod="openshift-insights/periodic-gathering-ftqrx-rt56g" Nov 03 19:37:56.205972 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.205950 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ftqrx-rt56g" Nov 03 19:37:56.249901 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.249854 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhrbz\" (UniqueName: \"kubernetes.io/projected/4b79231d-e954-41b2-924d-67808a734ecf-kube-api-access-nhrbz\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:56.249901 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.249877 5059 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b79231d-e954-41b2-924d-67808a734ecf-service-ca-bundle\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:37:56.297484 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.297439 5059 generic.go:356] "Generic (PLEG): container finished" podID="4b79231d-e954-41b2-924d-67808a734ecf" containerID="41d3b07ff68a6e28275ab819a514afbe80d479b939d817bbf8e5094b3a82d99c" exitCode=2 Nov 03 19:37:56.297924 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.297587 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" event={"ID":"4b79231d-e954-41b2-924d-67808a734ecf","Type":"ContainerDied","Data":"41d3b07ff68a6e28275ab819a514afbe80d479b939d817bbf8e5094b3a82d99c"} Nov 03 19:37:56.297924 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.297686 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" event={"ID":"4b79231d-e954-41b2-924d-67808a734ecf","Type":"ContainerDied","Data":"84dcb8f62c1c8e86cbc9805e06cd0f30068afc5169cf3efd95ae1cfd8d7d1404"} Nov 03 19:37:56.297924 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.297726 5059 scope.go:122] "RemoveContainer" containerID="41d3b07ff68a6e28275ab819a514afbe80d479b939d817bbf8e5094b3a82d99c" Nov 03 19:37:56.297924 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.297839 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bhdxh-d8v5q" Nov 03 19:37:56.301372 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.300795 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wdls7" event={"ID":"40834bff-2db0-4690-9cb9-7b4aacdab3ab","Type":"ContainerStarted","Data":"47e9f7c4c5f384b67047856a4f8a68a893ee773a78da4bbefeec8901821a2e6f"} Nov 03 19:37:56.304042 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.303656 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" event={"ID":"c99b0c19-004e-450c-852a-e38338635b61","Type":"ContainerStarted","Data":"b742b7b0202501adeba598c21e37a746e39b623b622c6c629096721511b0d73e"} Nov 03 19:37:56.308857 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.308832 5059 scope.go:122] "RemoveContainer" containerID="41d3b07ff68a6e28275ab819a514afbe80d479b939d817bbf8e5094b3a82d99c" Nov 03 19:37:56.309105 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:37:56.309081 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d3b07ff68a6e28275ab819a514afbe80d479b939d817bbf8e5094b3a82d99c\": container with ID starting with 41d3b07ff68a6e28275ab819a514afbe80d479b939d817bbf8e5094b3a82d99c not found: ID does not exist" containerID="41d3b07ff68a6e28275ab819a514afbe80d479b939d817bbf8e5094b3a82d99c" Nov 03 19:37:56.309191 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.309116 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d3b07ff68a6e28275ab819a514afbe80d479b939d817bbf8e5094b3a82d99c"} err="failed to get container status \"41d3b07ff68a6e28275ab819a514afbe80d479b939d817bbf8e5094b3a82d99c\": rpc error: code = NotFound desc = could not find container \"41d3b07ff68a6e28275ab819a514afbe80d479b939d817bbf8e5094b3a82d99c\": container with ID starting with 41d3b07ff68a6e28275ab819a514afbe80d479b939d817bbf8e5094b3a82d99c not found: ID does not exist" Nov 03 19:37:56.359477 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.359449 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-bhdxh-d8v5q"] Nov 03 19:37:56.366294 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.366272 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-bhdxh-d8v5q"] Nov 03 19:37:56.498491 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:56.498469 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-ftqrx-rt56g"] Nov 03 19:37:56.500883 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:37:56.500826 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60d7aa8_156a_4c7e_964c_3c20241c02ec.slice/crio-65dc305ce566405148e2450d163e39034937571d7cd1bc48476aac678578e3c8 WatchSource:0}: Error finding container 65dc305ce566405148e2450d163e39034937571d7cd1bc48476aac678578e3c8: Status 404 returned error can't find the container with id 65dc305ce566405148e2450d163e39034937571d7cd1bc48476aac678578e3c8 Nov 03 19:37:57.308922 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:57.308880 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ftqrx-rt56g" event={"ID":"a60d7aa8-156a-4c7e-964c-3c20241c02ec","Type":"ContainerStarted","Data":"3ed08a9a04c4bb41c1603c5fbc1967f1823b1e4aefed23d4755b36d6e13d899e"} Nov 03 19:37:57.309356 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:57.308944 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ftqrx-rt56g" event={"ID":"a60d7aa8-156a-4c7e-964c-3c20241c02ec","Type":"ContainerStarted","Data":"65dc305ce566405148e2450d163e39034937571d7cd1bc48476aac678578e3c8"} Nov 03 19:37:57.328233 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:57.328189 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-ftqrx-rt56g" podStartSLOduration=2.328172146 podStartE2EDuration="2.328172146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:57.327445505 +0000 UTC m=+71.585242759" watchObservedRunningTime="2026-11-03 19:37:57.328172146 +0000 UTC m=+71.585969403" Nov 03 19:37:57.854754 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:37:57.854715 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b79231d-e954-41b2-924d-67808a734ecf" path="/var/lib/kubelet/pods/4b79231d-e954-41b2-924d-67808a734ecf/volumes" Nov 03 19:38:10.602271 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:38:10.602228 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:38:40.602377 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:38:40.602337 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:39:10.604230 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:39:10.604194 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:39:19.635174 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:39:19.635144 5059 generic.go:356] "Generic (PLEG): container finished" podID="a60d7aa8-156a-4c7e-964c-3c20241c02ec" containerID="3ed08a9a04c4bb41c1603c5fbc1967f1823b1e4aefed23d4755b36d6e13d899e" exitCode=0 Nov 03 19:39:19.635539 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:39:19.635216 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ftqrx-rt56g" event={"ID":"a60d7aa8-156a-4c7e-964c-3c20241c02ec","Type":"ContainerDied","Data":"3ed08a9a04c4bb41c1603c5fbc1967f1823b1e4aefed23d4755b36d6e13d899e"} Nov 03 19:39:20.776475 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:39:20.776449 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ftqrx-rt56g" Nov 03 19:39:20.890399 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:39:20.890340 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a60d7aa8-156a-4c7e-964c-3c20241c02ec-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a60d7aa8-156a-4c7e-964c-3c20241c02ec-archives-path\") pod \"a60d7aa8-156a-4c7e-964c-3c20241c02ec\" (UID: \"a60d7aa8-156a-4c7e-964c-3c20241c02ec\") " Nov 03 19:39:20.890399 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:39:20.890383 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a60d7aa8-156a-4c7e-964c-3c20241c02ec-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a60d7aa8-156a-4c7e-964c-3c20241c02ec-trusted-ca-bundle\") pod \"a60d7aa8-156a-4c7e-964c-3c20241c02ec\" (UID: \"a60d7aa8-156a-4c7e-964c-3c20241c02ec\") " Nov 03 19:39:20.890551 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:39:20.890430 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a60d7aa8-156a-4c7e-964c-3c20241c02ec-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a60d7aa8-156a-4c7e-964c-3c20241c02ec-service-ca-bundle\") pod \"a60d7aa8-156a-4c7e-964c-3c20241c02ec\" (UID: \"a60d7aa8-156a-4c7e-964c-3c20241c02ec\") " Nov 03 19:39:20.890551 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:39:20.890461 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a60d7aa8-156a-4c7e-964c-3c20241c02ec-kube-api-access-7t6c7\" (UniqueName: \"kubernetes.io/projected/a60d7aa8-156a-4c7e-964c-3c20241c02ec-kube-api-access-7t6c7\") pod \"a60d7aa8-156a-4c7e-964c-3c20241c02ec\" (UID: \"a60d7aa8-156a-4c7e-964c-3c20241c02ec\") " Nov 03 19:39:20.890650 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:39:20.890626 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60d7aa8-156a-4c7e-964c-3c20241c02ec-archives-path" pod "a60d7aa8-156a-4c7e-964c-3c20241c02ec" (UID: "a60d7aa8-156a-4c7e-964c-3c20241c02ec"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:39:20.890842 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:39:20.890824 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60d7aa8-156a-4c7e-964c-3c20241c02ec-trusted-ca-bundle" pod "a60d7aa8-156a-4c7e-964c-3c20241c02ec" (UID: "a60d7aa8-156a-4c7e-964c-3c20241c02ec"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:39:20.890886 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:39:20.890863 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60d7aa8-156a-4c7e-964c-3c20241c02ec-service-ca-bundle" pod "a60d7aa8-156a-4c7e-964c-3c20241c02ec" (UID: "a60d7aa8-156a-4c7e-964c-3c20241c02ec"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:39:20.892536 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:39:20.892514 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60d7aa8-156a-4c7e-964c-3c20241c02ec-kube-api-access-7t6c7" pod "a60d7aa8-156a-4c7e-964c-3c20241c02ec" (UID: "a60d7aa8-156a-4c7e-964c-3c20241c02ec"). InnerVolumeSpecName "kube-api-access-7t6c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:39:20.991400 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:39:20.991375 5059 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a60d7aa8-156a-4c7e-964c-3c20241c02ec-archives-path\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:39:20.991400 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:39:20.991398 5059 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a60d7aa8-156a-4c7e-964c-3c20241c02ec-trusted-ca-bundle\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:39:20.991532 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:39:20.991412 5059 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a60d7aa8-156a-4c7e-964c-3c20241c02ec-service-ca-bundle\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:39:20.991532 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:39:20.991425 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t6c7\" (UniqueName: \"kubernetes.io/projected/a60d7aa8-156a-4c7e-964c-3c20241c02ec-kube-api-access-7t6c7\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:39:21.643408 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:39:21.643378 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ftqrx-rt56g" Nov 03 19:39:21.643597 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:39:21.643411 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ftqrx-rt56g" event={"ID":"a60d7aa8-156a-4c7e-964c-3c20241c02ec","Type":"ContainerDied","Data":"65dc305ce566405148e2450d163e39034937571d7cd1bc48476aac678578e3c8"} Nov 03 19:39:21.643597 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:39:21.643453 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65dc305ce566405148e2450d163e39034937571d7cd1bc48476aac678578e3c8" Nov 03 19:39:40.603169 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:39:40.603125 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:40:10.602867 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:40:10.602829 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:40:40.601443 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:40:40.601363 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:41:10.603081 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:41:10.603042 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:41:23.670957 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.670924 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4"] Nov 03 19:41:23.671401 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.671242 5059 cpu_manager.go:414] "containerMap: removing container" podUID="4b79231d-e954-41b2-924d-67808a734ecf" containerName="insights-gathering" Nov 03 19:41:23.671401 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.671256 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b79231d-e954-41b2-924d-67808a734ecf" containerName="insights-gathering" Nov 03 19:41:23.671401 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.671275 5059 cpu_manager.go:414] "containerMap: removing container" podUID="a60d7aa8-156a-4c7e-964c-3c20241c02ec" containerName="insights-gathering" Nov 03 19:41:23.671401 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.671281 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a60d7aa8-156a-4c7e-964c-3c20241c02ec" containerName="insights-gathering" Nov 03 19:41:23.671401 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.671330 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a60d7aa8-156a-4c7e-964c-3c20241c02ec" containerName="insights-gathering" Nov 03 19:41:23.671401 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.671341 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b79231d-e954-41b2-924d-67808a734ecf" containerName="insights-gathering" Nov 03 19:41:23.674618 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.674585 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.688968 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.688946 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 19:41:23.689761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.689741 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-rb4sg\"" Nov 03 19:41:23.691190 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.691167 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4"] Nov 03 19:41:23.865959 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.865926 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.866132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.865964 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.866132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.865991 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.866132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.866085 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.866275 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.866160 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdlh2\" (UniqueName: \"kubernetes.io/projected/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-kube-api-access-kdlh2\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.866275 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.866182 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.866275 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.866207 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.866366 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.866276 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.866366 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.866303 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.967574 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.967497 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.967574 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.967528 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.967574 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.967551 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.967846 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.967578 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.967846 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.967619 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.967846 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.967654 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.967846 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.967714 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.967846 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.967739 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.967846 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.967774 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdlh2\" (UniqueName: \"kubernetes.io/projected/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-kube-api-access-kdlh2\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.968307 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.967999 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.968307 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.968007 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.968307 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.968176 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.968307 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.968203 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.968494 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.968474 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.969986 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.969965 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.970091 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.970048 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.980024 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.980000 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.980222 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.980203 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdlh2\" (UniqueName: \"kubernetes.io/projected/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-kube-api-access-kdlh2\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:23.988079 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:23.988056 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:24.265625 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:24.265599 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4"] Nov 03 19:41:24.267964 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:41:24.267934 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d45aa9_bfa8_43c6_8674_6f8f14a3097e.slice/crio-f5d4f211d31c87466d57af904dc64201bfbfa2b8af799af82610b17982f3e0d7 WatchSource:0}: Error finding container f5d4f211d31c87466d57af904dc64201bfbfa2b8af799af82610b17982f3e0d7: Status 404 returned error can't find the container with id f5d4f211d31c87466d57af904dc64201bfbfa2b8af799af82610b17982f3e0d7 Nov 03 19:41:25.129557 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:25.129523 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" event={"ID":"70d45aa9-bfa8-43c6-8674-6f8f14a3097e","Type":"ContainerStarted","Data":"f5d4f211d31c87466d57af904dc64201bfbfa2b8af799af82610b17982f3e0d7"} Nov 03 19:41:28.838938 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:28.838902 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:41:28.839224 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:28.838973 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:41:28.839224 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:28.839003 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:41:29.147583 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:29.147498 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" event={"ID":"70d45aa9-bfa8-43c6-8674-6f8f14a3097e","Type":"ContainerStarted","Data":"6a213e69e74bbe139a2688d93452e7d59e19f98c3c9a8e404648454c84993ae7"} Nov 03 19:41:29.172991 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:29.172946 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" podStartSLOduration=1.6046658900000002 podStartE2EDuration="6.172932532s" totalImagesPullingTime="4.568266642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:29.171945162 +0000 UTC m=+283.429742432" watchObservedRunningTime="2026-11-03 19:41:29.172932532 +0000 UTC m=+283.430729781" Nov 03 19:41:29.988994 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:29.988959 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:29.993570 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:29.993546 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:30.151642 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:30.151612 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:30.152617 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:30.152601 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 19:41:30.909155 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:30.909118 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz"] Nov 03 19:41:30.912813 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:30.912797 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz" Nov 03 19:41:30.916455 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:30.916429 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3853f0d-7f5a-41a8-b2b9-dc1b730753f2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz\" (UID: \"e3853f0d-7f5a-41a8-b2b9-dc1b730753f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz" Nov 03 19:41:30.916571 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:30.916542 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3853f0d-7f5a-41a8-b2b9-dc1b730753f2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz\" (UID: \"e3853f0d-7f5a-41a8-b2b9-dc1b730753f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz" Nov 03 19:41:30.916633 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:30.916612 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-969qw\" (UniqueName: \"kubernetes.io/projected/e3853f0d-7f5a-41a8-b2b9-dc1b730753f2-kube-api-access-969qw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz\" (UID: \"e3853f0d-7f5a-41a8-b2b9-dc1b730753f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz" Nov 03 19:41:30.917887 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:30.917872 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-z4kf9\"" Nov 03 19:41:30.921415 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:30.921390 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz"] Nov 03 19:41:30.998026 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:30.998002 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 19:41:30.998344 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:30.998044 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 19:41:31.016949 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:31.016926 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3853f0d-7f5a-41a8-b2b9-dc1b730753f2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz\" (UID: \"e3853f0d-7f5a-41a8-b2b9-dc1b730753f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz" Nov 03 19:41:31.017075 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:31.016988 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3853f0d-7f5a-41a8-b2b9-dc1b730753f2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz\" (UID: \"e3853f0d-7f5a-41a8-b2b9-dc1b730753f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz" Nov 03 19:41:31.017075 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:31.017050 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-969qw\" (UniqueName: \"kubernetes.io/projected/e3853f0d-7f5a-41a8-b2b9-dc1b730753f2-kube-api-access-969qw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz\" (UID: \"e3853f0d-7f5a-41a8-b2b9-dc1b730753f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz" Nov 03 19:41:31.017297 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:31.017276 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3853f0d-7f5a-41a8-b2b9-dc1b730753f2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz\" (UID: \"e3853f0d-7f5a-41a8-b2b9-dc1b730753f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz" Nov 03 19:41:31.017363 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:31.017300 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3853f0d-7f5a-41a8-b2b9-dc1b730753f2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz\" (UID: \"e3853f0d-7f5a-41a8-b2b9-dc1b730753f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz" Nov 03 19:41:31.030129 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:31.030109 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-969qw\" (UniqueName: \"kubernetes.io/projected/e3853f0d-7f5a-41a8-b2b9-dc1b730753f2-kube-api-access-969qw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz\" (UID: \"e3853f0d-7f5a-41a8-b2b9-dc1b730753f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz" Nov 03 19:41:31.229436 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:31.229383 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz" Nov 03 19:41:31.501909 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:31.501883 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz"] Nov 03 19:41:31.503902 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:41:31.503875 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3853f0d_7f5a_41a8_b2b9_dc1b730753f2.slice/crio-4523b069274df7698d1aee35de461d856182a9d5e73ca8d99a11d11b361e85da WatchSource:0}: Error finding container 4523b069274df7698d1aee35de461d856182a9d5e73ca8d99a11d11b361e85da: Status 404 returned error can't find the container with id 4523b069274df7698d1aee35de461d856182a9d5e73ca8d99a11d11b361e85da Nov 03 19:41:32.161186 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:32.161139 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz" event={"ID":"e3853f0d-7f5a-41a8-b2b9-dc1b730753f2","Type":"ContainerStarted","Data":"4523b069274df7698d1aee35de461d856182a9d5e73ca8d99a11d11b361e85da"} Nov 03 19:41:37.185558 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:37.185523 5059 generic.go:356] "Generic (PLEG): container finished" podID="e3853f0d-7f5a-41a8-b2b9-dc1b730753f2" containerID="7bcd4a82a69aea94af0759459ddbccaebf9527ac1e3814ef792bac83392b4aa6" exitCode=0 Nov 03 19:41:37.186021 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:37.185562 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz" event={"ID":"e3853f0d-7f5a-41a8-b2b9-dc1b730753f2","Type":"ContainerDied","Data":"7bcd4a82a69aea94af0759459ddbccaebf9527ac1e3814ef792bac83392b4aa6"} Nov 03 19:41:40.201301 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:40.201269 5059 generic.go:356] "Generic (PLEG): container finished" podID="e3853f0d-7f5a-41a8-b2b9-dc1b730753f2" containerID="6abcd4b68690b0791ce17fbaaa16ead4fd929c811755fa8d2211289ee770f9bd" exitCode=0 Nov 03 19:41:40.201712 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:40.201309 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz" event={"ID":"e3853f0d-7f5a-41a8-b2b9-dc1b730753f2","Type":"ContainerDied","Data":"6abcd4b68690b0791ce17fbaaa16ead4fd929c811755fa8d2211289ee770f9bd"} Nov 03 19:41:40.603505 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:41:40.603465 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:41:45.884041 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:45.884015 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 19:41:45.885098 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:45.885081 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 19:41:46.230557 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:46.230474 5059 generic.go:356] "Generic (PLEG): container finished" podID="e3853f0d-7f5a-41a8-b2b9-dc1b730753f2" containerID="386f7e5573035b544a49ee0875358940b7fed0c314140f35957f5ac8a63fe021" exitCode=0 Nov 03 19:41:46.230715 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:46.230560 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz" event={"ID":"e3853f0d-7f5a-41a8-b2b9-dc1b730753f2","Type":"ContainerDied","Data":"386f7e5573035b544a49ee0875358940b7fed0c314140f35957f5ac8a63fe021"} Nov 03 19:41:47.361245 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:47.361224 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz" Nov 03 19:41:47.462472 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:47.462447 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3853f0d-7f5a-41a8-b2b9-dc1b730753f2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3853f0d-7f5a-41a8-b2b9-dc1b730753f2-bundle\") pod \"e3853f0d-7f5a-41a8-b2b9-dc1b730753f2\" (UID: \"e3853f0d-7f5a-41a8-b2b9-dc1b730753f2\") " Nov 03 19:41:47.462616 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:47.462487 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3853f0d-7f5a-41a8-b2b9-dc1b730753f2-util\" (UniqueName: \"kubernetes.io/empty-dir/e3853f0d-7f5a-41a8-b2b9-dc1b730753f2-util\") pod \"e3853f0d-7f5a-41a8-b2b9-dc1b730753f2\" (UID: \"e3853f0d-7f5a-41a8-b2b9-dc1b730753f2\") " Nov 03 19:41:47.462616 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:47.462601 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3853f0d-7f5a-41a8-b2b9-dc1b730753f2-kube-api-access-969qw\" (UniqueName: \"kubernetes.io/projected/e3853f0d-7f5a-41a8-b2b9-dc1b730753f2-kube-api-access-969qw\") pod \"e3853f0d-7f5a-41a8-b2b9-dc1b730753f2\" (UID: \"e3853f0d-7f5a-41a8-b2b9-dc1b730753f2\") " Nov 03 19:41:47.462886 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:47.462857 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3853f0d-7f5a-41a8-b2b9-dc1b730753f2-bundle" pod "e3853f0d-7f5a-41a8-b2b9-dc1b730753f2" (UID: "e3853f0d-7f5a-41a8-b2b9-dc1b730753f2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:41:47.464599 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:47.464581 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3853f0d-7f5a-41a8-b2b9-dc1b730753f2-kube-api-access-969qw" pod "e3853f0d-7f5a-41a8-b2b9-dc1b730753f2" (UID: "e3853f0d-7f5a-41a8-b2b9-dc1b730753f2"). InnerVolumeSpecName "kube-api-access-969qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:41:47.467769 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:47.467741 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3853f0d-7f5a-41a8-b2b9-dc1b730753f2-util" pod "e3853f0d-7f5a-41a8-b2b9-dc1b730753f2" (UID: "e3853f0d-7f5a-41a8-b2b9-dc1b730753f2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:41:47.563332 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:47.563265 5059 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3853f0d-7f5a-41a8-b2b9-dc1b730753f2-bundle\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:47.563332 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:47.563292 5059 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3853f0d-7f5a-41a8-b2b9-dc1b730753f2-util\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:47.563332 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:47.563307 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-969qw\" (UniqueName: \"kubernetes.io/projected/e3853f0d-7f5a-41a8-b2b9-dc1b730753f2-kube-api-access-969qw\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:48.240836 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:48.240765 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz" Nov 03 19:41:48.240969 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:48.240757 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz" event={"ID":"e3853f0d-7f5a-41a8-b2b9-dc1b730753f2","Type":"ContainerDied","Data":"4523b069274df7698d1aee35de461d856182a9d5e73ca8d99a11d11b361e85da"} Nov 03 19:41:48.240969 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:41:48.240880 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4523b069274df7698d1aee35de461d856182a9d5e73ca8d99a11d11b361e85da" Nov 03 19:42:00.646161 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.646129 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5"] Nov 03 19:42:00.646767 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.646601 5059 cpu_manager.go:414] "containerMap: removing container" podUID="e3853f0d-7f5a-41a8-b2b9-dc1b730753f2" containerName="extract" Nov 03 19:42:00.646767 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.646621 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3853f0d-7f5a-41a8-b2b9-dc1b730753f2" containerName="extract" Nov 03 19:42:00.646767 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.646644 5059 cpu_manager.go:414] "containerMap: removing container" podUID="e3853f0d-7f5a-41a8-b2b9-dc1b730753f2" containerName="pull" Nov 03 19:42:00.646767 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.646652 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3853f0d-7f5a-41a8-b2b9-dc1b730753f2" containerName="pull" Nov 03 19:42:00.646767 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.646709 5059 cpu_manager.go:414] "containerMap: removing container" podUID="e3853f0d-7f5a-41a8-b2b9-dc1b730753f2" containerName="util" Nov 03 19:42:00.646767 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.646718 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3853f0d-7f5a-41a8-b2b9-dc1b730753f2" containerName="util" Nov 03 19:42:00.647072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.646826 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3853f0d-7f5a-41a8-b2b9-dc1b730753f2" containerName="extract" Nov 03 19:42:00.649018 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.648995 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 19:42:00.656857 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.656837 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 19:42:00.656940 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.656893 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 19:42:00.658364 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.658340 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 19:42:00.658466 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.658450 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 19:42:00.658534 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.658514 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 19:42:00.658639 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.658618 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-847xh\"" Nov 03 19:42:00.691616 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.691592 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5"] Nov 03 19:42:00.767203 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.767172 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6785e24f-6fb3-43ab-a469-84d30d94b2f0-manifests\") pod \"opendatahub-operator-controller-manager-8464c976fd-9b8p5\" (UID: \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 19:42:00.767203 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.767203 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6785e24f-6fb3-43ab-a469-84d30d94b2f0-webhook-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-9b8p5\" (UID: \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 19:42:00.767381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.767296 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6785e24f-6fb3-43ab-a469-84d30d94b2f0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-9b8p5\" (UID: \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 19:42:00.767381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.767341 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n4kg\" (UniqueName: \"kubernetes.io/projected/6785e24f-6fb3-43ab-a469-84d30d94b2f0-kube-api-access-4n4kg\") pod \"opendatahub-operator-controller-manager-8464c976fd-9b8p5\" (UID: \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 19:42:00.767381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.767363 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6785e24f-6fb3-43ab-a469-84d30d94b2f0-metrics-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-9b8p5\" (UID: \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 19:42:00.767493 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.767473 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6785e24f-6fb3-43ab-a469-84d30d94b2f0-tmp\") pod \"opendatahub-operator-controller-manager-8464c976fd-9b8p5\" (UID: \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 19:42:00.868544 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.868517 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6785e24f-6fb3-43ab-a469-84d30d94b2f0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-9b8p5\" (UID: \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 19:42:00.868700 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.868553 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n4kg\" (UniqueName: \"kubernetes.io/projected/6785e24f-6fb3-43ab-a469-84d30d94b2f0-kube-api-access-4n4kg\") pod \"opendatahub-operator-controller-manager-8464c976fd-9b8p5\" (UID: \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 19:42:00.868700 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.868570 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6785e24f-6fb3-43ab-a469-84d30d94b2f0-metrics-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-9b8p5\" (UID: \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 19:42:00.868700 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.868596 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6785e24f-6fb3-43ab-a469-84d30d94b2f0-tmp\") pod \"opendatahub-operator-controller-manager-8464c976fd-9b8p5\" (UID: \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 19:42:00.868700 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.868630 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6785e24f-6fb3-43ab-a469-84d30d94b2f0-manifests\") pod \"opendatahub-operator-controller-manager-8464c976fd-9b8p5\" (UID: \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 19:42:00.868700 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.868645 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6785e24f-6fb3-43ab-a469-84d30d94b2f0-webhook-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-9b8p5\" (UID: \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 19:42:00.869109 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.869083 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6785e24f-6fb3-43ab-a469-84d30d94b2f0-manifests\") pod \"opendatahub-operator-controller-manager-8464c976fd-9b8p5\" (UID: \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 19:42:00.871197 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.871169 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6785e24f-6fb3-43ab-a469-84d30d94b2f0-tmp\") pod \"opendatahub-operator-controller-manager-8464c976fd-9b8p5\" (UID: \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 19:42:00.871296 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.871174 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6785e24f-6fb3-43ab-a469-84d30d94b2f0-webhook-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-9b8p5\" (UID: \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 19:42:00.871296 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.871283 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6785e24f-6fb3-43ab-a469-84d30d94b2f0-metrics-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-9b8p5\" (UID: \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 19:42:00.871370 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.871301 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6785e24f-6fb3-43ab-a469-84d30d94b2f0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-9b8p5\" (UID: \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 19:42:00.884026 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.884007 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n4kg\" (UniqueName: \"kubernetes.io/projected/6785e24f-6fb3-43ab-a469-84d30d94b2f0-kube-api-access-4n4kg\") pod \"opendatahub-operator-controller-manager-8464c976fd-9b8p5\" (UID: \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 19:42:00.965441 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:00.965363 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 19:42:01.271165 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:01.271137 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5"] Nov 03 19:42:01.272728 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:42:01.272700 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6785e24f_6fb3_43ab_a469_84d30d94b2f0.slice/crio-0d86ec4f25a9afa09a34a67ce566ad22504c2a23ba3bf73a57cc65382e5607c3 WatchSource:0}: Error finding container 0d86ec4f25a9afa09a34a67ce566ad22504c2a23ba3bf73a57cc65382e5607c3: Status 404 returned error can't find the container with id 0d86ec4f25a9afa09a34a67ce566ad22504c2a23ba3bf73a57cc65382e5607c3 Nov 03 19:42:01.274348 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:01.274330 5059 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:42:01.294348 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:01.294321 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" event={"ID":"6785e24f-6fb3-43ab-a469-84d30d94b2f0","Type":"ContainerStarted","Data":"0d86ec4f25a9afa09a34a67ce566ad22504c2a23ba3bf73a57cc65382e5607c3"} Nov 03 19:42:05.315272 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:05.315194 5059 generic.go:356] "Generic (PLEG): container finished" podID="6785e24f-6fb3-43ab-a469-84d30d94b2f0" containerID="3eedb7b821c263449d6f28d722073293785400637faa42e6848b5007c54e889e" exitCode=0 Nov 03 19:42:05.315272 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:05.315256 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" event={"ID":"6785e24f-6fb3-43ab-a469-84d30d94b2f0","Type":"ContainerDied","Data":"3eedb7b821c263449d6f28d722073293785400637faa42e6848b5007c54e889e"} Nov 03 19:42:06.322053 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:06.322014 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" event={"ID":"6785e24f-6fb3-43ab-a469-84d30d94b2f0","Type":"ContainerStarted","Data":"24306abe36f01f202cf4225862e6256f61b8116c2b16568f30c02b610183da1c"} Nov 03 19:42:06.322552 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:06.322130 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 19:42:06.370480 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:06.370421 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" podStartSLOduration=2.239772006 podStartE2EDuration="6.370404034s" totalImagesPullingTime="3.638390079s" totalInitContainerRuntime="492.241949ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:06.36527041 +0000 UTC m=+320.623067669" watchObservedRunningTime="2026-11-03 19:42:06.370404034 +0000 UTC m=+320.628201293" Nov 03 19:42:06.553467 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:06.553430 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw"] Nov 03 19:42:06.557421 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:06.557401 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" Nov 03 19:42:06.561448 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:06.561425 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 19:42:06.561579 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:06.561490 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 19:42:06.561855 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:06.561833 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-lw7w7\"" Nov 03 19:42:06.567322 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:06.567296 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw"] Nov 03 19:42:06.720163 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:06.720072 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwhh5\" (UniqueName: \"kubernetes.io/projected/270c522c-ef72-46a7-9e65-3be2ebdbcfa6-kube-api-access-rwhh5\") pod \"limitador-operator-controller-manager-5778bb4449-slpzw\" (UID: \"270c522c-ef72-46a7-9e65-3be2ebdbcfa6\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" Nov 03 19:42:06.821087 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:06.821054 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwhh5\" (UniqueName: \"kubernetes.io/projected/270c522c-ef72-46a7-9e65-3be2ebdbcfa6-kube-api-access-rwhh5\") pod \"limitador-operator-controller-manager-5778bb4449-slpzw\" (UID: \"270c522c-ef72-46a7-9e65-3be2ebdbcfa6\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" Nov 03 19:42:06.838028 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:06.837997 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwhh5\" (UniqueName: \"kubernetes.io/projected/270c522c-ef72-46a7-9e65-3be2ebdbcfa6-kube-api-access-rwhh5\") pod \"limitador-operator-controller-manager-5778bb4449-slpzw\" (UID: \"270c522c-ef72-46a7-9e65-3be2ebdbcfa6\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" Nov 03 19:42:06.871933 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:06.871905 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" Nov 03 19:42:07.157480 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:07.157451 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw"] Nov 03 19:42:07.159461 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:42:07.159432 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod270c522c_ef72_46a7_9e65_3be2ebdbcfa6.slice/crio-20bf4fd0a808dbaefe1d4e12aae0124a790da36e28b26676cb2056a8cd542a42 WatchSource:0}: Error finding container 20bf4fd0a808dbaefe1d4e12aae0124a790da36e28b26676cb2056a8cd542a42: Status 404 returned error can't find the container with id 20bf4fd0a808dbaefe1d4e12aae0124a790da36e28b26676cb2056a8cd542a42 Nov 03 19:42:07.327721 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:07.327682 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" event={"ID":"270c522c-ef72-46a7-9e65-3be2ebdbcfa6","Type":"ContainerStarted","Data":"20bf4fd0a808dbaefe1d4e12aae0124a790da36e28b26676cb2056a8cd542a42"} Nov 03 19:42:10.346411 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:10.346379 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" event={"ID":"270c522c-ef72-46a7-9e65-3be2ebdbcfa6","Type":"ContainerStarted","Data":"a4f29644d81f5639b20431cb580b7160e30c9a7f0f42d00f2d940270329ea0f4"} Nov 03 19:42:10.346829 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:10.346518 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" Nov 03 19:42:10.375843 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:10.375795 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" podStartSLOduration=1.566558841 podStartE2EDuration="4.375783786s" totalImagesPullingTime="2.809224945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:10.373375505 +0000 UTC m=+324.631172762" watchObservedRunningTime="2026-11-03 19:42:10.375783786 +0000 UTC m=+324.633581103" Nov 03 19:42:10.602704 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:42:10.602607 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:42:17.330505 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:17.330476 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 19:42:18.511423 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:18.511391 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mm85p"] Nov 03 19:42:18.547492 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:18.547467 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mm85p"] Nov 03 19:42:18.547656 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:18.547554 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mm85p" Nov 03 19:42:18.580582 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:18.580557 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:42:18.580870 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:18.580793 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 19:42:18.583957 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:18.583938 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-w5t25\"" Nov 03 19:42:18.722157 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:18.722128 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmzpr\" (UniqueName: \"kubernetes.io/projected/532de14d-f2dd-4c66-ba6f-8b103f7c9375-kube-api-access-zmzpr\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mm85p\" (UID: \"532de14d-f2dd-4c66-ba6f-8b103f7c9375\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mm85p" Nov 03 19:42:18.722309 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:18.722168 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/532de14d-f2dd-4c66-ba6f-8b103f7c9375-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mm85p\" (UID: \"532de14d-f2dd-4c66-ba6f-8b103f7c9375\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mm85p" Nov 03 19:42:18.823402 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:18.823376 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmzpr\" (UniqueName: \"kubernetes.io/projected/532de14d-f2dd-4c66-ba6f-8b103f7c9375-kube-api-access-zmzpr\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mm85p\" (UID: \"532de14d-f2dd-4c66-ba6f-8b103f7c9375\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mm85p" Nov 03 19:42:18.823535 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:18.823413 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/532de14d-f2dd-4c66-ba6f-8b103f7c9375-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mm85p\" (UID: \"532de14d-f2dd-4c66-ba6f-8b103f7c9375\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mm85p" Nov 03 19:42:18.823766 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:18.823737 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/532de14d-f2dd-4c66-ba6f-8b103f7c9375-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mm85p\" (UID: \"532de14d-f2dd-4c66-ba6f-8b103f7c9375\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mm85p" Nov 03 19:42:18.840616 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:18.840580 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmzpr\" (UniqueName: \"kubernetes.io/projected/532de14d-f2dd-4c66-ba6f-8b103f7c9375-kube-api-access-zmzpr\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mm85p\" (UID: \"532de14d-f2dd-4c66-ba6f-8b103f7c9375\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mm85p" Nov 03 19:42:18.878476 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:18.878455 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mm85p" Nov 03 19:42:19.159888 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:19.159866 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mm85p"] Nov 03 19:42:19.161613 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:42:19.161588 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532de14d_f2dd_4c66_ba6f_8b103f7c9375.slice/crio-059483529a6e51a7600c519628a33dc2f35af25b5257d5868669225c1c3bef09 WatchSource:0}: Error finding container 059483529a6e51a7600c519628a33dc2f35af25b5257d5868669225c1c3bef09: Status 404 returned error can't find the container with id 059483529a6e51a7600c519628a33dc2f35af25b5257d5868669225c1c3bef09 Nov 03 19:42:19.384434 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:19.384395 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mm85p" event={"ID":"532de14d-f2dd-4c66-ba6f-8b103f7c9375","Type":"ContainerStarted","Data":"059483529a6e51a7600c519628a33dc2f35af25b5257d5868669225c1c3bef09"} Nov 03 19:42:19.831385 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:19.831352 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-nj52h"] Nov 03 19:42:19.834150 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:19.834132 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nj52h" Nov 03 19:42:19.838220 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:42:19.838178 5059 status_manager.go:1045] "Failed to get status for pod" err="pods \"authorino-operator-7b7bb4bc77-nj52h\" is forbidden: User \"system:node:ip-10-0-132-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-83.us-east-2.compute.internal' and this object" podUID="941b16dd-0550-4434-86f9-26e38efb68ee" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nj52h" Nov 03 19:42:19.841183 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:19.841073 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-dq8cl\"" Nov 03 19:42:19.854762 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:19.854743 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-nj52h"] Nov 03 19:42:19.932329 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:19.932297 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7dng\" (UniqueName: \"kubernetes.io/projected/941b16dd-0550-4434-86f9-26e38efb68ee-kube-api-access-p7dng\") pod \"authorino-operator-7b7bb4bc77-nj52h\" (UID: \"941b16dd-0550-4434-86f9-26e38efb68ee\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-nj52h" Nov 03 19:42:20.033494 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:20.033465 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7dng\" (UniqueName: \"kubernetes.io/projected/941b16dd-0550-4434-86f9-26e38efb68ee-kube-api-access-p7dng\") pod \"authorino-operator-7b7bb4bc77-nj52h\" (UID: \"941b16dd-0550-4434-86f9-26e38efb68ee\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-nj52h" Nov 03 19:42:20.046247 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:20.046219 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7dng\" (UniqueName: \"kubernetes.io/projected/941b16dd-0550-4434-86f9-26e38efb68ee-kube-api-access-p7dng\") pod \"authorino-operator-7b7bb4bc77-nj52h\" (UID: \"941b16dd-0550-4434-86f9-26e38efb68ee\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-nj52h" Nov 03 19:42:20.146066 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:20.146008 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nj52h" Nov 03 19:42:20.480403 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:20.480373 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-nj52h"] Nov 03 19:42:20.483522 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:42:20.483490 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941b16dd_0550_4434_86f9_26e38efb68ee.slice/crio-856432fd57e3ebd630fbb3bc3ed8e855f477cf255bc3d852e79fb8e43e9b5835 WatchSource:0}: Error finding container 856432fd57e3ebd630fbb3bc3ed8e855f477cf255bc3d852e79fb8e43e9b5835: Status 404 returned error can't find the container with id 856432fd57e3ebd630fbb3bc3ed8e855f477cf255bc3d852e79fb8e43e9b5835 Nov 03 19:42:21.353886 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:21.353217 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" Nov 03 19:42:21.399783 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:21.399751 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nj52h" event={"ID":"941b16dd-0550-4434-86f9-26e38efb68ee","Type":"ContainerStarted","Data":"856432fd57e3ebd630fbb3bc3ed8e855f477cf255bc3d852e79fb8e43e9b5835"} Nov 03 19:42:22.414854 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:22.414798 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nj52h" event={"ID":"941b16dd-0550-4434-86f9-26e38efb68ee","Type":"ContainerStarted","Data":"ac8fdfdbcf156bbdd591e8f7042d8b960bce286b35806ac720a68b8da48a5940"} Nov 03 19:42:22.415287 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:22.414912 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nj52h" Nov 03 19:42:22.416594 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:22.416566 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mm85p" event={"ID":"532de14d-f2dd-4c66-ba6f-8b103f7c9375","Type":"ContainerStarted","Data":"111ab0145bc09ea2159df08b45a75e0a16c06fb447f17cd20b906e1b1a7c091f"} Nov 03 19:42:22.446521 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:22.446475 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nj52h" podStartSLOduration=1.687010775 podStartE2EDuration="3.446464699s" totalImagesPullingTime="1.759453924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:22.444347658 +0000 UTC m=+336.702144914" watchObservedRunningTime="2026-11-03 19:42:22.446464699 +0000 UTC m=+336.704261956" Nov 03 19:42:22.483633 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:22.483576 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mm85p" podStartSLOduration=1.342107666 podStartE2EDuration="4.483560561s" totalImagesPullingTime="3.141452895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:22.482639094 +0000 UTC m=+336.740436351" watchObservedRunningTime="2026-11-03 19:42:22.483560561 +0000 UTC m=+336.741357821" Nov 03 19:42:23.781475 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:23.781443 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84"] Nov 03 19:42:23.784999 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:23.784981 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84" Nov 03 19:42:23.798491 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:23.798469 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-vx824\"" Nov 03 19:42:23.802248 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:23.802223 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 19:42:23.804012 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:23.803993 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84"] Nov 03 19:42:23.861715 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:23.861680 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2454\" (UniqueName: \"kubernetes.io/projected/4a35329c-82fb-479f-abca-bff9ed082558-kube-api-access-p2454\") pod \"dns-operator-controller-manager-8557cbdb84-c2m84\" (UID: \"4a35329c-82fb-479f-abca-bff9ed082558\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84" Nov 03 19:42:23.962405 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:23.962372 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2454\" (UniqueName: \"kubernetes.io/projected/4a35329c-82fb-479f-abca-bff9ed082558-kube-api-access-p2454\") pod \"dns-operator-controller-manager-8557cbdb84-c2m84\" (UID: \"4a35329c-82fb-479f-abca-bff9ed082558\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84" Nov 03 19:42:23.986087 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:23.986058 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2454\" (UniqueName: \"kubernetes.io/projected/4a35329c-82fb-479f-abca-bff9ed082558-kube-api-access-p2454\") pod \"dns-operator-controller-manager-8557cbdb84-c2m84\" (UID: \"4a35329c-82fb-479f-abca-bff9ed082558\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84" Nov 03 19:42:24.097147 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:24.097120 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84" Nov 03 19:42:24.424976 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:24.424951 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84"] Nov 03 19:42:24.426902 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:42:24.426868 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a35329c_82fb_479f_abca_bff9ed082558.slice/crio-43465fcec0008095aec34e37b1b670001c5a5d5b990377e6dd6dce8357fd8e9d WatchSource:0}: Error finding container 43465fcec0008095aec34e37b1b670001c5a5d5b990377e6dd6dce8357fd8e9d: Status 404 returned error can't find the container with id 43465fcec0008095aec34e37b1b670001c5a5d5b990377e6dd6dce8357fd8e9d Nov 03 19:42:25.433184 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:25.433147 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84" event={"ID":"4a35329c-82fb-479f-abca-bff9ed082558","Type":"ContainerStarted","Data":"43465fcec0008095aec34e37b1b670001c5a5d5b990377e6dd6dce8357fd8e9d"} Nov 03 19:42:26.439059 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:26.438984 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84" event={"ID":"4a35329c-82fb-479f-abca-bff9ed082558","Type":"ContainerStarted","Data":"667c8d1cf790b30fc65fdbeb2860e35758fd0e7eb1773d3f331c5e009252f137"} Nov 03 19:42:26.439059 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:26.439044 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84" Nov 03 19:42:26.473325 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:26.473276 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84" podStartSLOduration=1.724423571 podStartE2EDuration="3.473264097s" totalImagesPullingTime="1.748840526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:26.472508145 +0000 UTC m=+340.730305401" watchObservedRunningTime="2026-11-03 19:42:26.473264097 +0000 UTC m=+340.731061353" Nov 03 19:42:33.422771 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:33.422733 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nj52h" Nov 03 19:42:37.445583 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:37.445544 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84" Nov 03 19:42:38.133983 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.133949 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm"] Nov 03 19:42:38.140626 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.140602 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.151239 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.151215 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-fsb74\"" Nov 03 19:42:38.155474 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.155439 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm"] Nov 03 19:42:38.178566 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.178537 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.178741 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.178599 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.178741 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.178687 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.178741 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.178716 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.178906 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.178754 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.178906 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.178843 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.178906 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.178875 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.179047 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.178901 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xfpq\" (UniqueName: \"kubernetes.io/projected/37e90ab0-e9c0-420c-a0f4-c7923aa00297-kube-api-access-7xfpq\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.179047 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.178951 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.280030 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.279994 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.280212 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.280043 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.280212 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.280062 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.280212 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.280185 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.280390 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.280241 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.280390 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.280281 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.280390 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.280314 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.280390 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.280339 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xfpq\" (UniqueName: \"kubernetes.io/projected/37e90ab0-e9c0-420c-a0f4-c7923aa00297-kube-api-access-7xfpq\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.280390 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.280370 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.280652 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.280438 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.281039 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.280778 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.281039 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.280803 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.281039 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.280921 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.281039 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.281032 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.283201 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.283175 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.283622 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.283602 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.331888 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.331855 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.340885 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.340859 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xfpq\" (UniqueName: \"kubernetes.io/projected/37e90ab0-e9c0-420c-a0f4-c7923aa00297-kube-api-access-7xfpq\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ztrnm\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.453215 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.453129 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:38.526583 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.524209 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-6b6mz"] Nov 03 19:42:38.533527 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.532354 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-6b6mz" Nov 03 19:42:38.548409 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.548301 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-6b6mz"] Nov 03 19:42:38.551103 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.551065 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 19:42:38.551590 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.551521 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-579hd\"" Nov 03 19:42:38.583364 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.583110 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655-config-file\") pod \"limitador-limitador-74dfcfb894-6b6mz\" (UID: \"9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-6b6mz" Nov 03 19:42:38.583364 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.583186 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2crt4\" (UniqueName: \"kubernetes.io/projected/9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655-kube-api-access-2crt4\") pod \"limitador-limitador-74dfcfb894-6b6mz\" (UID: \"9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-6b6mz" Nov 03 19:42:38.608942 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.608912 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-6b6mz"] Nov 03 19:42:38.684480 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.684448 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655-config-file\") pod \"limitador-limitador-74dfcfb894-6b6mz\" (UID: \"9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-6b6mz" Nov 03 19:42:38.684626 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.684490 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2crt4\" (UniqueName: \"kubernetes.io/projected/9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655-kube-api-access-2crt4\") pod \"limitador-limitador-74dfcfb894-6b6mz\" (UID: \"9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-6b6mz" Nov 03 19:42:38.685337 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.685313 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655-config-file\") pod \"limitador-limitador-74dfcfb894-6b6mz\" (UID: \"9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-6b6mz" Nov 03 19:42:38.697219 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.697183 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2crt4\" (UniqueName: \"kubernetes.io/projected/9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655-kube-api-access-2crt4\") pod \"limitador-limitador-74dfcfb894-6b6mz\" (UID: \"9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-6b6mz" Nov 03 19:42:38.740594 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.740562 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm"] Nov 03 19:42:38.742470 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:42:38.742440 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e90ab0_e9c0_420c_a0f4_c7923aa00297.slice/crio-1be2351effe8da8c655ee906e4857030a8714afdf23aa891ad55cfcc73d26f5d WatchSource:0}: Error finding container 1be2351effe8da8c655ee906e4857030a8714afdf23aa891ad55cfcc73d26f5d: Status 404 returned error can't find the container with id 1be2351effe8da8c655ee906e4857030a8714afdf23aa891ad55cfcc73d26f5d Nov 03 19:42:38.744964 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.744929 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:42:38.745067 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.745021 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:42:38.745136 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.745089 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:42:38.854987 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:38.854948 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-6b6mz" Nov 03 19:42:39.180926 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:39.180894 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-6b6mz"] Nov 03 19:42:39.181946 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:42:39.181917 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd6a9e6_bba4_4c13_8b4f_a4b4c0997655.slice/crio-bbabda3e36bbb13da47c87a31a937f5bfa13dc93b22bd8051a8c898fcc6159b8 WatchSource:0}: Error finding container bbabda3e36bbb13da47c87a31a937f5bfa13dc93b22bd8051a8c898fcc6159b8: Status 404 returned error can't find the container with id bbabda3e36bbb13da47c87a31a937f5bfa13dc93b22bd8051a8c898fcc6159b8 Nov 03 19:42:39.502028 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:39.501936 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-6b6mz" event={"ID":"9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655","Type":"ContainerStarted","Data":"bbabda3e36bbb13da47c87a31a937f5bfa13dc93b22bd8051a8c898fcc6159b8"} Nov 03 19:42:39.503580 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:39.503548 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" event={"ID":"37e90ab0-e9c0-420c-a0f4-c7923aa00297","Type":"ContainerStarted","Data":"f8323dfa4e2c1a85ed81f607b727475c16361a359dbb2bb285fe28a219d7a6eb"} Nov 03 19:42:39.503738 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:39.503607 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" event={"ID":"37e90ab0-e9c0-420c-a0f4-c7923aa00297","Type":"ContainerStarted","Data":"1be2351effe8da8c655ee906e4857030a8714afdf23aa891ad55cfcc73d26f5d"} Nov 03 19:42:39.537494 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:39.537427 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" podStartSLOduration=1.5374104709999998 podStartE2EDuration="1.537410471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:39.535349241 +0000 UTC m=+353.793146556" watchObservedRunningTime="2026-11-03 19:42:39.537410471 +0000 UTC m=+353.795207729" Nov 03 19:42:39.986321 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:39.986282 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-cshnh"] Nov 03 19:42:39.990637 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:39.990612 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-cshnh" Nov 03 19:42:40.006817 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:40.006787 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sgdb8\"" Nov 03 19:42:40.008874 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:40.008851 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-cshnh"] Nov 03 19:42:40.098655 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:40.098630 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9c99\" (UniqueName: \"kubernetes.io/projected/93d6f72b-56eb-4b61-bd96-a43e7ce41032-kube-api-access-p9c99\") pod \"authorino-57f7f8659f-cshnh\" (UID: \"93d6f72b-56eb-4b61-bd96-a43e7ce41032\") " pod="kuadrant-system/authorino-57f7f8659f-cshnh" Nov 03 19:42:40.199542 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:40.199504 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-x5lg7"] Nov 03 19:42:40.199734 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:40.199542 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9c99\" (UniqueName: \"kubernetes.io/projected/93d6f72b-56eb-4b61-bd96-a43e7ce41032-kube-api-access-p9c99\") pod \"authorino-57f7f8659f-cshnh\" (UID: \"93d6f72b-56eb-4b61-bd96-a43e7ce41032\") " pod="kuadrant-system/authorino-57f7f8659f-cshnh" Nov 03 19:42:40.202604 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:40.202579 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-x5lg7" Nov 03 19:42:40.220819 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:40.220793 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-x5lg7"] Nov 03 19:42:40.229777 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:40.229747 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9c99\" (UniqueName: \"kubernetes.io/projected/93d6f72b-56eb-4b61-bd96-a43e7ce41032-kube-api-access-p9c99\") pod \"authorino-57f7f8659f-cshnh\" (UID: \"93d6f72b-56eb-4b61-bd96-a43e7ce41032\") " pod="kuadrant-system/authorino-57f7f8659f-cshnh" Nov 03 19:42:40.300395 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:40.300312 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ld4s\" (UniqueName: \"kubernetes.io/projected/f865f05b-9a1f-4ff0-b784-228049b1f26a-kube-api-access-6ld4s\") pod \"authorino-56c6c9ddd5-x5lg7\" (UID: \"f865f05b-9a1f-4ff0-b784-228049b1f26a\") " pod="kuadrant-system/authorino-56c6c9ddd5-x5lg7" Nov 03 19:42:40.313369 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:40.313338 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-cshnh" Nov 03 19:42:40.401594 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:40.401560 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ld4s\" (UniqueName: \"kubernetes.io/projected/f865f05b-9a1f-4ff0-b784-228049b1f26a-kube-api-access-6ld4s\") pod \"authorino-56c6c9ddd5-x5lg7\" (UID: \"f865f05b-9a1f-4ff0-b784-228049b1f26a\") " pod="kuadrant-system/authorino-56c6c9ddd5-x5lg7" Nov 03 19:42:40.416054 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:40.416026 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ld4s\" (UniqueName: \"kubernetes.io/projected/f865f05b-9a1f-4ff0-b784-228049b1f26a-kube-api-access-6ld4s\") pod \"authorino-56c6c9ddd5-x5lg7\" (UID: \"f865f05b-9a1f-4ff0-b784-228049b1f26a\") " pod="kuadrant-system/authorino-56c6c9ddd5-x5lg7" Nov 03 19:42:40.453616 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:40.453584 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:40.459075 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:40.459048 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:40.508852 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:40.508822 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:40.510487 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:40.510461 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 19:42:40.514900 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:40.514869 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-x5lg7" Nov 03 19:42:40.597750 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:40.597721 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-cshnh"] Nov 03 19:42:40.618293 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:42:40.618227 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:42:40.805574 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:40.805543 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-x5lg7"] Nov 03 19:42:40.808160 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:42:40.808128 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf865f05b_9a1f_4ff0_b784_228049b1f26a.slice/crio-b06ca15e27d4d5f70e597a3873ea8d6a07ef1cc5e744c34d76277694bac90bfc WatchSource:0}: Error finding container b06ca15e27d4d5f70e597a3873ea8d6a07ef1cc5e744c34d76277694bac90bfc: Status 404 returned error can't find the container with id b06ca15e27d4d5f70e597a3873ea8d6a07ef1cc5e744c34d76277694bac90bfc Nov 03 19:42:41.514987 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:41.514947 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-6b6mz" event={"ID":"9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655","Type":"ContainerStarted","Data":"d25dba9a218f01ff0a633f1d06641a43f20b140452abcc833479c5d5b462d9ff"} Nov 03 19:42:41.515473 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:41.515021 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-6b6mz" Nov 03 19:42:41.516362 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:41.516329 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-x5lg7" event={"ID":"f865f05b-9a1f-4ff0-b784-228049b1f26a","Type":"ContainerStarted","Data":"b06ca15e27d4d5f70e597a3873ea8d6a07ef1cc5e744c34d76277694bac90bfc"} Nov 03 19:42:41.517724 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:41.517693 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-cshnh" event={"ID":"93d6f72b-56eb-4b61-bd96-a43e7ce41032","Type":"ContainerStarted","Data":"a1dc241a7c6f7f16a2f09c0eb84d9dfb1f48eefc90f11664b82ca14fd7d99d33"} Nov 03 19:42:41.536814 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:41.536762 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-6b6mz" podStartSLOduration=1.7138593279999998 podStartE2EDuration="3.536746295s" totalImagesPullingTime="1.822886967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:41.534286186 +0000 UTC m=+355.792083443" watchObservedRunningTime="2026-11-03 19:42:41.536746295 +0000 UTC m=+355.794543553" Nov 03 19:42:44.544923 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:44.544832 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-x5lg7" event={"ID":"f865f05b-9a1f-4ff0-b784-228049b1f26a","Type":"ContainerStarted","Data":"ccf3ede4ac80c83f3fef9e234a15ac9a7e53c669cdb18b5ae67940e9e01c3cae"} Nov 03 19:42:44.562390 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:44.562342 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-x5lg7" podStartSLOduration=1.180878644 podStartE2EDuration="4.56232937s" totalImagesPullingTime="3.381450726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:44.561303657 +0000 UTC m=+358.819100915" watchObservedRunningTime="2026-11-03 19:42:44.56232937 +0000 UTC m=+358.820126660" Nov 03 19:42:44.587432 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:44.587402 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-cshnh"] Nov 03 19:42:46.556717 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:46.556680 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-cshnh" event={"ID":"93d6f72b-56eb-4b61-bd96-a43e7ce41032","Type":"ContainerStarted","Data":"3376a0a19027b7fb37e88a76e0699a81db77578cce3a548358ed0f5804bd620e"} Nov 03 19:42:46.557144 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:46.556718 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-cshnh" podUID="93d6f72b-56eb-4b61-bd96-a43e7ce41032" containerName="authorino" containerID="cri-o://3376a0a19027b7fb37e88a76e0699a81db77578cce3a548358ed0f5804bd620e" gracePeriod=30 Nov 03 19:42:46.572983 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:46.572937 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-cshnh" podStartSLOduration=1.743069852 podStartE2EDuration="7.572924671s" totalImagesPullingTime="5.829854819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:46.572551413 +0000 UTC m=+360.830348690" watchObservedRunningTime="2026-11-03 19:42:46.572924671 +0000 UTC m=+360.830721928" Nov 03 19:42:46.812564 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:46.812543 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-cshnh" Nov 03 19:42:46.863116 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:46.863093 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93d6f72b-56eb-4b61-bd96-a43e7ce41032-kube-api-access-p9c99\" (UniqueName: \"kubernetes.io/projected/93d6f72b-56eb-4b61-bd96-a43e7ce41032-kube-api-access-p9c99\") pod \"93d6f72b-56eb-4b61-bd96-a43e7ce41032\" (UID: \"93d6f72b-56eb-4b61-bd96-a43e7ce41032\") " Nov 03 19:42:46.865170 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:46.865143 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d6f72b-56eb-4b61-bd96-a43e7ce41032-kube-api-access-p9c99" pod "93d6f72b-56eb-4b61-bd96-a43e7ce41032" (UID: "93d6f72b-56eb-4b61-bd96-a43e7ce41032"). InnerVolumeSpecName "kube-api-access-p9c99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:42:46.963841 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:46.963817 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9c99\" (UniqueName: \"kubernetes.io/projected/93d6f72b-56eb-4b61-bd96-a43e7ce41032-kube-api-access-p9c99\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:47.563845 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:47.563805 5059 generic.go:356] "Generic (PLEG): container finished" podID="93d6f72b-56eb-4b61-bd96-a43e7ce41032" containerID="3376a0a19027b7fb37e88a76e0699a81db77578cce3a548358ed0f5804bd620e" exitCode=0 Nov 03 19:42:47.564312 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:47.563848 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-cshnh" event={"ID":"93d6f72b-56eb-4b61-bd96-a43e7ce41032","Type":"ContainerDied","Data":"3376a0a19027b7fb37e88a76e0699a81db77578cce3a548358ed0f5804bd620e"} Nov 03 19:42:47.564312 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:47.563879 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-cshnh" Nov 03 19:42:47.564312 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:47.563900 5059 scope.go:122] "RemoveContainer" containerID="3376a0a19027b7fb37e88a76e0699a81db77578cce3a548358ed0f5804bd620e" Nov 03 19:42:47.564312 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:47.563891 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-cshnh" event={"ID":"93d6f72b-56eb-4b61-bd96-a43e7ce41032","Type":"ContainerDied","Data":"a1dc241a7c6f7f16a2f09c0eb84d9dfb1f48eefc90f11664b82ca14fd7d99d33"} Nov 03 19:42:47.574710 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:47.574691 5059 scope.go:122] "RemoveContainer" containerID="3376a0a19027b7fb37e88a76e0699a81db77578cce3a548358ed0f5804bd620e" Nov 03 19:42:47.575001 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:42:47.574969 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3376a0a19027b7fb37e88a76e0699a81db77578cce3a548358ed0f5804bd620e\": container with ID starting with 3376a0a19027b7fb37e88a76e0699a81db77578cce3a548358ed0f5804bd620e not found: ID does not exist" containerID="3376a0a19027b7fb37e88a76e0699a81db77578cce3a548358ed0f5804bd620e" Nov 03 19:42:47.575096 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:47.575012 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3376a0a19027b7fb37e88a76e0699a81db77578cce3a548358ed0f5804bd620e"} err="failed to get container status \"3376a0a19027b7fb37e88a76e0699a81db77578cce3a548358ed0f5804bd620e\": rpc error: code = NotFound desc = could not find container \"3376a0a19027b7fb37e88a76e0699a81db77578cce3a548358ed0f5804bd620e\": container with ID starting with 3376a0a19027b7fb37e88a76e0699a81db77578cce3a548358ed0f5804bd620e not found: ID does not exist" Nov 03 19:42:47.587957 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:47.587929 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-cshnh"] Nov 03 19:42:47.593598 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:47.593576 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-cshnh"] Nov 03 19:42:47.861624 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:47.856603 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93d6f72b-56eb-4b61-bd96-a43e7ce41032" path="/var/lib/kubelet/pods/93d6f72b-56eb-4b61-bd96-a43e7ce41032/volumes" Nov 03 19:42:52.528325 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:52.528296 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-6b6mz" Nov 03 19:42:54.600040 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:54.600007 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-6b6mz"] Nov 03 19:42:54.600545 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:54.600173 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-6b6mz" podUID="9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655" containerName="limitador" containerID="cri-o://d25dba9a218f01ff0a633f1d06641a43f20b140452abcc833479c5d5b462d9ff" gracePeriod=30 Nov 03 19:42:55.155058 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:55.155033 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-6b6mz" Nov 03 19:42:55.230739 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:55.230644 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655-config-file\" (UniqueName: \"kubernetes.io/configmap/9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655-config-file\") pod \"9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655\" (UID: \"9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655\") " Nov 03 19:42:55.230874 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:55.230828 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655-kube-api-access-2crt4\" (UniqueName: \"kubernetes.io/projected/9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655-kube-api-access-2crt4\") pod \"9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655\" (UID: \"9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655\") " Nov 03 19:42:55.230972 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:55.230949 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655-config-file" pod "9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655" (UID: "9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:42:55.231115 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:55.231099 5059 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655-config-file\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:55.232768 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:55.232740 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655-kube-api-access-2crt4" pod "9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655" (UID: "9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655"). InnerVolumeSpecName "kube-api-access-2crt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:42:55.332217 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:55.332189 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2crt4\" (UniqueName: \"kubernetes.io/projected/9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655-kube-api-access-2crt4\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:42:55.602487 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:55.602452 5059 generic.go:356] "Generic (PLEG): container finished" podID="9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655" containerID="d25dba9a218f01ff0a633f1d06641a43f20b140452abcc833479c5d5b462d9ff" exitCode=0 Nov 03 19:42:55.602927 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:55.602498 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-6b6mz" Nov 03 19:42:55.602927 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:55.602534 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-6b6mz" event={"ID":"9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655","Type":"ContainerDied","Data":"d25dba9a218f01ff0a633f1d06641a43f20b140452abcc833479c5d5b462d9ff"} Nov 03 19:42:55.602927 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:55.602590 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-6b6mz" event={"ID":"9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655","Type":"ContainerDied","Data":"bbabda3e36bbb13da47c87a31a937f5bfa13dc93b22bd8051a8c898fcc6159b8"} Nov 03 19:42:55.602927 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:55.602614 5059 scope.go:122] "RemoveContainer" containerID="d25dba9a218f01ff0a633f1d06641a43f20b140452abcc833479c5d5b462d9ff" Nov 03 19:42:55.611334 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:55.611320 5059 scope.go:122] "RemoveContainer" containerID="d25dba9a218f01ff0a633f1d06641a43f20b140452abcc833479c5d5b462d9ff" Nov 03 19:42:55.611586 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:42:55.611568 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d25dba9a218f01ff0a633f1d06641a43f20b140452abcc833479c5d5b462d9ff\": container with ID starting with d25dba9a218f01ff0a633f1d06641a43f20b140452abcc833479c5d5b462d9ff not found: ID does not exist" containerID="d25dba9a218f01ff0a633f1d06641a43f20b140452abcc833479c5d5b462d9ff" Nov 03 19:42:55.611626 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:55.611594 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d25dba9a218f01ff0a633f1d06641a43f20b140452abcc833479c5d5b462d9ff"} err="failed to get container status \"d25dba9a218f01ff0a633f1d06641a43f20b140452abcc833479c5d5b462d9ff\": rpc error: code = NotFound desc = could not find container \"d25dba9a218f01ff0a633f1d06641a43f20b140452abcc833479c5d5b462d9ff\": container with ID starting with d25dba9a218f01ff0a633f1d06641a43f20b140452abcc833479c5d5b462d9ff not found: ID does not exist" Nov 03 19:42:55.634424 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:55.634396 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-6b6mz"] Nov 03 19:42:55.638230 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:55.638206 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-6b6mz"] Nov 03 19:42:55.854856 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:42:55.854794 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655" path="/var/lib/kubelet/pods/9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655/volumes" Nov 03 19:43:01.040689 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.040627 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts"] Nov 03 19:43:01.041495 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.041471 5059 cpu_manager.go:414] "containerMap: removing container" podUID="93d6f72b-56eb-4b61-bd96-a43e7ce41032" containerName="authorino" Nov 03 19:43:01.041588 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.041497 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93d6f72b-56eb-4b61-bd96-a43e7ce41032" containerName="authorino" Nov 03 19:43:01.041588 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.041581 5059 cpu_manager.go:414] "containerMap: removing container" podUID="9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655" containerName="limitador" Nov 03 19:43:01.041719 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.041589 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655" containerName="limitador" Nov 03 19:43:01.041799 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.041784 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="93d6f72b-56eb-4b61-bd96-a43e7ce41032" containerName="authorino" Nov 03 19:43:01.041858 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.041813 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cd6a9e6-bba4-4c13-8b4f-a4b4c0997655" containerName="limitador" Nov 03 19:43:01.047864 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.047821 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts" Nov 03 19:43:01.052701 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.052655 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:43:01.053155 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.053134 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 19:43:01.060454 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.060431 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-w84zh\"" Nov 03 19:43:01.062012 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.061991 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts"] Nov 03 19:43:01.153783 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.153757 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:43:01.180277 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.180249 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9rhb\" (UniqueName: \"kubernetes.io/projected/633c352c-f07e-4032-b962-23a148297e45-kube-api-access-p9rhb\") pod \"postgres-7cdbf6dc44-x8zts\" (UID: \"633c352c-f07e-4032-b962-23a148297e45\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts" Nov 03 19:43:01.180381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.180285 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/633c352c-f07e-4032-b962-23a148297e45-data\") pod \"postgres-7cdbf6dc44-x8zts\" (UID: \"633c352c-f07e-4032-b962-23a148297e45\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts" Nov 03 19:43:01.281657 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.281623 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/633c352c-f07e-4032-b962-23a148297e45-data\") pod \"postgres-7cdbf6dc44-x8zts\" (UID: \"633c352c-f07e-4032-b962-23a148297e45\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts" Nov 03 19:43:01.281843 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.281777 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9rhb\" (UniqueName: \"kubernetes.io/projected/633c352c-f07e-4032-b962-23a148297e45-kube-api-access-p9rhb\") pod \"postgres-7cdbf6dc44-x8zts\" (UID: \"633c352c-f07e-4032-b962-23a148297e45\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts" Nov 03 19:43:01.282018 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.281995 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/633c352c-f07e-4032-b962-23a148297e45-data\") pod \"postgres-7cdbf6dc44-x8zts\" (UID: \"633c352c-f07e-4032-b962-23a148297e45\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts" Nov 03 19:43:01.318575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.318505 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9rhb\" (UniqueName: \"kubernetes.io/projected/633c352c-f07e-4032-b962-23a148297e45-kube-api-access-p9rhb\") pod \"postgres-7cdbf6dc44-x8zts\" (UID: \"633c352c-f07e-4032-b962-23a148297e45\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts" Nov 03 19:43:01.359802 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.359777 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts" Nov 03 19:43:01.543180 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.543148 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7"] Nov 03 19:43:01.546119 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.546098 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" Nov 03 19:43:01.558313 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.558290 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:43:01.558313 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.558289 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 19:43:01.558453 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.558291 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 19:43:01.558453 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.558341 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 19:43:01.558453 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.558346 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 19:43:01.558549 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.558503 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-k5vf6\"" Nov 03 19:43:01.577825 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.577803 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7"] Nov 03 19:43:01.641721 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.641697 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts"] Nov 03 19:43:01.643536 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:43:01.643505 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod633c352c_f07e_4032_b962_23a148297e45.slice/crio-885f346c447707f93bb08eda59e11a44a56b8387e22202e8922cad337479c936 WatchSource:0}: Error finding container 885f346c447707f93bb08eda59e11a44a56b8387e22202e8922cad337479c936: Status 404 returned error can't find the container with id 885f346c447707f93bb08eda59e11a44a56b8387e22202e8922cad337479c936 Nov 03 19:43:01.685510 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.685481 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1072628f-f513-4b4e-8954-afee5a1aa733-manager-config\") pod \"jobset-controller-manager-5d9f95b69-9vwq7\" (UID: \"1072628f-f513-4b4e-8954-afee5a1aa733\") " pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" Nov 03 19:43:01.685636 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.685528 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1072628f-f513-4b4e-8954-afee5a1aa733-metrics-certs\") pod \"jobset-controller-manager-5d9f95b69-9vwq7\" (UID: \"1072628f-f513-4b4e-8954-afee5a1aa733\") " pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" Nov 03 19:43:01.685698 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.685677 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1072628f-f513-4b4e-8954-afee5a1aa733-cert\") pod \"jobset-controller-manager-5d9f95b69-9vwq7\" (UID: \"1072628f-f513-4b4e-8954-afee5a1aa733\") " pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" Nov 03 19:43:01.685740 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.685720 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w522z\" (UniqueName: \"kubernetes.io/projected/1072628f-f513-4b4e-8954-afee5a1aa733-kube-api-access-w522z\") pod \"jobset-controller-manager-5d9f95b69-9vwq7\" (UID: \"1072628f-f513-4b4e-8954-afee5a1aa733\") " pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" Nov 03 19:43:01.786904 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.786866 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1072628f-f513-4b4e-8954-afee5a1aa733-cert\") pod \"jobset-controller-manager-5d9f95b69-9vwq7\" (UID: \"1072628f-f513-4b4e-8954-afee5a1aa733\") " pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" Nov 03 19:43:01.787064 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.786907 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w522z\" (UniqueName: \"kubernetes.io/projected/1072628f-f513-4b4e-8954-afee5a1aa733-kube-api-access-w522z\") pod \"jobset-controller-manager-5d9f95b69-9vwq7\" (UID: \"1072628f-f513-4b4e-8954-afee5a1aa733\") " pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" Nov 03 19:43:01.787064 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.786992 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1072628f-f513-4b4e-8954-afee5a1aa733-manager-config\") pod \"jobset-controller-manager-5d9f95b69-9vwq7\" (UID: \"1072628f-f513-4b4e-8954-afee5a1aa733\") " pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" Nov 03 19:43:01.787064 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.787045 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1072628f-f513-4b4e-8954-afee5a1aa733-metrics-certs\") pod \"jobset-controller-manager-5d9f95b69-9vwq7\" (UID: \"1072628f-f513-4b4e-8954-afee5a1aa733\") " pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" Nov 03 19:43:01.787685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.787642 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1072628f-f513-4b4e-8954-afee5a1aa733-manager-config\") pod \"jobset-controller-manager-5d9f95b69-9vwq7\" (UID: \"1072628f-f513-4b4e-8954-afee5a1aa733\") " pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" Nov 03 19:43:01.789519 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.789489 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1072628f-f513-4b4e-8954-afee5a1aa733-metrics-certs\") pod \"jobset-controller-manager-5d9f95b69-9vwq7\" (UID: \"1072628f-f513-4b4e-8954-afee5a1aa733\") " pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" Nov 03 19:43:01.789608 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.789565 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1072628f-f513-4b4e-8954-afee5a1aa733-cert\") pod \"jobset-controller-manager-5d9f95b69-9vwq7\" (UID: \"1072628f-f513-4b4e-8954-afee5a1aa733\") " pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" Nov 03 19:43:01.803768 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.803745 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w522z\" (UniqueName: \"kubernetes.io/projected/1072628f-f513-4b4e-8954-afee5a1aa733-kube-api-access-w522z\") pod \"jobset-controller-manager-5d9f95b69-9vwq7\" (UID: \"1072628f-f513-4b4e-8954-afee5a1aa733\") " pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" Nov 03 19:43:01.859010 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:01.858954 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" Nov 03 19:43:02.147643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:02.147617 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7"] Nov 03 19:43:02.149326 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:43:02.149303 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1072628f_f513_4b4e_8954_afee5a1aa733.slice/crio-2a05791624cadc6e4f146b5ee449504344b96fd16c4bce70bef096744cab69b4 WatchSource:0}: Error finding container 2a05791624cadc6e4f146b5ee449504344b96fd16c4bce70bef096744cab69b4: Status 404 returned error can't find the container with id 2a05791624cadc6e4f146b5ee449504344b96fd16c4bce70bef096744cab69b4 Nov 03 19:43:02.636945 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:02.636903 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts" event={"ID":"633c352c-f07e-4032-b962-23a148297e45","Type":"ContainerStarted","Data":"885f346c447707f93bb08eda59e11a44a56b8387e22202e8922cad337479c936"} Nov 03 19:43:02.638008 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:02.637985 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" event={"ID":"1072628f-f513-4b4e-8954-afee5a1aa733","Type":"ContainerStarted","Data":"2a05791624cadc6e4f146b5ee449504344b96fd16c4bce70bef096744cab69b4"} Nov 03 19:43:07.667590 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:07.667545 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts" event={"ID":"633c352c-f07e-4032-b962-23a148297e45","Type":"ContainerStarted","Data":"e882b42c99b4f73bf517ede2116273cfd6eda2d61af90817e8c3c5603097f5b8"} Nov 03 19:43:07.668020 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:07.667790 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts" Nov 03 19:43:07.688673 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:07.688616 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts" podStartSLOduration=1.118927074 podStartE2EDuration="6.688603116s" totalImagesPullingTime="5.569676042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:07.686250233 +0000 UTC m=+381.944047491" watchObservedRunningTime="2026-11-03 19:43:07.688603116 +0000 UTC m=+381.946400376" Nov 03 19:43:10.603164 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:43:10.603129 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:43:13.702178 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:13.702153 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts" Nov 03 19:43:33.790571 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:33.790527 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" event={"ID":"1072628f-f513-4b4e-8954-afee5a1aa733","Type":"ContainerStarted","Data":"cadce9a446527c8c3f13706fda86af3620c7a7c77a7b43d35d1a7f7d6579e0e8"} Nov 03 19:43:33.791231 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:33.790719 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" Nov 03 19:43:33.825047 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:33.824996 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" podStartSLOduration=1.423823563 podStartE2EDuration="32.824983915s" totalImagesPullingTime="31.401160352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:33.824078835 +0000 UTC m=+408.081876124" watchObservedRunningTime="2026-11-03 19:43:33.824983915 +0000 UTC m=+408.082781171" Nov 03 19:43:40.603022 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:43:40.602985 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:43:44.799866 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:43:44.799835 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" Nov 03 19:44:10.603101 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:44:10.603007 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:44:10.668710 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:10.668674 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6dd76875b5-sf54c"] Nov 03 19:44:10.676301 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:10.676278 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-sf54c" Nov 03 19:44:10.680511 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:10.680414 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-sf54c"] Nov 03 19:44:10.680642 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:10.680531 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 19:44:10.703518 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:10.703493 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-sf54c"] Nov 03 19:44:10.703705 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:44:10.703686 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-clkjs tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6dd76875b5-sf54c" podUID="3b9b3a85-bf92-4ab6-9f45-2866ad77faa2" Nov 03 19:44:10.740166 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:10.740140 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-nlh5d"] Nov 03 19:44:10.743972 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:10.743954 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-nlh5d" Nov 03 19:44:10.755056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:10.755034 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-nlh5d"] Nov 03 19:44:10.775590 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:10.775566 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clkjs\" (UniqueName: \"kubernetes.io/projected/3b9b3a85-bf92-4ab6-9f45-2866ad77faa2-kube-api-access-clkjs\") pod \"authorino-6dd76875b5-sf54c\" (UID: \"3b9b3a85-bf92-4ab6-9f45-2866ad77faa2\") " pod="kuadrant-system/authorino-6dd76875b5-sf54c" Nov 03 19:44:10.775765 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:10.775746 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3b9b3a85-bf92-4ab6-9f45-2866ad77faa2-tls-cert\") pod \"authorino-6dd76875b5-sf54c\" (UID: \"3b9b3a85-bf92-4ab6-9f45-2866ad77faa2\") " pod="kuadrant-system/authorino-6dd76875b5-sf54c" Nov 03 19:44:10.876478 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:10.876401 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/209d692f-1f73-436d-b5cf-7bcffc02b5e7-tls-cert\") pod \"authorino-6889cbd8c4-nlh5d\" (UID: \"209d692f-1f73-436d-b5cf-7bcffc02b5e7\") " pod="kuadrant-system/authorino-6889cbd8c4-nlh5d" Nov 03 19:44:10.876605 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:10.876526 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tpqq\" (UniqueName: \"kubernetes.io/projected/209d692f-1f73-436d-b5cf-7bcffc02b5e7-kube-api-access-8tpqq\") pod \"authorino-6889cbd8c4-nlh5d\" (UID: \"209d692f-1f73-436d-b5cf-7bcffc02b5e7\") " pod="kuadrant-system/authorino-6889cbd8c4-nlh5d" Nov 03 19:44:10.876605 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:10.876594 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3b9b3a85-bf92-4ab6-9f45-2866ad77faa2-tls-cert\") pod \"authorino-6dd76875b5-sf54c\" (UID: \"3b9b3a85-bf92-4ab6-9f45-2866ad77faa2\") " pod="kuadrant-system/authorino-6dd76875b5-sf54c" Nov 03 19:44:10.876732 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:10.876637 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clkjs\" (UniqueName: \"kubernetes.io/projected/3b9b3a85-bf92-4ab6-9f45-2866ad77faa2-kube-api-access-clkjs\") pod \"authorino-6dd76875b5-sf54c\" (UID: \"3b9b3a85-bf92-4ab6-9f45-2866ad77faa2\") " pod="kuadrant-system/authorino-6dd76875b5-sf54c" Nov 03 19:44:10.879724 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:10.879704 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3b9b3a85-bf92-4ab6-9f45-2866ad77faa2-tls-cert\") pod \"authorino-6dd76875b5-sf54c\" (UID: \"3b9b3a85-bf92-4ab6-9f45-2866ad77faa2\") " pod="kuadrant-system/authorino-6dd76875b5-sf54c" Nov 03 19:44:10.893201 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:10.893180 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clkjs\" (UniqueName: \"kubernetes.io/projected/3b9b3a85-bf92-4ab6-9f45-2866ad77faa2-kube-api-access-clkjs\") pod \"authorino-6dd76875b5-sf54c\" (UID: \"3b9b3a85-bf92-4ab6-9f45-2866ad77faa2\") " pod="kuadrant-system/authorino-6dd76875b5-sf54c" Nov 03 19:44:10.961004 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:10.960981 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-sf54c" Nov 03 19:44:10.968284 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:10.968265 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-sf54c" Nov 03 19:44:10.978069 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:10.978035 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/209d692f-1f73-436d-b5cf-7bcffc02b5e7-tls-cert\") pod \"authorino-6889cbd8c4-nlh5d\" (UID: \"209d692f-1f73-436d-b5cf-7bcffc02b5e7\") " pod="kuadrant-system/authorino-6889cbd8c4-nlh5d" Nov 03 19:44:10.978238 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:10.978133 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tpqq\" (UniqueName: \"kubernetes.io/projected/209d692f-1f73-436d-b5cf-7bcffc02b5e7-kube-api-access-8tpqq\") pod \"authorino-6889cbd8c4-nlh5d\" (UID: \"209d692f-1f73-436d-b5cf-7bcffc02b5e7\") " pod="kuadrant-system/authorino-6889cbd8c4-nlh5d" Nov 03 19:44:10.980424 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:10.980400 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/209d692f-1f73-436d-b5cf-7bcffc02b5e7-tls-cert\") pod \"authorino-6889cbd8c4-nlh5d\" (UID: \"209d692f-1f73-436d-b5cf-7bcffc02b5e7\") " pod="kuadrant-system/authorino-6889cbd8c4-nlh5d" Nov 03 19:44:10.990213 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:10.990180 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tpqq\" (UniqueName: \"kubernetes.io/projected/209d692f-1f73-436d-b5cf-7bcffc02b5e7-kube-api-access-8tpqq\") pod \"authorino-6889cbd8c4-nlh5d\" (UID: \"209d692f-1f73-436d-b5cf-7bcffc02b5e7\") " pod="kuadrant-system/authorino-6889cbd8c4-nlh5d" Nov 03 19:44:11.045958 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:11.045933 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-nlh5d"] Nov 03 19:44:11.046172 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:11.046161 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-nlh5d" Nov 03 19:44:11.076429 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:11.076399 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-77f7f55747-slnb6"] Nov 03 19:44:11.078791 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:11.078768 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b9b3a85-bf92-4ab6-9f45-2866ad77faa2-tls-cert\" (UniqueName: \"kubernetes.io/secret/3b9b3a85-bf92-4ab6-9f45-2866ad77faa2-tls-cert\") pod \"3b9b3a85-bf92-4ab6-9f45-2866ad77faa2\" (UID: \"3b9b3a85-bf92-4ab6-9f45-2866ad77faa2\") " Nov 03 19:44:11.078911 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:11.078826 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b9b3a85-bf92-4ab6-9f45-2866ad77faa2-kube-api-access-clkjs\" (UniqueName: \"kubernetes.io/projected/3b9b3a85-bf92-4ab6-9f45-2866ad77faa2-kube-api-access-clkjs\") pod \"3b9b3a85-bf92-4ab6-9f45-2866ad77faa2\" (UID: \"3b9b3a85-bf92-4ab6-9f45-2866ad77faa2\") " Nov 03 19:44:11.080938 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:11.080908 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b9b3a85-bf92-4ab6-9f45-2866ad77faa2-tls-cert" pod "3b9b3a85-bf92-4ab6-9f45-2866ad77faa2" (UID: "3b9b3a85-bf92-4ab6-9f45-2866ad77faa2"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:44:11.081051 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:11.080930 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b9b3a85-bf92-4ab6-9f45-2866ad77faa2-kube-api-access-clkjs" pod "3b9b3a85-bf92-4ab6-9f45-2866ad77faa2" (UID: "3b9b3a85-bf92-4ab6-9f45-2866ad77faa2"). InnerVolumeSpecName "kube-api-access-clkjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:44:11.081347 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:11.081331 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77f7f55747-slnb6" Nov 03 19:44:11.088892 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:11.088871 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77f7f55747-slnb6"] Nov 03 19:44:11.180375 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:11.180306 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw656\" (UniqueName: \"kubernetes.io/projected/9a62ecb7-cacf-4cb4-85b1-edf64bd88b57-kube-api-access-cw656\") pod \"authorino-77f7f55747-slnb6\" (UID: \"9a62ecb7-cacf-4cb4-85b1-edf64bd88b57\") " pod="kuadrant-system/authorino-77f7f55747-slnb6" Nov 03 19:44:11.180375 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:11.180347 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9a62ecb7-cacf-4cb4-85b1-edf64bd88b57-tls-cert\") pod \"authorino-77f7f55747-slnb6\" (UID: \"9a62ecb7-cacf-4cb4-85b1-edf64bd88b57\") " pod="kuadrant-system/authorino-77f7f55747-slnb6" Nov 03 19:44:11.180534 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:11.180466 5059 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3b9b3a85-bf92-4ab6-9f45-2866ad77faa2-tls-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:11.180534 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:11.180486 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clkjs\" (UniqueName: \"kubernetes.io/projected/3b9b3a85-bf92-4ab6-9f45-2866ad77faa2-kube-api-access-clkjs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:11.281287 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:11.281256 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw656\" (UniqueName: \"kubernetes.io/projected/9a62ecb7-cacf-4cb4-85b1-edf64bd88b57-kube-api-access-cw656\") pod \"authorino-77f7f55747-slnb6\" (UID: \"9a62ecb7-cacf-4cb4-85b1-edf64bd88b57\") " pod="kuadrant-system/authorino-77f7f55747-slnb6" Nov 03 19:44:11.281416 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:11.281296 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9a62ecb7-cacf-4cb4-85b1-edf64bd88b57-tls-cert\") pod \"authorino-77f7f55747-slnb6\" (UID: \"9a62ecb7-cacf-4cb4-85b1-edf64bd88b57\") " pod="kuadrant-system/authorino-77f7f55747-slnb6" Nov 03 19:44:11.283512 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:11.283487 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9a62ecb7-cacf-4cb4-85b1-edf64bd88b57-tls-cert\") pod \"authorino-77f7f55747-slnb6\" (UID: \"9a62ecb7-cacf-4cb4-85b1-edf64bd88b57\") " pod="kuadrant-system/authorino-77f7f55747-slnb6" Nov 03 19:44:11.293996 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:11.293972 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw656\" (UniqueName: \"kubernetes.io/projected/9a62ecb7-cacf-4cb4-85b1-edf64bd88b57-kube-api-access-cw656\") pod \"authorino-77f7f55747-slnb6\" (UID: \"9a62ecb7-cacf-4cb4-85b1-edf64bd88b57\") " pod="kuadrant-system/authorino-77f7f55747-slnb6" Nov 03 19:44:11.321935 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:11.321913 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-nlh5d"] Nov 03 19:44:11.323977 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:44:11.323957 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod209d692f_1f73_436d_b5cf_7bcffc02b5e7.slice/crio-3174c56e3b47db289187d93ac12dbede11d9c9c3455507248d3515d497014471 WatchSource:0}: Error finding container 3174c56e3b47db289187d93ac12dbede11d9c9c3455507248d3515d497014471: Status 404 returned error can't find the container with id 3174c56e3b47db289187d93ac12dbede11d9c9c3455507248d3515d497014471 Nov 03 19:44:11.400782 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:11.400761 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77f7f55747-slnb6" Nov 03 19:44:11.673370 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:11.673328 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77f7f55747-slnb6"] Nov 03 19:44:11.676602 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:44:11.676568 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a62ecb7_cacf_4cb4_85b1_edf64bd88b57.slice/crio-24380e6febe14b23c04fd040fdec16ba92f4324ffbb45989e250df344a309d13 WatchSource:0}: Error finding container 24380e6febe14b23c04fd040fdec16ba92f4324ffbb45989e250df344a309d13: Status 404 returned error can't find the container with id 24380e6febe14b23c04fd040fdec16ba92f4324ffbb45989e250df344a309d13 Nov 03 19:44:11.967698 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:11.967637 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77f7f55747-slnb6" event={"ID":"9a62ecb7-cacf-4cb4-85b1-edf64bd88b57","Type":"ContainerStarted","Data":"24380e6febe14b23c04fd040fdec16ba92f4324ffbb45989e250df344a309d13"} Nov 03 19:44:11.968914 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:11.968876 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-nlh5d" event={"ID":"209d692f-1f73-436d-b5cf-7bcffc02b5e7","Type":"ContainerStarted","Data":"3174c56e3b47db289187d93ac12dbede11d9c9c3455507248d3515d497014471"} Nov 03 19:44:11.968914 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:11.968911 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-sf54c" Nov 03 19:44:12.001706 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:12.001649 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-sf54c"] Nov 03 19:44:12.006376 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:12.006344 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-sf54c"] Nov 03 19:44:12.977635 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:12.977575 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77f7f55747-slnb6" event={"ID":"9a62ecb7-cacf-4cb4-85b1-edf64bd88b57","Type":"ContainerStarted","Data":"541003290bed4c2c3f046a04db72b50a764ee23e87cef458f102893006bcd6d9"} Nov 03 19:44:12.979764 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:12.979738 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-nlh5d" event={"ID":"209d692f-1f73-436d-b5cf-7bcffc02b5e7","Type":"ContainerStarted","Data":"a4254cc798c413bcc9902fd41ea30c23926e5542991b18b8a68afb6ca7cda240"} Nov 03 19:44:12.979901 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:12.979819 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-nlh5d" podUID="209d692f-1f73-436d-b5cf-7bcffc02b5e7" containerName="authorino" containerID="cri-o://a4254cc798c413bcc9902fd41ea30c23926e5542991b18b8a68afb6ca7cda240" gracePeriod=30 Nov 03 19:44:13.019678 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.018712 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-nlh5d" podStartSLOduration=2.2741496789999998 podStartE2EDuration="3.018693489s" totalImagesPullingTime="744.54381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:13.018020474 +0000 UTC m=+447.275817731" watchObservedRunningTime="2026-11-03 19:44:13.018693489 +0000 UTC m=+447.276490747" Nov 03 19:44:13.019678 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.019511 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-77f7f55747-slnb6" podStartSLOduration=1.252022006 podStartE2EDuration="2.019503479s" totalImagesPullingTime="767.481473ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:12.997996221 +0000 UTC m=+447.255793478" watchObservedRunningTime="2026-11-03 19:44:13.019503479 +0000 UTC m=+447.277300771" Nov 03 19:44:13.028185 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.028154 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-x5lg7"] Nov 03 19:44:13.028365 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.028341 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-x5lg7" podUID="f865f05b-9a1f-4ff0-b784-228049b1f26a" containerName="authorino" containerID="cri-o://ccf3ede4ac80c83f3fef9e234a15ac9a7e53c669cdb18b5ae67940e9e01c3cae" gracePeriod=30 Nov 03 19:44:13.304200 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.304174 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-nlh5d" Nov 03 19:44:13.307149 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.307128 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-x5lg7" Nov 03 19:44:13.405478 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.405443 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/209d692f-1f73-436d-b5cf-7bcffc02b5e7-tls-cert\" (UniqueName: \"kubernetes.io/secret/209d692f-1f73-436d-b5cf-7bcffc02b5e7-tls-cert\") pod \"209d692f-1f73-436d-b5cf-7bcffc02b5e7\" (UID: \"209d692f-1f73-436d-b5cf-7bcffc02b5e7\") " Nov 03 19:44:13.405643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.405552 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f865f05b-9a1f-4ff0-b784-228049b1f26a-kube-api-access-6ld4s\" (UniqueName: \"kubernetes.io/projected/f865f05b-9a1f-4ff0-b784-228049b1f26a-kube-api-access-6ld4s\") pod \"f865f05b-9a1f-4ff0-b784-228049b1f26a\" (UID: \"f865f05b-9a1f-4ff0-b784-228049b1f26a\") " Nov 03 19:44:13.405643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.405604 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/209d692f-1f73-436d-b5cf-7bcffc02b5e7-kube-api-access-8tpqq\" (UniqueName: \"kubernetes.io/projected/209d692f-1f73-436d-b5cf-7bcffc02b5e7-kube-api-access-8tpqq\") pod \"209d692f-1f73-436d-b5cf-7bcffc02b5e7\" (UID: \"209d692f-1f73-436d-b5cf-7bcffc02b5e7\") " Nov 03 19:44:13.407610 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.407582 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f865f05b-9a1f-4ff0-b784-228049b1f26a-kube-api-access-6ld4s" pod "f865f05b-9a1f-4ff0-b784-228049b1f26a" (UID: "f865f05b-9a1f-4ff0-b784-228049b1f26a"). InnerVolumeSpecName "kube-api-access-6ld4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:44:13.407855 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.407828 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209d692f-1f73-436d-b5cf-7bcffc02b5e7-kube-api-access-8tpqq" pod "209d692f-1f73-436d-b5cf-7bcffc02b5e7" (UID: "209d692f-1f73-436d-b5cf-7bcffc02b5e7"). InnerVolumeSpecName "kube-api-access-8tpqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:44:13.416395 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.416373 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/209d692f-1f73-436d-b5cf-7bcffc02b5e7-tls-cert" pod "209d692f-1f73-436d-b5cf-7bcffc02b5e7" (UID: "209d692f-1f73-436d-b5cf-7bcffc02b5e7"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:44:13.506852 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.506787 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ld4s\" (UniqueName: \"kubernetes.io/projected/f865f05b-9a1f-4ff0-b784-228049b1f26a-kube-api-access-6ld4s\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:13.506852 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.506811 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tpqq\" (UniqueName: \"kubernetes.io/projected/209d692f-1f73-436d-b5cf-7bcffc02b5e7-kube-api-access-8tpqq\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:13.506852 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.506820 5059 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/209d692f-1f73-436d-b5cf-7bcffc02b5e7-tls-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:44:13.855373 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.855340 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b9b3a85-bf92-4ab6-9f45-2866ad77faa2" path="/var/lib/kubelet/pods/3b9b3a85-bf92-4ab6-9f45-2866ad77faa2/volumes" Nov 03 19:44:13.985300 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.985261 5059 generic.go:356] "Generic (PLEG): container finished" podID="f865f05b-9a1f-4ff0-b784-228049b1f26a" containerID="ccf3ede4ac80c83f3fef9e234a15ac9a7e53c669cdb18b5ae67940e9e01c3cae" exitCode=0 Nov 03 19:44:13.985648 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.985307 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-x5lg7" Nov 03 19:44:13.985648 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.985318 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-x5lg7" event={"ID":"f865f05b-9a1f-4ff0-b784-228049b1f26a","Type":"ContainerDied","Data":"ccf3ede4ac80c83f3fef9e234a15ac9a7e53c669cdb18b5ae67940e9e01c3cae"} Nov 03 19:44:13.985648 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.985356 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-x5lg7" event={"ID":"f865f05b-9a1f-4ff0-b784-228049b1f26a","Type":"ContainerDied","Data":"b06ca15e27d4d5f70e597a3873ea8d6a07ef1cc5e744c34d76277694bac90bfc"} Nov 03 19:44:13.985648 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.985372 5059 scope.go:122] "RemoveContainer" containerID="ccf3ede4ac80c83f3fef9e234a15ac9a7e53c669cdb18b5ae67940e9e01c3cae" Nov 03 19:44:13.986691 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.986653 5059 generic.go:356] "Generic (PLEG): container finished" podID="209d692f-1f73-436d-b5cf-7bcffc02b5e7" containerID="a4254cc798c413bcc9902fd41ea30c23926e5542991b18b8a68afb6ca7cda240" exitCode=0 Nov 03 19:44:13.986797 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.986692 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-nlh5d" event={"ID":"209d692f-1f73-436d-b5cf-7bcffc02b5e7","Type":"ContainerDied","Data":"a4254cc798c413bcc9902fd41ea30c23926e5542991b18b8a68afb6ca7cda240"} Nov 03 19:44:13.986797 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.986731 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-nlh5d" event={"ID":"209d692f-1f73-436d-b5cf-7bcffc02b5e7","Type":"ContainerDied","Data":"3174c56e3b47db289187d93ac12dbede11d9c9c3455507248d3515d497014471"} Nov 03 19:44:13.986797 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.986702 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-nlh5d" Nov 03 19:44:13.994110 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.994094 5059 scope.go:122] "RemoveContainer" containerID="ccf3ede4ac80c83f3fef9e234a15ac9a7e53c669cdb18b5ae67940e9e01c3cae" Nov 03 19:44:13.994361 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:44:13.994341 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccf3ede4ac80c83f3fef9e234a15ac9a7e53c669cdb18b5ae67940e9e01c3cae\": container with ID starting with ccf3ede4ac80c83f3fef9e234a15ac9a7e53c669cdb18b5ae67940e9e01c3cae not found: ID does not exist" containerID="ccf3ede4ac80c83f3fef9e234a15ac9a7e53c669cdb18b5ae67940e9e01c3cae" Nov 03 19:44:13.994432 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.994373 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccf3ede4ac80c83f3fef9e234a15ac9a7e53c669cdb18b5ae67940e9e01c3cae"} err="failed to get container status \"ccf3ede4ac80c83f3fef9e234a15ac9a7e53c669cdb18b5ae67940e9e01c3cae\": rpc error: code = NotFound desc = could not find container \"ccf3ede4ac80c83f3fef9e234a15ac9a7e53c669cdb18b5ae67940e9e01c3cae\": container with ID starting with ccf3ede4ac80c83f3fef9e234a15ac9a7e53c669cdb18b5ae67940e9e01c3cae not found: ID does not exist" Nov 03 19:44:13.994432 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:13.994398 5059 scope.go:122] "RemoveContainer" containerID="a4254cc798c413bcc9902fd41ea30c23926e5542991b18b8a68afb6ca7cda240" Nov 03 19:44:14.001892 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:14.001859 5059 scope.go:122] "RemoveContainer" containerID="a4254cc798c413bcc9902fd41ea30c23926e5542991b18b8a68afb6ca7cda240" Nov 03 19:44:14.002228 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:44:14.002209 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4254cc798c413bcc9902fd41ea30c23926e5542991b18b8a68afb6ca7cda240\": container with ID starting with a4254cc798c413bcc9902fd41ea30c23926e5542991b18b8a68afb6ca7cda240 not found: ID does not exist" containerID="a4254cc798c413bcc9902fd41ea30c23926e5542991b18b8a68afb6ca7cda240" Nov 03 19:44:14.002294 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:14.002234 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4254cc798c413bcc9902fd41ea30c23926e5542991b18b8a68afb6ca7cda240"} err="failed to get container status \"a4254cc798c413bcc9902fd41ea30c23926e5542991b18b8a68afb6ca7cda240\": rpc error: code = NotFound desc = could not find container \"a4254cc798c413bcc9902fd41ea30c23926e5542991b18b8a68afb6ca7cda240\": container with ID starting with a4254cc798c413bcc9902fd41ea30c23926e5542991b18b8a68afb6ca7cda240 not found: ID does not exist" Nov 03 19:44:14.003470 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:14.003451 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-nlh5d"] Nov 03 19:44:14.008983 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:14.008963 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-nlh5d"] Nov 03 19:44:14.020397 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:14.020378 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-x5lg7"] Nov 03 19:44:14.024568 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:14.024549 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-x5lg7"] Nov 03 19:44:15.855361 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:15.855323 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="209d692f-1f73-436d-b5cf-7bcffc02b5e7" path="/var/lib/kubelet/pods/209d692f-1f73-436d-b5cf-7bcffc02b5e7/volumes" Nov 03 19:44:15.855880 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:44:15.855789 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f865f05b-9a1f-4ff0-b784-228049b1f26a" path="/var/lib/kubelet/pods/f865f05b-9a1f-4ff0-b784-228049b1f26a/volumes" Nov 03 19:44:40.605322 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:44:40.605280 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:45:10.603914 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:45:10.603874 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:45:40.603449 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:45:40.603412 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:46:10.602309 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:46:10.602274 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:46:12.621642 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.621609 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l"] Nov 03 19:46:12.624001 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.622107 5059 cpu_manager.go:414] "containerMap: removing container" podUID="209d692f-1f73-436d-b5cf-7bcffc02b5e7" containerName="authorino" Nov 03 19:46:12.624001 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.622122 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="209d692f-1f73-436d-b5cf-7bcffc02b5e7" containerName="authorino" Nov 03 19:46:12.624001 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.622138 5059 cpu_manager.go:414] "containerMap: removing container" podUID="f865f05b-9a1f-4ff0-b784-228049b1f26a" containerName="authorino" Nov 03 19:46:12.624001 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.622143 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f865f05b-9a1f-4ff0-b784-228049b1f26a" containerName="authorino" Nov 03 19:46:12.624001 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.622225 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="209d692f-1f73-436d-b5cf-7bcffc02b5e7" containerName="authorino" Nov 03 19:46:12.624001 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.622234 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f865f05b-9a1f-4ff0-b784-228049b1f26a" containerName="authorino" Nov 03 19:46:12.625208 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.625192 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 19:46:12.629677 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.629630 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 19:46:12.630938 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.630918 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8tzmp\"" Nov 03 19:46:12.631053 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.630941 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 19:46:12.636292 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.636268 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l"] Nov 03 19:46:12.637967 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.637947 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 19:46:12.728321 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.728292 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde430c2-2518-49a0-9851-42356d3f2fe9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tf75l\" (UID: \"cde430c2-2518-49a0-9851-42356d3f2fe9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 19:46:12.728424 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.728330 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tf75l\" (UID: \"cde430c2-2518-49a0-9851-42356d3f2fe9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 19:46:12.728424 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.728352 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tf75l\" (UID: \"cde430c2-2518-49a0-9851-42356d3f2fe9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 19:46:12.728424 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.728367 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tf75l\" (UID: \"cde430c2-2518-49a0-9851-42356d3f2fe9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 19:46:12.728526 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.728502 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfms2\" (UniqueName: \"kubernetes.io/projected/cde430c2-2518-49a0-9851-42356d3f2fe9-kube-api-access-wfms2\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tf75l\" (UID: \"cde430c2-2518-49a0-9851-42356d3f2fe9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 19:46:12.728559 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.728531 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tf75l\" (UID: \"cde430c2-2518-49a0-9851-42356d3f2fe9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 19:46:12.829807 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.829779 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde430c2-2518-49a0-9851-42356d3f2fe9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tf75l\" (UID: \"cde430c2-2518-49a0-9851-42356d3f2fe9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 19:46:12.830010 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.829826 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tf75l\" (UID: \"cde430c2-2518-49a0-9851-42356d3f2fe9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 19:46:12.830010 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.829857 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tf75l\" (UID: \"cde430c2-2518-49a0-9851-42356d3f2fe9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 19:46:12.830010 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.829879 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tf75l\" (UID: \"cde430c2-2518-49a0-9851-42356d3f2fe9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 19:46:12.830010 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.829974 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfms2\" (UniqueName: \"kubernetes.io/projected/cde430c2-2518-49a0-9851-42356d3f2fe9-kube-api-access-wfms2\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tf75l\" (UID: \"cde430c2-2518-49a0-9851-42356d3f2fe9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 19:46:12.830247 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.830011 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tf75l\" (UID: \"cde430c2-2518-49a0-9851-42356d3f2fe9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 19:46:12.830428 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.830408 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tf75l\" (UID: \"cde430c2-2518-49a0-9851-42356d3f2fe9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 19:46:12.830497 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.830459 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tf75l\" (UID: \"cde430c2-2518-49a0-9851-42356d3f2fe9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 19:46:12.830532 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.830497 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tf75l\" (UID: \"cde430c2-2518-49a0-9851-42356d3f2fe9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 19:46:12.832199 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.832177 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tf75l\" (UID: \"cde430c2-2518-49a0-9851-42356d3f2fe9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 19:46:12.832371 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.832353 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde430c2-2518-49a0-9851-42356d3f2fe9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tf75l\" (UID: \"cde430c2-2518-49a0-9851-42356d3f2fe9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 19:46:12.842327 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.842300 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfms2\" (UniqueName: \"kubernetes.io/projected/cde430c2-2518-49a0-9851-42356d3f2fe9-kube-api-access-wfms2\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tf75l\" (UID: \"cde430c2-2518-49a0-9851-42356d3f2fe9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 19:46:12.939793 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:12.939741 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 19:46:13.231001 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:13.230973 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l"] Nov 03 19:46:13.233400 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:46:13.233356 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde430c2_2518_49a0_9851_42356d3f2fe9.slice/crio-4c35e8e3486d38738c9026d17e0fc6245f178ee71991f27f201b46ee79442abe WatchSource:0}: Error finding container 4c35e8e3486d38738c9026d17e0fc6245f178ee71991f27f201b46ee79442abe: Status 404 returned error can't find the container with id 4c35e8e3486d38738c9026d17e0fc6245f178ee71991f27f201b46ee79442abe Nov 03 19:46:13.538058 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:13.537975 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" event={"ID":"cde430c2-2518-49a0-9851-42356d3f2fe9","Type":"ContainerStarted","Data":"4c35e8e3486d38738c9026d17e0fc6245f178ee71991f27f201b46ee79442abe"} Nov 03 19:46:15.551738 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:15.551694 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" event={"ID":"cde430c2-2518-49a0-9851-42356d3f2fe9","Type":"ContainerStarted","Data":"1317df3f930488af045a4ea5f2ffd5b2d0b6bac1972a73208461be0692b6e2b2"} Nov 03 19:46:15.572384 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:15.572334 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" podStartSLOduration=2.233936618 podStartE2EDuration="3.572322435s" totalImagesPullingTime="1.338385817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:15.571993099 +0000 UTC m=+569.829790357" watchObservedRunningTime="2026-11-03 19:46:15.572322435 +0000 UTC m=+569.830119688" Nov 03 19:46:15.940435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:15.940386 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 19:46:18.522241 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:18.522199 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp"] Nov 03 19:46:18.525859 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:18.525835 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 19:46:18.529775 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:18.529754 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 19:46:18.536974 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:18.536949 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp"] Nov 03 19:46:18.580545 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:18.580502 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp\" (UID: \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 19:46:18.580688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:18.580566 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp\" (UID: \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 19:46:18.580736 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:18.580707 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp\" (UID: \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 19:46:18.580779 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:18.580758 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkvf4\" (UniqueName: \"kubernetes.io/projected/e7cf1267-7d06-459b-9df7-fc7c477d36e1-kube-api-access-rkvf4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp\" (UID: \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 19:46:18.580924 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:18.580891 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp\" (UID: \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 19:46:18.581053 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:18.581038 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7cf1267-7d06-459b-9df7-fc7c477d36e1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp\" (UID: \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 19:46:18.681738 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:18.681707 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp\" (UID: \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 19:46:18.681738 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:18.681743 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp\" (UID: \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 19:46:18.681965 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:18.681765 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp\" (UID: \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 19:46:18.681965 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:18.681792 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkvf4\" (UniqueName: \"kubernetes.io/projected/e7cf1267-7d06-459b-9df7-fc7c477d36e1-kube-api-access-rkvf4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp\" (UID: \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 19:46:18.681965 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:18.681828 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp\" (UID: \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 19:46:18.681965 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:18.681880 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7cf1267-7d06-459b-9df7-fc7c477d36e1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp\" (UID: \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 19:46:18.682198 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:18.682165 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp\" (UID: \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 19:46:18.682310 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:18.682286 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp\" (UID: \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 19:46:18.682448 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:18.682427 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp\" (UID: \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 19:46:18.684022 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:18.683996 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp\" (UID: \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 19:46:18.684329 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:18.684308 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7cf1267-7d06-459b-9df7-fc7c477d36e1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp\" (UID: \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 19:46:18.694706 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:18.694684 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkvf4\" (UniqueName: \"kubernetes.io/projected/e7cf1267-7d06-459b-9df7-fc7c477d36e1-kube-api-access-rkvf4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp\" (UID: \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 19:46:18.839795 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:18.839769 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 19:46:19.113796 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.113772 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp"] Nov 03 19:46:19.115625 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:46:19.115597 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7cf1267_7d06_459b_9df7_fc7c477d36e1.slice/crio-6ce4ab0a96f501ffbbc5268b102246760445565e5b43e450a568003ae42adf0c WatchSource:0}: Error finding container 6ce4ab0a96f501ffbbc5268b102246760445565e5b43e450a568003ae42adf0c: Status 404 returned error can't find the container with id 6ce4ab0a96f501ffbbc5268b102246760445565e5b43e450a568003ae42adf0c Nov 03 19:46:19.209267 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.209231 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-d7wsq"] Nov 03 19:46:19.214722 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.214698 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 19:46:19.223385 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.223342 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 19:46:19.224179 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.224154 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-d7wsq"] Nov 03 19:46:19.287375 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.287344 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-home\") pod \"echo-simulator-kserve-64b6f8f96c-d7wsq\" (UID: \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 19:46:19.287498 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.287380 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-d7wsq\" (UID: \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 19:46:19.287498 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.287451 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-d7wsq\" (UID: \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 19:46:19.287498 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.287481 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n58hn\" (UniqueName: \"kubernetes.io/projected/5ac27a9c-459f-4310-aef7-8dc643b46c6c-kube-api-access-n58hn\") pod \"echo-simulator-kserve-64b6f8f96c-d7wsq\" (UID: \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 19:46:19.287593 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.287576 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ac27a9c-459f-4310-aef7-8dc643b46c6c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-d7wsq\" (UID: \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 19:46:19.287628 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.287605 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-d7wsq\" (UID: \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 19:46:19.388858 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.388774 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ac27a9c-459f-4310-aef7-8dc643b46c6c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-d7wsq\" (UID: \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 19:46:19.388858 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.388821 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-d7wsq\" (UID: \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 19:46:19.389068 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.388866 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-home\") pod \"echo-simulator-kserve-64b6f8f96c-d7wsq\" (UID: \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 19:46:19.389068 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.388891 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-d7wsq\" (UID: \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 19:46:19.389068 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.388936 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-d7wsq\" (UID: \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 19:46:19.389068 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.388968 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n58hn\" (UniqueName: \"kubernetes.io/projected/5ac27a9c-459f-4310-aef7-8dc643b46c6c-kube-api-access-n58hn\") pod \"echo-simulator-kserve-64b6f8f96c-d7wsq\" (UID: \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 19:46:19.389523 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.389433 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-home\") pod \"echo-simulator-kserve-64b6f8f96c-d7wsq\" (UID: \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 19:46:19.389523 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.389452 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-d7wsq\" (UID: \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 19:46:19.389722 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.389632 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-d7wsq\" (UID: \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 19:46:19.391713 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.391688 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-d7wsq\" (UID: \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 19:46:19.391992 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.391975 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ac27a9c-459f-4310-aef7-8dc643b46c6c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-d7wsq\" (UID: \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 19:46:19.406442 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.406414 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n58hn\" (UniqueName: \"kubernetes.io/projected/5ac27a9c-459f-4310-aef7-8dc643b46c6c-kube-api-access-n58hn\") pod \"echo-simulator-kserve-64b6f8f96c-d7wsq\" (UID: \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 19:46:19.532874 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.532833 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 19:46:19.579466 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.579426 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" event={"ID":"e7cf1267-7d06-459b-9df7-fc7c477d36e1","Type":"ContainerStarted","Data":"90b3ff7ea089036c1955493d6ad07b652eb65d8c482b1abd7f0dc8c58bea2751"} Nov 03 19:46:19.579641 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.579504 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" event={"ID":"e7cf1267-7d06-459b-9df7-fc7c477d36e1","Type":"ContainerStarted","Data":"6ce4ab0a96f501ffbbc5268b102246760445565e5b43e450a568003ae42adf0c"} Nov 03 19:46:19.605353 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.605291 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" podStartSLOduration=1.2796636559999999 podStartE2EDuration="1.605269601s" totalImagesPullingTime="325.605945ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:19.602353712 +0000 UTC m=+573.860150972" watchObservedRunningTime="2026-11-03 19:46:19.605269601 +0000 UTC m=+573.863066860" Nov 03 19:46:19.818710 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.818681 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-d7wsq"] Nov 03 19:46:19.820961 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:46:19.820933 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac27a9c_459f_4310_aef7_8dc643b46c6c.slice/crio-504e87c07e2e0fb48957e4f70d79e1670ae33f82206e608d6d77f1eb0478480d WatchSource:0}: Error finding container 504e87c07e2e0fb48957e4f70d79e1670ae33f82206e608d6d77f1eb0478480d: Status 404 returned error can't find the container with id 504e87c07e2e0fb48957e4f70d79e1670ae33f82206e608d6d77f1eb0478480d Nov 03 19:46:19.839873 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:19.839848 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 19:46:20.486638 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:20.486602 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5"] Nov 03 19:46:20.490216 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:20.490187 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:46:20.499417 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:20.499382 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 03 19:46:20.501561 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:20.501539 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5"] Nov 03 19:46:20.586596 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:20.586558 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" event={"ID":"5ac27a9c-459f-4310-aef7-8dc643b46c6c","Type":"ContainerStarted","Data":"504e87c07e2e0fb48957e4f70d79e1670ae33f82206e608d6d77f1eb0478480d"} Nov 03 19:46:20.602381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:20.602347 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5\" (UID: \"541bc63e-66f1-42ca-a946-aedc34beb62f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:46:20.602575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:20.602401 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/541bc63e-66f1-42ca-a946-aedc34beb62f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5\" (UID: \"541bc63e-66f1-42ca-a946-aedc34beb62f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:46:20.602575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:20.602447 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5\" (UID: \"541bc63e-66f1-42ca-a946-aedc34beb62f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:46:20.602575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:20.602465 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5\" (UID: \"541bc63e-66f1-42ca-a946-aedc34beb62f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:46:20.602575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:20.602496 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5\" (UID: \"541bc63e-66f1-42ca-a946-aedc34beb62f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:46:20.602575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:20.602530 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9lk5\" (UniqueName: \"kubernetes.io/projected/541bc63e-66f1-42ca-a946-aedc34beb62f-kube-api-access-g9lk5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5\" (UID: \"541bc63e-66f1-42ca-a946-aedc34beb62f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:46:20.704028 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:20.703991 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5\" (UID: \"541bc63e-66f1-42ca-a946-aedc34beb62f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:46:20.704203 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:20.704065 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9lk5\" (UniqueName: \"kubernetes.io/projected/541bc63e-66f1-42ca-a946-aedc34beb62f-kube-api-access-g9lk5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5\" (UID: \"541bc63e-66f1-42ca-a946-aedc34beb62f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:46:20.704203 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:20.704169 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5\" (UID: \"541bc63e-66f1-42ca-a946-aedc34beb62f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:46:20.704203 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:20.704195 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/541bc63e-66f1-42ca-a946-aedc34beb62f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5\" (UID: \"541bc63e-66f1-42ca-a946-aedc34beb62f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:46:20.704369 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:20.704226 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5\" (UID: \"541bc63e-66f1-42ca-a946-aedc34beb62f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:46:20.704369 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:20.704255 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5\" (UID: \"541bc63e-66f1-42ca-a946-aedc34beb62f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:46:20.704633 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:20.704611 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5\" (UID: \"541bc63e-66f1-42ca-a946-aedc34beb62f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:46:20.705023 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:20.704997 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5\" (UID: \"541bc63e-66f1-42ca-a946-aedc34beb62f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:46:20.705275 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:20.705253 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5\" (UID: \"541bc63e-66f1-42ca-a946-aedc34beb62f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:46:20.707117 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:20.707090 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5\" (UID: \"541bc63e-66f1-42ca-a946-aedc34beb62f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:46:20.707874 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:20.707852 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/541bc63e-66f1-42ca-a946-aedc34beb62f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5\" (UID: \"541bc63e-66f1-42ca-a946-aedc34beb62f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:46:20.717168 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:20.717148 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9lk5\" (UniqueName: \"kubernetes.io/projected/541bc63e-66f1-42ca-a946-aedc34beb62f-kube-api-access-g9lk5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5\" (UID: \"541bc63e-66f1-42ca-a946-aedc34beb62f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:46:20.805234 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:20.805152 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:46:21.090245 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:21.090116 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5"] Nov 03 19:46:21.092747 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:46:21.092705 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod541bc63e_66f1_42ca_a946_aedc34beb62f.slice/crio-1c219679b70309c8c555a19601b051b1cb61e2cff65c58137d12b1f924d60147 WatchSource:0}: Error finding container 1c219679b70309c8c555a19601b051b1cb61e2cff65c58137d12b1f924d60147: Status 404 returned error can't find the container with id 1c219679b70309c8c555a19601b051b1cb61e2cff65c58137d12b1f924d60147 Nov 03 19:46:21.593401 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:21.593368 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" event={"ID":"541bc63e-66f1-42ca-a946-aedc34beb62f","Type":"ContainerStarted","Data":"1c69adcd1fc6db5a42ddeb00863f24a144f0cf528222cad083ac5a885e389943"} Nov 03 19:46:21.593780 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:21.593423 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" event={"ID":"541bc63e-66f1-42ca-a946-aedc34beb62f","Type":"ContainerStarted","Data":"1c219679b70309c8c555a19601b051b1cb61e2cff65c58137d12b1f924d60147"} Nov 03 19:46:21.663045 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:21.662995 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podStartSLOduration=1.379891765 podStartE2EDuration="1.662980641s" totalImagesPullingTime="283.088876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:21.660539376 +0000 UTC m=+575.918336647" watchObservedRunningTime="2026-11-03 19:46:21.662980641 +0000 UTC m=+575.920777895" Nov 03 19:46:21.806222 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:21.806178 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:46:22.599654 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:22.599610 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" event={"ID":"5ac27a9c-459f-4310-aef7-8dc643b46c6c","Type":"ContainerStarted","Data":"3fd1b473caf0e5fcbbbf5a97f92155614da76dd2857a8df4e1895ff6ca147711"} Nov 03 19:46:22.621587 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:22.621530 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" podStartSLOduration=1.860170694 podStartE2EDuration="3.62151691s" totalImagesPullingTime="1.761346216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:22.620050945 +0000 UTC m=+576.877848204" watchObservedRunningTime="2026-11-03 19:46:22.62151691 +0000 UTC m=+576.879314166" Nov 03 19:46:22.940466 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:22.940379 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 19:46:22.955235 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:22.955207 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 19:46:23.533739 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:23.533703 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 19:46:23.616402 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:23.616362 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 19:46:28.840370 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:28.840336 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 19:46:28.852564 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:28.852539 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 19:46:29.533479 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:29.533438 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 19:46:29.546012 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:29.545986 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 19:46:29.648307 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:29.648281 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 19:46:29.649135 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:29.649116 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 19:46:30.806344 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:30.806314 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:46:30.818756 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:30.818725 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:46:31.660472 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:31.660439 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:46:33.917169 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:33.917134 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-66b59d468d-5bfd2"] Nov 03 19:46:33.920791 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:33.920771 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66b59d468d-5bfd2" Nov 03 19:46:33.930180 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:33.930159 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66b59d468d-5bfd2"] Nov 03 19:46:34.024859 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:34.024824 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62z44\" (UniqueName: \"kubernetes.io/projected/c7fce352-1850-4048-be2e-fa458fd4d2f3-kube-api-access-62z44\") pod \"authorino-66b59d468d-5bfd2\" (UID: \"c7fce352-1850-4048-be2e-fa458fd4d2f3\") " pod="kuadrant-system/authorino-66b59d468d-5bfd2" Nov 03 19:46:34.024984 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:34.024929 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7fce352-1850-4048-be2e-fa458fd4d2f3-tls-cert\") pod \"authorino-66b59d468d-5bfd2\" (UID: \"c7fce352-1850-4048-be2e-fa458fd4d2f3\") " pod="kuadrant-system/authorino-66b59d468d-5bfd2" Nov 03 19:46:34.125879 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:34.125847 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7fce352-1850-4048-be2e-fa458fd4d2f3-tls-cert\") pod \"authorino-66b59d468d-5bfd2\" (UID: \"c7fce352-1850-4048-be2e-fa458fd4d2f3\") " pod="kuadrant-system/authorino-66b59d468d-5bfd2" Nov 03 19:46:34.125985 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:34.125912 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62z44\" (UniqueName: \"kubernetes.io/projected/c7fce352-1850-4048-be2e-fa458fd4d2f3-kube-api-access-62z44\") pod \"authorino-66b59d468d-5bfd2\" (UID: \"c7fce352-1850-4048-be2e-fa458fd4d2f3\") " pod="kuadrant-system/authorino-66b59d468d-5bfd2" Nov 03 19:46:34.128279 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:34.128260 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7fce352-1850-4048-be2e-fa458fd4d2f3-tls-cert\") pod \"authorino-66b59d468d-5bfd2\" (UID: \"c7fce352-1850-4048-be2e-fa458fd4d2f3\") " pod="kuadrant-system/authorino-66b59d468d-5bfd2" Nov 03 19:46:34.138712 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:34.138688 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62z44\" (UniqueName: \"kubernetes.io/projected/c7fce352-1850-4048-be2e-fa458fd4d2f3-kube-api-access-62z44\") pod \"authorino-66b59d468d-5bfd2\" (UID: \"c7fce352-1850-4048-be2e-fa458fd4d2f3\") " pod="kuadrant-system/authorino-66b59d468d-5bfd2" Nov 03 19:46:34.235932 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:34.235867 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66b59d468d-5bfd2" Nov 03 19:46:34.507789 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:34.507766 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66b59d468d-5bfd2"] Nov 03 19:46:34.509910 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:46:34.509884 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7fce352_1850_4048_be2e_fa458fd4d2f3.slice/crio-54963a6f8070c198ca5235d825b024fa48be551889691ecb18a8d272bc1c0398 WatchSource:0}: Error finding container 54963a6f8070c198ca5235d825b024fa48be551889691ecb18a8d272bc1c0398: Status 404 returned error can't find the container with id 54963a6f8070c198ca5235d825b024fa48be551889691ecb18a8d272bc1c0398 Nov 03 19:46:34.665821 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:34.665792 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66b59d468d-5bfd2" event={"ID":"c7fce352-1850-4048-be2e-fa458fd4d2f3","Type":"ContainerStarted","Data":"54963a6f8070c198ca5235d825b024fa48be551889691ecb18a8d272bc1c0398"} Nov 03 19:46:35.675978 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:35.675938 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66b59d468d-5bfd2" event={"ID":"c7fce352-1850-4048-be2e-fa458fd4d2f3","Type":"ContainerStarted","Data":"8121a3441525651ca2c833f0f9236c162d2c111dd09ec30bb62b251a8accdd62"} Nov 03 19:46:35.734560 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:35.734496 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-66b59d468d-5bfd2" podStartSLOduration=2.106551559 podStartE2EDuration="2.734474598s" totalImagesPullingTime="627.923039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:35.70110087 +0000 UTC m=+589.958898127" watchObservedRunningTime="2026-11-03 19:46:35.734474598 +0000 UTC m=+589.992271857" Nov 03 19:46:35.736064 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:35.736036 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-77f7f55747-slnb6"] Nov 03 19:46:35.736415 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:35.736389 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-77f7f55747-slnb6" podUID="9a62ecb7-cacf-4cb4-85b1-edf64bd88b57" containerName="authorino" containerID="cri-o://541003290bed4c2c3f046a04db72b50a764ee23e87cef458f102893006bcd6d9" gracePeriod=30 Nov 03 19:46:35.993216 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:35.993185 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77f7f55747-slnb6" Nov 03 19:46:36.044443 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:36.043019 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a62ecb7-cacf-4cb4-85b1-edf64bd88b57-kube-api-access-cw656\" (UniqueName: \"kubernetes.io/projected/9a62ecb7-cacf-4cb4-85b1-edf64bd88b57-kube-api-access-cw656\") pod \"9a62ecb7-cacf-4cb4-85b1-edf64bd88b57\" (UID: \"9a62ecb7-cacf-4cb4-85b1-edf64bd88b57\") " Nov 03 19:46:36.044443 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:36.043175 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a62ecb7-cacf-4cb4-85b1-edf64bd88b57-tls-cert\" (UniqueName: \"kubernetes.io/secret/9a62ecb7-cacf-4cb4-85b1-edf64bd88b57-tls-cert\") pod \"9a62ecb7-cacf-4cb4-85b1-edf64bd88b57\" (UID: \"9a62ecb7-cacf-4cb4-85b1-edf64bd88b57\") " Nov 03 19:46:36.045852 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:36.045824 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a62ecb7-cacf-4cb4-85b1-edf64bd88b57-kube-api-access-cw656" pod "9a62ecb7-cacf-4cb4-85b1-edf64bd88b57" (UID: "9a62ecb7-cacf-4cb4-85b1-edf64bd88b57"). InnerVolumeSpecName "kube-api-access-cw656". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:46:36.056791 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:36.056766 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a62ecb7-cacf-4cb4-85b1-edf64bd88b57-tls-cert" pod "9a62ecb7-cacf-4cb4-85b1-edf64bd88b57" (UID: "9a62ecb7-cacf-4cb4-85b1-edf64bd88b57"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:46:36.144260 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:36.144227 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw656\" (UniqueName: \"kubernetes.io/projected/9a62ecb7-cacf-4cb4-85b1-edf64bd88b57-kube-api-access-cw656\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:46:36.144260 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:36.144256 5059 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9a62ecb7-cacf-4cb4-85b1-edf64bd88b57-tls-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:46:36.682992 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:36.682954 5059 generic.go:356] "Generic (PLEG): container finished" podID="9a62ecb7-cacf-4cb4-85b1-edf64bd88b57" containerID="541003290bed4c2c3f046a04db72b50a764ee23e87cef458f102893006bcd6d9" exitCode=0 Nov 03 19:46:36.683478 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:36.683037 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77f7f55747-slnb6" Nov 03 19:46:36.683478 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:36.683060 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77f7f55747-slnb6" event={"ID":"9a62ecb7-cacf-4cb4-85b1-edf64bd88b57","Type":"ContainerDied","Data":"541003290bed4c2c3f046a04db72b50a764ee23e87cef458f102893006bcd6d9"} Nov 03 19:46:36.683478 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:36.683109 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77f7f55747-slnb6" event={"ID":"9a62ecb7-cacf-4cb4-85b1-edf64bd88b57","Type":"ContainerDied","Data":"24380e6febe14b23c04fd040fdec16ba92f4324ffbb45989e250df344a309d13"} Nov 03 19:46:36.683478 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:36.683134 5059 scope.go:122] "RemoveContainer" containerID="541003290bed4c2c3f046a04db72b50a764ee23e87cef458f102893006bcd6d9" Nov 03 19:46:36.693885 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:36.693863 5059 scope.go:122] "RemoveContainer" containerID="541003290bed4c2c3f046a04db72b50a764ee23e87cef458f102893006bcd6d9" Nov 03 19:46:36.694216 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:46:36.694183 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"541003290bed4c2c3f046a04db72b50a764ee23e87cef458f102893006bcd6d9\": container with ID starting with 541003290bed4c2c3f046a04db72b50a764ee23e87cef458f102893006bcd6d9 not found: ID does not exist" containerID="541003290bed4c2c3f046a04db72b50a764ee23e87cef458f102893006bcd6d9" Nov 03 19:46:36.694276 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:36.694231 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"541003290bed4c2c3f046a04db72b50a764ee23e87cef458f102893006bcd6d9"} err="failed to get container status \"541003290bed4c2c3f046a04db72b50a764ee23e87cef458f102893006bcd6d9\": rpc error: code = NotFound desc = could not find container \"541003290bed4c2c3f046a04db72b50a764ee23e87cef458f102893006bcd6d9\": container with ID starting with 541003290bed4c2c3f046a04db72b50a764ee23e87cef458f102893006bcd6d9 not found: ID does not exist" Nov 03 19:46:36.709037 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:36.709014 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-77f7f55747-slnb6"] Nov 03 19:46:36.713005 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:36.712984 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-77f7f55747-slnb6"] Nov 03 19:46:37.856584 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:37.856552 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a62ecb7-cacf-4cb4-85b1-edf64bd88b57" path="/var/lib/kubelet/pods/9a62ecb7-cacf-4cb4-85b1-edf64bd88b57/volumes" Nov 03 19:46:38.986582 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:38.986549 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr"] Nov 03 19:46:38.986970 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:38.986956 5059 cpu_manager.go:414] "containerMap: removing container" podUID="9a62ecb7-cacf-4cb4-85b1-edf64bd88b57" containerName="authorino" Nov 03 19:46:38.987024 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:38.986970 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a62ecb7-cacf-4cb4-85b1-edf64bd88b57" containerName="authorino" Nov 03 19:46:38.987059 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:38.987043 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a62ecb7-cacf-4cb4-85b1-edf64bd88b57" containerName="authorino" Nov 03 19:46:38.991467 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:38.991436 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.000353 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.000325 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-65hnm\"" Nov 03 19:46:39.005967 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.005933 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr"] Nov 03 19:46:39.012719 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.011153 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm"] Nov 03 19:46:39.015271 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.015247 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.035317 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.035287 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm"] Nov 03 19:46:39.086065 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.086021 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.086261 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.086083 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.086335 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.086260 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.086335 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.086301 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.086452 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.086390 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.086452 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.086419 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1f91b8-973a-4345-acd7-ea3325faf638-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.086452 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.086441 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brprv\" (UniqueName: \"kubernetes.io/projected/1b1f91b8-973a-4345-acd7-ea3325faf638-kube-api-access-brprv\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.086608 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.086463 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.086608 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.086530 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.188081 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.188038 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.188081 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.188086 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.188311 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.188211 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.188359 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.188309 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.188359 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.188339 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.188437 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.188377 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.188437 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.188399 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.188535 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.188456 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.188535 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.188470 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.188535 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.188515 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1f91b8-973a-4345-acd7-ea3325faf638-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.188702 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.188534 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.188702 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.188564 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brprv\" (UniqueName: \"kubernetes.io/projected/1b1f91b8-973a-4345-acd7-ea3325faf638-kube-api-access-brprv\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.188702 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.188583 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.188702 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.188656 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.188913 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.188703 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.188913 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.188728 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.188913 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.188733 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.188913 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.188770 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.189115 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.188924 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.189115 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.189001 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.189115 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.189061 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgszk\" (UniqueName: \"kubernetes.io/projected/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-kube-api-access-qgszk\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.189323 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.189301 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1f91b8-973a-4345-acd7-ea3325faf638-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.189382 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.189355 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.190882 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.190856 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.191006 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.190960 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.201636 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.201610 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.201755 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.201700 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brprv\" (UniqueName: \"kubernetes.io/projected/1b1f91b8-973a-4345-acd7-ea3325faf638-kube-api-access-brprv\") pod \"maas-default-gateway-openshift-default-855f9574cf-cnppr\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.289602 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.289507 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.289602 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.289563 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.289602 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.289593 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.289894 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.289635 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.289894 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.289688 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.289894 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.289719 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.289894 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.289763 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.289894 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.289828 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgszk\" (UniqueName: \"kubernetes.io/projected/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-kube-api-access-qgszk\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.289894 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.289860 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.290188 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.289978 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.290188 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.290134 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.290460 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.290432 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.290559 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.290435 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.290720 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.290702 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.292212 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.292186 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.292858 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.292835 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.311916 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.305830 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:39.323565 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.323241 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.323565 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.323367 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgszk\" (UniqueName: \"kubernetes.io/projected/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-kube-api-access-qgszk\") pod \"maas-default-gateway-openshift-default-855f9574cf-vznnm\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.331960 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.331872 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:39.653104 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.653072 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr"] Nov 03 19:46:39.657680 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:46:39.657627 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1f91b8_973a_4345_acd7_ea3325faf638.slice/crio-e4d63cbbad213bec6820394b5b1cd23984fdd2c9fd732c27187a960835ddce84 WatchSource:0}: Error finding container e4d63cbbad213bec6820394b5b1cd23984fdd2c9fd732c27187a960835ddce84: Status 404 returned error can't find the container with id e4d63cbbad213bec6820394b5b1cd23984fdd2c9fd732c27187a960835ddce84 Nov 03 19:46:39.660277 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.660243 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:46:39.660390 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.660307 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:46:39.660390 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.660336 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:46:39.701338 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.701309 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm"] Nov 03 19:46:39.703840 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.703808 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" event={"ID":"1b1f91b8-973a-4345-acd7-ea3325faf638","Type":"ContainerStarted","Data":"e4d63cbbad213bec6820394b5b1cd23984fdd2c9fd732c27187a960835ddce84"} Nov 03 19:46:39.706843 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:46:39.706811 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171c7288_3157_4fe0_80ee_94cf5b2ca2dd.slice/crio-5356f35de0393003597c8867655dc2f00fd424a96cbcb72e79de3ca84e22e177 WatchSource:0}: Error finding container 5356f35de0393003597c8867655dc2f00fd424a96cbcb72e79de3ca84e22e177: Status 404 returned error can't find the container with id 5356f35de0393003597c8867655dc2f00fd424a96cbcb72e79de3ca84e22e177 Nov 03 19:46:39.709142 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.709110 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:46:39.709220 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.709181 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:46:39.709220 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:39.709209 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:46:40.633572 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:46:40.633531 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:46:40.718377 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:40.718341 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" event={"ID":"1b1f91b8-973a-4345-acd7-ea3325faf638","Type":"ContainerStarted","Data":"0957e98fb98eca19314edc21a5a36bb3335ce419f10eaf36789a50068a116b53"} Nov 03 19:46:40.719707 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:40.719683 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" event={"ID":"171c7288-3157-4fe0-80ee-94cf5b2ca2dd","Type":"ContainerStarted","Data":"07485e5a1a9bd8f3e2278bef20b0f805688a601c1ec005e9ee1b9f8abb449cd3"} Nov 03 19:46:40.719760 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:40.719728 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" event={"ID":"171c7288-3157-4fe0-80ee-94cf5b2ca2dd","Type":"ContainerStarted","Data":"5356f35de0393003597c8867655dc2f00fd424a96cbcb72e79de3ca84e22e177"} Nov 03 19:46:40.746325 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:40.746261 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" podStartSLOduration=2.746247498 podStartE2EDuration="2.746247498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:40.743313686 +0000 UTC m=+595.001110956" watchObservedRunningTime="2026-11-03 19:46:40.746247498 +0000 UTC m=+595.004044754" Nov 03 19:46:41.309527 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:41.309236 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:41.310697 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:41.310645 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cnppr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 19:46:41.310791 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:41.310727 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" podUID="1b1f91b8-973a-4345-acd7-ea3325faf638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 19:46:41.336350 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:41.336325 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:41.337547 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:41.337521 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vznnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 19:46:41.337592 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:41.337576 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" podUID="171c7288-3157-4fe0-80ee-94cf5b2ca2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 19:46:42.310072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:42.310037 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cnppr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 19:46:42.310985 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:42.310636 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" podUID="1b1f91b8-973a-4345-acd7-ea3325faf638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 19:46:42.336009 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:42.335984 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vznnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 19:46:42.336122 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:42.336026 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" podUID="171c7288-3157-4fe0-80ee-94cf5b2ca2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 19:46:43.307143 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:43.307099 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cnppr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 19:46:43.307325 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:43.307168 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" podUID="1b1f91b8-973a-4345-acd7-ea3325faf638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 19:46:43.337000 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:43.336965 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vznnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 19:46:43.337387 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:43.337015 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" podUID="171c7288-3157-4fe0-80ee-94cf5b2ca2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 19:46:44.308054 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:44.308001 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cnppr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 19:46:44.308278 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:44.308064 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" podUID="1b1f91b8-973a-4345-acd7-ea3325faf638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 19:46:44.337012 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:44.336981 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vznnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 19:46:44.337372 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:44.337032 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" podUID="171c7288-3157-4fe0-80ee-94cf5b2ca2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 19:46:45.307130 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:45.307092 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cnppr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 19:46:45.307310 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:45.307159 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" podUID="1b1f91b8-973a-4345-acd7-ea3325faf638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 19:46:45.336795 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:45.336764 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vznnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 19:46:45.336925 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:45.336804 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" podUID="171c7288-3157-4fe0-80ee-94cf5b2ca2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 19:46:45.963369 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:45.963339 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 19:46:45.964197 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:45.964168 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 19:46:46.307182 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:46.307109 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cnppr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 19:46:46.307182 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:46.307161 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" podUID="1b1f91b8-973a-4345-acd7-ea3325faf638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 19:46:46.337684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:46.336966 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vznnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 19:46:46.337684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:46.337021 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" podUID="171c7288-3157-4fe0-80ee-94cf5b2ca2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 19:46:47.307268 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:47.307226 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cnppr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 19:46:47.307690 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:47.307294 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" podUID="1b1f91b8-973a-4345-acd7-ea3325faf638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 19:46:47.335989 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:47.335956 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vznnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 19:46:47.336116 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:47.336028 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" podUID="171c7288-3157-4fe0-80ee-94cf5b2ca2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 19:46:48.382684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:48.382603 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:48.383121 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:48.383061 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:48.384100 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:48.384076 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 19:46:48.407391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:48.407360 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:48.408578 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:48.408557 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:48.409396 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:48.409370 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 19:46:48.413525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:46:48.413461 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" podStartSLOduration=10.413445327 podStartE2EDuration="10.413445327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:40.776518424 +0000 UTC m=+595.034315680" watchObservedRunningTime="2026-11-03 19:46:48.413445327 +0000 UTC m=+602.671242584" Nov 03 19:47:04.992274 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:04.992238 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb"] Nov 03 19:47:04.996687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:04.996648 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.007252 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.007228 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-745a88f9-openshift-default-dockercfg-4g4k7\"" Nov 03 19:47:05.010360 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.010338 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb"] Nov 03 19:47:05.097588 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.097539 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istiod-ca-cert\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.097786 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.097598 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-token\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.097786 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.097766 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-data\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.097918 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.097822 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-workload-socket\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.097918 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.097859 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-credential-socket\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.097918 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.097897 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc4mc\" (UniqueName: \"kubernetes.io/projected/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-kube-api-access-hc4mc\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.098078 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.097928 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-podinfo\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.098078 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.097955 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-envoy\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.098078 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.098064 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-workload-certs\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.199097 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.199055 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc4mc\" (UniqueName: \"kubernetes.io/projected/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-kube-api-access-hc4mc\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.199294 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.199115 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-podinfo\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.199294 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.199147 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-envoy\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.199294 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.199248 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-workload-certs\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.199468 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.199302 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istiod-ca-cert\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.199468 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.199334 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-token\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.199468 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.199379 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-data\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.199468 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.199417 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-workload-socket\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.199468 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.199451 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-credential-socket\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.199923 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.199895 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-credential-socket\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.200228 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.200200 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istiod-ca-cert\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.200427 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.200401 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-data\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.200619 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.200594 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-workload-certs\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.201356 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.201332 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-workload-socket\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.203494 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.203467 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-podinfo\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.203646 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.203623 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-envoy\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.215017 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.214932 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc4mc\" (UniqueName: \"kubernetes.io/projected/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-kube-api-access-hc4mc\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.219332 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.219305 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-token\") pod \"e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.245829 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.245762 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx"] Nov 03 19:47:05.251367 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.251348 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.260856 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.260835 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-9959e5f4-gw-openshift-default-dockercfg-zb284\"" Nov 03 19:47:05.263751 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.263688 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx"] Nov 03 19:47:05.300392 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.300363 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-workload-certs\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.300496 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.300401 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-data\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.300552 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.300526 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-envoy\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.300642 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.300623 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-credential-socket\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.300726 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.300696 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-podinfo\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.300726 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.300716 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-workload-socket\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.300822 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.300735 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istiod-ca-cert\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.300861 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.300841 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm8d9\" (UniqueName: \"kubernetes.io/projected/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-kube-api-access-nm8d9\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.300917 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.300878 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-token\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.311418 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.311397 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:05.402443 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.402402 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-credential-socket\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.402570 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.402477 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-podinfo\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.402570 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.402505 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-workload-socket\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.402570 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.402532 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istiod-ca-cert\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.402712 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.402573 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm8d9\" (UniqueName: \"kubernetes.io/projected/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-kube-api-access-nm8d9\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.402712 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.402607 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-token\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.402712 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.402679 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-workload-certs\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.402872 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.402718 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-data\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.402872 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.402778 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-credential-socket\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.402872 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.402786 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-envoy\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.403084 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.403064 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-workload-socket\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.403379 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.403360 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-data\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.403471 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.403360 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-workload-certs\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.403927 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.403908 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istiod-ca-cert\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.405694 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.405628 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-podinfo\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.405796 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.405784 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-envoy\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.422864 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.422839 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm8d9\" (UniqueName: \"kubernetes.io/projected/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-kube-api-access-nm8d9\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.425381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.425356 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-token\") pod \"e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.567649 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.567566 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:05.612107 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.612083 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb"] Nov 03 19:47:05.614838 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:47:05.614806 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329073b3_fd81_4e4c_8ea6_cebbd5f45c7d.slice/crio-536d9ffc69aeb979adc1ca1d0c71c3c79e3d533186acea52f136d16f2de74046 WatchSource:0}: Error finding container 536d9ffc69aeb979adc1ca1d0c71c3c79e3d533186acea52f136d16f2de74046: Status 404 returned error can't find the container with id 536d9ffc69aeb979adc1ca1d0c71c3c79e3d533186acea52f136d16f2de74046 Nov 03 19:47:05.617401 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.617359 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:47:05.617561 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.617435 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:47:05.617561 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.617471 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:47:05.848724 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.848702 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx"] Nov 03 19:47:05.851226 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:47:05.851196 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbce3ead_79e2_4d3e_8caa_8a35a3640e75.slice/crio-80c1a4918c7f81e626f5a44c16094248783d8140ce796e04bd90a35988d1396d WatchSource:0}: Error finding container 80c1a4918c7f81e626f5a44c16094248783d8140ce796e04bd90a35988d1396d: Status 404 returned error can't find the container with id 80c1a4918c7f81e626f5a44c16094248783d8140ce796e04bd90a35988d1396d Nov 03 19:47:05.853959 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.853924 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:47:05.854051 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.853983 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:47:05.854051 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.854010 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:47:05.857696 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.857645 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" event={"ID":"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d","Type":"ContainerStarted","Data":"b2c8777e659022d83bc00a050326e45373018d9c4d228874596bf49434c6a751"} Nov 03 19:47:05.857786 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.857725 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" event={"ID":"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d","Type":"ContainerStarted","Data":"536d9ffc69aeb979adc1ca1d0c71c3c79e3d533186acea52f136d16f2de74046"} Nov 03 19:47:05.879221 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:05.879180 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" podStartSLOduration=1.879167964 podStartE2EDuration="1.879167964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:05.876548821 +0000 UTC m=+620.134346077" watchObservedRunningTime="2026-11-03 19:47:05.879167964 +0000 UTC m=+620.136965221" Nov 03 19:47:06.312220 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:06.312132 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:06.317130 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:06.317106 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:06.860144 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:06.860101 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" event={"ID":"cbce3ead-79e2-4d3e-8caa-8a35a3640e75","Type":"ContainerStarted","Data":"cc59a5dc60887acb9a2ac3411bb69ac6aace5f5f03de774f152d126699d06328"} Nov 03 19:47:06.860341 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:06.860156 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" event={"ID":"cbce3ead-79e2-4d3e-8caa-8a35a3640e75","Type":"ContainerStarted","Data":"80c1a4918c7f81e626f5a44c16094248783d8140ce796e04bd90a35988d1396d"} Nov 03 19:47:06.860341 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:06.860317 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:06.861291 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:06.861271 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:06.883878 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:06.883818 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" podStartSLOduration=1.883803289 podStartE2EDuration="1.883803289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:06.883102312 +0000 UTC m=+621.140899570" watchObservedRunningTime="2026-11-03 19:47:06.883803289 +0000 UTC m=+621.141600547" Nov 03 19:47:07.568211 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:07.568172 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:07.572950 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:07.572923 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:07.872594 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:07.872562 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:07.873535 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:07.873511 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:10.281269 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.281238 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b"] Nov 03 19:47:10.284794 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.284777 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b" Nov 03 19:47:10.296094 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.295984 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-9959e5f4\"" Nov 03 19:47:10.297438 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.297405 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-9959e5f4-dockercfg-bvqpc\"" Nov 03 19:47:10.299787 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.299766 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b"] Nov 03 19:47:10.329139 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.329113 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp"] Nov 03 19:47:10.332610 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.332595 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp" Nov 03 19:47:10.348395 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.348372 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp"] Nov 03 19:47:10.356293 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.356264 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695e7ed2-53a7-4add-b4ba-74effcc52faf-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b\" (UID: \"695e7ed2-53a7-4add-b4ba-74effcc52faf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b" Nov 03 19:47:10.356476 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.356451 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5wjn\" (UniqueName: \"kubernetes.io/projected/695e7ed2-53a7-4add-b4ba-74effcc52faf-kube-api-access-t5wjn\") pod \"payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b\" (UID: \"695e7ed2-53a7-4add-b4ba-74effcc52faf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b" Nov 03 19:47:10.457731 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.457695 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzvcq\" (UniqueName: \"kubernetes.io/projected/d1824e75-de08-4007-8ff9-200f3bb369d3-kube-api-access-jzvcq\") pod \"payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp\" (UID: \"d1824e75-de08-4007-8ff9-200f3bb369d3\") " pod="openshift-ingress/payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp" Nov 03 19:47:10.457896 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.457841 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5wjn\" (UniqueName: \"kubernetes.io/projected/695e7ed2-53a7-4add-b4ba-74effcc52faf-kube-api-access-t5wjn\") pod \"payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b\" (UID: \"695e7ed2-53a7-4add-b4ba-74effcc52faf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b" Nov 03 19:47:10.457896 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.457872 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1824e75-de08-4007-8ff9-200f3bb369d3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp\" (UID: \"d1824e75-de08-4007-8ff9-200f3bb369d3\") " pod="openshift-ingress/payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp" Nov 03 19:47:10.458072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.457952 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695e7ed2-53a7-4add-b4ba-74effcc52faf-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b\" (UID: \"695e7ed2-53a7-4add-b4ba-74effcc52faf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b" Nov 03 19:47:10.458626 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.458606 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695e7ed2-53a7-4add-b4ba-74effcc52faf-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b\" (UID: \"695e7ed2-53a7-4add-b4ba-74effcc52faf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b" Nov 03 19:47:10.473210 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.473187 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5wjn\" (UniqueName: \"kubernetes.io/projected/695e7ed2-53a7-4add-b4ba-74effcc52faf-kube-api-access-t5wjn\") pod \"payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b\" (UID: \"695e7ed2-53a7-4add-b4ba-74effcc52faf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b" Nov 03 19:47:10.559009 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.558947 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzvcq\" (UniqueName: \"kubernetes.io/projected/d1824e75-de08-4007-8ff9-200f3bb369d3-kube-api-access-jzvcq\") pod \"payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp\" (UID: \"d1824e75-de08-4007-8ff9-200f3bb369d3\") " pod="openshift-ingress/payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp" Nov 03 19:47:10.559124 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.559010 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1824e75-de08-4007-8ff9-200f3bb369d3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp\" (UID: \"d1824e75-de08-4007-8ff9-200f3bb369d3\") " pod="openshift-ingress/payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp" Nov 03 19:47:10.559683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.559647 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1824e75-de08-4007-8ff9-200f3bb369d3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp\" (UID: \"d1824e75-de08-4007-8ff9-200f3bb369d3\") " pod="openshift-ingress/payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp" Nov 03 19:47:10.571598 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.571572 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzvcq\" (UniqueName: \"kubernetes.io/projected/d1824e75-de08-4007-8ff9-200f3bb369d3-kube-api-access-jzvcq\") pod \"payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp\" (UID: \"d1824e75-de08-4007-8ff9-200f3bb369d3\") " pod="openshift-ingress/payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp" Nov 03 19:47:10.597729 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.597705 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b" Nov 03 19:47:10.602969 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:47:10.602939 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:47:10.648977 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.647224 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp" Nov 03 19:47:10.874094 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.874071 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b"] Nov 03 19:47:10.875642 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:47:10.875613 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695e7ed2_53a7_4add_b4ba_74effcc52faf.slice/crio-fbc080ac2e861e62858f1768dfe6fc85ac911b093fafe9d835aa176aaafd0368 WatchSource:0}: Error finding container fbc080ac2e861e62858f1768dfe6fc85ac911b093fafe9d835aa176aaafd0368: Status 404 returned error can't find the container with id fbc080ac2e861e62858f1768dfe6fc85ac911b093fafe9d835aa176aaafd0368 Nov 03 19:47:10.877715 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.877695 5059 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:47:10.887876 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.887853 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b" event={"ID":"695e7ed2-53a7-4add-b4ba-74effcc52faf","Type":"ContainerStarted","Data":"fbc080ac2e861e62858f1768dfe6fc85ac911b093fafe9d835aa176aaafd0368"} Nov 03 19:47:10.944156 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:10.944132 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp"] Nov 03 19:47:10.945521 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:47:10.945497 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1824e75_de08_4007_8ff9_200f3bb369d3.slice/crio-03566886203db2d622177761b14acdb8e200d37b3eb124d01f09f4ba2212462d WatchSource:0}: Error finding container 03566886203db2d622177761b14acdb8e200d37b3eb124d01f09f4ba2212462d: Status 404 returned error can't find the container with id 03566886203db2d622177761b14acdb8e200d37b3eb124d01f09f4ba2212462d Nov 03 19:47:11.273361 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:11.273290 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh"] Nov 03 19:47:11.278350 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:11.278329 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh" Nov 03 19:47:11.283961 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:11.283593 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3f7e10b8\"" Nov 03 19:47:11.285104 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:11.285065 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3f7e10b8-dockercfg-6m2qs\"" Nov 03 19:47:11.287316 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:11.286530 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh"] Nov 03 19:47:11.308684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:11.308645 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf"] Nov 03 19:47:11.312999 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:11.312979 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf" Nov 03 19:47:11.321611 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:11.321583 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf"] Nov 03 19:47:11.371983 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:11.371931 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087fa86d-7f7a-45a8-b368-b0052d5f5ec0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh\" (UID: \"087fa86d-7f7a-45a8-b368-b0052d5f5ec0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh" Nov 03 19:47:11.372170 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:11.372124 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f57f\" (UniqueName: \"kubernetes.io/projected/087fa86d-7f7a-45a8-b368-b0052d5f5ec0-kube-api-access-8f57f\") pod \"payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh\" (UID: \"087fa86d-7f7a-45a8-b368-b0052d5f5ec0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh" Nov 03 19:47:11.473101 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:11.473063 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzlkl\" (UniqueName: \"kubernetes.io/projected/f337606e-fc9b-43c6-8ac4-205f2e011dde-kube-api-access-kzlkl\") pod \"payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf\" (UID: \"f337606e-fc9b-43c6-8ac4-205f2e011dde\") " pod="openshift-ingress/payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf" Nov 03 19:47:11.473273 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:11.473206 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087fa86d-7f7a-45a8-b368-b0052d5f5ec0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh\" (UID: \"087fa86d-7f7a-45a8-b368-b0052d5f5ec0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh" Nov 03 19:47:11.473273 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:11.473261 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f337606e-fc9b-43c6-8ac4-205f2e011dde-plugins-config-volume\") pod \"payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf\" (UID: \"f337606e-fc9b-43c6-8ac4-205f2e011dde\") " pod="openshift-ingress/payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf" Nov 03 19:47:11.473702 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:11.473397 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f57f\" (UniqueName: \"kubernetes.io/projected/087fa86d-7f7a-45a8-b368-b0052d5f5ec0-kube-api-access-8f57f\") pod \"payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh\" (UID: \"087fa86d-7f7a-45a8-b368-b0052d5f5ec0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh" Nov 03 19:47:11.474100 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:11.474064 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087fa86d-7f7a-45a8-b368-b0052d5f5ec0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh\" (UID: \"087fa86d-7f7a-45a8-b368-b0052d5f5ec0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh" Nov 03 19:47:11.486491 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:11.486467 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f57f\" (UniqueName: \"kubernetes.io/projected/087fa86d-7f7a-45a8-b368-b0052d5f5ec0-kube-api-access-8f57f\") pod \"payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh\" (UID: \"087fa86d-7f7a-45a8-b368-b0052d5f5ec0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh" Nov 03 19:47:11.574546 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:11.574276 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzlkl\" (UniqueName: \"kubernetes.io/projected/f337606e-fc9b-43c6-8ac4-205f2e011dde-kube-api-access-kzlkl\") pod \"payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf\" (UID: \"f337606e-fc9b-43c6-8ac4-205f2e011dde\") " pod="openshift-ingress/payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf" Nov 03 19:47:11.574546 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:11.574363 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f337606e-fc9b-43c6-8ac4-205f2e011dde-plugins-config-volume\") pod \"payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf\" (UID: \"f337606e-fc9b-43c6-8ac4-205f2e011dde\") " pod="openshift-ingress/payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf" Nov 03 19:47:11.579685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:11.575281 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f337606e-fc9b-43c6-8ac4-205f2e011dde-plugins-config-volume\") pod \"payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf\" (UID: \"f337606e-fc9b-43c6-8ac4-205f2e011dde\") " pod="openshift-ingress/payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf" Nov 03 19:47:11.602637 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:11.602399 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh" Nov 03 19:47:11.602890 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:11.602863 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzlkl\" (UniqueName: \"kubernetes.io/projected/f337606e-fc9b-43c6-8ac4-205f2e011dde-kube-api-access-kzlkl\") pod \"payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf\" (UID: \"f337606e-fc9b-43c6-8ac4-205f2e011dde\") " pod="openshift-ingress/payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf" Nov 03 19:47:11.629381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:11.629356 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf" Nov 03 19:47:11.897469 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:11.897330 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp" event={"ID":"d1824e75-de08-4007-8ff9-200f3bb369d3","Type":"ContainerStarted","Data":"03566886203db2d622177761b14acdb8e200d37b3eb124d01f09f4ba2212462d"} Nov 03 19:47:11.959592 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:11.959558 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh"] Nov 03 19:47:11.965393 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:47:11.965346 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087fa86d_7f7a_45a8_b368_b0052d5f5ec0.slice/crio-0ca372c1281d9058bf76ae8e15097135e00416887c56f9bee86728e973dbff84 WatchSource:0}: Error finding container 0ca372c1281d9058bf76ae8e15097135e00416887c56f9bee86728e973dbff84: Status 404 returned error can't find the container with id 0ca372c1281d9058bf76ae8e15097135e00416887c56f9bee86728e973dbff84 Nov 03 19:47:12.004746 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:12.004673 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf"] Nov 03 19:47:12.007097 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:47:12.007067 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf337606e_fc9b_43c6_8ac4_205f2e011dde.slice/crio-15ea57f997225d69d43afb1aeaf47701842b34016040f98784269ca61cfe0497 WatchSource:0}: Error finding container 15ea57f997225d69d43afb1aeaf47701842b34016040f98784269ca61cfe0497: Status 404 returned error can't find the container with id 15ea57f997225d69d43afb1aeaf47701842b34016040f98784269ca61cfe0497 Nov 03 19:47:12.909108 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:12.909021 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf" event={"ID":"f337606e-fc9b-43c6-8ac4-205f2e011dde","Type":"ContainerStarted","Data":"15ea57f997225d69d43afb1aeaf47701842b34016040f98784269ca61cfe0497"} Nov 03 19:47:12.911056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:12.910988 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh" event={"ID":"087fa86d-7f7a-45a8-b368-b0052d5f5ec0","Type":"ContainerStarted","Data":"0ca372c1281d9058bf76ae8e15097135e00416887c56f9bee86728e973dbff84"} Nov 03 19:47:13.772011 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:13.770637 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv"] Nov 03 19:47:13.786083 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:13.785372 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:13.787222 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:13.787127 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv"] Nov 03 19:47:13.793524 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:13.793491 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-8c14f244-openshift-default-dockercfg-9zgpv\"" Nov 03 19:47:13.909137 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:13.909097 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-workload-socket\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:13.909529 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:13.909214 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-istio-envoy\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:13.909529 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:13.909319 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-workload-certs\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:13.909529 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:13.909354 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/497df71a-3372-4792-9cc4-980950a8de12-istiod-ca-cert\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:13.909529 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:13.909371 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-credential-socket\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:13.909529 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:13.909398 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/497df71a-3372-4792-9cc4-980950a8de12-istio-token\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:13.909529 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:13.909440 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-istio-data\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:13.909529 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:13.909457 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czwh7\" (UniqueName: \"kubernetes.io/projected/497df71a-3372-4792-9cc4-980950a8de12-kube-api-access-czwh7\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:13.909529 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:13.909490 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/497df71a-3372-4792-9cc4-980950a8de12-istio-podinfo\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:14.012080 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.010916 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/497df71a-3372-4792-9cc4-980950a8de12-istio-token\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:14.012080 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.011007 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-istio-data\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:14.012080 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.011041 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czwh7\" (UniqueName: \"kubernetes.io/projected/497df71a-3372-4792-9cc4-980950a8de12-kube-api-access-czwh7\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:14.012080 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.011096 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/497df71a-3372-4792-9cc4-980950a8de12-istio-podinfo\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:14.012080 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.011145 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-workload-socket\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:14.012080 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.011185 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-istio-envoy\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:14.012080 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.011304 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-workload-certs\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:14.012080 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.011358 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/497df71a-3372-4792-9cc4-980950a8de12-istiod-ca-cert\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:14.012080 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.011386 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-credential-socket\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:14.012080 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.011818 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-credential-socket\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:14.012080 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.011917 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-istio-data\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:14.012999 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.012931 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-workload-certs\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:14.012999 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.012957 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-workload-socket\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:14.013174 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.013153 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/497df71a-3372-4792-9cc4-980950a8de12-istiod-ca-cert\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:14.017695 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.015255 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/497df71a-3372-4792-9cc4-980950a8de12-istio-podinfo\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:14.021879 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.021855 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-istio-envoy\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:14.024839 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.024787 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/497df71a-3372-4792-9cc4-980950a8de12-istio-token\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:14.025169 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.025126 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czwh7\" (UniqueName: \"kubernetes.io/projected/497df71a-3372-4792-9cc4-980950a8de12-kube-api-access-czwh7\") pod \"e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:14.110047 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.109818 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:14.437353 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.437313 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv"] Nov 03 19:47:14.440469 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.440340 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:47:14.440469 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.440428 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:47:14.440839 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.440494 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:47:14.926617 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.926547 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b" event={"ID":"695e7ed2-53a7-4add-b4ba-74effcc52faf","Type":"ContainerStarted","Data":"c137788f9c8799ab826a726643a2f95bc2a243573463da0cef4a805c16354b09"} Nov 03 19:47:14.927099 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.926738 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b" Nov 03 19:47:14.928994 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.928963 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" event={"ID":"497df71a-3372-4792-9cc4-980950a8de12","Type":"ContainerStarted","Data":"512a43fcddf2d69a21758c6b29f9d5edb489c0faf84aaeea5a81243ac933224f"} Nov 03 19:47:14.929133 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.929016 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" event={"ID":"497df71a-3372-4792-9cc4-980950a8de12","Type":"ContainerStarted","Data":"84119ccf9aa5290f6ac0565371f91a6952b59f3355f3c2ffc84ba51b025f5a64"} Nov 03 19:47:14.931820 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.931797 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp" event={"ID":"d1824e75-de08-4007-8ff9-200f3bb369d3","Type":"ContainerStarted","Data":"afd4e3bafa034ca292543211cd99e43e6c1305ec32edbc5be195052cf1e9d34d"} Nov 03 19:47:14.931991 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.931971 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp" Nov 03 19:47:14.948557 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.948504 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b" podStartSLOduration=1.322236628 podStartE2EDuration="4.948488527s" totalImagesPullingTime="3.626251899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:14.946071066 +0000 UTC m=+629.203868329" watchObservedRunningTime="2026-11-03 19:47:14.948488527 +0000 UTC m=+629.206285796" Nov 03 19:47:14.965394 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.965308 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp" podStartSLOduration=1.402672635 podStartE2EDuration="4.965297081s" totalImagesPullingTime="3.562624446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:14.964055028 +0000 UTC m=+629.221852287" watchObservedRunningTime="2026-11-03 19:47:14.965297081 +0000 UTC m=+629.223094340" Nov 03 19:47:14.993210 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:14.993168 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" podStartSLOduration=1.993153079 podStartE2EDuration="1.993153079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:14.991018424 +0000 UTC m=+629.248815680" watchObservedRunningTime="2026-11-03 19:47:14.993153079 +0000 UTC m=+629.250950374" Nov 03 19:47:15.111117 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:15.111026 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:15.129382 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:15.129345 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:16.019287 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:16.018293 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf" event={"ID":"f337606e-fc9b-43c6-8ac4-205f2e011dde","Type":"ContainerStarted","Data":"4a87a2828e5c0893df365342163e5e0af79fb4f530a6e6eae56429e17ff408d9"} Nov 03 19:47:16.019287 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:16.019229 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf" Nov 03 19:47:16.024756 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:16.023211 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh" event={"ID":"087fa86d-7f7a-45a8-b368-b0052d5f5ec0","Type":"ContainerStarted","Data":"fdbbd7c38271db09868eb23ad9c7d917e41b8e03ebd2717394e707ddbe1fc99d"} Nov 03 19:47:16.024756 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:16.023261 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:16.024756 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:16.023288 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh" Nov 03 19:47:16.031450 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:16.031428 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:16.041222 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:16.041182 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf" podStartSLOduration=1.84647713 podStartE2EDuration="5.041168641s" totalImagesPullingTime="3.194691511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:16.039935927 +0000 UTC m=+630.297733197" watchObservedRunningTime="2026-11-03 19:47:16.041168641 +0000 UTC m=+630.298965894" Nov 03 19:47:16.084711 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:16.084488 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh" podStartSLOduration=1.872424177 podStartE2EDuration="5.084468162s" totalImagesPullingTime="3.212043985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:16.057588211 +0000 UTC m=+630.315385468" watchObservedRunningTime="2026-11-03 19:47:16.084468162 +0000 UTC m=+630.342265420" Nov 03 19:47:17.809482 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:17.809442 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m"] Nov 03 19:47:17.814014 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:17.813994 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m" Nov 03 19:47:17.831912 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:17.831889 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m"] Nov 03 19:47:17.832335 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:17.832294 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-8c14f244\"" Nov 03 19:47:17.832474 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:17.832344 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-8c14f244-dockercfg-pcpfw\"" Nov 03 19:47:17.901328 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:17.901236 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn"] Nov 03 19:47:17.907714 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:17.907466 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn"] Nov 03 19:47:17.907714 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:17.907586 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn" Nov 03 19:47:17.909056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:17.909023 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c49a14-9ecc-4de2-a81d-94621505ca70-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m\" (UID: \"d1c49a14-9ecc-4de2-a81d-94621505ca70\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m" Nov 03 19:47:17.909164 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:17.909097 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmcl7\" (UniqueName: \"kubernetes.io/projected/3f7e4716-e0cb-4170-82f4-aa74ef088ef3-kube-api-access-cmcl7\") pod \"payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn\" (UID: \"3f7e4716-e0cb-4170-82f4-aa74ef088ef3\") " pod="openshift-ingress/payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn" Nov 03 19:47:17.909213 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:17.909199 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7e4716-e0cb-4170-82f4-aa74ef088ef3-plugins-config-volume\") pod \"payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn\" (UID: \"3f7e4716-e0cb-4170-82f4-aa74ef088ef3\") " pod="openshift-ingress/payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn" Nov 03 19:47:17.909263 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:17.909247 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhp66\" (UniqueName: \"kubernetes.io/projected/d1c49a14-9ecc-4de2-a81d-94621505ca70-kube-api-access-xhp66\") pod \"payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m\" (UID: \"d1c49a14-9ecc-4de2-a81d-94621505ca70\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m" Nov 03 19:47:17.927356 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:47:17.925495 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7e4716_e0cb_4170_82f4_aa74ef088ef3.slice\": RecentStats: unable to find data in memory cache]" Nov 03 19:47:18.010757 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:18.010582 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7e4716-e0cb-4170-82f4-aa74ef088ef3-plugins-config-volume\") pod \"payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn\" (UID: \"3f7e4716-e0cb-4170-82f4-aa74ef088ef3\") " pod="openshift-ingress/payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn" Nov 03 19:47:18.010757 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:18.010631 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhp66\" (UniqueName: \"kubernetes.io/projected/d1c49a14-9ecc-4de2-a81d-94621505ca70-kube-api-access-xhp66\") pod \"payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m\" (UID: \"d1c49a14-9ecc-4de2-a81d-94621505ca70\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m" Nov 03 19:47:18.010757 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:18.010695 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c49a14-9ecc-4de2-a81d-94621505ca70-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m\" (UID: \"d1c49a14-9ecc-4de2-a81d-94621505ca70\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m" Nov 03 19:47:18.011444 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:18.011082 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmcl7\" (UniqueName: \"kubernetes.io/projected/3f7e4716-e0cb-4170-82f4-aa74ef088ef3-kube-api-access-cmcl7\") pod \"payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn\" (UID: \"3f7e4716-e0cb-4170-82f4-aa74ef088ef3\") " pod="openshift-ingress/payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn" Nov 03 19:47:18.011611 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:18.011533 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7e4716-e0cb-4170-82f4-aa74ef088ef3-plugins-config-volume\") pod \"payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn\" (UID: \"3f7e4716-e0cb-4170-82f4-aa74ef088ef3\") " pod="openshift-ingress/payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn" Nov 03 19:47:18.011977 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:18.011951 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c49a14-9ecc-4de2-a81d-94621505ca70-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m\" (UID: \"d1c49a14-9ecc-4de2-a81d-94621505ca70\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m" Nov 03 19:47:18.052598 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:18.052140 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhp66\" (UniqueName: \"kubernetes.io/projected/d1c49a14-9ecc-4de2-a81d-94621505ca70-kube-api-access-xhp66\") pod \"payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m\" (UID: \"d1c49a14-9ecc-4de2-a81d-94621505ca70\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m" Nov 03 19:47:18.055818 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:18.054759 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmcl7\" (UniqueName: \"kubernetes.io/projected/3f7e4716-e0cb-4170-82f4-aa74ef088ef3-kube-api-access-cmcl7\") pod \"payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn\" (UID: \"3f7e4716-e0cb-4170-82f4-aa74ef088ef3\") " pod="openshift-ingress/payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn" Nov 03 19:47:18.128725 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:18.128696 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m" Nov 03 19:47:18.241586 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:18.228378 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn" Nov 03 19:47:18.451647 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:18.451579 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m"] Nov 03 19:47:18.454338 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:47:18.454297 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c49a14_9ecc_4de2_a81d_94621505ca70.slice/crio-a8d15d1a1a9f8a25f1b7e2976211b4c10fe2af65e3035dada57725a6579b2773 WatchSource:0}: Error finding container a8d15d1a1a9f8a25f1b7e2976211b4c10fe2af65e3035dada57725a6579b2773: Status 404 returned error can't find the container with id a8d15d1a1a9f8a25f1b7e2976211b4c10fe2af65e3035dada57725a6579b2773 Nov 03 19:47:18.623310 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:18.623283 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn"] Nov 03 19:47:18.624844 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:47:18.624813 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7e4716_e0cb_4170_82f4_aa74ef088ef3.slice/crio-7d90f7b653c24e702c89ec09b167286d40b3ce05062f62ac554503fa344973fe WatchSource:0}: Error finding container 7d90f7b653c24e702c89ec09b167286d40b3ce05062f62ac554503fa344973fe: Status 404 returned error can't find the container with id 7d90f7b653c24e702c89ec09b167286d40b3ce05062f62ac554503fa344973fe Nov 03 19:47:19.081690 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:19.077341 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn" event={"ID":"3f7e4716-e0cb-4170-82f4-aa74ef088ef3","Type":"ContainerStarted","Data":"b53bf546a51a84e4668237abf191a7f8072957a666b12f2b83bceaf5b150492a"} Nov 03 19:47:19.081690 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:19.077406 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn" event={"ID":"3f7e4716-e0cb-4170-82f4-aa74ef088ef3","Type":"ContainerStarted","Data":"7d90f7b653c24e702c89ec09b167286d40b3ce05062f62ac554503fa344973fe"} Nov 03 19:47:19.082545 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:19.082289 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn" Nov 03 19:47:19.084981 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:19.084467 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m" event={"ID":"d1c49a14-9ecc-4de2-a81d-94621505ca70","Type":"ContainerStarted","Data":"5c84b0527753f42c973b20350fa943627a491bc9f4cd2f01e6c16a9dca20b405"} Nov 03 19:47:19.084981 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:19.084505 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m" event={"ID":"d1c49a14-9ecc-4de2-a81d-94621505ca70","Type":"ContainerStarted","Data":"a8d15d1a1a9f8a25f1b7e2976211b4c10fe2af65e3035dada57725a6579b2773"} Nov 03 19:47:19.084981 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:19.084729 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m" Nov 03 19:47:19.101457 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:19.101413 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn" podStartSLOduration=2.101401196 podStartE2EDuration="2.101401196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:19.100605006 +0000 UTC m=+633.358402267" watchObservedRunningTime="2026-11-03 19:47:19.101401196 +0000 UTC m=+633.359198453" Nov 03 19:47:19.119422 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:19.119367 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m" podStartSLOduration=2.119352608 podStartE2EDuration="2.119352608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:19.118143515 +0000 UTC m=+633.375940771" watchObservedRunningTime="2026-11-03 19:47:19.119352608 +0000 UTC m=+633.377149865" Nov 03 19:47:23.931803 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:23.931766 5059 patch_prober.go:29] interesting pod/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:47:23.932197 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:23.931819 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" podUID="cbce3ead-79e2-4d3e-8caa-8a35a3640e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:26.027405 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:26.027326 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b" Nov 03 19:47:26.027405 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:26.027396 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp" Nov 03 19:47:26.135651 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:26.135611 5059 generic.go:356] "Generic (PLEG): container finished" podID="cbce3ead-79e2-4d3e-8caa-8a35a3640e75" containerID="cc59a5dc60887acb9a2ac3411bb69ac6aace5f5f03de774f152d126699d06328" exitCode=0 Nov 03 19:47:26.135867 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:26.135657 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" event={"ID":"cbce3ead-79e2-4d3e-8caa-8a35a3640e75","Type":"ContainerDied","Data":"cc59a5dc60887acb9a2ac3411bb69ac6aace5f5f03de774f152d126699d06328"} Nov 03 19:47:26.136369 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:26.136348 5059 scope.go:122] "RemoveContainer" containerID="cc59a5dc60887acb9a2ac3411bb69ac6aace5f5f03de774f152d126699d06328" Nov 03 19:47:26.137897 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:26.137860 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:47:26.138058 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:26.137952 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:47:26.138058 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:26.138003 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:47:27.034616 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:27.034587 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh" Nov 03 19:47:27.151894 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:27.151858 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" event={"ID":"cbce3ead-79e2-4d3e-8caa-8a35a3640e75","Type":"ContainerStarted","Data":"d37a4cd3eecbd37a2b70844c7ae297fc283a99464149be8dea809712753f7a83"} Nov 03 19:47:27.570115 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:27.567797 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:27.570115 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:27.568212 5059 patch_prober.go:29] interesting pod/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 19:47:27.570115 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:27.568274 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" podUID="cbce3ead-79e2-4d3e-8caa-8a35a3640e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 19:47:28.082752 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.078959 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf" Nov 03 19:47:28.145954 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.145923 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6"] Nov 03 19:47:28.150596 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.150573 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6" Nov 03 19:47:28.155789 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.155766 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 19:47:28.156038 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.156023 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:47:28.162691 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.160016 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6"] Nov 03 19:47:28.221730 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.203240 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32fecea-2421-4b40-81b4-cada3ad747b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6\" (UID: \"a32fecea-2421-4b40-81b4-cada3ad747b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6" Nov 03 19:47:28.221730 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.203324 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lx47\" (UniqueName: \"kubernetes.io/projected/a32fecea-2421-4b40-81b4-cada3ad747b0-kube-api-access-5lx47\") pod \"maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6\" (UID: \"a32fecea-2421-4b40-81b4-cada3ad747b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6" Nov 03 19:47:28.221730 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.203374 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32fecea-2421-4b40-81b4-cada3ad747b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6\" (UID: \"a32fecea-2421-4b40-81b4-cada3ad747b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6" Nov 03 19:47:28.305245 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.305207 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32fecea-2421-4b40-81b4-cada3ad747b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6\" (UID: \"a32fecea-2421-4b40-81b4-cada3ad747b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6" Nov 03 19:47:28.305594 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.305571 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lx47\" (UniqueName: \"kubernetes.io/projected/a32fecea-2421-4b40-81b4-cada3ad747b0-kube-api-access-5lx47\") pod \"maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6\" (UID: \"a32fecea-2421-4b40-81b4-cada3ad747b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6" Nov 03 19:47:28.305750 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.305733 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32fecea-2421-4b40-81b4-cada3ad747b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6\" (UID: \"a32fecea-2421-4b40-81b4-cada3ad747b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6" Nov 03 19:47:28.306784 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.306756 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32fecea-2421-4b40-81b4-cada3ad747b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6\" (UID: \"a32fecea-2421-4b40-81b4-cada3ad747b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6" Nov 03 19:47:28.310291 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.310269 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32fecea-2421-4b40-81b4-cada3ad747b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6\" (UID: \"a32fecea-2421-4b40-81b4-cada3ad747b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6" Nov 03 19:47:28.319970 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.319946 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lx47\" (UniqueName: \"kubernetes.io/projected/a32fecea-2421-4b40-81b4-cada3ad747b0-kube-api-access-5lx47\") pod \"maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6\" (UID: \"a32fecea-2421-4b40-81b4-cada3ad747b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6" Nov 03 19:47:28.519420 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.519337 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6" Nov 03 19:47:28.588191 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.588158 5059 patch_prober.go:29] interesting pod/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 19:47:28.588481 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.588444 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" podUID="cbce3ead-79e2-4d3e-8caa-8a35a3640e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 19:47:28.651072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.650056 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl"] Nov 03 19:47:28.723350 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.723317 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl" Nov 03 19:47:28.724580 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.724159 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl"] Nov 03 19:47:28.813577 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.813474 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f78ce38-1dbf-42b1-bd02-510fe259b11c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl\" (UID: \"6f78ce38-1dbf-42b1-bd02-510fe259b11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl" Nov 03 19:47:28.813577 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.813553 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f78ce38-1dbf-42b1-bd02-510fe259b11c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl\" (UID: \"6f78ce38-1dbf-42b1-bd02-510fe259b11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl" Nov 03 19:47:28.816282 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.813610 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm4l4\" (UniqueName: \"kubernetes.io/projected/6f78ce38-1dbf-42b1-bd02-510fe259b11c-kube-api-access-dm4l4\") pod \"maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl\" (UID: \"6f78ce38-1dbf-42b1-bd02-510fe259b11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl" Nov 03 19:47:28.847869 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.847837 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6"] Nov 03 19:47:28.852850 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:47:28.852808 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda32fecea_2421_4b40_81b4_cada3ad747b0.slice/crio-00ebe954989cb847d6cdcc56edca6faed648ee2db90342d4f9aebd5f119a4f3e WatchSource:0}: Error finding container 00ebe954989cb847d6cdcc56edca6faed648ee2db90342d4f9aebd5f119a4f3e: Status 404 returned error can't find the container with id 00ebe954989cb847d6cdcc56edca6faed648ee2db90342d4f9aebd5f119a4f3e Nov 03 19:47:28.918361 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.914740 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f78ce38-1dbf-42b1-bd02-510fe259b11c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl\" (UID: \"6f78ce38-1dbf-42b1-bd02-510fe259b11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl" Nov 03 19:47:28.918361 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.915245 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f78ce38-1dbf-42b1-bd02-510fe259b11c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl\" (UID: \"6f78ce38-1dbf-42b1-bd02-510fe259b11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl" Nov 03 19:47:28.918361 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.915317 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm4l4\" (UniqueName: \"kubernetes.io/projected/6f78ce38-1dbf-42b1-bd02-510fe259b11c-kube-api-access-dm4l4\") pod \"maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl\" (UID: \"6f78ce38-1dbf-42b1-bd02-510fe259b11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl" Nov 03 19:47:28.918361 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.916004 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f78ce38-1dbf-42b1-bd02-510fe259b11c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl\" (UID: \"6f78ce38-1dbf-42b1-bd02-510fe259b11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl" Nov 03 19:47:28.918361 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.917643 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f78ce38-1dbf-42b1-bd02-510fe259b11c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl\" (UID: \"6f78ce38-1dbf-42b1-bd02-510fe259b11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl" Nov 03 19:47:28.927395 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:28.927364 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm4l4\" (UniqueName: \"kubernetes.io/projected/6f78ce38-1dbf-42b1-bd02-510fe259b11c-kube-api-access-dm4l4\") pod \"maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl\" (UID: \"6f78ce38-1dbf-42b1-bd02-510fe259b11c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl" Nov 03 19:47:29.041323 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:29.041293 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl" Nov 03 19:47:29.197801 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:29.197756 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6" event={"ID":"a32fecea-2421-4b40-81b4-cada3ad747b0","Type":"ContainerStarted","Data":"00ebe954989cb847d6cdcc56edca6faed648ee2db90342d4f9aebd5f119a4f3e"} Nov 03 19:47:29.347653 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:29.347625 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl"] Nov 03 19:47:29.348627 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:47:29.348597 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f78ce38_1dbf_42b1_bd02_510fe259b11c.slice/crio-d3698f6faaa26584860650c298f45ffd1913ba3c802e7683cac8a4f062550ba9 WatchSource:0}: Error finding container d3698f6faaa26584860650c298f45ffd1913ba3c802e7683cac8a4f062550ba9: Status 404 returned error can't find the container with id d3698f6faaa26584860650c298f45ffd1913ba3c802e7683cac8a4f062550ba9 Nov 03 19:47:29.578685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:29.577759 5059 patch_prober.go:29] interesting pod/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 19:47:29.578685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:29.577820 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" podUID="cbce3ead-79e2-4d3e-8caa-8a35a3640e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 19:47:30.093712 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:30.093676 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn" Nov 03 19:47:30.095068 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:30.095042 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m" Nov 03 19:47:30.207802 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:30.207652 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl" event={"ID":"6f78ce38-1dbf-42b1-bd02-510fe259b11c","Type":"ContainerStarted","Data":"d3698f6faaa26584860650c298f45ffd1913ba3c802e7683cac8a4f062550ba9"} Nov 03 19:47:30.587739 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:30.571037 5059 patch_prober.go:29] interesting pod/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 19:47:30.587739 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:30.571092 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" podUID="cbce3ead-79e2-4d3e-8caa-8a35a3640e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 19:47:31.581687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:31.581065 5059 patch_prober.go:29] interesting pod/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 19:47:31.581687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:31.581128 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" podUID="cbce3ead-79e2-4d3e-8caa-8a35a3640e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 19:47:32.082754 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:32.081781 5059 patch_prober.go:29] interesting pod/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:47:32.082754 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:32.081844 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" podUID="497df71a-3372-4792-9cc4-980950a8de12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:32.572985 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:32.572949 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:32.573885 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:32.573712 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:32.574472 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:32.574420 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:33.231180 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:33.231111 5059 generic.go:356] "Generic (PLEG): container finished" podID="329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" containerID="b2c8777e659022d83bc00a050326e45373018d9c4d228874596bf49434c6a751" exitCode=0 Nov 03 19:47:33.231499 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:33.231183 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" event={"ID":"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d","Type":"ContainerDied","Data":"b2c8777e659022d83bc00a050326e45373018d9c4d228874596bf49434c6a751"} Nov 03 19:47:33.231770 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:33.231753 5059 scope.go:122] "RemoveContainer" containerID="b2c8777e659022d83bc00a050326e45373018d9c4d228874596bf49434c6a751" Nov 03 19:47:33.232953 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:33.232920 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:47:33.233061 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:33.233005 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:47:33.233061 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:33.233038 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:47:33.637067 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:33.637002 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k"] Nov 03 19:47:33.640165 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:33.640144 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k" Nov 03 19:47:33.651734 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:33.651715 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k"] Nov 03 19:47:33.678014 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:33.677971 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c88bb62-ccb1-4b52-b155-5782364bd32e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k\" (UID: \"1c88bb62-ccb1-4b52-b155-5782364bd32e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k" Nov 03 19:47:33.678014 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:33.678091 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjcvs\" (UniqueName: \"kubernetes.io/projected/1c88bb62-ccb1-4b52-b155-5782364bd32e-kube-api-access-vjcvs\") pod \"maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k\" (UID: \"1c88bb62-ccb1-4b52-b155-5782364bd32e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k" Nov 03 19:47:33.678014 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:33.678178 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c88bb62-ccb1-4b52-b155-5782364bd32e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k\" (UID: \"1c88bb62-ccb1-4b52-b155-5782364bd32e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k" Nov 03 19:47:33.781227 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:33.780349 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c88bb62-ccb1-4b52-b155-5782364bd32e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k\" (UID: \"1c88bb62-ccb1-4b52-b155-5782364bd32e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k" Nov 03 19:47:33.781227 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:33.780431 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjcvs\" (UniqueName: \"kubernetes.io/projected/1c88bb62-ccb1-4b52-b155-5782364bd32e-kube-api-access-vjcvs\") pod \"maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k\" (UID: \"1c88bb62-ccb1-4b52-b155-5782364bd32e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k" Nov 03 19:47:33.781227 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:33.780497 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c88bb62-ccb1-4b52-b155-5782364bd32e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k\" (UID: \"1c88bb62-ccb1-4b52-b155-5782364bd32e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k" Nov 03 19:47:33.782802 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:33.782773 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c88bb62-ccb1-4b52-b155-5782364bd32e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k\" (UID: \"1c88bb62-ccb1-4b52-b155-5782364bd32e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k" Nov 03 19:47:33.785731 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:33.785680 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c88bb62-ccb1-4b52-b155-5782364bd32e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k\" (UID: \"1c88bb62-ccb1-4b52-b155-5782364bd32e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k" Nov 03 19:47:33.797963 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:33.797934 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjcvs\" (UniqueName: \"kubernetes.io/projected/1c88bb62-ccb1-4b52-b155-5782364bd32e-kube-api-access-vjcvs\") pod \"maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k\" (UID: \"1c88bb62-ccb1-4b52-b155-5782364bd32e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k" Nov 03 19:47:33.975177 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:33.969879 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k" Nov 03 19:47:34.238549 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:34.238490 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" event={"ID":"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d","Type":"ContainerStarted","Data":"1bbd05af50b32d0c319dc05258f2cf0eb6e21cd5bc159a6db3e622dd227dc8fe"} Nov 03 19:47:34.240029 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:34.240004 5059 generic.go:356] "Generic (PLEG): container finished" podID="a32fecea-2421-4b40-81b4-cada3ad747b0" containerID="a549776ace8f57bb78ee6a6a39e01694e2d70ae5714b8bbbc9ac2905d6aa41e4" exitCode=0 Nov 03 19:47:34.240136 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:34.240076 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6" event={"ID":"a32fecea-2421-4b40-81b4-cada3ad747b0","Type":"ContainerDied","Data":"a549776ace8f57bb78ee6a6a39e01694e2d70ae5714b8bbbc9ac2905d6aa41e4"} Nov 03 19:47:34.241689 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:34.241653 5059 generic.go:356] "Generic (PLEG): container finished" podID="497df71a-3372-4792-9cc4-980950a8de12" containerID="512a43fcddf2d69a21758c6b29f9d5edb489c0faf84aaeea5a81243ac933224f" exitCode=0 Nov 03 19:47:34.241806 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:34.241687 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" event={"ID":"497df71a-3372-4792-9cc4-980950a8de12","Type":"ContainerDied","Data":"512a43fcddf2d69a21758c6b29f9d5edb489c0faf84aaeea5a81243ac933224f"} Nov 03 19:47:34.242002 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:34.241988 5059 scope.go:122] "RemoveContainer" containerID="512a43fcddf2d69a21758c6b29f9d5edb489c0faf84aaeea5a81243ac933224f" Nov 03 19:47:34.243219 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:34.243190 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:47:34.243331 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:34.243277 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:47:34.243394 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:34.243327 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:47:34.243455 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:34.243388 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl" event={"ID":"6f78ce38-1dbf-42b1-bd02-510fe259b11c","Type":"ContainerStarted","Data":"8592dd0724f9bcb2e4cf1bf26a434c67f9bf07e3467400b6041c3941b8890523"} Nov 03 19:47:34.252296 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:34.252272 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k"] Nov 03 19:47:34.311652 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:34.311625 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:34.312438 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:34.312419 5059 patch_prober.go:29] interesting pod/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 19:47:34.312511 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:34.312457 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" podUID="329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 19:47:35.250296 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:35.250264 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" event={"ID":"497df71a-3372-4792-9cc4-980950a8de12","Type":"ContainerStarted","Data":"4ee9f309a44572d20ebfd12bc5c187c007f46324e6283bcb18d338e3705d8df8"} Nov 03 19:47:35.251835 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:35.251786 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k" event={"ID":"1c88bb62-ccb1-4b52-b155-5782364bd32e","Type":"ContainerStarted","Data":"98a4dc4f78b76af12391a8da0febef1c9605aa92341e584a518f3e67e5f06dd0"} Nov 03 19:47:35.251835 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:35.251834 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k" event={"ID":"1c88bb62-ccb1-4b52-b155-5782364bd32e","Type":"ContainerStarted","Data":"3b8ef3d826070c190d426a6b0d3c106db6d7ecee2ee0e544f7f0a715798d9c9f"} Nov 03 19:47:35.253393 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:35.253358 5059 generic.go:356] "Generic (PLEG): container finished" podID="6f78ce38-1dbf-42b1-bd02-510fe259b11c" containerID="8592dd0724f9bcb2e4cf1bf26a434c67f9bf07e3467400b6041c3941b8890523" exitCode=0 Nov 03 19:47:35.253503 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:35.253436 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl" event={"ID":"6f78ce38-1dbf-42b1-bd02-510fe259b11c","Type":"ContainerDied","Data":"8592dd0724f9bcb2e4cf1bf26a434c67f9bf07e3467400b6041c3941b8890523"} Nov 03 19:47:35.307687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:35.307386 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl" podStartSLOduration=3.548716438 podStartE2EDuration="7.307364925s" totalImagesPullingTime="3.758648487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:34.383286268 +0000 UTC m=+648.641083506" watchObservedRunningTime="2026-11-03 19:47:35.307364925 +0000 UTC m=+649.565162184" Nov 03 19:47:35.313037 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:35.312604 5059 patch_prober.go:29] interesting pod/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 19:47:35.313037 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:35.312674 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" podUID="329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 19:47:35.344325 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:35.344272 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k" podStartSLOduration=2.344260543 podStartE2EDuration="2.344260543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:35.341019807 +0000 UTC m=+649.598817115" watchObservedRunningTime="2026-11-03 19:47:35.344260543 +0000 UTC m=+649.602057800" Nov 03 19:47:35.611713 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:35.610326 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6" Nov 03 19:47:35.723947 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:35.723864 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a32fecea-2421-4b40-81b4-cada3ad747b0-kube-api-access-5lx47\" (UniqueName: \"kubernetes.io/projected/a32fecea-2421-4b40-81b4-cada3ad747b0-kube-api-access-5lx47\") pod \"a32fecea-2421-4b40-81b4-cada3ad747b0\" (UID: \"a32fecea-2421-4b40-81b4-cada3ad747b0\") " Nov 03 19:47:35.723947 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:35.723913 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a32fecea-2421-4b40-81b4-cada3ad747b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32fecea-2421-4b40-81b4-cada3ad747b0-maas-api-xks-ca\") pod \"a32fecea-2421-4b40-81b4-cada3ad747b0\" (UID: \"a32fecea-2421-4b40-81b4-cada3ad747b0\") " Nov 03 19:47:35.724135 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:35.724014 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a32fecea-2421-4b40-81b4-cada3ad747b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32fecea-2421-4b40-81b4-cada3ad747b0-maas-api-openshift-ca\") pod \"a32fecea-2421-4b40-81b4-cada3ad747b0\" (UID: \"a32fecea-2421-4b40-81b4-cada3ad747b0\") " Nov 03 19:47:35.724381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:35.724350 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a32fecea-2421-4b40-81b4-cada3ad747b0-maas-api-openshift-ca" pod "a32fecea-2421-4b40-81b4-cada3ad747b0" (UID: "a32fecea-2421-4b40-81b4-cada3ad747b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:47:35.726806 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:35.726784 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a32fecea-2421-4b40-81b4-cada3ad747b0-maas-api-xks-ca" pod "a32fecea-2421-4b40-81b4-cada3ad747b0" (UID: "a32fecea-2421-4b40-81b4-cada3ad747b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:47:35.726883 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:35.726784 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a32fecea-2421-4b40-81b4-cada3ad747b0-kube-api-access-5lx47" pod "a32fecea-2421-4b40-81b4-cada3ad747b0" (UID: "a32fecea-2421-4b40-81b4-cada3ad747b0"). InnerVolumeSpecName "kube-api-access-5lx47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:35.825587 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:35.825550 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32fecea-2421-4b40-81b4-cada3ad747b0-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:35.825587 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:35.825581 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lx47\" (UniqueName: \"kubernetes.io/projected/a32fecea-2421-4b40-81b4-cada3ad747b0-kube-api-access-5lx47\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:35.825587 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:35.825591 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32fecea-2421-4b40-81b4-cada3ad747b0-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:36.110232 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:36.110184 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:36.110426 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:36.110289 5059 patch_prober.go:29] interesting pod/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:47:36.110426 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:36.110322 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" podUID="497df71a-3372-4792-9cc4-980950a8de12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 19:47:36.282686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:36.282547 5059 generic.go:356] "Generic (PLEG): container finished" podID="1c88bb62-ccb1-4b52-b155-5782364bd32e" containerID="98a4dc4f78b76af12391a8da0febef1c9605aa92341e584a518f3e67e5f06dd0" exitCode=0 Nov 03 19:47:36.283197 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:36.282807 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k" event={"ID":"1c88bb62-ccb1-4b52-b155-5782364bd32e","Type":"ContainerDied","Data":"98a4dc4f78b76af12391a8da0febef1c9605aa92341e584a518f3e67e5f06dd0"} Nov 03 19:47:36.304794 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:36.285025 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6" Nov 03 19:47:36.304794 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:36.285574 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6" event={"ID":"a32fecea-2421-4b40-81b4-cada3ad747b0","Type":"ContainerDied","Data":"00ebe954989cb847d6cdcc56edca6faed648ee2db90342d4f9aebd5f119a4f3e"} Nov 03 19:47:36.304794 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:36.285611 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00ebe954989cb847d6cdcc56edca6faed648ee2db90342d4f9aebd5f119a4f3e" Nov 03 19:47:36.316314 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:36.316024 5059 patch_prober.go:29] interesting pod/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 19:47:36.316314 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:36.316063 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" podUID="329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 19:47:36.613863 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:36.613836 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl" Nov 03 19:47:36.635413 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:36.635382 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f78ce38-1dbf-42b1-bd02-510fe259b11c-kube-api-access-dm4l4\" (UniqueName: \"kubernetes.io/projected/6f78ce38-1dbf-42b1-bd02-510fe259b11c-kube-api-access-dm4l4\") pod \"6f78ce38-1dbf-42b1-bd02-510fe259b11c\" (UID: \"6f78ce38-1dbf-42b1-bd02-510fe259b11c\") " Nov 03 19:47:36.635563 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:36.635462 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f78ce38-1dbf-42b1-bd02-510fe259b11c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f78ce38-1dbf-42b1-bd02-510fe259b11c-maas-api-xks-ca\") pod \"6f78ce38-1dbf-42b1-bd02-510fe259b11c\" (UID: \"6f78ce38-1dbf-42b1-bd02-510fe259b11c\") " Nov 03 19:47:36.636187 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:36.636152 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f78ce38-1dbf-42b1-bd02-510fe259b11c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f78ce38-1dbf-42b1-bd02-510fe259b11c-maas-api-openshift-ca\") pod \"6f78ce38-1dbf-42b1-bd02-510fe259b11c\" (UID: \"6f78ce38-1dbf-42b1-bd02-510fe259b11c\") " Nov 03 19:47:36.637965 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:36.636845 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f78ce38-1dbf-42b1-bd02-510fe259b11c-maas-api-openshift-ca" pod "6f78ce38-1dbf-42b1-bd02-510fe259b11c" (UID: "6f78ce38-1dbf-42b1-bd02-510fe259b11c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:47:36.640094 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:36.640055 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f78ce38-1dbf-42b1-bd02-510fe259b11c-kube-api-access-dm4l4" pod "6f78ce38-1dbf-42b1-bd02-510fe259b11c" (UID: "6f78ce38-1dbf-42b1-bd02-510fe259b11c"). InnerVolumeSpecName "kube-api-access-dm4l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:36.640388 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:36.640371 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f78ce38-1dbf-42b1-bd02-510fe259b11c-maas-api-xks-ca" pod "6f78ce38-1dbf-42b1-bd02-510fe259b11c" (UID: "6f78ce38-1dbf-42b1-bd02-510fe259b11c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:47:36.737776 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:36.737692 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm4l4\" (UniqueName: \"kubernetes.io/projected/6f78ce38-1dbf-42b1-bd02-510fe259b11c-kube-api-access-dm4l4\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:36.737776 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:36.737720 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f78ce38-1dbf-42b1-bd02-510fe259b11c-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:36.737776 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:36.737730 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f78ce38-1dbf-42b1-bd02-510fe259b11c-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:36.861788 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:36.861749 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:37.111223 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:37.111185 5059 patch_prober.go:29] interesting pod/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:47:37.111419 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:37.111245 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" podUID="497df71a-3372-4792-9cc4-980950a8de12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 19:47:37.310714 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:37.306411 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl" Nov 03 19:47:37.310714 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:37.307755 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl" event={"ID":"6f78ce38-1dbf-42b1-bd02-510fe259b11c","Type":"ContainerDied","Data":"d3698f6faaa26584860650c298f45ffd1913ba3c802e7683cac8a4f062550ba9"} Nov 03 19:47:37.310714 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:37.307809 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3698f6faaa26584860650c298f45ffd1913ba3c802e7683cac8a4f062550ba9" Nov 03 19:47:37.311888 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:37.311858 5059 patch_prober.go:29] interesting pod/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 19:47:37.311962 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:37.311911 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" podUID="329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 19:47:37.541413 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:37.541382 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k" Nov 03 19:47:37.549570 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:37.549549 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c88bb62-ccb1-4b52-b155-5782364bd32e-kube-api-access-vjcvs\" (UniqueName: \"kubernetes.io/projected/1c88bb62-ccb1-4b52-b155-5782364bd32e-kube-api-access-vjcvs\") pod \"1c88bb62-ccb1-4b52-b155-5782364bd32e\" (UID: \"1c88bb62-ccb1-4b52-b155-5782364bd32e\") " Nov 03 19:47:37.549707 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:37.549688 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c88bb62-ccb1-4b52-b155-5782364bd32e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c88bb62-ccb1-4b52-b155-5782364bd32e-maas-api-xks-ca\") pod \"1c88bb62-ccb1-4b52-b155-5782364bd32e\" (UID: \"1c88bb62-ccb1-4b52-b155-5782364bd32e\") " Nov 03 19:47:37.550010 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:37.549769 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c88bb62-ccb1-4b52-b155-5782364bd32e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c88bb62-ccb1-4b52-b155-5782364bd32e-maas-api-openshift-ca\") pod \"1c88bb62-ccb1-4b52-b155-5782364bd32e\" (UID: \"1c88bb62-ccb1-4b52-b155-5782364bd32e\") " Nov 03 19:47:37.550283 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:37.550258 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c88bb62-ccb1-4b52-b155-5782364bd32e-maas-api-openshift-ca" pod "1c88bb62-ccb1-4b52-b155-5782364bd32e" (UID: "1c88bb62-ccb1-4b52-b155-5782364bd32e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:47:37.551700 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:37.551682 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c88bb62-ccb1-4b52-b155-5782364bd32e-kube-api-access-vjcvs" pod "1c88bb62-ccb1-4b52-b155-5782364bd32e" (UID: "1c88bb62-ccb1-4b52-b155-5782364bd32e"). InnerVolumeSpecName "kube-api-access-vjcvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:37.551830 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:37.551802 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c88bb62-ccb1-4b52-b155-5782364bd32e-maas-api-xks-ca" pod "1c88bb62-ccb1-4b52-b155-5782364bd32e" (UID: "1c88bb62-ccb1-4b52-b155-5782364bd32e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:47:37.651174 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:37.651086 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjcvs\" (UniqueName: \"kubernetes.io/projected/1c88bb62-ccb1-4b52-b155-5782364bd32e-kube-api-access-vjcvs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:37.651174 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:37.651114 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c88bb62-ccb1-4b52-b155-5782364bd32e-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:37.651174 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:37.651130 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c88bb62-ccb1-4b52-b155-5782364bd32e-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:38.111089 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:38.111046 5059 patch_prober.go:29] interesting pod/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:47:38.111292 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:38.111097 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" podUID="497df71a-3372-4792-9cc4-980950a8de12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 19:47:38.315738 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:38.315112 5059 patch_prober.go:29] interesting pod/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 19:47:38.315738 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:38.315175 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" podUID="329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 19:47:38.324047 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:38.324001 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k" event={"ID":"1c88bb62-ccb1-4b52-b155-5782364bd32e","Type":"ContainerDied","Data":"3b8ef3d826070c190d426a6b0d3c106db6d7ecee2ee0e544f7f0a715798d9c9f"} Nov 03 19:47:38.324181 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:38.324117 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b8ef3d826070c190d426a6b0d3c106db6d7ecee2ee0e544f7f0a715798d9c9f" Nov 03 19:47:38.324221 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:38.324208 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k" Nov 03 19:47:38.823380 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:38.823353 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp"] Nov 03 19:47:38.838245 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:38.838200 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b"] Nov 03 19:47:39.114766 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:39.114125 5059 patch_prober.go:29] interesting pod/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:47:39.114766 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:39.114184 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" podUID="497df71a-3372-4792-9cc4-980950a8de12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 19:47:39.311909 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:39.311869 5059 patch_prober.go:29] interesting pod/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 19:47:39.312195 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:39.312162 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" podUID="329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 19:47:40.111041 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:40.111003 5059 patch_prober.go:29] interesting pod/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:47:40.111442 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:40.111059 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" podUID="497df71a-3372-4792-9cc4-980950a8de12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 19:47:40.316152 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:40.316120 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:40.317277 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:40.317259 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:40.456079 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:40.455995 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf"] Nov 03 19:47:40.469801 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:40.469771 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh"] Nov 03 19:47:40.604357 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:47:40.604316 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:47:41.114533 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:41.114505 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:41.114959 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:41.114870 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:41.115753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:41.115736 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:42.516407 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:42.516371 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6"] Nov 03 19:47:42.520403 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:42.520373 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745a88f9-pb4g6"] Nov 03 19:47:42.682089 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:42.682059 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb"] Nov 03 19:47:42.682357 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:42.682329 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" podUID="329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" containerName="istio-proxy" containerID="cri-o://1bbd05af50b32d0c319dc05258f2cf0eb6e21cd5bc159a6db3e622dd227dc8fe" gracePeriod=30 Nov 03 19:47:42.901394 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:42.901358 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl"] Nov 03 19:47:42.908529 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:42.908502 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9959e5f4-zrvvl"] Nov 03 19:47:43.080926 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:43.080895 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx"] Nov 03 19:47:43.081138 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:43.081108 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" podUID="cbce3ead-79e2-4d3e-8caa-8a35a3640e75" containerName="istio-proxy" containerID="cri-o://d37a4cd3eecbd37a2b70844c7ae297fc283a99464149be8dea809712753f7a83" gracePeriod=30 Nov 03 19:47:43.824289 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:43.824241 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp" podUID="d1824e75-de08-4007-8ff9-200f3bb369d3" containerName="payload-processing" containerID="cri-o://afd4e3bafa034ca292543211cd99e43e6c1305ec32edbc5be195052cf1e9d34d" gracePeriod=25 Nov 03 19:47:43.838892 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:43.838816 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b" podUID="695e7ed2-53a7-4add-b4ba-74effcc52faf" containerName="payload-pre-processing" containerID="cri-o://c137788f9c8799ab826a726643a2f95bc2a243573463da0cef4a805c16354b09" gracePeriod=25 Nov 03 19:47:43.856488 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:43.856462 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f78ce38-1dbf-42b1-bd02-510fe259b11c" path="/var/lib/kubelet/pods/6f78ce38-1dbf-42b1-bd02-510fe259b11c/volumes" Nov 03 19:47:43.857501 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:43.857484 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a32fecea-2421-4b40-81b4-cada3ad747b0" path="/var/lib/kubelet/pods/a32fecea-2421-4b40-81b4-cada3ad747b0/volumes" Nov 03 19:47:44.138440 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.138405 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp" Nov 03 19:47:44.158329 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.158304 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b" Nov 03 19:47:44.262412 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.262376 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/695e7ed2-53a7-4add-b4ba-74effcc52faf-kube-api-access-t5wjn\" (UniqueName: \"kubernetes.io/projected/695e7ed2-53a7-4add-b4ba-74effcc52faf-kube-api-access-t5wjn\") pod \"695e7ed2-53a7-4add-b4ba-74effcc52faf\" (UID: \"695e7ed2-53a7-4add-b4ba-74effcc52faf\") " Nov 03 19:47:44.262601 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.262471 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/695e7ed2-53a7-4add-b4ba-74effcc52faf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695e7ed2-53a7-4add-b4ba-74effcc52faf-plugins-config-volume\") pod \"695e7ed2-53a7-4add-b4ba-74effcc52faf\" (UID: \"695e7ed2-53a7-4add-b4ba-74effcc52faf\") " Nov 03 19:47:44.262601 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.262510 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1824e75-de08-4007-8ff9-200f3bb369d3-kube-api-access-jzvcq\" (UniqueName: \"kubernetes.io/projected/d1824e75-de08-4007-8ff9-200f3bb369d3-kube-api-access-jzvcq\") pod \"d1824e75-de08-4007-8ff9-200f3bb369d3\" (UID: \"d1824e75-de08-4007-8ff9-200f3bb369d3\") " Nov 03 19:47:44.262601 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.262546 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1824e75-de08-4007-8ff9-200f3bb369d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1824e75-de08-4007-8ff9-200f3bb369d3-plugins-config-volume\") pod \"d1824e75-de08-4007-8ff9-200f3bb369d3\" (UID: \"d1824e75-de08-4007-8ff9-200f3bb369d3\") " Nov 03 19:47:44.262903 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.262880 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695e7ed2-53a7-4add-b4ba-74effcc52faf-plugins-config-volume" pod "695e7ed2-53a7-4add-b4ba-74effcc52faf" (UID: "695e7ed2-53a7-4add-b4ba-74effcc52faf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:47:44.262972 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.262880 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1824e75-de08-4007-8ff9-200f3bb369d3-plugins-config-volume" pod "d1824e75-de08-4007-8ff9-200f3bb369d3" (UID: "d1824e75-de08-4007-8ff9-200f3bb369d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:47:44.263027 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.262974 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695e7ed2-53a7-4add-b4ba-74effcc52faf-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:44.263027 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.262995 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1824e75-de08-4007-8ff9-200f3bb369d3-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:44.264783 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.264759 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695e7ed2-53a7-4add-b4ba-74effcc52faf-kube-api-access-t5wjn" pod "695e7ed2-53a7-4add-b4ba-74effcc52faf" (UID: "695e7ed2-53a7-4add-b4ba-74effcc52faf"). InnerVolumeSpecName "kube-api-access-t5wjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:44.264877 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.264785 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1824e75-de08-4007-8ff9-200f3bb369d3-kube-api-access-jzvcq" pod "d1824e75-de08-4007-8ff9-200f3bb369d3" (UID: "d1824e75-de08-4007-8ff9-200f3bb369d3"). InnerVolumeSpecName "kube-api-access-jzvcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:44.363070 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.362982 5059 generic.go:356] "Generic (PLEG): container finished" podID="695e7ed2-53a7-4add-b4ba-74effcc52faf" containerID="c137788f9c8799ab826a726643a2f95bc2a243573463da0cef4a805c16354b09" exitCode=0 Nov 03 19:47:44.363070 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.363039 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b" Nov 03 19:47:44.363276 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.363074 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b" event={"ID":"695e7ed2-53a7-4add-b4ba-74effcc52faf","Type":"ContainerDied","Data":"c137788f9c8799ab826a726643a2f95bc2a243573463da0cef4a805c16354b09"} Nov 03 19:47:44.363276 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.363138 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b" event={"ID":"695e7ed2-53a7-4add-b4ba-74effcc52faf","Type":"ContainerDied","Data":"fbc080ac2e861e62858f1768dfe6fc85ac911b093fafe9d835aa176aaafd0368"} Nov 03 19:47:44.363276 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.363172 5059 scope.go:122] "RemoveContainer" containerID="c137788f9c8799ab826a726643a2f95bc2a243573463da0cef4a805c16354b09" Nov 03 19:47:44.363654 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.363634 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5wjn\" (UniqueName: \"kubernetes.io/projected/695e7ed2-53a7-4add-b4ba-74effcc52faf-kube-api-access-t5wjn\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:44.363654 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.363654 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzvcq\" (UniqueName: \"kubernetes.io/projected/d1824e75-de08-4007-8ff9-200f3bb369d3-kube-api-access-jzvcq\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:44.364578 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.364552 5059 generic.go:356] "Generic (PLEG): container finished" podID="d1824e75-de08-4007-8ff9-200f3bb369d3" containerID="afd4e3bafa034ca292543211cd99e43e6c1305ec32edbc5be195052cf1e9d34d" exitCode=0 Nov 03 19:47:44.364729 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.364581 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp" event={"ID":"d1824e75-de08-4007-8ff9-200f3bb369d3","Type":"ContainerDied","Data":"afd4e3bafa034ca292543211cd99e43e6c1305ec32edbc5be195052cf1e9d34d"} Nov 03 19:47:44.364729 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.364610 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp" Nov 03 19:47:44.364729 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.364627 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp" event={"ID":"d1824e75-de08-4007-8ff9-200f3bb369d3","Type":"ContainerDied","Data":"03566886203db2d622177761b14acdb8e200d37b3eb124d01f09f4ba2212462d"} Nov 03 19:47:44.374395 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.374380 5059 scope.go:122] "RemoveContainer" containerID="c137788f9c8799ab826a726643a2f95bc2a243573463da0cef4a805c16354b09" Nov 03 19:47:44.374701 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:47:44.374681 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c137788f9c8799ab826a726643a2f95bc2a243573463da0cef4a805c16354b09\": container with ID starting with c137788f9c8799ab826a726643a2f95bc2a243573463da0cef4a805c16354b09 not found: ID does not exist" containerID="c137788f9c8799ab826a726643a2f95bc2a243573463da0cef4a805c16354b09" Nov 03 19:47:44.374810 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.374711 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c137788f9c8799ab826a726643a2f95bc2a243573463da0cef4a805c16354b09"} err="failed to get container status \"c137788f9c8799ab826a726643a2f95bc2a243573463da0cef4a805c16354b09\": rpc error: code = NotFound desc = could not find container \"c137788f9c8799ab826a726643a2f95bc2a243573463da0cef4a805c16354b09\": container with ID starting with c137788f9c8799ab826a726643a2f95bc2a243573463da0cef4a805c16354b09 not found: ID does not exist" Nov 03 19:47:44.374810 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.374734 5059 scope.go:122] "RemoveContainer" containerID="afd4e3bafa034ca292543211cd99e43e6c1305ec32edbc5be195052cf1e9d34d" Nov 03 19:47:44.383508 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.383488 5059 scope.go:122] "RemoveContainer" containerID="afd4e3bafa034ca292543211cd99e43e6c1305ec32edbc5be195052cf1e9d34d" Nov 03 19:47:44.383789 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:47:44.383770 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd4e3bafa034ca292543211cd99e43e6c1305ec32edbc5be195052cf1e9d34d\": container with ID starting with afd4e3bafa034ca292543211cd99e43e6c1305ec32edbc5be195052cf1e9d34d not found: ID does not exist" containerID="afd4e3bafa034ca292543211cd99e43e6c1305ec32edbc5be195052cf1e9d34d" Nov 03 19:47:44.383850 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.383795 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd4e3bafa034ca292543211cd99e43e6c1305ec32edbc5be195052cf1e9d34d"} err="failed to get container status \"afd4e3bafa034ca292543211cd99e43e6c1305ec32edbc5be195052cf1e9d34d\": rpc error: code = NotFound desc = could not find container \"afd4e3bafa034ca292543211cd99e43e6c1305ec32edbc5be195052cf1e9d34d\": container with ID starting with afd4e3bafa034ca292543211cd99e43e6c1305ec32edbc5be195052cf1e9d34d not found: ID does not exist" Nov 03 19:47:44.391442 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.391421 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp"] Nov 03 19:47:44.395298 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.395276 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9959e5f4-6488566cf6-9gscp"] Nov 03 19:47:44.429475 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.429183 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b"] Nov 03 19:47:44.429475 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:44.429243 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9959e5f4-7564dbc89f-8vl4b"] Nov 03 19:47:45.457313 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:45.457275 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf" podUID="f337606e-fc9b-43c6-8ac4-205f2e011dde" containerName="payload-processing" containerID="cri-o://4a87a2828e5c0893df365342163e5e0af79fb4f530a6e6eae56429e17ff408d9" gracePeriod=25 Nov 03 19:47:45.470766 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:45.470713 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh" podUID="087fa86d-7f7a-45a8-b368-b0052d5f5ec0" containerName="payload-pre-processing" containerID="cri-o://fdbbd7c38271db09868eb23ad9c7d917e41b8e03ebd2717394e707ddbe1fc99d" gracePeriod=25 Nov 03 19:47:45.737005 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:45.736979 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf" Nov 03 19:47:45.748839 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:45.748815 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh" Nov 03 19:47:45.856154 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:45.856125 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="695e7ed2-53a7-4add-b4ba-74effcc52faf" path="/var/lib/kubelet/pods/695e7ed2-53a7-4add-b4ba-74effcc52faf/volumes" Nov 03 19:47:45.856517 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:45.856504 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1824e75-de08-4007-8ff9-200f3bb369d3" path="/var/lib/kubelet/pods/d1824e75-de08-4007-8ff9-200f3bb369d3/volumes" Nov 03 19:47:45.879731 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:45.879710 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f337606e-fc9b-43c6-8ac4-205f2e011dde-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f337606e-fc9b-43c6-8ac4-205f2e011dde-plugins-config-volume\") pod \"f337606e-fc9b-43c6-8ac4-205f2e011dde\" (UID: \"f337606e-fc9b-43c6-8ac4-205f2e011dde\") " Nov 03 19:47:45.879843 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:45.879762 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f337606e-fc9b-43c6-8ac4-205f2e011dde-kube-api-access-kzlkl\" (UniqueName: \"kubernetes.io/projected/f337606e-fc9b-43c6-8ac4-205f2e011dde-kube-api-access-kzlkl\") pod \"f337606e-fc9b-43c6-8ac4-205f2e011dde\" (UID: \"f337606e-fc9b-43c6-8ac4-205f2e011dde\") " Nov 03 19:47:45.879843 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:45.879835 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/087fa86d-7f7a-45a8-b368-b0052d5f5ec0-kube-api-access-8f57f\" (UniqueName: \"kubernetes.io/projected/087fa86d-7f7a-45a8-b368-b0052d5f5ec0-kube-api-access-8f57f\") pod \"087fa86d-7f7a-45a8-b368-b0052d5f5ec0\" (UID: \"087fa86d-7f7a-45a8-b368-b0052d5f5ec0\") " Nov 03 19:47:45.879910 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:45.879878 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/087fa86d-7f7a-45a8-b368-b0052d5f5ec0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087fa86d-7f7a-45a8-b368-b0052d5f5ec0-plugins-config-volume\") pod \"087fa86d-7f7a-45a8-b368-b0052d5f5ec0\" (UID: \"087fa86d-7f7a-45a8-b368-b0052d5f5ec0\") " Nov 03 19:47:45.880106 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:45.880076 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f337606e-fc9b-43c6-8ac4-205f2e011dde-plugins-config-volume" pod "f337606e-fc9b-43c6-8ac4-205f2e011dde" (UID: "f337606e-fc9b-43c6-8ac4-205f2e011dde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:47:45.880303 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:45.880276 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/087fa86d-7f7a-45a8-b368-b0052d5f5ec0-plugins-config-volume" pod "087fa86d-7f7a-45a8-b368-b0052d5f5ec0" (UID: "087fa86d-7f7a-45a8-b368-b0052d5f5ec0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:47:45.880361 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:45.880309 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f337606e-fc9b-43c6-8ac4-205f2e011dde-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:45.881794 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:45.881770 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/087fa86d-7f7a-45a8-b368-b0052d5f5ec0-kube-api-access-8f57f" pod "087fa86d-7f7a-45a8-b368-b0052d5f5ec0" (UID: "087fa86d-7f7a-45a8-b368-b0052d5f5ec0"). InnerVolumeSpecName "kube-api-access-8f57f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:45.881897 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:45.881863 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f337606e-fc9b-43c6-8ac4-205f2e011dde-kube-api-access-kzlkl" pod "f337606e-fc9b-43c6-8ac4-205f2e011dde" (UID: "f337606e-fc9b-43c6-8ac4-205f2e011dde"). InnerVolumeSpecName "kube-api-access-kzlkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:45.981528 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:45.981502 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f57f\" (UniqueName: \"kubernetes.io/projected/087fa86d-7f7a-45a8-b368-b0052d5f5ec0-kube-api-access-8f57f\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:45.981528 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:45.981528 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/087fa86d-7f7a-45a8-b368-b0052d5f5ec0-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:45.981688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:45.981537 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzlkl\" (UniqueName: \"kubernetes.io/projected/f337606e-fc9b-43c6-8ac4-205f2e011dde-kube-api-access-kzlkl\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:46.087987 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.087947 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj"] Nov 03 19:47:46.088428 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.088408 5059 cpu_manager.go:414] "containerMap: removing container" podUID="6f78ce38-1dbf-42b1-bd02-510fe259b11c" containerName="revoke-keys" Nov 03 19:47:46.088428 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.088423 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f78ce38-1dbf-42b1-bd02-510fe259b11c" containerName="revoke-keys" Nov 03 19:47:46.088627 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.088448 5059 cpu_manager.go:414] "containerMap: removing container" podUID="695e7ed2-53a7-4add-b4ba-74effcc52faf" containerName="payload-pre-processing" Nov 03 19:47:46.088627 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.088453 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="695e7ed2-53a7-4add-b4ba-74effcc52faf" containerName="payload-pre-processing" Nov 03 19:47:46.088627 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.088460 5059 cpu_manager.go:414] "containerMap: removing container" podUID="f337606e-fc9b-43c6-8ac4-205f2e011dde" containerName="payload-processing" Nov 03 19:47:46.088627 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.088465 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f337606e-fc9b-43c6-8ac4-205f2e011dde" containerName="payload-processing" Nov 03 19:47:46.088627 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.088471 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1c88bb62-ccb1-4b52-b155-5782364bd32e" containerName="revoke-keys" Nov 03 19:47:46.088627 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.088476 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c88bb62-ccb1-4b52-b155-5782364bd32e" containerName="revoke-keys" Nov 03 19:47:46.088627 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.088491 5059 cpu_manager.go:414] "containerMap: removing container" podUID="087fa86d-7f7a-45a8-b368-b0052d5f5ec0" containerName="payload-pre-processing" Nov 03 19:47:46.088627 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.088499 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="087fa86d-7f7a-45a8-b368-b0052d5f5ec0" containerName="payload-pre-processing" Nov 03 19:47:46.088627 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.088516 5059 cpu_manager.go:414] "containerMap: removing container" podUID="a32fecea-2421-4b40-81b4-cada3ad747b0" containerName="revoke-keys" Nov 03 19:47:46.088627 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.088522 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a32fecea-2421-4b40-81b4-cada3ad747b0" containerName="revoke-keys" Nov 03 19:47:46.088627 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.088540 5059 cpu_manager.go:414] "containerMap: removing container" podUID="d1824e75-de08-4007-8ff9-200f3bb369d3" containerName="payload-processing" Nov 03 19:47:46.088627 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.088548 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1824e75-de08-4007-8ff9-200f3bb369d3" containerName="payload-processing" Nov 03 19:47:46.089444 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.088655 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="087fa86d-7f7a-45a8-b368-b0052d5f5ec0" containerName="payload-pre-processing" Nov 03 19:47:46.089444 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.088685 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a32fecea-2421-4b40-81b4-cada3ad747b0" containerName="revoke-keys" Nov 03 19:47:46.089444 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.088695 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1824e75-de08-4007-8ff9-200f3bb369d3" containerName="payload-processing" Nov 03 19:47:46.089444 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.088705 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f78ce38-1dbf-42b1-bd02-510fe259b11c" containerName="revoke-keys" Nov 03 19:47:46.089444 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.088718 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="695e7ed2-53a7-4add-b4ba-74effcc52faf" containerName="payload-pre-processing" Nov 03 19:47:46.089444 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.088726 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f337606e-fc9b-43c6-8ac4-205f2e011dde" containerName="payload-processing" Nov 03 19:47:46.089444 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.088733 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c88bb62-ccb1-4b52-b155-5782364bd32e" containerName="revoke-keys" Nov 03 19:47:46.093802 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.093782 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.100705 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.100681 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d3d0df-openshift-default-dockercfg-m4lvp\"" Nov 03 19:47:46.109338 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.109313 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj"] Nov 03 19:47:46.140151 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.140117 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k"] Nov 03 19:47:46.146361 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.146333 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c20ed791-v9r2k"] Nov 03 19:47:46.183259 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.183226 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.183388 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.183281 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnp2q\" (UniqueName: \"kubernetes.io/projected/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-kube-api-access-bnp2q\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.183441 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.183403 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.183478 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.183456 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.183516 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.183487 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.183550 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.183514 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.183591 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.183578 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.183626 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.183602 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.183681 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.183626 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.284778 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.284690 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.284778 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.284738 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.285000 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.284781 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.285000 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.284828 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.285000 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.284894 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnp2q\" (UniqueName: \"kubernetes.io/projected/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-kube-api-access-bnp2q\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.285000 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.284942 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.285000 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.284998 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.285257 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.285040 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.285257 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.285083 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.285373 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.285338 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.285528 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.285374 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.285528 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.285420 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.285528 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.285520 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.285716 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.285537 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.287627 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.287593 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.287731 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.287682 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.297386 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.297359 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.298299 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.298280 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnp2q\" (UniqueName: \"kubernetes.io/projected/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-kube-api-access-bnp2q\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.379245 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.379207 5059 generic.go:356] "Generic (PLEG): container finished" podID="f337606e-fc9b-43c6-8ac4-205f2e011dde" containerID="4a87a2828e5c0893df365342163e5e0af79fb4f530a6e6eae56429e17ff408d9" exitCode=0 Nov 03 19:47:46.379438 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.379268 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf" Nov 03 19:47:46.379438 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.379289 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf" event={"ID":"f337606e-fc9b-43c6-8ac4-205f2e011dde","Type":"ContainerDied","Data":"4a87a2828e5c0893df365342163e5e0af79fb4f530a6e6eae56429e17ff408d9"} Nov 03 19:47:46.379438 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.379335 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf" event={"ID":"f337606e-fc9b-43c6-8ac4-205f2e011dde","Type":"ContainerDied","Data":"15ea57f997225d69d43afb1aeaf47701842b34016040f98784269ca61cfe0497"} Nov 03 19:47:46.379438 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.379367 5059 scope.go:122] "RemoveContainer" containerID="4a87a2828e5c0893df365342163e5e0af79fb4f530a6e6eae56429e17ff408d9" Nov 03 19:47:46.380623 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.380604 5059 generic.go:356] "Generic (PLEG): container finished" podID="087fa86d-7f7a-45a8-b368-b0052d5f5ec0" containerID="fdbbd7c38271db09868eb23ad9c7d917e41b8e03ebd2717394e707ddbe1fc99d" exitCode=0 Nov 03 19:47:46.380742 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.380650 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh" event={"ID":"087fa86d-7f7a-45a8-b368-b0052d5f5ec0","Type":"ContainerDied","Data":"fdbbd7c38271db09868eb23ad9c7d917e41b8e03ebd2717394e707ddbe1fc99d"} Nov 03 19:47:46.380742 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.380655 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh" Nov 03 19:47:46.380742 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.380708 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh" event={"ID":"087fa86d-7f7a-45a8-b368-b0052d5f5ec0","Type":"ContainerDied","Data":"0ca372c1281d9058bf76ae8e15097135e00416887c56f9bee86728e973dbff84"} Nov 03 19:47:46.389226 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.389211 5059 scope.go:122] "RemoveContainer" containerID="4a87a2828e5c0893df365342163e5e0af79fb4f530a6e6eae56429e17ff408d9" Nov 03 19:47:46.389509 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:47:46.389492 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a87a2828e5c0893df365342163e5e0af79fb4f530a6e6eae56429e17ff408d9\": container with ID starting with 4a87a2828e5c0893df365342163e5e0af79fb4f530a6e6eae56429e17ff408d9 not found: ID does not exist" containerID="4a87a2828e5c0893df365342163e5e0af79fb4f530a6e6eae56429e17ff408d9" Nov 03 19:47:46.389552 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.389517 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a87a2828e5c0893df365342163e5e0af79fb4f530a6e6eae56429e17ff408d9"} err="failed to get container status \"4a87a2828e5c0893df365342163e5e0af79fb4f530a6e6eae56429e17ff408d9\": rpc error: code = NotFound desc = could not find container \"4a87a2828e5c0893df365342163e5e0af79fb4f530a6e6eae56429e17ff408d9\": container with ID starting with 4a87a2828e5c0893df365342163e5e0af79fb4f530a6e6eae56429e17ff408d9 not found: ID does not exist" Nov 03 19:47:46.389552 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.389530 5059 scope.go:122] "RemoveContainer" containerID="fdbbd7c38271db09868eb23ad9c7d917e41b8e03ebd2717394e707ddbe1fc99d" Nov 03 19:47:46.398008 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.397990 5059 scope.go:122] "RemoveContainer" containerID="fdbbd7c38271db09868eb23ad9c7d917e41b8e03ebd2717394e707ddbe1fc99d" Nov 03 19:47:46.398285 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:47:46.398268 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdbbd7c38271db09868eb23ad9c7d917e41b8e03ebd2717394e707ddbe1fc99d\": container with ID starting with fdbbd7c38271db09868eb23ad9c7d917e41b8e03ebd2717394e707ddbe1fc99d not found: ID does not exist" containerID="fdbbd7c38271db09868eb23ad9c7d917e41b8e03ebd2717394e707ddbe1fc99d" Nov 03 19:47:46.398338 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.398289 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdbbd7c38271db09868eb23ad9c7d917e41b8e03ebd2717394e707ddbe1fc99d"} err="failed to get container status \"fdbbd7c38271db09868eb23ad9c7d917e41b8e03ebd2717394e707ddbe1fc99d\": rpc error: code = NotFound desc = could not find container \"fdbbd7c38271db09868eb23ad9c7d917e41b8e03ebd2717394e707ddbe1fc99d\": container with ID starting with fdbbd7c38271db09868eb23ad9c7d917e41b8e03ebd2717394e707ddbe1fc99d not found: ID does not exist" Nov 03 19:47:46.406235 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.406208 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh"] Nov 03 19:47:46.410323 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.410114 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:46.412481 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.412459 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3f7e10b8-7c7f97f9-h8rsh"] Nov 03 19:47:46.423394 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.423371 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf"] Nov 03 19:47:46.430864 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.430842 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3f7e10b8-6dd8f5869d-tkccf"] Nov 03 19:47:46.607449 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.607420 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn"] Nov 03 19:47:46.622560 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.622530 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m"] Nov 03 19:47:46.693090 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.693065 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj"] Nov 03 19:47:46.696836 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.696801 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:47:46.696924 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.696865 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:47:46.696924 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:46.696894 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:47:47.389517 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:47.389484 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" event={"ID":"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b","Type":"ContainerStarted","Data":"6f8ee91ca327bbf5a0c3704b9e3ce856daebe10db7f411298f5e612e7a7a2a8f"} Nov 03 19:47:47.389744 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:47.389532 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" event={"ID":"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b","Type":"ContainerStarted","Data":"7e98325901ed5796a82a8f7bf31137a3ae839778e2b1640d33607242ae0eec8f"} Nov 03 19:47:47.411047 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:47.411020 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:47.416080 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:47.416024 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" podStartSLOduration=1.416007579 podStartE2EDuration="1.416007579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:47.41530014 +0000 UTC m=+661.673097396" watchObservedRunningTime="2026-11-03 19:47:47.416007579 +0000 UTC m=+661.673804837" Nov 03 19:47:47.416474 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:47.416456 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:47.555459 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:47.555427 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv"] Nov 03 19:47:47.555703 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:47.555638 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" podUID="497df71a-3372-4792-9cc4-980950a8de12" containerName="istio-proxy" containerID="cri-o://4ee9f309a44572d20ebfd12bc5c187c007f46324e6283bcb18d338e3705d8df8" gracePeriod=30 Nov 03 19:47:47.576125 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:47.576008 5059 patch_prober.go:29] interesting pod/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:47:47.576284 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:47.576154 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" podUID="cbce3ead-79e2-4d3e-8caa-8a35a3640e75" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:47:47.870169 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:47.870131 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="087fa86d-7f7a-45a8-b368-b0052d5f5ec0" path="/var/lib/kubelet/pods/087fa86d-7f7a-45a8-b368-b0052d5f5ec0/volumes" Nov 03 19:47:47.870630 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:47.870610 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c88bb62-ccb1-4b52-b155-5782364bd32e" path="/var/lib/kubelet/pods/1c88bb62-ccb1-4b52-b155-5782364bd32e/volumes" Nov 03 19:47:47.871094 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:47.871066 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f337606e-fc9b-43c6-8ac4-205f2e011dde" path="/var/lib/kubelet/pods/f337606e-fc9b-43c6-8ac4-205f2e011dde/volumes" Nov 03 19:47:47.945851 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:47.945790 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:48.111178 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.111154 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-workload-socket\") pod \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " Nov 03 19:47:48.111267 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.111188 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-credential-socket\") pod \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " Nov 03 19:47:48.111267 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.111216 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-workload-certs\") pod \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " Nov 03 19:47:48.111359 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.111275 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-data\") pod \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " Nov 03 19:47:48.111359 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.111317 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-kube-api-access-hc4mc\" (UniqueName: \"kubernetes.io/projected/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-kube-api-access-hc4mc\") pod \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " Nov 03 19:47:48.111359 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.111352 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-podinfo\") pod \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " Nov 03 19:47:48.111523 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.111394 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-envoy\") pod \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " Nov 03 19:47:48.111523 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.111418 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-workload-socket" pod "329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" (UID: "329073b3-fd81-4e4c-8ea6-cebbd5f45c7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:48.111523 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.111439 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istiod-ca-cert\") pod \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " Nov 03 19:47:48.111523 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.111450 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-credential-socket" pod "329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" (UID: "329073b3-fd81-4e4c-8ea6-cebbd5f45c7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:48.111523 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.111487 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-token\" (UniqueName: \"kubernetes.io/projected/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-token\") pod \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\" (UID: \"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d\") " Nov 03 19:47:48.111785 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.111579 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-data" pod "329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" (UID: "329073b3-fd81-4e4c-8ea6-cebbd5f45c7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:48.111785 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.111597 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-workload-certs" pod "329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" (UID: "329073b3-fd81-4e4c-8ea6-cebbd5f45c7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:48.111886 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.111870 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.111939 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.111890 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.111939 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.111905 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.111939 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.111918 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.112092 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.112030 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istiod-ca-cert" pod "329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" (UID: "329073b3-fd81-4e4c-8ea6-cebbd5f45c7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:47:48.113603 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.113576 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-envoy" pod "329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" (UID: "329073b3-fd81-4e4c-8ea6-cebbd5f45c7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:48.113603 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.113578 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-podinfo" pod "329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" (UID: "329073b3-fd81-4e4c-8ea6-cebbd5f45c7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:47:48.113955 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.113935 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-token" pod "329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" (UID: "329073b3-fd81-4e4c-8ea6-cebbd5f45c7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:48.114009 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.113963 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-kube-api-access-hc4mc" pod "329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" (UID: "329073b3-fd81-4e4c-8ea6-cebbd5f45c7d"). InnerVolumeSpecName "kube-api-access-hc4mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:48.217148 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.217114 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc4mc\" (UniqueName: \"kubernetes.io/projected/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-kube-api-access-hc4mc\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.217148 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.217149 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.217304 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.217163 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.217304 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.217176 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.217304 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.217188 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.237083 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:47:48.237037 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf337606e_fc9b_43c6_8ac4_205f2e011dde.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbce3ead_79e2_4d3e_8caa_8a35a3640e75.slice/crio-80c1a4918c7f81e626f5a44c16094248783d8140ce796e04bd90a35988d1396d\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329073b3_fd81_4e4c_8ea6_cebbd5f45c7d.slice/crio-conmon-1bbd05af50b32d0c319dc05258f2cf0eb6e21cd5bc159a6db3e622dd227dc8fe.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329073b3_fd81_4e4c_8ea6_cebbd5f45c7d.slice/crio-1bbd05af50b32d0c319dc05258f2cf0eb6e21cd5bc159a6db3e622dd227dc8fe.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf337606e_fc9b_43c6_8ac4_205f2e011dde.slice/crio-15ea57f997225d69d43afb1aeaf47701842b34016040f98784269ca61cfe0497\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087fa86d_7f7a_45a8_b368_b0052d5f5ec0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbce3ead_79e2_4d3e_8caa_8a35a3640e75.slice/crio-d37a4cd3eecbd37a2b70844c7ae297fc283a99464149be8dea809712753f7a83.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbce3ead_79e2_4d3e_8caa_8a35a3640e75.slice/crio-conmon-d37a4cd3eecbd37a2b70844c7ae297fc283a99464149be8dea809712753f7a83.scope\": RecentStats: unable to find data in memory cache]" Nov 03 19:47:48.242151 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:47:48.242125 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087fa86d_7f7a_45a8_b368_b0052d5f5ec0.slice/crio-0ca372c1281d9058bf76ae8e15097135e00416887c56f9bee86728e973dbff84\": RecentStats: unable to find data in memory cache]" Nov 03 19:47:48.332792 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.332769 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:48.396375 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.396343 5059 generic.go:356] "Generic (PLEG): container finished" podID="cbce3ead-79e2-4d3e-8caa-8a35a3640e75" containerID="d37a4cd3eecbd37a2b70844c7ae297fc283a99464149be8dea809712753f7a83" exitCode=0 Nov 03 19:47:48.396554 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.396408 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" Nov 03 19:47:48.396554 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.396420 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" event={"ID":"cbce3ead-79e2-4d3e-8caa-8a35a3640e75","Type":"ContainerDied","Data":"d37a4cd3eecbd37a2b70844c7ae297fc283a99464149be8dea809712753f7a83"} Nov 03 19:47:48.396554 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.396513 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx" event={"ID":"cbce3ead-79e2-4d3e-8caa-8a35a3640e75","Type":"ContainerDied","Data":"80c1a4918c7f81e626f5a44c16094248783d8140ce796e04bd90a35988d1396d"} Nov 03 19:47:48.396554 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.396540 5059 scope.go:122] "RemoveContainer" containerID="d37a4cd3eecbd37a2b70844c7ae297fc283a99464149be8dea809712753f7a83" Nov 03 19:47:48.398117 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.398097 5059 generic.go:356] "Generic (PLEG): container finished" podID="329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" containerID="1bbd05af50b32d0c319dc05258f2cf0eb6e21cd5bc159a6db3e622dd227dc8fe" exitCode=0 Nov 03 19:47:48.398233 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.398146 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" Nov 03 19:47:48.398233 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.398184 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" event={"ID":"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d","Type":"ContainerDied","Data":"1bbd05af50b32d0c319dc05258f2cf0eb6e21cd5bc159a6db3e622dd227dc8fe"} Nov 03 19:47:48.398310 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.398243 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb" event={"ID":"329073b3-fd81-4e4c-8ea6-cebbd5f45c7d","Type":"ContainerDied","Data":"536d9ffc69aeb979adc1ca1d0c71c3c79e3d533186acea52f136d16f2de74046"} Nov 03 19:47:48.398550 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.398526 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:48.399474 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.399457 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:47:48.408608 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.408340 5059 scope.go:122] "RemoveContainer" containerID="cc59a5dc60887acb9a2ac3411bb69ac6aace5f5f03de774f152d126699d06328" Nov 03 19:47:48.417569 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.417553 5059 scope.go:122] "RemoveContainer" containerID="d37a4cd3eecbd37a2b70844c7ae297fc283a99464149be8dea809712753f7a83" Nov 03 19:47:48.417863 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:47:48.417846 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37a4cd3eecbd37a2b70844c7ae297fc283a99464149be8dea809712753f7a83\": container with ID starting with d37a4cd3eecbd37a2b70844c7ae297fc283a99464149be8dea809712753f7a83 not found: ID does not exist" containerID="d37a4cd3eecbd37a2b70844c7ae297fc283a99464149be8dea809712753f7a83" Nov 03 19:47:48.417906 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.417872 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37a4cd3eecbd37a2b70844c7ae297fc283a99464149be8dea809712753f7a83"} err="failed to get container status \"d37a4cd3eecbd37a2b70844c7ae297fc283a99464149be8dea809712753f7a83\": rpc error: code = NotFound desc = could not find container \"d37a4cd3eecbd37a2b70844c7ae297fc283a99464149be8dea809712753f7a83\": container with ID starting with d37a4cd3eecbd37a2b70844c7ae297fc283a99464149be8dea809712753f7a83 not found: ID does not exist" Nov 03 19:47:48.417906 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.417887 5059 scope.go:122] "RemoveContainer" containerID="cc59a5dc60887acb9a2ac3411bb69ac6aace5f5f03de774f152d126699d06328" Nov 03 19:47:48.418092 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:47:48.418076 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc59a5dc60887acb9a2ac3411bb69ac6aace5f5f03de774f152d126699d06328\": container with ID starting with cc59a5dc60887acb9a2ac3411bb69ac6aace5f5f03de774f152d126699d06328 not found: ID does not exist" containerID="cc59a5dc60887acb9a2ac3411bb69ac6aace5f5f03de774f152d126699d06328" Nov 03 19:47:48.418136 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.418095 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc59a5dc60887acb9a2ac3411bb69ac6aace5f5f03de774f152d126699d06328"} err="failed to get container status \"cc59a5dc60887acb9a2ac3411bb69ac6aace5f5f03de774f152d126699d06328\": rpc error: code = NotFound desc = could not find container \"cc59a5dc60887acb9a2ac3411bb69ac6aace5f5f03de774f152d126699d06328\": container with ID starting with cc59a5dc60887acb9a2ac3411bb69ac6aace5f5f03de774f152d126699d06328 not found: ID does not exist" Nov 03 19:47:48.418136 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.418108 5059 scope.go:122] "RemoveContainer" containerID="1bbd05af50b32d0c319dc05258f2cf0eb6e21cd5bc159a6db3e622dd227dc8fe" Nov 03 19:47:48.418555 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.418537 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-envoy\") pod \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " Nov 03 19:47:48.418616 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.418566 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-credential-socket\") pod \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " Nov 03 19:47:48.418616 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.418588 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-kube-api-access-nm8d9\" (UniqueName: \"kubernetes.io/projected/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-kube-api-access-nm8d9\") pod \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " Nov 03 19:47:48.418738 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.418626 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-workload-certs\") pod \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " Nov 03 19:47:48.418738 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.418690 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-podinfo\") pod \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " Nov 03 19:47:48.418840 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.418736 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-token\" (UniqueName: \"kubernetes.io/projected/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-token\") pod \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " Nov 03 19:47:48.418840 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.418785 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-workload-socket\") pod \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " Nov 03 19:47:48.418942 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.418875 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-credential-socket" pod "cbce3ead-79e2-4d3e-8caa-8a35a3640e75" (UID: "cbce3ead-79e2-4d3e-8caa-8a35a3640e75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:48.418942 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.418912 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-data\") pod \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " Nov 03 19:47:48.419044 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.418944 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-workload-certs" pod "cbce3ead-79e2-4d3e-8caa-8a35a3640e75" (UID: "cbce3ead-79e2-4d3e-8caa-8a35a3640e75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:48.419044 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.419014 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istiod-ca-cert\") pod \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\" (UID: \"cbce3ead-79e2-4d3e-8caa-8a35a3640e75\") " Nov 03 19:47:48.419148 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.419131 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-data" pod "cbce3ead-79e2-4d3e-8caa-8a35a3640e75" (UID: "cbce3ead-79e2-4d3e-8caa-8a35a3640e75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:48.419209 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.419161 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-workload-socket" pod "cbce3ead-79e2-4d3e-8caa-8a35a3640e75" (UID: "cbce3ead-79e2-4d3e-8caa-8a35a3640e75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:48.419378 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.419358 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.419441 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.419387 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.419441 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.419403 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.419441 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.419406 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istiod-ca-cert" pod "cbce3ead-79e2-4d3e-8caa-8a35a3640e75" (UID: "cbce3ead-79e2-4d3e-8caa-8a35a3640e75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:47:48.419441 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.419418 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.420785 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.420757 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-envoy" pod "cbce3ead-79e2-4d3e-8caa-8a35a3640e75" (UID: "cbce3ead-79e2-4d3e-8caa-8a35a3640e75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:48.421074 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.421058 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-token" pod "cbce3ead-79e2-4d3e-8caa-8a35a3640e75" (UID: "cbce3ead-79e2-4d3e-8caa-8a35a3640e75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:48.421189 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.421176 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-podinfo" pod "cbce3ead-79e2-4d3e-8caa-8a35a3640e75" (UID: "cbce3ead-79e2-4d3e-8caa-8a35a3640e75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:47:48.421269 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.421255 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-kube-api-access-nm8d9" pod "cbce3ead-79e2-4d3e-8caa-8a35a3640e75" (UID: "cbce3ead-79e2-4d3e-8caa-8a35a3640e75"). InnerVolumeSpecName "kube-api-access-nm8d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:48.427230 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.427199 5059 scope.go:122] "RemoveContainer" containerID="b2c8777e659022d83bc00a050326e45373018d9c4d228874596bf49434c6a751" Nov 03 19:47:48.437529 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.437503 5059 scope.go:122] "RemoveContainer" containerID="1bbd05af50b32d0c319dc05258f2cf0eb6e21cd5bc159a6db3e622dd227dc8fe" Nov 03 19:47:48.437945 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:47:48.437917 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bbd05af50b32d0c319dc05258f2cf0eb6e21cd5bc159a6db3e622dd227dc8fe\": container with ID starting with 1bbd05af50b32d0c319dc05258f2cf0eb6e21cd5bc159a6db3e622dd227dc8fe not found: ID does not exist" containerID="1bbd05af50b32d0c319dc05258f2cf0eb6e21cd5bc159a6db3e622dd227dc8fe" Nov 03 19:47:48.438043 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.437961 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbd05af50b32d0c319dc05258f2cf0eb6e21cd5bc159a6db3e622dd227dc8fe"} err="failed to get container status \"1bbd05af50b32d0c319dc05258f2cf0eb6e21cd5bc159a6db3e622dd227dc8fe\": rpc error: code = NotFound desc = could not find container \"1bbd05af50b32d0c319dc05258f2cf0eb6e21cd5bc159a6db3e622dd227dc8fe\": container with ID starting with 1bbd05af50b32d0c319dc05258f2cf0eb6e21cd5bc159a6db3e622dd227dc8fe not found: ID does not exist" Nov 03 19:47:48.438043 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.437988 5059 scope.go:122] "RemoveContainer" containerID="b2c8777e659022d83bc00a050326e45373018d9c4d228874596bf49434c6a751" Nov 03 19:47:48.438593 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:47:48.438565 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2c8777e659022d83bc00a050326e45373018d9c4d228874596bf49434c6a751\": container with ID starting with b2c8777e659022d83bc00a050326e45373018d9c4d228874596bf49434c6a751 not found: ID does not exist" containerID="b2c8777e659022d83bc00a050326e45373018d9c4d228874596bf49434c6a751" Nov 03 19:47:48.438698 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.438600 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c8777e659022d83bc00a050326e45373018d9c4d228874596bf49434c6a751"} err="failed to get container status \"b2c8777e659022d83bc00a050326e45373018d9c4d228874596bf49434c6a751\": rpc error: code = NotFound desc = could not find container \"b2c8777e659022d83bc00a050326e45373018d9c4d228874596bf49434c6a751\": container with ID starting with b2c8777e659022d83bc00a050326e45373018d9c4d228874596bf49434c6a751 not found: ID does not exist" Nov 03 19:47:48.457020 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.456937 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb"] Nov 03 19:47:48.460690 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.460651 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-745a88f9-openshift-default-7fd449db85-wj6bb"] Nov 03 19:47:48.520431 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.520400 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.520585 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.520441 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.520585 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.520457 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm8d9\" (UniqueName: \"kubernetes.io/projected/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-kube-api-access-nm8d9\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.520585 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.520470 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.520585 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.520482 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbce3ead-79e2-4d3e-8caa-8a35a3640e75-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:48.725489 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.725426 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx"] Nov 03 19:47:48.730991 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.730968 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-9959e5f4-gw-openshift-default-c5d65d75c-h47mx"] Nov 03 19:47:48.776440 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.776406 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6"] Nov 03 19:47:48.776873 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.776858 5059 cpu_manager.go:414] "containerMap: removing container" podUID="cbce3ead-79e2-4d3e-8caa-8a35a3640e75" containerName="istio-proxy" Nov 03 19:47:48.776873 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.776874 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbce3ead-79e2-4d3e-8caa-8a35a3640e75" containerName="istio-proxy" Nov 03 19:47:48.776977 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.776886 5059 cpu_manager.go:414] "containerMap: removing container" podUID="cbce3ead-79e2-4d3e-8caa-8a35a3640e75" containerName="istio-proxy" Nov 03 19:47:48.776977 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.776892 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbce3ead-79e2-4d3e-8caa-8a35a3640e75" containerName="istio-proxy" Nov 03 19:47:48.776977 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.776898 5059 cpu_manager.go:414] "containerMap: removing container" podUID="329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" containerName="istio-proxy" Nov 03 19:47:48.776977 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.776903 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" containerName="istio-proxy" Nov 03 19:47:48.776977 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.776918 5059 cpu_manager.go:414] "containerMap: removing container" podUID="329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" containerName="istio-proxy" Nov 03 19:47:48.776977 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.776923 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" containerName="istio-proxy" Nov 03 19:47:48.777248 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.777013 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbce3ead-79e2-4d3e-8caa-8a35a3640e75" containerName="istio-proxy" Nov 03 19:47:48.777248 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.777021 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbce3ead-79e2-4d3e-8caa-8a35a3640e75" containerName="istio-proxy" Nov 03 19:47:48.777248 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.777026 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" containerName="istio-proxy" Nov 03 19:47:48.782029 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.782002 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:48.800139 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.800101 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-32e482-openshift-default-dockercfg-5pv2v\"" Nov 03 19:47:48.803130 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.802981 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6"] Nov 03 19:47:48.891549 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.891517 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl"] Nov 03 19:47:48.892398 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.892376 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" containerName="istio-proxy" Nov 03 19:47:48.896181 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.896148 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl" Nov 03 19:47:48.909391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.909085 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl"] Nov 03 19:47:48.925340 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.925294 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:48.925497 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.925379 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:48.925497 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.925425 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:48.925497 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.925454 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:48.925677 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.925504 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5jnb\" (UniqueName: \"kubernetes.io/projected/eac98813-7b4c-4c4a-8bf9-444bba2fa644-kube-api-access-q5jnb\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:48.925677 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.925541 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:48.925677 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.925580 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:48.925677 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.925614 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:48.925677 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.925645 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:48.935528 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.935502 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9"] Nov 03 19:47:48.939998 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.939978 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9" Nov 03 19:47:48.949874 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.948917 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-9f46e5-dockercfg-6g47t\"" Nov 03 19:47:48.953525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.953501 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9"] Nov 03 19:47:48.960081 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:48.960047 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-9f46e5\"" Nov 03 19:47:49.027379 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.027287 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g2ll\" (UniqueName: \"kubernetes.io/projected/9c3d72b6-73c4-402f-8308-ee5d87a9dd83-kube-api-access-8g2ll\") pod \"payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl\" (UID: \"9c3d72b6-73c4-402f-8308-ee5d87a9dd83\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl" Nov 03 19:47:49.027379 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.027336 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j788w\" (UniqueName: \"kubernetes.io/projected/890f130b-cdeb-43da-acd4-1654673271cc-kube-api-access-j788w\") pod \"payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9\" (UID: \"890f130b-cdeb-43da-acd4-1654673271cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9" Nov 03 19:47:49.027582 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.027439 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5jnb\" (UniqueName: \"kubernetes.io/projected/eac98813-7b4c-4c4a-8bf9-444bba2fa644-kube-api-access-q5jnb\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:49.027582 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.027490 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:49.027582 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.027529 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:49.027582 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.027562 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:49.027844 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.027591 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:49.027844 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.027711 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:49.027844 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.027777 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/890f130b-cdeb-43da-acd4-1654673271cc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9\" (UID: \"890f130b-cdeb-43da-acd4-1654673271cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9" Nov 03 19:47:49.027844 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.027827 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:49.028034 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.027881 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:49.028034 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.027909 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3d72b6-73c4-402f-8308-ee5d87a9dd83-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl\" (UID: \"9c3d72b6-73c4-402f-8308-ee5d87a9dd83\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl" Nov 03 19:47:49.028034 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.027943 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:49.028034 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.027967 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:49.028289 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.028262 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:49.028375 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.028285 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:49.028375 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.028352 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:49.028969 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.028944 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:49.030716 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.030691 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:49.031144 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.031119 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:49.051028 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.050992 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:49.051137 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.050998 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5jnb\" (UniqueName: \"kubernetes.io/projected/eac98813-7b4c-4c4a-8bf9-444bba2fa644-kube-api-access-q5jnb\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:49.096472 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.096443 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:49.141729 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.129711 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/890f130b-cdeb-43da-acd4-1654673271cc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9\" (UID: \"890f130b-cdeb-43da-acd4-1654673271cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9" Nov 03 19:47:49.141729 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.129811 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3d72b6-73c4-402f-8308-ee5d87a9dd83-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl\" (UID: \"9c3d72b6-73c4-402f-8308-ee5d87a9dd83\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl" Nov 03 19:47:49.141729 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.129868 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g2ll\" (UniqueName: \"kubernetes.io/projected/9c3d72b6-73c4-402f-8308-ee5d87a9dd83-kube-api-access-8g2ll\") pod \"payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl\" (UID: \"9c3d72b6-73c4-402f-8308-ee5d87a9dd83\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl" Nov 03 19:47:49.141729 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.129893 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j788w\" (UniqueName: \"kubernetes.io/projected/890f130b-cdeb-43da-acd4-1654673271cc-kube-api-access-j788w\") pod \"payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9\" (UID: \"890f130b-cdeb-43da-acd4-1654673271cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9" Nov 03 19:47:49.141729 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.131421 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3d72b6-73c4-402f-8308-ee5d87a9dd83-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl\" (UID: \"9c3d72b6-73c4-402f-8308-ee5d87a9dd83\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl" Nov 03 19:47:49.141729 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.131891 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/890f130b-cdeb-43da-acd4-1654673271cc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9\" (UID: \"890f130b-cdeb-43da-acd4-1654673271cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9" Nov 03 19:47:49.152474 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.152412 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j788w\" (UniqueName: \"kubernetes.io/projected/890f130b-cdeb-43da-acd4-1654673271cc-kube-api-access-j788w\") pod \"payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9\" (UID: \"890f130b-cdeb-43da-acd4-1654673271cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9" Nov 03 19:47:49.180252 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.177846 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g2ll\" (UniqueName: \"kubernetes.io/projected/9c3d72b6-73c4-402f-8308-ee5d87a9dd83-kube-api-access-8g2ll\") pod \"payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl\" (UID: \"9c3d72b6-73c4-402f-8308-ee5d87a9dd83\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl" Nov 03 19:47:49.212771 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.212736 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl" Nov 03 19:47:49.258471 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.258424 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9" Nov 03 19:47:49.426016 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.425989 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6"] Nov 03 19:47:49.427237 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:47:49.427208 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeac98813_7b4c_4c4a_8bf9_444bba2fa644.slice/crio-139eb572a391573df18b064fe99053fa177236d3732ef6a9676a36d718e52a4a WatchSource:0}: Error finding container 139eb572a391573df18b064fe99053fa177236d3732ef6a9676a36d718e52a4a: Status 404 returned error can't find the container with id 139eb572a391573df18b064fe99053fa177236d3732ef6a9676a36d718e52a4a Nov 03 19:47:49.429371 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.429340 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:47:49.429437 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.429403 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:47:49.429437 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.429430 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:47:49.508161 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.508137 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl"] Nov 03 19:47:49.509981 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:47:49.509957 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3d72b6_73c4_402f_8308_ee5d87a9dd83.slice/crio-79ebb628096fb75f5b9ad65ace257111a2c6249c36bb44d175dca428fee2c503 WatchSource:0}: Error finding container 79ebb628096fb75f5b9ad65ace257111a2c6249c36bb44d175dca428fee2c503: Status 404 returned error can't find the container with id 79ebb628096fb75f5b9ad65ace257111a2c6249c36bb44d175dca428fee2c503 Nov 03 19:47:49.563753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.555221 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9"] Nov 03 19:47:49.579286 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:47:49.579252 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890f130b_cdeb_43da_acd4_1654673271cc.slice/crio-ef2df6bf30fd3d11eec890703aa6deaf10a3cd2e8b8333e43dae23ca4c91de09 WatchSource:0}: Error finding container ef2df6bf30fd3d11eec890703aa6deaf10a3cd2e8b8333e43dae23ca4c91de09: Status 404 returned error can't find the container with id ef2df6bf30fd3d11eec890703aa6deaf10a3cd2e8b8333e43dae23ca4c91de09 Nov 03 19:47:49.782821 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.782789 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn"] Nov 03 19:47:49.786736 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.786719 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn" Nov 03 19:47:49.797214 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.797192 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn"] Nov 03 19:47:49.856960 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.845122 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-2b5703\"" Nov 03 19:47:49.856960 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.845197 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-2b5703-dockercfg-b84j2\"" Nov 03 19:47:49.866995 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.864772 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="329073b3-fd81-4e4c-8ea6-cebbd5f45c7d" path="/var/lib/kubelet/pods/329073b3-fd81-4e4c-8ea6-cebbd5f45c7d/volumes" Nov 03 19:47:49.866995 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.865507 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbce3ead-79e2-4d3e-8caa-8a35a3640e75" path="/var/lib/kubelet/pods/cbce3ead-79e2-4d3e-8caa-8a35a3640e75/volumes" Nov 03 19:47:49.937599 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.937567 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpnqh\" (UniqueName: \"kubernetes.io/projected/25ceab3d-6344-4512-a471-d91aec0cb48c-kube-api-access-hpnqh\") pod \"payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn\" (UID: \"25ceab3d-6344-4512-a471-d91aec0cb48c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn" Nov 03 19:47:49.938039 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:49.937610 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ceab3d-6344-4512-a471-d91aec0cb48c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn\" (UID: \"25ceab3d-6344-4512-a471-d91aec0cb48c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn" Nov 03 19:47:50.038619 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:50.038583 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpnqh\" (UniqueName: \"kubernetes.io/projected/25ceab3d-6344-4512-a471-d91aec0cb48c-kube-api-access-hpnqh\") pod \"payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn\" (UID: \"25ceab3d-6344-4512-a471-d91aec0cb48c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn" Nov 03 19:47:50.038619 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:50.038630 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ceab3d-6344-4512-a471-d91aec0cb48c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn\" (UID: \"25ceab3d-6344-4512-a471-d91aec0cb48c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn" Nov 03 19:47:50.039354 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:50.039333 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ceab3d-6344-4512-a471-d91aec0cb48c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn\" (UID: \"25ceab3d-6344-4512-a471-d91aec0cb48c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn" Nov 03 19:47:50.052104 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:50.052079 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpnqh\" (UniqueName: \"kubernetes.io/projected/25ceab3d-6344-4512-a471-d91aec0cb48c-kube-api-access-hpnqh\") pod \"payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn\" (UID: \"25ceab3d-6344-4512-a471-d91aec0cb48c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn" Nov 03 19:47:50.101196 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:50.101168 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn" Nov 03 19:47:50.395060 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:50.395027 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn"] Nov 03 19:47:50.397274 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:47:50.397248 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ceab3d_6344_4512_a471_d91aec0cb48c.slice/crio-78cef308b00fc9217bde938ee3c21f496c6ebd2c6184bb774beaa70f0faf95db WatchSource:0}: Error finding container 78cef308b00fc9217bde938ee3c21f496c6ebd2c6184bb774beaa70f0faf95db: Status 404 returned error can't find the container with id 78cef308b00fc9217bde938ee3c21f496c6ebd2c6184bb774beaa70f0faf95db Nov 03 19:47:50.413626 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:50.413596 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" event={"ID":"eac98813-7b4c-4c4a-8bf9-444bba2fa644","Type":"ContainerStarted","Data":"89b4df7aa727c470a64d52c6160fcb987049e7d0bf5bc45f6489c13773b39202"} Nov 03 19:47:50.413788 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:50.413656 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" event={"ID":"eac98813-7b4c-4c4a-8bf9-444bba2fa644","Type":"ContainerStarted","Data":"139eb572a391573df18b064fe99053fa177236d3732ef6a9676a36d718e52a4a"} Nov 03 19:47:50.415266 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:50.415237 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn" event={"ID":"25ceab3d-6344-4512-a471-d91aec0cb48c","Type":"ContainerStarted","Data":"78cef308b00fc9217bde938ee3c21f496c6ebd2c6184bb774beaa70f0faf95db"} Nov 03 19:47:50.417181 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:50.417152 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl" event={"ID":"9c3d72b6-73c4-402f-8308-ee5d87a9dd83","Type":"ContainerStarted","Data":"3d68de58f4e860642d4d4255a14fe00815183f88da48e5f851f0a0c70e971419"} Nov 03 19:47:50.417287 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:50.417196 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl" event={"ID":"9c3d72b6-73c4-402f-8308-ee5d87a9dd83","Type":"ContainerStarted","Data":"79ebb628096fb75f5b9ad65ace257111a2c6249c36bb44d175dca428fee2c503"} Nov 03 19:47:50.417287 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:50.417219 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl" Nov 03 19:47:50.419031 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:50.419006 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9" event={"ID":"890f130b-cdeb-43da-acd4-1654673271cc","Type":"ContainerStarted","Data":"80563e166a6378b34d09a54f08d0ed8f1e665107abe95933f0fa76b31fc439e7"} Nov 03 19:47:50.419127 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:50.419055 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9" event={"ID":"890f130b-cdeb-43da-acd4-1654673271cc","Type":"ContainerStarted","Data":"ef2df6bf30fd3d11eec890703aa6deaf10a3cd2e8b8333e43dae23ca4c91de09"} Nov 03 19:47:50.419243 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:50.419230 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9" Nov 03 19:47:50.470697 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:50.469453 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" podStartSLOduration=2.469435494 podStartE2EDuration="2.469435494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:50.445886452 +0000 UTC m=+664.703683711" watchObservedRunningTime="2026-11-03 19:47:50.469435494 +0000 UTC m=+664.727232752" Nov 03 19:47:50.471753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:50.471714 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl" podStartSLOduration=2.471702793 podStartE2EDuration="2.471702793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:50.469167384 +0000 UTC m=+664.726964642" watchObservedRunningTime="2026-11-03 19:47:50.471702793 +0000 UTC m=+664.729500047" Nov 03 19:47:50.497398 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:50.497357 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9" podStartSLOduration=2.497344242 podStartE2EDuration="2.497344242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:50.49549667 +0000 UTC m=+664.753293928" watchObservedRunningTime="2026-11-03 19:47:50.497344242 +0000 UTC m=+664.755141498" Nov 03 19:47:51.097004 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:51.096966 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:51.101797 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:51.101773 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:51.426163 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:51.426077 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn" event={"ID":"25ceab3d-6344-4512-a471-d91aec0cb48c","Type":"ContainerStarted","Data":"b13a3b8cfa23ddc9815549092296b81b15bde977434b7a3b30d1385e768e8fb0"} Nov 03 19:47:51.426458 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:51.426435 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn" Nov 03 19:47:51.426786 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:51.426764 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:51.427746 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:51.427727 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:47:51.447280 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:51.447234 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn" podStartSLOduration=2.447220298 podStartE2EDuration="2.447220298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:51.444779385 +0000 UTC m=+665.702576634" watchObservedRunningTime="2026-11-03 19:47:51.447220298 +0000 UTC m=+665.705017551" Nov 03 19:47:51.608235 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:51.608197 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn" podUID="3f7e4716-e0cb-4170-82f4-aa74ef088ef3" containerName="payload-processing" containerID="cri-o://b53bf546a51a84e4668237abf191a7f8072957a666b12f2b83bceaf5b150492a" gracePeriod=25 Nov 03 19:47:51.622986 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:51.622960 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m" podUID="d1c49a14-9ecc-4de2-a81d-94621505ca70" containerName="payload-pre-processing" containerID="cri-o://5c84b0527753f42c973b20350fa943627a491bc9f4cd2f01e6c16a9dca20b405" gracePeriod=25 Nov 03 19:47:51.906786 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:51.906762 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn" Nov 03 19:47:51.936028 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:51.935971 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m" Nov 03 19:47:52.064118 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.064089 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c49a14-9ecc-4de2-a81d-94621505ca70-kube-api-access-xhp66\" (UniqueName: \"kubernetes.io/projected/d1c49a14-9ecc-4de2-a81d-94621505ca70-kube-api-access-xhp66\") pod \"d1c49a14-9ecc-4de2-a81d-94621505ca70\" (UID: \"d1c49a14-9ecc-4de2-a81d-94621505ca70\") " Nov 03 19:47:52.064253 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.064142 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1c49a14-9ecc-4de2-a81d-94621505ca70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c49a14-9ecc-4de2-a81d-94621505ca70-plugins-config-volume\") pod \"d1c49a14-9ecc-4de2-a81d-94621505ca70\" (UID: \"d1c49a14-9ecc-4de2-a81d-94621505ca70\") " Nov 03 19:47:52.064253 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.064238 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7e4716-e0cb-4170-82f4-aa74ef088ef3-kube-api-access-cmcl7\" (UniqueName: \"kubernetes.io/projected/3f7e4716-e0cb-4170-82f4-aa74ef088ef3-kube-api-access-cmcl7\") pod \"3f7e4716-e0cb-4170-82f4-aa74ef088ef3\" (UID: \"3f7e4716-e0cb-4170-82f4-aa74ef088ef3\") " Nov 03 19:47:52.064357 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.064335 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f7e4716-e0cb-4170-82f4-aa74ef088ef3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7e4716-e0cb-4170-82f4-aa74ef088ef3-plugins-config-volume\") pod \"3f7e4716-e0cb-4170-82f4-aa74ef088ef3\" (UID: \"3f7e4716-e0cb-4170-82f4-aa74ef088ef3\") " Nov 03 19:47:52.064646 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.064623 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c49a14-9ecc-4de2-a81d-94621505ca70-plugins-config-volume" pod "d1c49a14-9ecc-4de2-a81d-94621505ca70" (UID: "d1c49a14-9ecc-4de2-a81d-94621505ca70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:47:52.064734 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.064646 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f7e4716-e0cb-4170-82f4-aa74ef088ef3-plugins-config-volume" pod "3f7e4716-e0cb-4170-82f4-aa74ef088ef3" (UID: "3f7e4716-e0cb-4170-82f4-aa74ef088ef3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:47:52.064819 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.064804 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c49a14-9ecc-4de2-a81d-94621505ca70-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:52.064886 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.064820 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7e4716-e0cb-4170-82f4-aa74ef088ef3-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:52.066306 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.066280 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7e4716-e0cb-4170-82f4-aa74ef088ef3-kube-api-access-cmcl7" pod "3f7e4716-e0cb-4170-82f4-aa74ef088ef3" (UID: "3f7e4716-e0cb-4170-82f4-aa74ef088ef3"). InnerVolumeSpecName "kube-api-access-cmcl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:52.066459 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.066442 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c49a14-9ecc-4de2-a81d-94621505ca70-kube-api-access-xhp66" pod "d1c49a14-9ecc-4de2-a81d-94621505ca70" (UID: "d1c49a14-9ecc-4de2-a81d-94621505ca70"). InnerVolumeSpecName "kube-api-access-xhp66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:52.165989 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.165963 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmcl7\" (UniqueName: \"kubernetes.io/projected/3f7e4716-e0cb-4170-82f4-aa74ef088ef3-kube-api-access-cmcl7\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:52.166304 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.165988 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhp66\" (UniqueName: \"kubernetes.io/projected/d1c49a14-9ecc-4de2-a81d-94621505ca70-kube-api-access-xhp66\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:52.439919 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.439883 5059 generic.go:356] "Generic (PLEG): container finished" podID="3f7e4716-e0cb-4170-82f4-aa74ef088ef3" containerID="b53bf546a51a84e4668237abf191a7f8072957a666b12f2b83bceaf5b150492a" exitCode=0 Nov 03 19:47:52.440117 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.439944 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn" Nov 03 19:47:52.440117 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.440061 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn" event={"ID":"3f7e4716-e0cb-4170-82f4-aa74ef088ef3","Type":"ContainerDied","Data":"b53bf546a51a84e4668237abf191a7f8072957a666b12f2b83bceaf5b150492a"} Nov 03 19:47:52.440117 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.440108 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn" event={"ID":"3f7e4716-e0cb-4170-82f4-aa74ef088ef3","Type":"ContainerDied","Data":"7d90f7b653c24e702c89ec09b167286d40b3ce05062f62ac554503fa344973fe"} Nov 03 19:47:52.440275 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.440139 5059 scope.go:122] "RemoveContainer" containerID="b53bf546a51a84e4668237abf191a7f8072957a666b12f2b83bceaf5b150492a" Nov 03 19:47:52.455686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.453671 5059 generic.go:356] "Generic (PLEG): container finished" podID="d1c49a14-9ecc-4de2-a81d-94621505ca70" containerID="5c84b0527753f42c973b20350fa943627a491bc9f4cd2f01e6c16a9dca20b405" exitCode=0 Nov 03 19:47:52.455686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.454772 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m" Nov 03 19:47:52.467941 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.467899 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m" event={"ID":"d1c49a14-9ecc-4de2-a81d-94621505ca70","Type":"ContainerDied","Data":"5c84b0527753f42c973b20350fa943627a491bc9f4cd2f01e6c16a9dca20b405"} Nov 03 19:47:52.468111 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.467962 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m" event={"ID":"d1c49a14-9ecc-4de2-a81d-94621505ca70","Type":"ContainerDied","Data":"a8d15d1a1a9f8a25f1b7e2976211b4c10fe2af65e3035dada57725a6579b2773"} Nov 03 19:47:52.474642 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.474618 5059 scope.go:122] "RemoveContainer" containerID="b53bf546a51a84e4668237abf191a7f8072957a666b12f2b83bceaf5b150492a" Nov 03 19:47:52.475272 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:47:52.475246 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b53bf546a51a84e4668237abf191a7f8072957a666b12f2b83bceaf5b150492a\": container with ID starting with b53bf546a51a84e4668237abf191a7f8072957a666b12f2b83bceaf5b150492a not found: ID does not exist" containerID="b53bf546a51a84e4668237abf191a7f8072957a666b12f2b83bceaf5b150492a" Nov 03 19:47:52.475377 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.475276 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b53bf546a51a84e4668237abf191a7f8072957a666b12f2b83bceaf5b150492a"} err="failed to get container status \"b53bf546a51a84e4668237abf191a7f8072957a666b12f2b83bceaf5b150492a\": rpc error: code = NotFound desc = could not find container \"b53bf546a51a84e4668237abf191a7f8072957a666b12f2b83bceaf5b150492a\": container with ID starting with b53bf546a51a84e4668237abf191a7f8072957a666b12f2b83bceaf5b150492a not found: ID does not exist" Nov 03 19:47:52.475377 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.475290 5059 scope.go:122] "RemoveContainer" containerID="5c84b0527753f42c973b20350fa943627a491bc9f4cd2f01e6c16a9dca20b405" Nov 03 19:47:52.486335 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.486317 5059 scope.go:122] "RemoveContainer" containerID="5c84b0527753f42c973b20350fa943627a491bc9f4cd2f01e6c16a9dca20b405" Nov 03 19:47:52.486605 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:47:52.486586 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c84b0527753f42c973b20350fa943627a491bc9f4cd2f01e6c16a9dca20b405\": container with ID starting with 5c84b0527753f42c973b20350fa943627a491bc9f4cd2f01e6c16a9dca20b405 not found: ID does not exist" containerID="5c84b0527753f42c973b20350fa943627a491bc9f4cd2f01e6c16a9dca20b405" Nov 03 19:47:52.486681 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.486611 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c84b0527753f42c973b20350fa943627a491bc9f4cd2f01e6c16a9dca20b405"} err="failed to get container status \"5c84b0527753f42c973b20350fa943627a491bc9f4cd2f01e6c16a9dca20b405\": rpc error: code = NotFound desc = could not find container \"5c84b0527753f42c973b20350fa943627a491bc9f4cd2f01e6c16a9dca20b405\": container with ID starting with 5c84b0527753f42c973b20350fa943627a491bc9f4cd2f01e6c16a9dca20b405 not found: ID does not exist" Nov 03 19:47:52.495473 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.495450 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn"] Nov 03 19:47:52.513331 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.513306 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8c14f244-696d54c4dd-dzszn"] Nov 03 19:47:52.531649 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.531623 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m"] Nov 03 19:47:52.544999 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.544974 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8c14f244-656fd8c655-c4l2m"] Nov 03 19:47:52.835609 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.835587 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:52.979856 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.979823 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-credential-socket\") pod \"497df71a-3372-4792-9cc4-980950a8de12\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " Nov 03 19:47:52.979856 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.979858 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-istio-data\") pod \"497df71a-3372-4792-9cc4-980950a8de12\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " Nov 03 19:47:52.980065 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.979884 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-workload-socket\") pod \"497df71a-3372-4792-9cc4-980950a8de12\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " Nov 03 19:47:52.980065 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.979907 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/497df71a-3372-4792-9cc4-980950a8de12-kube-api-access-czwh7\" (UniqueName: \"kubernetes.io/projected/497df71a-3372-4792-9cc4-980950a8de12-kube-api-access-czwh7\") pod \"497df71a-3372-4792-9cc4-980950a8de12\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " Nov 03 19:47:52.980065 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.979964 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/497df71a-3372-4792-9cc4-980950a8de12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/497df71a-3372-4792-9cc4-980950a8de12-istio-podinfo\") pod \"497df71a-3372-4792-9cc4-980950a8de12\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " Nov 03 19:47:52.980065 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.980028 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/497df71a-3372-4792-9cc4-980950a8de12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/497df71a-3372-4792-9cc4-980950a8de12-istiod-ca-cert\") pod \"497df71a-3372-4792-9cc4-980950a8de12\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " Nov 03 19:47:52.980278 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.980080 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-workload-certs\") pod \"497df71a-3372-4792-9cc4-980950a8de12\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " Nov 03 19:47:52.980278 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.980127 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-istio-envoy\") pod \"497df71a-3372-4792-9cc4-980950a8de12\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " Nov 03 19:47:52.980278 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.980173 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/497df71a-3372-4792-9cc4-980950a8de12-istio-token\" (UniqueName: \"kubernetes.io/projected/497df71a-3372-4792-9cc4-980950a8de12-istio-token\") pod \"497df71a-3372-4792-9cc4-980950a8de12\" (UID: \"497df71a-3372-4792-9cc4-980950a8de12\") " Nov 03 19:47:52.980278 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.980177 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-istio-data" pod "497df71a-3372-4792-9cc4-980950a8de12" (UID: "497df71a-3372-4792-9cc4-980950a8de12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:52.980278 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.980180 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-credential-socket" pod "497df71a-3372-4792-9cc4-980950a8de12" (UID: "497df71a-3372-4792-9cc4-980950a8de12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:52.980278 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.980228 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-workload-socket" pod "497df71a-3372-4792-9cc4-980950a8de12" (UID: "497df71a-3372-4792-9cc4-980950a8de12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:52.980566 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.980429 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-workload-certs" pod "497df71a-3372-4792-9cc4-980950a8de12" (UID: "497df71a-3372-4792-9cc4-980950a8de12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:52.980566 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.980526 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/497df71a-3372-4792-9cc4-980950a8de12-istiod-ca-cert" pod "497df71a-3372-4792-9cc4-980950a8de12" (UID: "497df71a-3372-4792-9cc4-980950a8de12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:47:52.980828 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.980792 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:52.980828 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.980815 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:52.980958 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.980829 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:52.980958 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.980842 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/497df71a-3372-4792-9cc4-980950a8de12-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:52.980958 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.980854 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:52.982421 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.982391 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-istio-envoy" pod "497df71a-3372-4792-9cc4-980950a8de12" (UID: "497df71a-3372-4792-9cc4-980950a8de12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:52.982560 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.982535 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497df71a-3372-4792-9cc4-980950a8de12-kube-api-access-czwh7" pod "497df71a-3372-4792-9cc4-980950a8de12" (UID: "497df71a-3372-4792-9cc4-980950a8de12"). InnerVolumeSpecName "kube-api-access-czwh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:52.982710 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.982685 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/497df71a-3372-4792-9cc4-980950a8de12-istio-podinfo" pod "497df71a-3372-4792-9cc4-980950a8de12" (UID: "497df71a-3372-4792-9cc4-980950a8de12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:47:52.982835 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:52.982813 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497df71a-3372-4792-9cc4-980950a8de12-istio-token" pod "497df71a-3372-4792-9cc4-980950a8de12" (UID: "497df71a-3372-4792-9cc4-980950a8de12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:53.082121 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:53.082083 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czwh7\" (UniqueName: \"kubernetes.io/projected/497df71a-3372-4792-9cc4-980950a8de12-kube-api-access-czwh7\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:53.082121 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:53.082122 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/497df71a-3372-4792-9cc4-980950a8de12-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:53.082308 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:53.082139 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/497df71a-3372-4792-9cc4-980950a8de12-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:53.082308 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:53.082154 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/497df71a-3372-4792-9cc4-980950a8de12-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:53.465162 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:53.465124 5059 generic.go:356] "Generic (PLEG): container finished" podID="497df71a-3372-4792-9cc4-980950a8de12" containerID="4ee9f309a44572d20ebfd12bc5c187c007f46324e6283bcb18d338e3705d8df8" exitCode=0 Nov 03 19:47:53.465691 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:53.465234 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" event={"ID":"497df71a-3372-4792-9cc4-980950a8de12","Type":"ContainerDied","Data":"4ee9f309a44572d20ebfd12bc5c187c007f46324e6283bcb18d338e3705d8df8"} Nov 03 19:47:53.465691 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:53.465303 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" event={"ID":"497df71a-3372-4792-9cc4-980950a8de12","Type":"ContainerDied","Data":"84119ccf9aa5290f6ac0565371f91a6952b59f3355f3c2ffc84ba51b025f5a64"} Nov 03 19:47:53.465691 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:53.465329 5059 scope.go:122] "RemoveContainer" containerID="4ee9f309a44572d20ebfd12bc5c187c007f46324e6283bcb18d338e3705d8df8" Nov 03 19:47:53.465691 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:53.465436 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv" Nov 03 19:47:53.485325 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:53.484362 5059 scope.go:122] "RemoveContainer" containerID="512a43fcddf2d69a21758c6b29f9d5edb489c0faf84aaeea5a81243ac933224f" Nov 03 19:47:53.494790 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:53.494767 5059 scope.go:122] "RemoveContainer" containerID="4ee9f309a44572d20ebfd12bc5c187c007f46324e6283bcb18d338e3705d8df8" Nov 03 19:47:53.495075 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:47:53.495052 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ee9f309a44572d20ebfd12bc5c187c007f46324e6283bcb18d338e3705d8df8\": container with ID starting with 4ee9f309a44572d20ebfd12bc5c187c007f46324e6283bcb18d338e3705d8df8 not found: ID does not exist" containerID="4ee9f309a44572d20ebfd12bc5c187c007f46324e6283bcb18d338e3705d8df8" Nov 03 19:47:53.495119 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:53.495085 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ee9f309a44572d20ebfd12bc5c187c007f46324e6283bcb18d338e3705d8df8"} err="failed to get container status \"4ee9f309a44572d20ebfd12bc5c187c007f46324e6283bcb18d338e3705d8df8\": rpc error: code = NotFound desc = could not find container \"4ee9f309a44572d20ebfd12bc5c187c007f46324e6283bcb18d338e3705d8df8\": container with ID starting with 4ee9f309a44572d20ebfd12bc5c187c007f46324e6283bcb18d338e3705d8df8 not found: ID does not exist" Nov 03 19:47:53.495119 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:53.495102 5059 scope.go:122] "RemoveContainer" containerID="512a43fcddf2d69a21758c6b29f9d5edb489c0faf84aaeea5a81243ac933224f" Nov 03 19:47:53.495351 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:47:53.495331 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"512a43fcddf2d69a21758c6b29f9d5edb489c0faf84aaeea5a81243ac933224f\": container with ID starting with 512a43fcddf2d69a21758c6b29f9d5edb489c0faf84aaeea5a81243ac933224f not found: ID does not exist" containerID="512a43fcddf2d69a21758c6b29f9d5edb489c0faf84aaeea5a81243ac933224f" Nov 03 19:47:53.495404 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:53.495360 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"512a43fcddf2d69a21758c6b29f9d5edb489c0faf84aaeea5a81243ac933224f"} err="failed to get container status \"512a43fcddf2d69a21758c6b29f9d5edb489c0faf84aaeea5a81243ac933224f\": rpc error: code = NotFound desc = could not find container \"512a43fcddf2d69a21758c6b29f9d5edb489c0faf84aaeea5a81243ac933224f\": container with ID starting with 512a43fcddf2d69a21758c6b29f9d5edb489c0faf84aaeea5a81243ac933224f not found: ID does not exist" Nov 03 19:47:53.507748 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:53.507724 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv"] Nov 03 19:47:53.512445 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:53.512426 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-8c14f244-openshift-default-9bfc77fc7-lz4lv"] Nov 03 19:47:53.856513 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:53.856473 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f7e4716-e0cb-4170-82f4-aa74ef088ef3" path="/var/lib/kubelet/pods/3f7e4716-e0cb-4170-82f4-aa74ef088ef3/volumes" Nov 03 19:47:53.857005 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:53.856982 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="497df71a-3372-4792-9cc4-980950a8de12" path="/var/lib/kubelet/pods/497df71a-3372-4792-9cc4-980950a8de12/volumes" Nov 03 19:47:53.857401 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:53.857385 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1c49a14-9ecc-4de2-a81d-94621505ca70" path="/var/lib/kubelet/pods/d1c49a14-9ecc-4de2-a81d-94621505ca70/volumes" Nov 03 19:47:54.265549 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.265466 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq"] Nov 03 19:47:54.266054 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.266033 5059 cpu_manager.go:414] "containerMap: removing container" podUID="497df71a-3372-4792-9cc4-980950a8de12" containerName="istio-proxy" Nov 03 19:47:54.266171 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.266054 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497df71a-3372-4792-9cc4-980950a8de12" containerName="istio-proxy" Nov 03 19:47:54.266171 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.266097 5059 cpu_manager.go:414] "containerMap: removing container" podUID="3f7e4716-e0cb-4170-82f4-aa74ef088ef3" containerName="payload-processing" Nov 03 19:47:54.266171 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.266105 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f7e4716-e0cb-4170-82f4-aa74ef088ef3" containerName="payload-processing" Nov 03 19:47:54.266171 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.266126 5059 cpu_manager.go:414] "containerMap: removing container" podUID="d1c49a14-9ecc-4de2-a81d-94621505ca70" containerName="payload-pre-processing" Nov 03 19:47:54.266171 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.266134 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1c49a14-9ecc-4de2-a81d-94621505ca70" containerName="payload-pre-processing" Nov 03 19:47:54.266409 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.266258 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1c49a14-9ecc-4de2-a81d-94621505ca70" containerName="payload-pre-processing" Nov 03 19:47:54.266409 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.266272 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="497df71a-3372-4792-9cc4-980950a8de12" containerName="istio-proxy" Nov 03 19:47:54.266409 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.266288 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f7e4716-e0cb-4170-82f4-aa74ef088ef3" containerName="payload-processing" Nov 03 19:47:54.271281 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.271251 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq" Nov 03 19:47:54.279549 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.279528 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-3fbb1a-dockercfg-b6ghv\"" Nov 03 19:47:54.279722 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.279629 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-3fbb1a\"" Nov 03 19:47:54.280873 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.280854 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq"] Nov 03 19:47:54.324416 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.324388 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt"] Nov 03 19:47:54.324850 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.324836 5059 cpu_manager.go:414] "containerMap: removing container" podUID="497df71a-3372-4792-9cc4-980950a8de12" containerName="istio-proxy" Nov 03 19:47:54.324898 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.324851 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497df71a-3372-4792-9cc4-980950a8de12" containerName="istio-proxy" Nov 03 19:47:54.324954 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.324945 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="497df71a-3372-4792-9cc4-980950a8de12" containerName="istio-proxy" Nov 03 19:47:54.328207 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.328189 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt" Nov 03 19:47:54.339497 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.339470 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt"] Nov 03 19:47:54.394257 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.394228 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7dv8\" (UniqueName: \"kubernetes.io/projected/0d960e4e-9539-4a53-9df7-1c520ab7d708-kube-api-access-q7dv8\") pod \"payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq\" (UID: \"0d960e4e-9539-4a53-9df7-1c520ab7d708\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq" Nov 03 19:47:54.394366 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.394353 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d960e4e-9539-4a53-9df7-1c520ab7d708-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq\" (UID: \"0d960e4e-9539-4a53-9df7-1c520ab7d708\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq" Nov 03 19:47:54.495728 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.495695 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d960e4e-9539-4a53-9df7-1c520ab7d708-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq\" (UID: \"0d960e4e-9539-4a53-9df7-1c520ab7d708\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq" Nov 03 19:47:54.496123 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.495784 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92b1be0-0890-4db6-ac75-968b6e5a3434-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt\" (UID: \"c92b1be0-0890-4db6-ac75-968b6e5a3434\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt" Nov 03 19:47:54.496123 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.495809 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4m8l\" (UniqueName: \"kubernetes.io/projected/c92b1be0-0890-4db6-ac75-968b6e5a3434-kube-api-access-b4m8l\") pod \"payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt\" (UID: \"c92b1be0-0890-4db6-ac75-968b6e5a3434\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt" Nov 03 19:47:54.496123 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.495827 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7dv8\" (UniqueName: \"kubernetes.io/projected/0d960e4e-9539-4a53-9df7-1c520ab7d708-kube-api-access-q7dv8\") pod \"payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq\" (UID: \"0d960e4e-9539-4a53-9df7-1c520ab7d708\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq" Nov 03 19:47:54.496375 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.496356 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d960e4e-9539-4a53-9df7-1c520ab7d708-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq\" (UID: \"0d960e4e-9539-4a53-9df7-1c520ab7d708\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq" Nov 03 19:47:54.508276 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.508247 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7dv8\" (UniqueName: \"kubernetes.io/projected/0d960e4e-9539-4a53-9df7-1c520ab7d708-kube-api-access-q7dv8\") pod \"payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq\" (UID: \"0d960e4e-9539-4a53-9df7-1c520ab7d708\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq" Nov 03 19:47:54.586861 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.586833 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq" Nov 03 19:47:54.597294 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.597268 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92b1be0-0890-4db6-ac75-968b6e5a3434-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt\" (UID: \"c92b1be0-0890-4db6-ac75-968b6e5a3434\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt" Nov 03 19:47:54.597412 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.597313 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4m8l\" (UniqueName: \"kubernetes.io/projected/c92b1be0-0890-4db6-ac75-968b6e5a3434-kube-api-access-b4m8l\") pod \"payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt\" (UID: \"c92b1be0-0890-4db6-ac75-968b6e5a3434\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt" Nov 03 19:47:54.598090 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.598030 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92b1be0-0890-4db6-ac75-968b6e5a3434-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt\" (UID: \"c92b1be0-0890-4db6-ac75-968b6e5a3434\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt" Nov 03 19:47:54.612524 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.612500 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4m8l\" (UniqueName: \"kubernetes.io/projected/c92b1be0-0890-4db6-ac75-968b6e5a3434-kube-api-access-b4m8l\") pod \"payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt\" (UID: \"c92b1be0-0890-4db6-ac75-968b6e5a3434\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt" Nov 03 19:47:54.645548 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.645518 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt" Nov 03 19:47:54.677687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.677635 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2"] Nov 03 19:47:54.682739 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.682718 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2" Nov 03 19:47:54.689460 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.689434 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-32e482-dockercfg-4mbnz\"" Nov 03 19:47:54.690387 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.690351 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-32e482\"" Nov 03 19:47:54.698831 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.698807 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2"] Nov 03 19:47:54.740202 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.740175 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p"] Nov 03 19:47:54.743841 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.743824 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p" Nov 03 19:47:54.754334 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.754314 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p"] Nov 03 19:47:54.800043 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.800008 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfcb49c5-4381-411e-9d64-d9dcd6181239-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2\" (UID: \"cfcb49c5-4381-411e-9d64-d9dcd6181239\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2" Nov 03 19:47:54.800186 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.800082 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d75d7\" (UniqueName: \"kubernetes.io/projected/cfcb49c5-4381-411e-9d64-d9dcd6181239-kube-api-access-d75d7\") pod \"payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2\" (UID: \"cfcb49c5-4381-411e-9d64-d9dcd6181239\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2" Nov 03 19:47:54.868946 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.868922 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq"] Nov 03 19:47:54.870528 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:47:54.870505 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d960e4e_9539_4a53_9df7_1c520ab7d708.slice/crio-a06ee8f1bbe5022a2a8de8dbe428f1689ba38c88a172963d6ef8b6f18ad320ef WatchSource:0}: Error finding container a06ee8f1bbe5022a2a8de8dbe428f1689ba38c88a172963d6ef8b6f18ad320ef: Status 404 returned error can't find the container with id a06ee8f1bbe5022a2a8de8dbe428f1689ba38c88a172963d6ef8b6f18ad320ef Nov 03 19:47:54.902563 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.902511 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfcb49c5-4381-411e-9d64-d9dcd6181239-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2\" (UID: \"cfcb49c5-4381-411e-9d64-d9dcd6181239\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2" Nov 03 19:47:54.902919 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.902869 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d75d7\" (UniqueName: \"kubernetes.io/projected/cfcb49c5-4381-411e-9d64-d9dcd6181239-kube-api-access-d75d7\") pod \"payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2\" (UID: \"cfcb49c5-4381-411e-9d64-d9dcd6181239\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2" Nov 03 19:47:54.903319 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.903253 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p\" (UID: \"ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p" Nov 03 19:47:54.903477 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.903293 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvmxs\" (UniqueName: \"kubernetes.io/projected/ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51-kube-api-access-tvmxs\") pod \"payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p\" (UID: \"ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p" Nov 03 19:47:54.904987 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.904946 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfcb49c5-4381-411e-9d64-d9dcd6181239-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2\" (UID: \"cfcb49c5-4381-411e-9d64-d9dcd6181239\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2" Nov 03 19:47:54.921749 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.921693 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d75d7\" (UniqueName: \"kubernetes.io/projected/cfcb49c5-4381-411e-9d64-d9dcd6181239-kube-api-access-d75d7\") pod \"payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2\" (UID: \"cfcb49c5-4381-411e-9d64-d9dcd6181239\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2" Nov 03 19:47:54.945109 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:54.945085 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt"] Nov 03 19:47:54.947084 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:47:54.947059 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc92b1be0_0890_4db6_ac75_968b6e5a3434.slice/crio-41ae12074f52cc4f3189ae21cd41ddc55c92d901d330aef39e9eebafc0c4a8aa WatchSource:0}: Error finding container 41ae12074f52cc4f3189ae21cd41ddc55c92d901d330aef39e9eebafc0c4a8aa: Status 404 returned error can't find the container with id 41ae12074f52cc4f3189ae21cd41ddc55c92d901d330aef39e9eebafc0c4a8aa Nov 03 19:47:55.003223 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:55.003197 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2" Nov 03 19:47:55.004602 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:55.004580 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p\" (UID: \"ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p" Nov 03 19:47:55.004720 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:55.004612 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvmxs\" (UniqueName: \"kubernetes.io/projected/ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51-kube-api-access-tvmxs\") pod \"payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p\" (UID: \"ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p" Nov 03 19:47:55.005236 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:55.005217 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p\" (UID: \"ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p" Nov 03 19:47:55.018901 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:55.018870 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvmxs\" (UniqueName: \"kubernetes.io/projected/ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51-kube-api-access-tvmxs\") pod \"payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p\" (UID: \"ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p" Nov 03 19:47:55.061990 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:55.061963 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p" Nov 03 19:47:55.282828 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:55.282795 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2"] Nov 03 19:47:55.284252 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:47:55.284220 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfcb49c5_4381_411e_9d64_d9dcd6181239.slice/crio-6eb6c3a433e2fe458436cce8e74c7f5d996106a0fc51f6eab2da16d2f07bc977 WatchSource:0}: Error finding container 6eb6c3a433e2fe458436cce8e74c7f5d996106a0fc51f6eab2da16d2f07bc977: Status 404 returned error can't find the container with id 6eb6c3a433e2fe458436cce8e74c7f5d996106a0fc51f6eab2da16d2f07bc977 Nov 03 19:47:55.350095 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:55.350072 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p"] Nov 03 19:47:55.351483 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:47:55.351456 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6d56f2_a6c4_4d0e_86d5_64360d5e4a51.slice/crio-c178def3a32d1f877a58690af646328cbb2be8a3529de23564ab32855263ec87 WatchSource:0}: Error finding container c178def3a32d1f877a58690af646328cbb2be8a3529de23564ab32855263ec87: Status 404 returned error can't find the container with id c178def3a32d1f877a58690af646328cbb2be8a3529de23564ab32855263ec87 Nov 03 19:47:55.487617 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:55.487583 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq" event={"ID":"0d960e4e-9539-4a53-9df7-1c520ab7d708","Type":"ContainerStarted","Data":"4436c68602fe345d6bb4fb9dfe6e2ac4dec763399336a2db74a54312609ee767"} Nov 03 19:47:55.487778 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:55.487642 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq" Nov 03 19:47:55.487778 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:55.487694 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq" event={"ID":"0d960e4e-9539-4a53-9df7-1c520ab7d708","Type":"ContainerStarted","Data":"a06ee8f1bbe5022a2a8de8dbe428f1689ba38c88a172963d6ef8b6f18ad320ef"} Nov 03 19:47:55.489216 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:55.489188 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p" event={"ID":"ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51","Type":"ContainerStarted","Data":"04da4e840db9b36ab74f969a24bc9303a68de3283b026a9e0deacd6cef05ae43"} Nov 03 19:47:55.489302 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:55.489245 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p" Nov 03 19:47:55.489302 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:55.489271 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p" event={"ID":"ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51","Type":"ContainerStarted","Data":"c178def3a32d1f877a58690af646328cbb2be8a3529de23564ab32855263ec87"} Nov 03 19:47:55.490712 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:55.490657 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt" event={"ID":"c92b1be0-0890-4db6-ac75-968b6e5a3434","Type":"ContainerStarted","Data":"ddfb36a61a4890d032f40a96fa7dc33ada9d7745461eee81dc563f215eda9eae"} Nov 03 19:47:55.490796 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:55.490731 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt" event={"ID":"c92b1be0-0890-4db6-ac75-968b6e5a3434","Type":"ContainerStarted","Data":"41ae12074f52cc4f3189ae21cd41ddc55c92d901d330aef39e9eebafc0c4a8aa"} Nov 03 19:47:55.490796 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:55.490758 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt" Nov 03 19:47:55.492229 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:55.492206 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2" event={"ID":"cfcb49c5-4381-411e-9d64-d9dcd6181239","Type":"ContainerStarted","Data":"06775cb2b6311ab6024c5cf0da37eef3631ba5e53f2a44ad31bd33c7d778d26f"} Nov 03 19:47:55.492316 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:55.492249 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2" event={"ID":"cfcb49c5-4381-411e-9d64-d9dcd6181239","Type":"ContainerStarted","Data":"6eb6c3a433e2fe458436cce8e74c7f5d996106a0fc51f6eab2da16d2f07bc977"} Nov 03 19:47:55.492316 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:55.492298 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2" Nov 03 19:47:55.513622 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:55.513573 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq" podStartSLOduration=1.513557212 podStartE2EDuration="1.513557212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:55.510995721 +0000 UTC m=+669.768792979" watchObservedRunningTime="2026-11-03 19:47:55.513557212 +0000 UTC m=+669.771354471" Nov 03 19:47:55.531702 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:55.531647 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt" podStartSLOduration=1.5316338520000001 podStartE2EDuration="1.531633852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:55.529699834 +0000 UTC m=+669.787497092" watchObservedRunningTime="2026-11-03 19:47:55.531633852 +0000 UTC m=+669.789431109" Nov 03 19:47:55.549343 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:55.549304 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2" podStartSLOduration=1.5492695159999998 podStartE2EDuration="1.549269516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:55.547908863 +0000 UTC m=+669.805706121" watchObservedRunningTime="2026-11-03 19:47:55.549269516 +0000 UTC m=+669.807066773" Nov 03 19:47:55.569801 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:47:55.569764 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p" podStartSLOduration=1.569749498 podStartE2EDuration="1.569749498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:55.568096575 +0000 UTC m=+669.825893833" watchObservedRunningTime="2026-11-03 19:47:55.569749498 +0000 UTC m=+669.827546755" Nov 03 19:47:58.327064 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:47:58.327027 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087fa86d_7f7a_45a8_b368_b0052d5f5ec0.slice/crio-0ca372c1281d9058bf76ae8e15097135e00416887c56f9bee86728e973dbff84\": RecentStats: unable to find data in memory cache]" Nov 03 19:48:00.976264 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:48:00.976219 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087fa86d_7f7a_45a8_b368_b0052d5f5ec0.slice/crio-0ca372c1281d9058bf76ae8e15097135e00416887c56f9bee86728e973dbff84\": RecentStats: unable to find data in memory cache]" Nov 03 19:48:01.429572 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:01.429538 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9" Nov 03 19:48:01.429761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:01.429612 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl" Nov 03 19:48:02.494353 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:02.494320 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn" Nov 03 19:48:04.439616 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:04.439556 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:48:04.440148 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:04.439677 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" podUID="d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:05.684638 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.684594 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22"] Nov 03 19:48:05.693451 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.693425 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:48:05.700075 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.699870 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22"] Nov 03 19:48:05.701406 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.701382 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"openshift-service-ca.crt\"" Nov 03 19:48:05.701540 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.701484 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"simulator-w3-9f46e5-kserve-self-signed-certs\"" Nov 03 19:48:05.702175 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.702154 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"default-dockercfg-zrxnc\"" Nov 03 19:48:05.702290 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.702209 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"kube-root-ca.crt\"" Nov 03 19:48:05.782690 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.782279 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-tmp-dir\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22\" (UID: \"f14445d5-52ed-487d-972c-4499966065fa\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:48:05.782690 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.782411 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-home\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22\" (UID: \"f14445d5-52ed-487d-972c-4499966065fa\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:48:05.782690 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.782450 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f14445d5-52ed-487d-972c-4499966065fa-tls-certs\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22\" (UID: \"f14445d5-52ed-487d-972c-4499966065fa\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:48:05.782690 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.782498 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-model-cache\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22\" (UID: \"f14445d5-52ed-487d-972c-4499966065fa\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:48:05.782690 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.782530 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-dshm\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22\" (UID: \"f14445d5-52ed-487d-972c-4499966065fa\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:48:05.782690 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.782566 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9s47\" (UniqueName: \"kubernetes.io/projected/f14445d5-52ed-487d-972c-4499966065fa-kube-api-access-g9s47\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22\" (UID: \"f14445d5-52ed-487d-972c-4499966065fa\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:48:05.887693 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.887287 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9s47\" (UniqueName: \"kubernetes.io/projected/f14445d5-52ed-487d-972c-4499966065fa-kube-api-access-g9s47\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22\" (UID: \"f14445d5-52ed-487d-972c-4499966065fa\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:48:05.888008 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.887985 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-tmp-dir\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22\" (UID: \"f14445d5-52ed-487d-972c-4499966065fa\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:48:05.888235 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.888219 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-home\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22\" (UID: \"f14445d5-52ed-487d-972c-4499966065fa\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:48:05.888355 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.888338 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f14445d5-52ed-487d-972c-4499966065fa-tls-certs\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22\" (UID: \"f14445d5-52ed-487d-972c-4499966065fa\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:48:05.888531 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.888506 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-model-cache\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22\" (UID: \"f14445d5-52ed-487d-972c-4499966065fa\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:48:05.888695 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.888655 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-dshm\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22\" (UID: \"f14445d5-52ed-487d-972c-4499966065fa\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:48:05.890793 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.890075 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-tmp-dir\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22\" (UID: \"f14445d5-52ed-487d-972c-4499966065fa\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:48:05.890946 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.890377 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-home\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22\" (UID: \"f14445d5-52ed-487d-972c-4499966065fa\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:48:05.891050 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.890935 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-model-cache\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22\" (UID: \"f14445d5-52ed-487d-972c-4499966065fa\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:48:05.893837 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.893816 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f14445d5-52ed-487d-972c-4499966065fa-tls-certs\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22\" (UID: \"f14445d5-52ed-487d-972c-4499966065fa\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:48:05.894342 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.894319 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-dshm\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22\" (UID: \"f14445d5-52ed-487d-972c-4499966065fa\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:48:05.903623 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:05.903598 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9s47\" (UniqueName: \"kubernetes.io/projected/f14445d5-52ed-487d-972c-4499966065fa-kube-api-access-g9s47\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22\" (UID: \"f14445d5-52ed-487d-972c-4499966065fa\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:48:06.011256 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:06.011180 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:48:06.315681 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:06.315576 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22"] Nov 03 19:48:06.323198 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:48:06.323169 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf14445d5_52ed_487d_972c_4499966065fa.slice/crio-a2b18e67c031d89367ba22383c2672c4ea5db2ef498701a9b608b8cf3af93bb9 WatchSource:0}: Error finding container a2b18e67c031d89367ba22383c2672c4ea5db2ef498701a9b608b8cf3af93bb9: Status 404 returned error can't find the container with id a2b18e67c031d89367ba22383c2672c4ea5db2ef498701a9b608b8cf3af93bb9 Nov 03 19:48:06.501024 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:06.500983 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p" Nov 03 19:48:06.502745 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:06.502719 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2" Nov 03 19:48:06.503789 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:06.503769 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt" Nov 03 19:48:06.504301 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:06.504279 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq" Nov 03 19:48:06.597204 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:06.595268 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" event={"ID":"f14445d5-52ed-487d-972c-4499966065fa","Type":"ContainerStarted","Data":"44d16c3b122733147748e39e219fb00f390ff6e37a49c47ed3e51ba2960a62c1"} Nov 03 19:48:06.597204 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:06.595316 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" event={"ID":"f14445d5-52ed-487d-972c-4499966065fa","Type":"ContainerStarted","Data":"a2b18e67c031d89367ba22383c2672c4ea5db2ef498701a9b608b8cf3af93bb9"} Nov 03 19:48:06.619567 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:06.619512 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podStartSLOduration=1.619494991 podStartE2EDuration="1.619494991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:06.618169216 +0000 UTC m=+680.875966473" watchObservedRunningTime="2026-11-03 19:48:06.619494991 +0000 UTC m=+680.877292250" Nov 03 19:48:07.011454 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:07.011427 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:48:07.602359 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:07.602325 5059 generic.go:356] "Generic (PLEG): container finished" podID="d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" containerID="6f8ee91ca327bbf5a0c3704b9e3ce856daebe10db7f411298f5e612e7a7a2a8f" exitCode=0 Nov 03 19:48:07.602574 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:07.602400 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" event={"ID":"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b","Type":"ContainerDied","Data":"6f8ee91ca327bbf5a0c3704b9e3ce856daebe10db7f411298f5e612e7a7a2a8f"} Nov 03 19:48:07.602997 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:07.602978 5059 scope.go:122] "RemoveContainer" containerID="6f8ee91ca327bbf5a0c3704b9e3ce856daebe10db7f411298f5e612e7a7a2a8f" Nov 03 19:48:07.604251 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:07.604223 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:48:07.604353 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:07.604307 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:48:07.604415 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:07.604358 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:48:08.343008 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:48:08.342973 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087fa86d_7f7a_45a8_b368_b0052d5f5ec0.slice/crio-0ca372c1281d9058bf76ae8e15097135e00416887c56f9bee86728e973dbff84\": RecentStats: unable to find data in memory cache]" Nov 03 19:48:08.611404 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:08.611322 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" event={"ID":"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b","Type":"ContainerStarted","Data":"5c19ca06fbb1d449a708204a75d839cc487a9ea18245ce5649dd7dfdd316e86f"} Nov 03 19:48:09.411686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:09.411382 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:48:09.411686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:09.411506 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:48:09.411686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:09.411548 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" podUID="d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 19:48:10.411009 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:10.410974 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:48:10.411185 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:10.411030 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" podUID="d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 19:48:10.614184 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:48:10.614137 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:48:11.410853 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:11.410812 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:48:11.411098 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:11.410890 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" podUID="d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 19:48:11.633822 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:11.633791 5059 generic.go:356] "Generic (PLEG): container finished" podID="eac98813-7b4c-4c4a-8bf9-444bba2fa644" containerID="89b4df7aa727c470a64d52c6160fcb987049e7d0bf5bc45f6489c13773b39202" exitCode=0 Nov 03 19:48:11.634225 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:11.633869 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" event={"ID":"eac98813-7b4c-4c4a-8bf9-444bba2fa644","Type":"ContainerDied","Data":"89b4df7aa727c470a64d52c6160fcb987049e7d0bf5bc45f6489c13773b39202"} Nov 03 19:48:11.634293 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:11.634279 5059 scope.go:122] "RemoveContainer" containerID="89b4df7aa727c470a64d52c6160fcb987049e7d0bf5bc45f6489c13773b39202" Nov 03 19:48:11.635628 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:11.635602 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:48:11.635713 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:11.635699 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:48:11.635759 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:11.635745 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:48:12.411288 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:12.411246 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:48:12.411493 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:12.411318 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" podUID="d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 19:48:12.642117 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:12.642078 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" event={"ID":"eac98813-7b4c-4c4a-8bf9-444bba2fa644","Type":"ContainerStarted","Data":"f0ca6f95c31b4ccad9131b19d71014e3f54afa9b79e7f53f9d77d4b861fb6af9"} Nov 03 19:48:13.097683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:13.097075 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:48:13.097683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:13.097210 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:48:13.097683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:13.097266 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" podUID="eac98813-7b4c-4c4a-8bf9-444bba2fa644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 19:48:13.412740 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:13.412173 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:48:13.412740 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:13.412236 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" podUID="d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 19:48:14.097758 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:14.097725 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:48:14.098151 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:14.097781 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" podUID="eac98813-7b4c-4c4a-8bf9-444bba2fa644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 19:48:14.414885 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:14.414807 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:48:14.415248 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:14.415225 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:48:14.415961 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:14.415940 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 19:48:15.097224 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:15.097188 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:48:15.097399 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:15.097236 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" podUID="eac98813-7b4c-4c4a-8bf9-444bba2fa644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 19:48:16.011340 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:16.011307 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:48:16.025081 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:16.025055 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:48:16.028693 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:48:16.028656 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087fa86d_7f7a_45a8_b368_b0052d5f5ec0.slice/crio-0ca372c1281d9058bf76ae8e15097135e00416887c56f9bee86728e973dbff84\": RecentStats: unable to find data in memory cache]" Nov 03 19:48:16.097719 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:16.097656 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:48:16.097884 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:16.097738 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" podUID="eac98813-7b4c-4c4a-8bf9-444bba2fa644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 19:48:16.702385 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:16.702356 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:48:17.097084 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:17.097049 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:48:17.097484 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:17.097103 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" podUID="eac98813-7b4c-4c4a-8bf9-444bba2fa644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 19:48:18.104375 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:18.104340 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:48:18.104943 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:18.104629 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:48:18.106074 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:18.106051 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 19:48:18.355373 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:48:18.355276 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087fa86d_7f7a_45a8_b368_b0052d5f5ec0.slice/crio-0ca372c1281d9058bf76ae8e15097135e00416887c56f9bee86728e973dbff84\": RecentStats: unable to find data in memory cache]" Nov 03 19:48:20.130744 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:48:20.130624 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087fa86d_7f7a_45a8_b368_b0052d5f5ec0.slice/crio-0ca372c1281d9058bf76ae8e15097135e00416887c56f9bee86728e973dbff84\": RecentStats: unable to find data in memory cache]" Nov 03 19:48:20.131196 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:48:20.130919 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087fa86d_7f7a_45a8_b368_b0052d5f5ec0.slice/crio-0ca372c1281d9058bf76ae8e15097135e00416887c56f9bee86728e973dbff84\": RecentStats: unable to find data in memory cache]" Nov 03 19:48:22.514317 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.514284 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w"] Nov 03 19:48:22.518318 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.518299 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:48:22.522674 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.522646 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"openshift-service-ca.crt\"" Nov 03 19:48:22.524110 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.524085 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"simulator-w4-3fbb1a-kserve-self-signed-certs\"" Nov 03 19:48:22.524110 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.524092 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"kube-root-ca.crt\"" Nov 03 19:48:22.524266 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.524161 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"default-dockercfg-lxnx2\"" Nov 03 19:48:22.535983 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.535962 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w"] Nov 03 19:48:22.621967 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.621938 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20690fd2-7022-4657-8c8c-bd738493ffc0-tls-certs\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-4892w\" (UID: \"20690fd2-7022-4657-8c8c-bd738493ffc0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:48:22.622104 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.622004 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-tmp-dir\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-4892w\" (UID: \"20690fd2-7022-4657-8c8c-bd738493ffc0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:48:22.622104 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.622059 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-dshm\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-4892w\" (UID: \"20690fd2-7022-4657-8c8c-bd738493ffc0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:48:22.622186 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.622136 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrhrx\" (UniqueName: \"kubernetes.io/projected/20690fd2-7022-4657-8c8c-bd738493ffc0-kube-api-access-jrhrx\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-4892w\" (UID: \"20690fd2-7022-4657-8c8c-bd738493ffc0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:48:22.622186 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.622177 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-home\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-4892w\" (UID: \"20690fd2-7022-4657-8c8c-bd738493ffc0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:48:22.622262 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.622198 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-model-cache\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-4892w\" (UID: \"20690fd2-7022-4657-8c8c-bd738493ffc0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:48:22.723621 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.723590 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20690fd2-7022-4657-8c8c-bd738493ffc0-tls-certs\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-4892w\" (UID: \"20690fd2-7022-4657-8c8c-bd738493ffc0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:48:22.723790 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.723650 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-tmp-dir\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-4892w\" (UID: \"20690fd2-7022-4657-8c8c-bd738493ffc0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:48:22.723790 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.723700 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-dshm\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-4892w\" (UID: \"20690fd2-7022-4657-8c8c-bd738493ffc0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:48:22.723790 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.723738 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrhrx\" (UniqueName: \"kubernetes.io/projected/20690fd2-7022-4657-8c8c-bd738493ffc0-kube-api-access-jrhrx\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-4892w\" (UID: \"20690fd2-7022-4657-8c8c-bd738493ffc0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:48:22.723790 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.723784 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-home\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-4892w\" (UID: \"20690fd2-7022-4657-8c8c-bd738493ffc0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:48:22.723979 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.723803 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-model-cache\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-4892w\" (UID: \"20690fd2-7022-4657-8c8c-bd738493ffc0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:48:22.724068 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.724038 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-tmp-dir\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-4892w\" (UID: \"20690fd2-7022-4657-8c8c-bd738493ffc0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:48:22.724130 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.724105 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-model-cache\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-4892w\" (UID: \"20690fd2-7022-4657-8c8c-bd738493ffc0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:48:22.724210 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.724183 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-home\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-4892w\" (UID: \"20690fd2-7022-4657-8c8c-bd738493ffc0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:48:22.726129 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.726101 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-dshm\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-4892w\" (UID: \"20690fd2-7022-4657-8c8c-bd738493ffc0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:48:22.726347 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.726326 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20690fd2-7022-4657-8c8c-bd738493ffc0-tls-certs\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-4892w\" (UID: \"20690fd2-7022-4657-8c8c-bd738493ffc0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:48:22.736143 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.736123 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrhrx\" (UniqueName: \"kubernetes.io/projected/20690fd2-7022-4657-8c8c-bd738493ffc0-kube-api-access-jrhrx\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-4892w\" (UID: \"20690fd2-7022-4657-8c8c-bd738493ffc0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:48:22.830628 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:22.830600 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:48:23.107798 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.107769 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w"] Nov 03 19:48:23.108711 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:48:23.108645 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20690fd2_7022_4657_8c8c_bd738493ffc0.slice/crio-74e01caa8834dc2b10b8f8ebf75574792274d55c381887f50b657f8472821284 WatchSource:0}: Error finding container 74e01caa8834dc2b10b8f8ebf75574792274d55c381887f50b657f8472821284: Status 404 returned error can't find the container with id 74e01caa8834dc2b10b8f8ebf75574792274d55c381887f50b657f8472821284 Nov 03 19:48:23.692479 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.692428 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55"] Nov 03 19:48:23.700539 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.700517 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:48:23.706163 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.706136 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55"] Nov 03 19:48:23.713810 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.713785 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"openshift-service-ca.crt\"" Nov 03 19:48:23.713810 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.713808 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"kube-root-ca.crt\"" Nov 03 19:48:23.715265 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.715246 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"simulator-w1-32e482-kserve-self-signed-certs\"" Nov 03 19:48:23.715265 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.715249 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"default-dockercfg-5r8nn\"" Nov 03 19:48:23.740526 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.740471 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" event={"ID":"20690fd2-7022-4657-8c8c-bd738493ffc0","Type":"ContainerStarted","Data":"5b05929b67ea95a6b627445e0a3fa63ef0a0d7ef033293a27edabc525d4bd1cd"} Nov 03 19:48:23.740526 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.740514 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" event={"ID":"20690fd2-7022-4657-8c8c-bd738493ffc0","Type":"ContainerStarted","Data":"74e01caa8834dc2b10b8f8ebf75574792274d55c381887f50b657f8472821284"} Nov 03 19:48:23.767511 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.767470 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podStartSLOduration=1.767457734 podStartE2EDuration="1.767457734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:23.765902078 +0000 UTC m=+698.023699336" watchObservedRunningTime="2026-11-03 19:48:23.767457734 +0000 UTC m=+698.025254990" Nov 03 19:48:23.831525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.831499 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:48:23.836009 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.835985 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-tls-certs\") pod \"simulator-w1-32e482-kserve-776bf6595d-mgx55\" (UID: \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:48:23.836109 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.836018 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-model-cache\") pod \"simulator-w1-32e482-kserve-776bf6595d-mgx55\" (UID: \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:48:23.836109 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.836043 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-home\") pod \"simulator-w1-32e482-kserve-776bf6595d-mgx55\" (UID: \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:48:23.837133 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.837111 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-dshm\") pod \"simulator-w1-32e482-kserve-776bf6595d-mgx55\" (UID: \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:48:23.837311 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.837275 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zjtd\" (UniqueName: \"kubernetes.io/projected/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-kube-api-access-7zjtd\") pod \"simulator-w1-32e482-kserve-776bf6595d-mgx55\" (UID: \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:48:23.837444 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.837429 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-tmp-dir\") pod \"simulator-w1-32e482-kserve-776bf6595d-mgx55\" (UID: \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:48:23.938520 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.938481 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-tmp-dir\") pod \"simulator-w1-32e482-kserve-776bf6595d-mgx55\" (UID: \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:48:23.938718 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.938558 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-tls-certs\") pod \"simulator-w1-32e482-kserve-776bf6595d-mgx55\" (UID: \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:48:23.938718 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.938574 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-model-cache\") pod \"simulator-w1-32e482-kserve-776bf6595d-mgx55\" (UID: \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:48:23.938718 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.938590 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-home\") pod \"simulator-w1-32e482-kserve-776bf6595d-mgx55\" (UID: \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:48:23.938718 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.938612 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-dshm\") pod \"simulator-w1-32e482-kserve-776bf6595d-mgx55\" (UID: \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:48:23.938718 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.938648 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zjtd\" (UniqueName: \"kubernetes.io/projected/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-kube-api-access-7zjtd\") pod \"simulator-w1-32e482-kserve-776bf6595d-mgx55\" (UID: \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:48:23.939009 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.938982 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-tmp-dir\") pod \"simulator-w1-32e482-kserve-776bf6595d-mgx55\" (UID: \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:48:23.939062 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.939024 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-model-cache\") pod \"simulator-w1-32e482-kserve-776bf6595d-mgx55\" (UID: \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:48:23.939101 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.939069 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-home\") pod \"simulator-w1-32e482-kserve-776bf6595d-mgx55\" (UID: \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:48:23.940994 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.940968 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-dshm\") pod \"simulator-w1-32e482-kserve-776bf6595d-mgx55\" (UID: \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:48:23.941168 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.941151 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-tls-certs\") pod \"simulator-w1-32e482-kserve-776bf6595d-mgx55\" (UID: \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:48:23.951442 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:23.951382 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zjtd\" (UniqueName: \"kubernetes.io/projected/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-kube-api-access-7zjtd\") pod \"simulator-w1-32e482-kserve-776bf6595d-mgx55\" (UID: \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:48:24.017203 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:24.017177 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:48:24.293918 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:24.293888 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55"] Nov 03 19:48:24.748124 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:24.748088 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" event={"ID":"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02","Type":"ContainerStarted","Data":"238b88770b80cec7d339089cb9d69cc1f7ef5a1fdcefa3eb790154cc7f3289f1"} Nov 03 19:48:24.748557 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:24.748139 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" event={"ID":"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02","Type":"ContainerStarted","Data":"04f4c299d03d7084481887555642aa0b3229a752661fb38bc15ee6075eb3e8c8"} Nov 03 19:48:24.770299 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:24.770253 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podStartSLOduration=1.770240586 podStartE2EDuration="1.770240586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:24.768042031 +0000 UTC m=+699.025839301" watchObservedRunningTime="2026-11-03 19:48:24.770240586 +0000 UTC m=+699.028037844" Nov 03 19:48:25.017358 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:25.017265 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:48:28.424656 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:48:28.424620 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087fa86d_7f7a_45a8_b368_b0052d5f5ec0.slice/crio-0ca372c1281d9058bf76ae8e15097135e00416887c56f9bee86728e973dbff84\": RecentStats: unable to find data in memory cache]" Nov 03 19:48:30.962556 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:48:30.962517 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087fa86d_7f7a_45a8_b368_b0052d5f5ec0.slice/crio-0ca372c1281d9058bf76ae8e15097135e00416887c56f9bee86728e973dbff84\": RecentStats: unable to find data in memory cache]" Nov 03 19:48:32.831491 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:32.831462 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:48:32.844311 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:32.844287 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:48:33.814166 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:33.814134 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:48:34.017786 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:34.017751 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:48:34.030088 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:34.030063 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:48:34.820095 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:48:34.820067 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:48:38.488864 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:48:38.488830 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087fa86d_7f7a_45a8_b368_b0052d5f5ec0.slice/crio-0ca372c1281d9058bf76ae8e15097135e00416887c56f9bee86728e973dbff84\": RecentStats: unable to find data in memory cache]" Nov 03 19:48:40.604125 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:48:40.604088 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:49:04.152803 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.152766 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc"] Nov 03 19:49:04.158983 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.158955 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.166676 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.166640 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3fbb1a-openshift-default-dockercfg-247ct\"" Nov 03 19:49:04.169945 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.169923 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc"] Nov 03 19:49:04.255250 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.255218 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.255391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.255258 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdvl4\" (UniqueName: \"kubernetes.io/projected/474b2e1d-832b-4edd-8765-31e0bc77cc2a-kube-api-access-hdvl4\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.255391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.255335 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.255391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.255368 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.255391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.255390 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.255563 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.255528 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.255600 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.255567 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.255713 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.255692 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.255758 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.255728 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.356618 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.356591 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.356618 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.356619 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdvl4\" (UniqueName: \"kubernetes.io/projected/474b2e1d-832b-4edd-8765-31e0bc77cc2a-kube-api-access-hdvl4\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.356822 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.356643 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.356822 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.356711 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.356822 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.356742 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.356822 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.356794 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.357024 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.356825 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.357024 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.356908 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.357024 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.356944 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.357177 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.357094 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.357235 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.357178 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.357235 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.357218 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.357341 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.357241 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.357405 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.357387 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.359631 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.359605 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.359819 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.359800 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.371324 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.371291 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.371839 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.371813 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdvl4\" (UniqueName: \"kubernetes.io/projected/474b2e1d-832b-4edd-8765-31e0bc77cc2a-kube-api-access-hdvl4\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.473026 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.472951 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:04.756167 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.756144 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc"] Nov 03 19:49:04.757741 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:49:04.757714 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474b2e1d_832b_4edd_8765_31e0bc77cc2a.slice/crio-15b8f003915ecc6ccbabe46fd4f8d1f9ea8654fc469c49b72288ac914824c2e8 WatchSource:0}: Error finding container 15b8f003915ecc6ccbabe46fd4f8d1f9ea8654fc469c49b72288ac914824c2e8: Status 404 returned error can't find the container with id 15b8f003915ecc6ccbabe46fd4f8d1f9ea8654fc469c49b72288ac914824c2e8 Nov 03 19:49:04.760056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.760015 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:49:04.760159 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.760095 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:49:04.760159 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.760136 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:49:04.998534 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.998490 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" event={"ID":"474b2e1d-832b-4edd-8765-31e0bc77cc2a","Type":"ContainerStarted","Data":"3513472a5d02bb21de815edd2ec4ca696e8e46e17be5d839916c093e17bd9499"} Nov 03 19:49:04.998700 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:04.998562 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" event={"ID":"474b2e1d-832b-4edd-8765-31e0bc77cc2a","Type":"ContainerStarted","Data":"15b8f003915ecc6ccbabe46fd4f8d1f9ea8654fc469c49b72288ac914824c2e8"} Nov 03 19:49:05.025022 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:05.024928 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" podStartSLOduration=1.024914565 podStartE2EDuration="1.024914565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:05.022090864 +0000 UTC m=+739.279888133" watchObservedRunningTime="2026-11-03 19:49:05.024914565 +0000 UTC m=+739.282711822" Nov 03 19:49:05.477688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:05.475712 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:05.481684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:05.479217 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:49:05.481684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:05.479275 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" podUID="474b2e1d-832b-4edd-8765-31e0bc77cc2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 19:49:06.483124 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:06.483083 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:49:06.483634 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:06.483162 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" podUID="474b2e1d-832b-4edd-8765-31e0bc77cc2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 19:49:07.479686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:07.478052 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:49:07.479686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:07.478111 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" podUID="474b2e1d-832b-4edd-8765-31e0bc77cc2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 19:49:08.479103 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:08.479061 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:49:08.479733 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:08.479699 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" podUID="474b2e1d-832b-4edd-8765-31e0bc77cc2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 19:49:09.234769 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:09.234725 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52c129de-8l2t4"] Nov 03 19:49:09.238824 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:09.238801 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c129de-8l2t4" Nov 03 19:49:09.244591 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:09.244558 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:49:09.246006 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:09.245988 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 19:49:09.249026 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:09.249005 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52c129de-8l2t4"] Nov 03 19:49:09.333360 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:09.333324 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9065a339-0b3b-49f5-ba8f-03ac69b21ca4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52c129de-8l2t4\" (UID: \"9065a339-0b3b-49f5-ba8f-03ac69b21ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c129de-8l2t4" Nov 03 19:49:09.333524 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:09.333365 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpqdg\" (UniqueName: \"kubernetes.io/projected/9065a339-0b3b-49f5-ba8f-03ac69b21ca4-kube-api-access-bpqdg\") pod \"maas-api-revoke-sub-52c129de-8l2t4\" (UID: \"9065a339-0b3b-49f5-ba8f-03ac69b21ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c129de-8l2t4" Nov 03 19:49:09.333628 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:09.333606 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9065a339-0b3b-49f5-ba8f-03ac69b21ca4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52c129de-8l2t4\" (UID: \"9065a339-0b3b-49f5-ba8f-03ac69b21ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c129de-8l2t4" Nov 03 19:49:09.434980 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:09.434943 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9065a339-0b3b-49f5-ba8f-03ac69b21ca4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52c129de-8l2t4\" (UID: \"9065a339-0b3b-49f5-ba8f-03ac69b21ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c129de-8l2t4" Nov 03 19:49:09.435151 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:09.434996 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9065a339-0b3b-49f5-ba8f-03ac69b21ca4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52c129de-8l2t4\" (UID: \"9065a339-0b3b-49f5-ba8f-03ac69b21ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c129de-8l2t4" Nov 03 19:49:09.435151 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:09.435026 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpqdg\" (UniqueName: \"kubernetes.io/projected/9065a339-0b3b-49f5-ba8f-03ac69b21ca4-kube-api-access-bpqdg\") pod \"maas-api-revoke-sub-52c129de-8l2t4\" (UID: \"9065a339-0b3b-49f5-ba8f-03ac69b21ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c129de-8l2t4" Nov 03 19:49:09.435775 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:09.435746 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9065a339-0b3b-49f5-ba8f-03ac69b21ca4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52c129de-8l2t4\" (UID: \"9065a339-0b3b-49f5-ba8f-03ac69b21ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c129de-8l2t4" Nov 03 19:49:09.437963 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:09.437944 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9065a339-0b3b-49f5-ba8f-03ac69b21ca4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52c129de-8l2t4\" (UID: \"9065a339-0b3b-49f5-ba8f-03ac69b21ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c129de-8l2t4" Nov 03 19:49:09.451283 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:09.451257 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpqdg\" (UniqueName: \"kubernetes.io/projected/9065a339-0b3b-49f5-ba8f-03ac69b21ca4-kube-api-access-bpqdg\") pod \"maas-api-revoke-sub-52c129de-8l2t4\" (UID: \"9065a339-0b3b-49f5-ba8f-03ac69b21ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c129de-8l2t4" Nov 03 19:49:09.479029 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:09.478998 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:49:09.479231 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:09.479203 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" podUID="474b2e1d-832b-4edd-8765-31e0bc77cc2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 19:49:09.555813 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:09.555754 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c129de-8l2t4" Nov 03 19:49:09.882747 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:09.882714 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52c129de-8l2t4"] Nov 03 19:49:09.885334 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:49:09.885299 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9065a339_0b3b_49f5_ba8f_03ac69b21ca4.slice/crio-a09f7c50e9949b24c5eb3329bd6c8ba5bdbee397f645b45224e37e4c9ce208dd WatchSource:0}: Error finding container a09f7c50e9949b24c5eb3329bd6c8ba5bdbee397f645b45224e37e4c9ce208dd: Status 404 returned error can't find the container with id a09f7c50e9949b24c5eb3329bd6c8ba5bdbee397f645b45224e37e4c9ce208dd Nov 03 19:49:10.038467 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:10.038414 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c129de-8l2t4" event={"ID":"9065a339-0b3b-49f5-ba8f-03ac69b21ca4","Type":"ContainerStarted","Data":"a09f7c50e9949b24c5eb3329bd6c8ba5bdbee397f645b45224e37e4c9ce208dd"} Nov 03 19:49:10.482687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:10.478048 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:49:10.482687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:10.478108 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" podUID="474b2e1d-832b-4edd-8765-31e0bc77cc2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 19:49:10.612949 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:49:10.610950 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:49:11.049682 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:11.048986 5059 generic.go:356] "Generic (PLEG): container finished" podID="9065a339-0b3b-49f5-ba8f-03ac69b21ca4" containerID="46c29a995f2d9acb49d9d32e9a972cc00c24a0c0d6178965b216e4a8ffe93a13" exitCode=0 Nov 03 19:49:11.049682 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:11.049382 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c129de-8l2t4" event={"ID":"9065a339-0b3b-49f5-ba8f-03ac69b21ca4","Type":"ContainerDied","Data":"46c29a995f2d9acb49d9d32e9a972cc00c24a0c0d6178965b216e4a8ffe93a13"} Nov 03 19:49:12.247041 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:12.247015 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c129de-8l2t4" Nov 03 19:49:12.383453 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:12.382969 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9065a339-0b3b-49f5-ba8f-03ac69b21ca4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9065a339-0b3b-49f5-ba8f-03ac69b21ca4-maas-api-openshift-ca\") pod \"9065a339-0b3b-49f5-ba8f-03ac69b21ca4\" (UID: \"9065a339-0b3b-49f5-ba8f-03ac69b21ca4\") " Nov 03 19:49:12.383453 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:12.383053 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9065a339-0b3b-49f5-ba8f-03ac69b21ca4-kube-api-access-bpqdg\" (UniqueName: \"kubernetes.io/projected/9065a339-0b3b-49f5-ba8f-03ac69b21ca4-kube-api-access-bpqdg\") pod \"9065a339-0b3b-49f5-ba8f-03ac69b21ca4\" (UID: \"9065a339-0b3b-49f5-ba8f-03ac69b21ca4\") " Nov 03 19:49:12.383453 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:12.383170 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9065a339-0b3b-49f5-ba8f-03ac69b21ca4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9065a339-0b3b-49f5-ba8f-03ac69b21ca4-maas-api-xks-ca\") pod \"9065a339-0b3b-49f5-ba8f-03ac69b21ca4\" (UID: \"9065a339-0b3b-49f5-ba8f-03ac69b21ca4\") " Nov 03 19:49:12.384043 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:12.384010 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9065a339-0b3b-49f5-ba8f-03ac69b21ca4-maas-api-openshift-ca" pod "9065a339-0b3b-49f5-ba8f-03ac69b21ca4" (UID: "9065a339-0b3b-49f5-ba8f-03ac69b21ca4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:49:12.408451 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:12.408417 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9065a339-0b3b-49f5-ba8f-03ac69b21ca4-kube-api-access-bpqdg" pod "9065a339-0b3b-49f5-ba8f-03ac69b21ca4" (UID: "9065a339-0b3b-49f5-ba8f-03ac69b21ca4"). InnerVolumeSpecName "kube-api-access-bpqdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:49:12.408692 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:12.408630 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9065a339-0b3b-49f5-ba8f-03ac69b21ca4-maas-api-xks-ca" pod "9065a339-0b3b-49f5-ba8f-03ac69b21ca4" (UID: "9065a339-0b3b-49f5-ba8f-03ac69b21ca4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:49:12.484737 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:12.484699 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9065a339-0b3b-49f5-ba8f-03ac69b21ca4-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:12.484737 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:12.484734 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpqdg\" (UniqueName: \"kubernetes.io/projected/9065a339-0b3b-49f5-ba8f-03ac69b21ca4-kube-api-access-bpqdg\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:12.484975 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:12.484748 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9065a339-0b3b-49f5-ba8f-03ac69b21ca4-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:12.492433 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:12.492394 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:49:12.492582 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:12.492474 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" podUID="474b2e1d-832b-4edd-8765-31e0bc77cc2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:49:12.494036 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:12.493989 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:12.494845 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:12.494821 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:12.496962 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:12.496926 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 19:49:13.066360 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:13.066330 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c129de-8l2t4" Nov 03 19:49:13.066360 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:13.066347 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c129de-8l2t4" event={"ID":"9065a339-0b3b-49f5-ba8f-03ac69b21ca4","Type":"ContainerDied","Data":"a09f7c50e9949b24c5eb3329bd6c8ba5bdbee397f645b45224e37e4c9ce208dd"} Nov 03 19:49:13.066560 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:13.066396 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a09f7c50e9949b24c5eb3329bd6c8ba5bdbee397f645b45224e37e4c9ce208dd" Nov 03 19:49:14.474158 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.474125 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d"] Nov 03 19:49:14.474708 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.474692 5059 cpu_manager.go:414] "containerMap: removing container" podUID="9065a339-0b3b-49f5-ba8f-03ac69b21ca4" containerName="revoke-keys" Nov 03 19:49:14.474765 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.474709 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9065a339-0b3b-49f5-ba8f-03ac69b21ca4" containerName="revoke-keys" Nov 03 19:49:14.474817 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.474807 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9065a339-0b3b-49f5-ba8f-03ac69b21ca4" containerName="revoke-keys" Nov 03 19:49:14.478315 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.478292 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.488393 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.488372 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-9f46e5-openshift-default-dockercfg-4djl8\"" Nov 03 19:49:14.491381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.491351 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d"] Nov 03 19:49:14.608961 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.608936 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.609103 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.608968 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.609103 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.608989 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5dsd\" (UniqueName: \"kubernetes.io/projected/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-kube-api-access-q5dsd\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.609193 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.609141 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.609193 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.609180 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.609262 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.609208 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.609331 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.609308 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.609423 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.609348 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.609483 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.609468 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.710898 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.710875 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.711015 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.710903 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.711015 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.710925 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5dsd\" (UniqueName: \"kubernetes.io/projected/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-kube-api-access-q5dsd\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.711015 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.710966 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.711015 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.710990 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.711015 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.711009 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.711265 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.711030 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.711265 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.711051 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.711265 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.711133 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.711545 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.711516 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.711695 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.711628 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.711695 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.711624 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.711791 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.711696 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.711791 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.711716 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.713513 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.713487 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.713739 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.713723 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.723406 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.723384 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.723863 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.723837 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5dsd\" (UniqueName: \"kubernetes.io/projected/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-kube-api-access-q5dsd\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:14.791872 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:14.791821 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:15.077139 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:15.077104 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d"] Nov 03 19:49:15.077922 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:49:15.077883 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b88c7d_08e9_4203_9ebd_2c9f69b884dc.slice/crio-d5fbecd3af8991e6c4ec43085889d7697d506a848fcc7a66a3f398d621b52d17 WatchSource:0}: Error finding container d5fbecd3af8991e6c4ec43085889d7697d506a848fcc7a66a3f398d621b52d17: Status 404 returned error can't find the container with id d5fbecd3af8991e6c4ec43085889d7697d506a848fcc7a66a3f398d621b52d17 Nov 03 19:49:15.080342 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:15.080308 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:49:15.080433 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:15.080376 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:49:15.080433 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:15.080402 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:49:16.089491 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:16.089454 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" event={"ID":"40b88c7d-08e9-4203-9ebd-2c9f69b884dc","Type":"ContainerStarted","Data":"6739ff178c81c67d823d4e8761f1f61d4d648a31a61e02195fb24f7ce8ca0bdf"} Nov 03 19:49:16.089951 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:16.089511 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" event={"ID":"40b88c7d-08e9-4203-9ebd-2c9f69b884dc","Type":"ContainerStarted","Data":"d5fbecd3af8991e6c4ec43085889d7697d506a848fcc7a66a3f398d621b52d17"} Nov 03 19:49:16.120776 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:16.120733 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" podStartSLOduration=2.12072008 podStartE2EDuration="2.12072008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:16.117951825 +0000 UTC m=+750.375749084" watchObservedRunningTime="2026-11-03 19:49:16.12072008 +0000 UTC m=+750.378517338" Nov 03 19:49:16.792263 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:16.792233 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:16.793725 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:16.793695 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:49:16.793857 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:16.793747 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" podUID="40b88c7d-08e9-4203-9ebd-2c9f69b884dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 19:49:17.792377 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:17.792344 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:49:17.792800 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:17.792394 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" podUID="40b88c7d-08e9-4203-9ebd-2c9f69b884dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 19:49:18.793118 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:18.793084 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:49:18.793475 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:18.793133 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" podUID="40b88c7d-08e9-4203-9ebd-2c9f69b884dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 19:49:19.792520 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:19.792481 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:49:19.792743 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:19.792539 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" podUID="40b88c7d-08e9-4203-9ebd-2c9f69b884dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 19:49:20.792363 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:20.792322 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:49:20.792799 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:20.792386 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" podUID="40b88c7d-08e9-4203-9ebd-2c9f69b884dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 19:49:21.797307 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:21.797271 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:21.797796 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:21.797773 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:21.798467 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:21.798449 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 19:49:29.122536 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.122500 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95"] Nov 03 19:49:29.126379 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.126357 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.130504 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.130484 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-2b5703-openshift-default-dockercfg-jmjbm\"" Nov 03 19:49:29.139485 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.139456 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95"] Nov 03 19:49:29.209998 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.209964 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.210137 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.210027 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d80995-5f1c-46db-8d71-a130cb92e279-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.210137 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.210103 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.210225 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.210155 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.210272 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.210253 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.210308 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.210292 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.210345 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.210312 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqzgx\" (UniqueName: \"kubernetes.io/projected/c2d80995-5f1c-46db-8d71-a130cb92e279-kube-api-access-cqzgx\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.210445 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.210430 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.210489 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.210476 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.311020 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.310987 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.311199 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.311031 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.311199 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.311131 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqzgx\" (UniqueName: \"kubernetes.io/projected/c2d80995-5f1c-46db-8d71-a130cb92e279-kube-api-access-cqzgx\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.311323 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.311207 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.311323 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.311241 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.311483 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.311462 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.311636 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.311618 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d80995-5f1c-46db-8d71-a130cb92e279-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.311733 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.311680 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.311798 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.311734 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.311798 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.311750 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.311798 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.311764 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.311952 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.311847 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.312037 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.312013 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.312340 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.312316 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d80995-5f1c-46db-8d71-a130cb92e279-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.313642 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.313616 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.314294 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.314270 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.314795 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.314773 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-371219cd-lt9kd"] Nov 03 19:49:29.318751 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.318734 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-371219cd-lt9kd" Nov 03 19:49:29.323731 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.323713 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.323847 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.323771 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqzgx\" (UniqueName: \"kubernetes.io/projected/c2d80995-5f1c-46db-8d71-a130cb92e279-kube-api-access-cqzgx\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.325220 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.325199 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 19:49:29.325333 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.325262 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:49:29.328793 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.328772 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-371219cd-lt9kd"] Nov 03 19:49:29.412765 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.412703 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36e9fc9c-cc8e-4775-9d55-11969c44792d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-371219cd-lt9kd\" (UID: \"36e9fc9c-cc8e-4775-9d55-11969c44792d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-371219cd-lt9kd" Nov 03 19:49:29.412765 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.412748 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36e9fc9c-cc8e-4775-9d55-11969c44792d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-371219cd-lt9kd\" (UID: \"36e9fc9c-cc8e-4775-9d55-11969c44792d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-371219cd-lt9kd" Nov 03 19:49:29.412915 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.412890 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5vp7\" (UniqueName: \"kubernetes.io/projected/36e9fc9c-cc8e-4775-9d55-11969c44792d-kube-api-access-p5vp7\") pod \"maas-api-revoke-sub-371219cd-lt9kd\" (UID: \"36e9fc9c-cc8e-4775-9d55-11969c44792d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-371219cd-lt9kd" Nov 03 19:49:29.443297 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.443272 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:29.514229 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.514196 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36e9fc9c-cc8e-4775-9d55-11969c44792d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-371219cd-lt9kd\" (UID: \"36e9fc9c-cc8e-4775-9d55-11969c44792d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-371219cd-lt9kd" Nov 03 19:49:29.514348 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.514260 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36e9fc9c-cc8e-4775-9d55-11969c44792d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-371219cd-lt9kd\" (UID: \"36e9fc9c-cc8e-4775-9d55-11969c44792d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-371219cd-lt9kd" Nov 03 19:49:29.514398 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.514382 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5vp7\" (UniqueName: \"kubernetes.io/projected/36e9fc9c-cc8e-4775-9d55-11969c44792d-kube-api-access-p5vp7\") pod \"maas-api-revoke-sub-371219cd-lt9kd\" (UID: \"36e9fc9c-cc8e-4775-9d55-11969c44792d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-371219cd-lt9kd" Nov 03 19:49:29.515180 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.515156 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36e9fc9c-cc8e-4775-9d55-11969c44792d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-371219cd-lt9kd\" (UID: \"36e9fc9c-cc8e-4775-9d55-11969c44792d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-371219cd-lt9kd" Nov 03 19:49:29.519159 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.519136 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36e9fc9c-cc8e-4775-9d55-11969c44792d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-371219cd-lt9kd\" (UID: \"36e9fc9c-cc8e-4775-9d55-11969c44792d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-371219cd-lt9kd" Nov 03 19:49:29.529603 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.529574 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5vp7\" (UniqueName: \"kubernetes.io/projected/36e9fc9c-cc8e-4775-9d55-11969c44792d-kube-api-access-p5vp7\") pod \"maas-api-revoke-sub-371219cd-lt9kd\" (UID: \"36e9fc9c-cc8e-4775-9d55-11969c44792d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-371219cd-lt9kd" Nov 03 19:49:29.654844 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.654813 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-371219cd-lt9kd" Nov 03 19:49:29.727820 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.727792 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95"] Nov 03 19:49:29.728645 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:49:29.728620 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d80995_5f1c_46db_8d71_a130cb92e279.slice/crio-7f4aac55cec05c0b92f14350c76f0428c0ca2ed33f0cd6f1220537cebab730b1 WatchSource:0}: Error finding container 7f4aac55cec05c0b92f14350c76f0428c0ca2ed33f0cd6f1220537cebab730b1: Status 404 returned error can't find the container with id 7f4aac55cec05c0b92f14350c76f0428c0ca2ed33f0cd6f1220537cebab730b1 Nov 03 19:49:29.730976 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.730942 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:49:29.731064 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.731022 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:49:29.731123 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.731066 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:49:29.934176 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:29.934136 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-371219cd-lt9kd"] Nov 03 19:49:29.934491 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:49:29.934468 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36e9fc9c_cc8e_4775_9d55_11969c44792d.slice/crio-dfcdea67f11b74a4f6fc7c2c5f6de39f422eea20fffce7edbb87a2e79cb4b0a0 WatchSource:0}: Error finding container dfcdea67f11b74a4f6fc7c2c5f6de39f422eea20fffce7edbb87a2e79cb4b0a0: Status 404 returned error can't find the container with id dfcdea67f11b74a4f6fc7c2c5f6de39f422eea20fffce7edbb87a2e79cb4b0a0 Nov 03 19:49:30.188896 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:30.188787 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" event={"ID":"c2d80995-5f1c-46db-8d71-a130cb92e279","Type":"ContainerStarted","Data":"39fd5a87217d3b8146ccfe78173093985a26712d776696de4ab3b763f50a762f"} Nov 03 19:49:30.188896 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:30.188860 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" event={"ID":"c2d80995-5f1c-46db-8d71-a130cb92e279","Type":"ContainerStarted","Data":"7f4aac55cec05c0b92f14350c76f0428c0ca2ed33f0cd6f1220537cebab730b1"} Nov 03 19:49:30.191746 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:30.191694 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-371219cd-lt9kd" event={"ID":"36e9fc9c-cc8e-4775-9d55-11969c44792d","Type":"ContainerStarted","Data":"dfcdea67f11b74a4f6fc7c2c5f6de39f422eea20fffce7edbb87a2e79cb4b0a0"} Nov 03 19:49:30.215198 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:30.215150 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" podStartSLOduration=1.215136989 podStartE2EDuration="1.215136989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:30.211864502 +0000 UTC m=+764.469661771" watchObservedRunningTime="2026-11-03 19:49:30.215136989 +0000 UTC m=+764.472934245" Nov 03 19:49:30.443653 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:30.443578 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:30.445314 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:30.445292 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:49:30.445368 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:30.445338 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" podUID="c2d80995-5f1c-46db-8d71-a130cb92e279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 19:49:30.489942 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:30.489904 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ce079d3-kn6pp"] Nov 03 19:49:30.493762 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:30.493745 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce079d3-kn6pp" Nov 03 19:49:30.503809 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:30.503591 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ce079d3-kn6pp"] Nov 03 19:49:30.525348 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:30.525319 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f72b2d-9363-46aa-a77e-7f1640eb085d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ce079d3-kn6pp\" (UID: \"95f72b2d-9363-46aa-a77e-7f1640eb085d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce079d3-kn6pp" Nov 03 19:49:30.525459 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:30.525365 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfgks\" (UniqueName: \"kubernetes.io/projected/95f72b2d-9363-46aa-a77e-7f1640eb085d-kube-api-access-pfgks\") pod \"maas-api-revoke-sub-0ce079d3-kn6pp\" (UID: \"95f72b2d-9363-46aa-a77e-7f1640eb085d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce079d3-kn6pp" Nov 03 19:49:30.525500 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:30.525472 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f72b2d-9363-46aa-a77e-7f1640eb085d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ce079d3-kn6pp\" (UID: \"95f72b2d-9363-46aa-a77e-7f1640eb085d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce079d3-kn6pp" Nov 03 19:49:30.627174 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:30.627132 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f72b2d-9363-46aa-a77e-7f1640eb085d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ce079d3-kn6pp\" (UID: \"95f72b2d-9363-46aa-a77e-7f1640eb085d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce079d3-kn6pp" Nov 03 19:49:30.627407 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:30.627180 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfgks\" (UniqueName: \"kubernetes.io/projected/95f72b2d-9363-46aa-a77e-7f1640eb085d-kube-api-access-pfgks\") pod \"maas-api-revoke-sub-0ce079d3-kn6pp\" (UID: \"95f72b2d-9363-46aa-a77e-7f1640eb085d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce079d3-kn6pp" Nov 03 19:49:30.627407 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:30.627244 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f72b2d-9363-46aa-a77e-7f1640eb085d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ce079d3-kn6pp\" (UID: \"95f72b2d-9363-46aa-a77e-7f1640eb085d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce079d3-kn6pp" Nov 03 19:49:30.627947 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:30.627922 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f72b2d-9363-46aa-a77e-7f1640eb085d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ce079d3-kn6pp\" (UID: \"95f72b2d-9363-46aa-a77e-7f1640eb085d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce079d3-kn6pp" Nov 03 19:49:30.629774 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:30.629752 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f72b2d-9363-46aa-a77e-7f1640eb085d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ce079d3-kn6pp\" (UID: \"95f72b2d-9363-46aa-a77e-7f1640eb085d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce079d3-kn6pp" Nov 03 19:49:30.642128 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:30.642104 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfgks\" (UniqueName: \"kubernetes.io/projected/95f72b2d-9363-46aa-a77e-7f1640eb085d-kube-api-access-pfgks\") pod \"maas-api-revoke-sub-0ce079d3-kn6pp\" (UID: \"95f72b2d-9363-46aa-a77e-7f1640eb085d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce079d3-kn6pp" Nov 03 19:49:30.809763 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:30.809652 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce079d3-kn6pp" Nov 03 19:49:31.113138 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:31.113111 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ce079d3-kn6pp"] Nov 03 19:49:31.117732 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:49:31.117543 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f72b2d_9363_46aa_a77e_7f1640eb085d.slice/crio-040d5bd6eecee17fd32a2d5f481cade010e23177fa18beb9f04a2c5cf8e71e38 WatchSource:0}: Error finding container 040d5bd6eecee17fd32a2d5f481cade010e23177fa18beb9f04a2c5cf8e71e38: Status 404 returned error can't find the container with id 040d5bd6eecee17fd32a2d5f481cade010e23177fa18beb9f04a2c5cf8e71e38 Nov 03 19:49:31.199317 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:31.199276 5059 generic.go:356] "Generic (PLEG): container finished" podID="36e9fc9c-cc8e-4775-9d55-11969c44792d" containerID="9507c865fc128e60fba96e8045e83025c012bb62e9fa3731771847f3a9d32af8" exitCode=0 Nov 03 19:49:31.199750 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:31.199360 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-371219cd-lt9kd" event={"ID":"36e9fc9c-cc8e-4775-9d55-11969c44792d","Type":"ContainerDied","Data":"9507c865fc128e60fba96e8045e83025c012bb62e9fa3731771847f3a9d32af8"} Nov 03 19:49:31.200747 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:31.200723 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce079d3-kn6pp" event={"ID":"95f72b2d-9363-46aa-a77e-7f1640eb085d","Type":"ContainerStarted","Data":"040d5bd6eecee17fd32a2d5f481cade010e23177fa18beb9f04a2c5cf8e71e38"} Nov 03 19:49:31.444252 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:31.444171 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:49:31.444252 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:31.444229 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" podUID="c2d80995-5f1c-46db-8d71-a130cb92e279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 19:49:32.208400 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.208363 5059 generic.go:356] "Generic (PLEG): container finished" podID="95f72b2d-9363-46aa-a77e-7f1640eb085d" containerID="3979c4877483574442f4dd4ba5cd89ba3e1d3def2157691d8be3e10485f5b6d6" exitCode=0 Nov 03 19:49:32.208889 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.208456 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce079d3-kn6pp" event={"ID":"95f72b2d-9363-46aa-a77e-7f1640eb085d","Type":"ContainerDied","Data":"3979c4877483574442f4dd4ba5cd89ba3e1d3def2157691d8be3e10485f5b6d6"} Nov 03 19:49:32.243158 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.243127 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b50ae9ac-whrv7"] Nov 03 19:49:32.247497 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.247472 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b50ae9ac-whrv7" Nov 03 19:49:32.256931 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.256903 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b50ae9ac-whrv7"] Nov 03 19:49:32.344096 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.344013 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470843af-fc92-4961-824f-7a862483a18b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b50ae9ac-whrv7\" (UID: \"470843af-fc92-4961-824f-7a862483a18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b50ae9ac-whrv7" Nov 03 19:49:32.344214 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.344165 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn4rl\" (UniqueName: \"kubernetes.io/projected/470843af-fc92-4961-824f-7a862483a18b-kube-api-access-pn4rl\") pod \"maas-api-revoke-sub-b50ae9ac-whrv7\" (UID: \"470843af-fc92-4961-824f-7a862483a18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b50ae9ac-whrv7" Nov 03 19:49:32.344256 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.344206 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470843af-fc92-4961-824f-7a862483a18b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b50ae9ac-whrv7\" (UID: \"470843af-fc92-4961-824f-7a862483a18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b50ae9ac-whrv7" Nov 03 19:49:32.420383 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.420356 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-371219cd-lt9kd" Nov 03 19:49:32.444111 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.444075 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:49:32.444257 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.444126 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" podUID="c2d80995-5f1c-46db-8d71-a130cb92e279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 19:49:32.444678 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.444638 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36e9fc9c-cc8e-4775-9d55-11969c44792d-kube-api-access-p5vp7\" (UniqueName: \"kubernetes.io/projected/36e9fc9c-cc8e-4775-9d55-11969c44792d-kube-api-access-p5vp7\") pod \"36e9fc9c-cc8e-4775-9d55-11969c44792d\" (UID: \"36e9fc9c-cc8e-4775-9d55-11969c44792d\") " Nov 03 19:49:32.444814 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.444798 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36e9fc9c-cc8e-4775-9d55-11969c44792d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36e9fc9c-cc8e-4775-9d55-11969c44792d-maas-api-openshift-ca\") pod \"36e9fc9c-cc8e-4775-9d55-11969c44792d\" (UID: \"36e9fc9c-cc8e-4775-9d55-11969c44792d\") " Nov 03 19:49:32.444897 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.444857 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36e9fc9c-cc8e-4775-9d55-11969c44792d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36e9fc9c-cc8e-4775-9d55-11969c44792d-maas-api-xks-ca\") pod \"36e9fc9c-cc8e-4775-9d55-11969c44792d\" (UID: \"36e9fc9c-cc8e-4775-9d55-11969c44792d\") " Nov 03 19:49:32.444994 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.444973 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470843af-fc92-4961-824f-7a862483a18b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b50ae9ac-whrv7\" (UID: \"470843af-fc92-4961-824f-7a862483a18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b50ae9ac-whrv7" Nov 03 19:49:32.445093 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.445062 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn4rl\" (UniqueName: \"kubernetes.io/projected/470843af-fc92-4961-824f-7a862483a18b-kube-api-access-pn4rl\") pod \"maas-api-revoke-sub-b50ae9ac-whrv7\" (UID: \"470843af-fc92-4961-824f-7a862483a18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b50ae9ac-whrv7" Nov 03 19:49:32.445201 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.445180 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36e9fc9c-cc8e-4775-9d55-11969c44792d-maas-api-openshift-ca" pod "36e9fc9c-cc8e-4775-9d55-11969c44792d" (UID: "36e9fc9c-cc8e-4775-9d55-11969c44792d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:49:32.445472 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.445448 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470843af-fc92-4961-824f-7a862483a18b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b50ae9ac-whrv7\" (UID: \"470843af-fc92-4961-824f-7a862483a18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b50ae9ac-whrv7" Nov 03 19:49:32.445769 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.445744 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36e9fc9c-cc8e-4775-9d55-11969c44792d-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:32.445865 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.445833 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470843af-fc92-4961-824f-7a862483a18b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b50ae9ac-whrv7\" (UID: \"470843af-fc92-4961-824f-7a862483a18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b50ae9ac-whrv7" Nov 03 19:49:32.447258 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.447210 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36e9fc9c-cc8e-4775-9d55-11969c44792d-kube-api-access-p5vp7" pod "36e9fc9c-cc8e-4775-9d55-11969c44792d" (UID: "36e9fc9c-cc8e-4775-9d55-11969c44792d"). InnerVolumeSpecName "kube-api-access-p5vp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:49:32.447358 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.447327 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36e9fc9c-cc8e-4775-9d55-11969c44792d-maas-api-xks-ca" pod "36e9fc9c-cc8e-4775-9d55-11969c44792d" (UID: "36e9fc9c-cc8e-4775-9d55-11969c44792d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:49:32.447785 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.447763 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470843af-fc92-4961-824f-7a862483a18b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b50ae9ac-whrv7\" (UID: \"470843af-fc92-4961-824f-7a862483a18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b50ae9ac-whrv7" Nov 03 19:49:32.488337 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.488302 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn4rl\" (UniqueName: \"kubernetes.io/projected/470843af-fc92-4961-824f-7a862483a18b-kube-api-access-pn4rl\") pod \"maas-api-revoke-sub-b50ae9ac-whrv7\" (UID: \"470843af-fc92-4961-824f-7a862483a18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b50ae9ac-whrv7" Nov 03 19:49:32.547307 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.547276 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5vp7\" (UniqueName: \"kubernetes.io/projected/36e9fc9c-cc8e-4775-9d55-11969c44792d-kube-api-access-p5vp7\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:32.547307 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.547305 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36e9fc9c-cc8e-4775-9d55-11969c44792d-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:32.592042 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.592009 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b50ae9ac-whrv7" Nov 03 19:49:32.885807 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:32.885719 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b50ae9ac-whrv7"] Nov 03 19:49:32.890932 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:49:32.890893 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod470843af_fc92_4961_824f_7a862483a18b.slice/crio-077747d88ea30b02ab7990f5f69ed340387338d461d4dcc744c6675ee0936110 WatchSource:0}: Error finding container 077747d88ea30b02ab7990f5f69ed340387338d461d4dcc744c6675ee0936110: Status 404 returned error can't find the container with id 077747d88ea30b02ab7990f5f69ed340387338d461d4dcc744c6675ee0936110 Nov 03 19:49:33.216130 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:33.216050 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-371219cd-lt9kd" Nov 03 19:49:33.216130 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:33.216083 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-371219cd-lt9kd" event={"ID":"36e9fc9c-cc8e-4775-9d55-11969c44792d","Type":"ContainerDied","Data":"dfcdea67f11b74a4f6fc7c2c5f6de39f422eea20fffce7edbb87a2e79cb4b0a0"} Nov 03 19:49:33.216557 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:33.216130 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfcdea67f11b74a4f6fc7c2c5f6de39f422eea20fffce7edbb87a2e79cb4b0a0" Nov 03 19:49:33.217437 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:33.217417 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b50ae9ac-whrv7" event={"ID":"470843af-fc92-4961-824f-7a862483a18b","Type":"ContainerStarted","Data":"077747d88ea30b02ab7990f5f69ed340387338d461d4dcc744c6675ee0936110"} Nov 03 19:49:33.414557 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:33.414533 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce079d3-kn6pp" Nov 03 19:49:33.444515 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:33.444485 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:49:33.444675 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:33.444544 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" podUID="c2d80995-5f1c-46db-8d71-a130cb92e279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 19:49:33.457290 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:33.457264 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95f72b2d-9363-46aa-a77e-7f1640eb085d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f72b2d-9363-46aa-a77e-7f1640eb085d-maas-api-xks-ca\") pod \"95f72b2d-9363-46aa-a77e-7f1640eb085d\" (UID: \"95f72b2d-9363-46aa-a77e-7f1640eb085d\") " Nov 03 19:49:33.457388 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:33.457307 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95f72b2d-9363-46aa-a77e-7f1640eb085d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f72b2d-9363-46aa-a77e-7f1640eb085d-maas-api-openshift-ca\") pod \"95f72b2d-9363-46aa-a77e-7f1640eb085d\" (UID: \"95f72b2d-9363-46aa-a77e-7f1640eb085d\") " Nov 03 19:49:33.457388 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:33.457371 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95f72b2d-9363-46aa-a77e-7f1640eb085d-kube-api-access-pfgks\" (UniqueName: \"kubernetes.io/projected/95f72b2d-9363-46aa-a77e-7f1640eb085d-kube-api-access-pfgks\") pod \"95f72b2d-9363-46aa-a77e-7f1640eb085d\" (UID: \"95f72b2d-9363-46aa-a77e-7f1640eb085d\") " Nov 03 19:49:33.482756 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:33.466787 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95f72b2d-9363-46aa-a77e-7f1640eb085d-maas-api-openshift-ca" pod "95f72b2d-9363-46aa-a77e-7f1640eb085d" (UID: "95f72b2d-9363-46aa-a77e-7f1640eb085d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:49:33.485146 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:33.485104 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95f72b2d-9363-46aa-a77e-7f1640eb085d-maas-api-xks-ca" pod "95f72b2d-9363-46aa-a77e-7f1640eb085d" (UID: "95f72b2d-9363-46aa-a77e-7f1640eb085d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:49:33.486897 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:33.486837 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f72b2d-9363-46aa-a77e-7f1640eb085d-kube-api-access-pfgks" pod "95f72b2d-9363-46aa-a77e-7f1640eb085d" (UID: "95f72b2d-9363-46aa-a77e-7f1640eb085d"). InnerVolumeSpecName "kube-api-access-pfgks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:49:33.579684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:33.579510 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f72b2d-9363-46aa-a77e-7f1640eb085d-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:33.579684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:33.579547 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f72b2d-9363-46aa-a77e-7f1640eb085d-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:33.579684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:33.579563 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfgks\" (UniqueName: \"kubernetes.io/projected/95f72b2d-9363-46aa-a77e-7f1640eb085d-kube-api-access-pfgks\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:34.224851 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:34.224761 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce079d3-kn6pp" Nov 03 19:49:34.224851 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:34.224789 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ce079d3-kn6pp" event={"ID":"95f72b2d-9363-46aa-a77e-7f1640eb085d","Type":"ContainerDied","Data":"040d5bd6eecee17fd32a2d5f481cade010e23177fa18beb9f04a2c5cf8e71e38"} Nov 03 19:49:34.224851 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:34.224837 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="040d5bd6eecee17fd32a2d5f481cade010e23177fa18beb9f04a2c5cf8e71e38" Nov 03 19:49:34.226554 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:34.226521 5059 generic.go:356] "Generic (PLEG): container finished" podID="470843af-fc92-4961-824f-7a862483a18b" containerID="d0e7c9ae0e393242a028e6fa8725fcdce924e125da4eedba4666cfa18ee8bd5f" exitCode=0 Nov 03 19:49:34.226723 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:34.226550 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b50ae9ac-whrv7" event={"ID":"470843af-fc92-4961-824f-7a862483a18b","Type":"ContainerDied","Data":"d0e7c9ae0e393242a028e6fa8725fcdce924e125da4eedba4666cfa18ee8bd5f"} Nov 03 19:49:34.449380 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:34.449335 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:49:34.449589 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:34.449414 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" podUID="c2d80995-5f1c-46db-8d71-a130cb92e279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 19:49:35.444325 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:35.444288 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:49:35.444998 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:35.444943 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" podUID="c2d80995-5f1c-46db-8d71-a130cb92e279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 19:49:35.549150 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:35.549125 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b50ae9ac-whrv7" Nov 03 19:49:35.631931 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:35.631891 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/470843af-fc92-4961-824f-7a862483a18b-kube-api-access-pn4rl\" (UniqueName: \"kubernetes.io/projected/470843af-fc92-4961-824f-7a862483a18b-kube-api-access-pn4rl\") pod \"470843af-fc92-4961-824f-7a862483a18b\" (UID: \"470843af-fc92-4961-824f-7a862483a18b\") " Nov 03 19:49:35.632122 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:35.632016 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/470843af-fc92-4961-824f-7a862483a18b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470843af-fc92-4961-824f-7a862483a18b-maas-api-xks-ca\") pod \"470843af-fc92-4961-824f-7a862483a18b\" (UID: \"470843af-fc92-4961-824f-7a862483a18b\") " Nov 03 19:49:35.632210 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:35.632121 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/470843af-fc92-4961-824f-7a862483a18b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470843af-fc92-4961-824f-7a862483a18b-maas-api-openshift-ca\") pod \"470843af-fc92-4961-824f-7a862483a18b\" (UID: \"470843af-fc92-4961-824f-7a862483a18b\") " Nov 03 19:49:35.633736 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:35.633705 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/470843af-fc92-4961-824f-7a862483a18b-maas-api-openshift-ca" pod "470843af-fc92-4961-824f-7a862483a18b" (UID: "470843af-fc92-4961-824f-7a862483a18b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:49:35.655935 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:35.655143 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/470843af-fc92-4961-824f-7a862483a18b-kube-api-access-pn4rl" pod "470843af-fc92-4961-824f-7a862483a18b" (UID: "470843af-fc92-4961-824f-7a862483a18b"). InnerVolumeSpecName "kube-api-access-pn4rl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:49:35.655935 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:35.655500 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/470843af-fc92-4961-824f-7a862483a18b-maas-api-xks-ca" pod "470843af-fc92-4961-824f-7a862483a18b" (UID: "470843af-fc92-4961-824f-7a862483a18b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:49:35.734347 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:35.734232 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470843af-fc92-4961-824f-7a862483a18b-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:35.734347 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:35.734271 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn4rl\" (UniqueName: \"kubernetes.io/projected/470843af-fc92-4961-824f-7a862483a18b-kube-api-access-pn4rl\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:35.734347 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:35.734287 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470843af-fc92-4961-824f-7a862483a18b-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:36.241604 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:36.241520 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b50ae9ac-whrv7" Nov 03 19:49:36.241797 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:36.241520 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b50ae9ac-whrv7" event={"ID":"470843af-fc92-4961-824f-7a862483a18b","Type":"ContainerDied","Data":"077747d88ea30b02ab7990f5f69ed340387338d461d4dcc744c6675ee0936110"} Nov 03 19:49:36.241797 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:36.241709 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="077747d88ea30b02ab7990f5f69ed340387338d461d4dcc744c6675ee0936110" Nov 03 19:49:37.125584 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:37.125551 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:37.126034 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:37.125849 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:37.127195 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:37.127167 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 19:49:40.610785 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:49:40.610749 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:49:42.566009 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:42.565976 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cdba36c-svnbd"] Nov 03 19:49:42.566500 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:42.566474 5059 cpu_manager.go:414] "containerMap: removing container" podUID="36e9fc9c-cc8e-4775-9d55-11969c44792d" containerName="revoke-keys" Nov 03 19:49:42.566500 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:42.566487 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36e9fc9c-cc8e-4775-9d55-11969c44792d" containerName="revoke-keys" Nov 03 19:49:42.566500 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:42.566495 5059 cpu_manager.go:414] "containerMap: removing container" podUID="470843af-fc92-4961-824f-7a862483a18b" containerName="revoke-keys" Nov 03 19:49:42.566500 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:42.566500 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="470843af-fc92-4961-824f-7a862483a18b" containerName="revoke-keys" Nov 03 19:49:42.566740 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:42.566527 5059 cpu_manager.go:414] "containerMap: removing container" podUID="95f72b2d-9363-46aa-a77e-7f1640eb085d" containerName="revoke-keys" Nov 03 19:49:42.566740 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:42.566532 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95f72b2d-9363-46aa-a77e-7f1640eb085d" containerName="revoke-keys" Nov 03 19:49:42.566740 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:42.566619 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="95f72b2d-9363-46aa-a77e-7f1640eb085d" containerName="revoke-keys" Nov 03 19:49:42.566740 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:42.566630 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="36e9fc9c-cc8e-4775-9d55-11969c44792d" containerName="revoke-keys" Nov 03 19:49:42.566740 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:42.566636 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="470843af-fc92-4961-824f-7a862483a18b" containerName="revoke-keys" Nov 03 19:49:42.571041 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:42.571023 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cdba36c-svnbd" Nov 03 19:49:42.578916 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:42.577628 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:49:42.578916 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:42.577758 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 19:49:42.579914 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:42.579890 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cdba36c-svnbd"] Nov 03 19:49:42.713752 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:42.713723 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a20d160-602b-4513-94a1-159916a46cb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cdba36c-svnbd\" (UID: \"6a20d160-602b-4513-94a1-159916a46cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cdba36c-svnbd" Nov 03 19:49:42.713902 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:42.713776 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpgh5\" (UniqueName: \"kubernetes.io/projected/6a20d160-602b-4513-94a1-159916a46cb8-kube-api-access-xpgh5\") pod \"maas-api-revoke-sub-4cdba36c-svnbd\" (UID: \"6a20d160-602b-4513-94a1-159916a46cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cdba36c-svnbd" Nov 03 19:49:42.713902 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:42.713856 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a20d160-602b-4513-94a1-159916a46cb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cdba36c-svnbd\" (UID: \"6a20d160-602b-4513-94a1-159916a46cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cdba36c-svnbd" Nov 03 19:49:42.815056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:42.815027 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a20d160-602b-4513-94a1-159916a46cb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cdba36c-svnbd\" (UID: \"6a20d160-602b-4513-94a1-159916a46cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cdba36c-svnbd" Nov 03 19:49:42.815212 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:42.815093 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpgh5\" (UniqueName: \"kubernetes.io/projected/6a20d160-602b-4513-94a1-159916a46cb8-kube-api-access-xpgh5\") pod \"maas-api-revoke-sub-4cdba36c-svnbd\" (UID: \"6a20d160-602b-4513-94a1-159916a46cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cdba36c-svnbd" Nov 03 19:49:42.815212 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:42.815132 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a20d160-602b-4513-94a1-159916a46cb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cdba36c-svnbd\" (UID: \"6a20d160-602b-4513-94a1-159916a46cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cdba36c-svnbd" Nov 03 19:49:42.815684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:42.815635 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a20d160-602b-4513-94a1-159916a46cb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cdba36c-svnbd\" (UID: \"6a20d160-602b-4513-94a1-159916a46cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cdba36c-svnbd" Nov 03 19:49:42.817596 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:42.817547 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a20d160-602b-4513-94a1-159916a46cb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cdba36c-svnbd\" (UID: \"6a20d160-602b-4513-94a1-159916a46cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cdba36c-svnbd" Nov 03 19:49:42.827726 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:42.827707 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpgh5\" (UniqueName: \"kubernetes.io/projected/6a20d160-602b-4513-94a1-159916a46cb8-kube-api-access-xpgh5\") pod \"maas-api-revoke-sub-4cdba36c-svnbd\" (UID: \"6a20d160-602b-4513-94a1-159916a46cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cdba36c-svnbd" Nov 03 19:49:42.886752 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:42.886730 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cdba36c-svnbd" Nov 03 19:49:43.184175 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:43.180569 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cdba36c-svnbd"] Nov 03 19:49:43.194346 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:49:43.194281 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a20d160_602b_4513_94a1_159916a46cb8.slice/crio-26c6637e88170d490473f0393e334887e3ca2fbe66f04c43b6e73233f43f95b1 WatchSource:0}: Error finding container 26c6637e88170d490473f0393e334887e3ca2fbe66f04c43b6e73233f43f95b1: Status 404 returned error can't find the container with id 26c6637e88170d490473f0393e334887e3ca2fbe66f04c43b6e73233f43f95b1 Nov 03 19:49:43.291638 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:43.291604 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cdba36c-svnbd" event={"ID":"6a20d160-602b-4513-94a1-159916a46cb8","Type":"ContainerStarted","Data":"26c6637e88170d490473f0393e334887e3ca2fbe66f04c43b6e73233f43f95b1"} Nov 03 19:49:44.299041 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:44.298963 5059 generic.go:356] "Generic (PLEG): container finished" podID="6a20d160-602b-4513-94a1-159916a46cb8" containerID="4934f20758fe1e6818e6cfdfdd7652937b28b05db9f7f37a3fe0771ca77b07ce" exitCode=0 Nov 03 19:49:44.299398 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:44.299049 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cdba36c-svnbd" event={"ID":"6a20d160-602b-4513-94a1-159916a46cb8","Type":"ContainerDied","Data":"4934f20758fe1e6818e6cfdfdd7652937b28b05db9f7f37a3fe0771ca77b07ce"} Nov 03 19:49:45.455185 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:45.455159 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cdba36c-svnbd" Nov 03 19:49:45.543065 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:45.543024 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a20d160-602b-4513-94a1-159916a46cb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a20d160-602b-4513-94a1-159916a46cb8-maas-api-openshift-ca\") pod \"6a20d160-602b-4513-94a1-159916a46cb8\" (UID: \"6a20d160-602b-4513-94a1-159916a46cb8\") " Nov 03 19:49:45.543065 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:45.543075 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a20d160-602b-4513-94a1-159916a46cb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a20d160-602b-4513-94a1-159916a46cb8-maas-api-xks-ca\") pod \"6a20d160-602b-4513-94a1-159916a46cb8\" (UID: \"6a20d160-602b-4513-94a1-159916a46cb8\") " Nov 03 19:49:45.543256 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:45.543103 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a20d160-602b-4513-94a1-159916a46cb8-kube-api-access-xpgh5\" (UniqueName: \"kubernetes.io/projected/6a20d160-602b-4513-94a1-159916a46cb8-kube-api-access-xpgh5\") pod \"6a20d160-602b-4513-94a1-159916a46cb8\" (UID: \"6a20d160-602b-4513-94a1-159916a46cb8\") " Nov 03 19:49:45.543363 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:45.543341 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a20d160-602b-4513-94a1-159916a46cb8-maas-api-openshift-ca" pod "6a20d160-602b-4513-94a1-159916a46cb8" (UID: "6a20d160-602b-4513-94a1-159916a46cb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:49:45.545256 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:45.545226 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a20d160-602b-4513-94a1-159916a46cb8-kube-api-access-xpgh5" pod "6a20d160-602b-4513-94a1-159916a46cb8" (UID: "6a20d160-602b-4513-94a1-159916a46cb8"). InnerVolumeSpecName "kube-api-access-xpgh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:49:45.545352 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:45.545276 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a20d160-602b-4513-94a1-159916a46cb8-maas-api-xks-ca" pod "6a20d160-602b-4513-94a1-159916a46cb8" (UID: "6a20d160-602b-4513-94a1-159916a46cb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:49:45.644058 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:45.644026 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a20d160-602b-4513-94a1-159916a46cb8-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:45.644058 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:45.644051 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a20d160-602b-4513-94a1-159916a46cb8-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:45.644241 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:45.644066 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpgh5\" (UniqueName: \"kubernetes.io/projected/6a20d160-602b-4513-94a1-159916a46cb8-kube-api-access-xpgh5\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:46.311689 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.311639 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cdba36c-svnbd" event={"ID":"6a20d160-602b-4513-94a1-159916a46cb8","Type":"ContainerDied","Data":"26c6637e88170d490473f0393e334887e3ca2fbe66f04c43b6e73233f43f95b1"} Nov 03 19:49:46.311689 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.311680 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cdba36c-svnbd" Nov 03 19:49:46.311914 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.311709 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26c6637e88170d490473f0393e334887e3ca2fbe66f04c43b6e73233f43f95b1" Nov 03 19:49:46.442127 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.442095 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6b644db666-cldhm"] Nov 03 19:49:46.442609 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.442595 5059 cpu_manager.go:414] "containerMap: removing container" podUID="6a20d160-602b-4513-94a1-159916a46cb8" containerName="revoke-keys" Nov 03 19:49:46.442692 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.442611 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a20d160-602b-4513-94a1-159916a46cb8" containerName="revoke-keys" Nov 03 19:49:46.442783 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.442771 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a20d160-602b-4513-94a1-159916a46cb8" containerName="revoke-keys" Nov 03 19:49:46.446249 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.446232 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" Nov 03 19:49:46.453298 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.453239 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b46da6fce236\"" Nov 03 19:49:46.454270 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.454248 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 19:49:46.454419 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.454396 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 19:49:46.454843 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.454824 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 19:49:46.455022 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.454996 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 19:49:46.455125 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.455110 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-bs8pz\"" Nov 03 19:49:46.455182 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.455148 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ee59ddb9f8a1\"" Nov 03 19:49:46.456715 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.456694 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b644db666-cldhm"] Nov 03 19:49:46.567263 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.567157 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0419a6-8280-42c3-be8b-12c18824688e-plugins-config-volume\") pod \"payload-processing-external-model-6b644db666-cldhm\" (UID: \"4a0419a6-8280-42c3-be8b-12c18824688e\") " pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" Nov 03 19:49:46.567452 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.567290 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpxhl\" (UniqueName: \"kubernetes.io/projected/4a0419a6-8280-42c3-be8b-12c18824688e-kube-api-access-xpxhl\") pod \"payload-processing-external-model-6b644db666-cldhm\" (UID: \"4a0419a6-8280-42c3-be8b-12c18824688e\") " pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" Nov 03 19:49:46.567452 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.567389 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a0419a6-8280-42c3-be8b-12c18824688e-routing-overlay-volume\") pod \"payload-processing-external-model-6b644db666-cldhm\" (UID: \"4a0419a6-8280-42c3-be8b-12c18824688e\") " pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" Nov 03 19:49:46.567452 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.567440 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a0419a6-8280-42c3-be8b-12c18824688e-provider-credentials\") pod \"payload-processing-external-model-6b644db666-cldhm\" (UID: \"4a0419a6-8280-42c3-be8b-12c18824688e\") " pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" Nov 03 19:49:46.668575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.668454 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a0419a6-8280-42c3-be8b-12c18824688e-routing-overlay-volume\") pod \"payload-processing-external-model-6b644db666-cldhm\" (UID: \"4a0419a6-8280-42c3-be8b-12c18824688e\") " pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" Nov 03 19:49:46.668575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.668545 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a0419a6-8280-42c3-be8b-12c18824688e-provider-credentials\") pod \"payload-processing-external-model-6b644db666-cldhm\" (UID: \"4a0419a6-8280-42c3-be8b-12c18824688e\") " pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" Nov 03 19:49:46.668575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.668581 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0419a6-8280-42c3-be8b-12c18824688e-plugins-config-volume\") pod \"payload-processing-external-model-6b644db666-cldhm\" (UID: \"4a0419a6-8280-42c3-be8b-12c18824688e\") " pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" Nov 03 19:49:46.668939 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.668650 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpxhl\" (UniqueName: \"kubernetes.io/projected/4a0419a6-8280-42c3-be8b-12c18824688e-kube-api-access-xpxhl\") pod \"payload-processing-external-model-6b644db666-cldhm\" (UID: \"4a0419a6-8280-42c3-be8b-12c18824688e\") " pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" Nov 03 19:49:46.669613 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.669574 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a0419a6-8280-42c3-be8b-12c18824688e-routing-overlay-volume\") pod \"payload-processing-external-model-6b644db666-cldhm\" (UID: \"4a0419a6-8280-42c3-be8b-12c18824688e\") " pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" Nov 03 19:49:46.669978 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.669950 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0419a6-8280-42c3-be8b-12c18824688e-plugins-config-volume\") pod \"payload-processing-external-model-6b644db666-cldhm\" (UID: \"4a0419a6-8280-42c3-be8b-12c18824688e\") " pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" Nov 03 19:49:46.671544 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.671520 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a0419a6-8280-42c3-be8b-12c18824688e-provider-credentials\") pod \"payload-processing-external-model-6b644db666-cldhm\" (UID: \"4a0419a6-8280-42c3-be8b-12c18824688e\") " pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" Nov 03 19:49:46.681134 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.681108 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpxhl\" (UniqueName: \"kubernetes.io/projected/4a0419a6-8280-42c3-be8b-12c18824688e-kube-api-access-xpxhl\") pod \"payload-processing-external-model-6b644db666-cldhm\" (UID: \"4a0419a6-8280-42c3-be8b-12c18824688e\") " pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" Nov 03 19:49:46.768547 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:46.768519 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" Nov 03 19:49:47.062215 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:49:47.062152 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0419a6_8280_42c3_be8b_12c18824688e.slice/crio-fa556354ef374468b57a8ce3ff62f535dbf4c7af5977f55453ab9add162f34c9 WatchSource:0}: Error finding container fa556354ef374468b57a8ce3ff62f535dbf4c7af5977f55453ab9add162f34c9: Status 404 returned error can't find the container with id fa556354ef374468b57a8ce3ff62f535dbf4c7af5977f55453ab9add162f34c9 Nov 03 19:49:47.063365 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:47.062938 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b644db666-cldhm"] Nov 03 19:49:47.322861 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:47.322811 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" event={"ID":"4a0419a6-8280-42c3-be8b-12c18824688e","Type":"ContainerStarted","Data":"d6cb71bb0883731596fa89d9915fca7ef38cecc343f69a539b42bebfbf8a7ad3"} Nov 03 19:49:47.323061 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:47.322880 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" event={"ID":"4a0419a6-8280-42c3-be8b-12c18824688e","Type":"ContainerStarted","Data":"fa556354ef374468b57a8ce3ff62f535dbf4c7af5977f55453ab9add162f34c9"} Nov 03 19:49:47.323061 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:47.322973 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" Nov 03 19:49:47.347551 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:47.347444 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" podStartSLOduration=1.34742731 podStartE2EDuration="1.34742731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:47.346902118 +0000 UTC m=+781.604699376" watchObservedRunningTime="2026-11-03 19:49:47.34742731 +0000 UTC m=+781.605224572" Nov 03 19:49:56.120253 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:56.120219 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6991b942-jnrzn"] Nov 03 19:49:56.125234 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:56.125217 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6991b942-jnrzn" Nov 03 19:49:56.130647 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:56.130622 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:49:56.130875 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:56.130731 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 19:49:56.132598 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:56.132577 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6991b942-jnrzn"] Nov 03 19:49:56.207972 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:56.207937 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzjjn\" (UniqueName: \"kubernetes.io/projected/ff5d8a5e-3638-4b29-a49c-3ef488b372f5-kube-api-access-dzjjn\") pod \"maas-api-revoke-sub-6991b942-jnrzn\" (UID: \"ff5d8a5e-3638-4b29-a49c-3ef488b372f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6991b942-jnrzn" Nov 03 19:49:56.208168 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:56.208004 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff5d8a5e-3638-4b29-a49c-3ef488b372f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6991b942-jnrzn\" (UID: \"ff5d8a5e-3638-4b29-a49c-3ef488b372f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6991b942-jnrzn" Nov 03 19:49:56.208168 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:56.208066 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff5d8a5e-3638-4b29-a49c-3ef488b372f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6991b942-jnrzn\" (UID: \"ff5d8a5e-3638-4b29-a49c-3ef488b372f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6991b942-jnrzn" Nov 03 19:49:56.315523 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:56.314380 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff5d8a5e-3638-4b29-a49c-3ef488b372f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6991b942-jnrzn\" (UID: \"ff5d8a5e-3638-4b29-a49c-3ef488b372f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6991b942-jnrzn" Nov 03 19:49:56.315523 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:56.314437 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff5d8a5e-3638-4b29-a49c-3ef488b372f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6991b942-jnrzn\" (UID: \"ff5d8a5e-3638-4b29-a49c-3ef488b372f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6991b942-jnrzn" Nov 03 19:49:56.315523 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:56.314740 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzjjn\" (UniqueName: \"kubernetes.io/projected/ff5d8a5e-3638-4b29-a49c-3ef488b372f5-kube-api-access-dzjjn\") pod \"maas-api-revoke-sub-6991b942-jnrzn\" (UID: \"ff5d8a5e-3638-4b29-a49c-3ef488b372f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6991b942-jnrzn" Nov 03 19:49:56.315876 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:56.315839 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff5d8a5e-3638-4b29-a49c-3ef488b372f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6991b942-jnrzn\" (UID: \"ff5d8a5e-3638-4b29-a49c-3ef488b372f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6991b942-jnrzn" Nov 03 19:49:56.325643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:56.325613 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff5d8a5e-3638-4b29-a49c-3ef488b372f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6991b942-jnrzn\" (UID: \"ff5d8a5e-3638-4b29-a49c-3ef488b372f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6991b942-jnrzn" Nov 03 19:49:56.328321 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:56.328292 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzjjn\" (UniqueName: \"kubernetes.io/projected/ff5d8a5e-3638-4b29-a49c-3ef488b372f5-kube-api-access-dzjjn\") pod \"maas-api-revoke-sub-6991b942-jnrzn\" (UID: \"ff5d8a5e-3638-4b29-a49c-3ef488b372f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6991b942-jnrzn" Nov 03 19:49:56.443201 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:56.443111 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6991b942-jnrzn" Nov 03 19:49:56.727918 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:56.727892 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6991b942-jnrzn"] Nov 03 19:49:56.729705 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:49:56.729638 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5d8a5e_3638_4b29_a49c_3ef488b372f5.slice/crio-16103de4d717ff3619a302d94f1be8ae4e08b01f22787dcf162833f66b7958c9 WatchSource:0}: Error finding container 16103de4d717ff3619a302d94f1be8ae4e08b01f22787dcf162833f66b7958c9: Status 404 returned error can't find the container with id 16103de4d717ff3619a302d94f1be8ae4e08b01f22787dcf162833f66b7958c9 Nov 03 19:49:57.387177 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:57.387122 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6991b942-jnrzn" event={"ID":"ff5d8a5e-3638-4b29-a49c-3ef488b372f5","Type":"ContainerStarted","Data":"37bf1ae52160ffa8a5696735fe05152485fb49ba63fbc5baad03b3b596af52f2"} Nov 03 19:49:57.387695 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:57.387231 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6991b942-jnrzn" event={"ID":"ff5d8a5e-3638-4b29-a49c-3ef488b372f5","Type":"ContainerStarted","Data":"16103de4d717ff3619a302d94f1be8ae4e08b01f22787dcf162833f66b7958c9"} Nov 03 19:49:57.408749 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:57.408687 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6991b942-jnrzn" podStartSLOduration=1.408651371 podStartE2EDuration="1.408651371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:57.406155559 +0000 UTC m=+791.663952828" watchObservedRunningTime="2026-11-03 19:49:57.408651371 +0000 UTC m=+791.666448628" Nov 03 19:49:58.333536 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:58.333501 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" Nov 03 19:49:58.435139 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:58.435105 5059 generic.go:356] "Generic (PLEG): container finished" podID="ff5d8a5e-3638-4b29-a49c-3ef488b372f5" containerID="37bf1ae52160ffa8a5696735fe05152485fb49ba63fbc5baad03b3b596af52f2" exitCode=0 Nov 03 19:49:58.435620 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:58.435179 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6991b942-jnrzn" event={"ID":"ff5d8a5e-3638-4b29-a49c-3ef488b372f5","Type":"ContainerDied","Data":"37bf1ae52160ffa8a5696735fe05152485fb49ba63fbc5baad03b3b596af52f2"} Nov 03 19:49:59.623431 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.623407 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6991b942-jnrzn" Nov 03 19:49:59.638490 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.638455 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w"] Nov 03 19:49:59.639498 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.639292 5059 cpu_manager.go:414] "containerMap: removing container" podUID="ff5d8a5e-3638-4b29-a49c-3ef488b372f5" containerName="revoke-keys" Nov 03 19:49:59.639498 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.639316 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff5d8a5e-3638-4b29-a49c-3ef488b372f5" containerName="revoke-keys" Nov 03 19:49:59.639498 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.639472 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff5d8a5e-3638-4b29-a49c-3ef488b372f5" containerName="revoke-keys" Nov 03 19:49:59.643303 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.643282 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.651624 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.651601 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff5d8a5e-3638-4b29-a49c-3ef488b372f5-kube-api-access-dzjjn\" (UniqueName: \"kubernetes.io/projected/ff5d8a5e-3638-4b29-a49c-3ef488b372f5-kube-api-access-dzjjn\") pod \"ff5d8a5e-3638-4b29-a49c-3ef488b372f5\" (UID: \"ff5d8a5e-3638-4b29-a49c-3ef488b372f5\") " Nov 03 19:49:59.651756 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.651740 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff5d8a5e-3638-4b29-a49c-3ef488b372f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff5d8a5e-3638-4b29-a49c-3ef488b372f5-maas-api-xks-ca\") pod \"ff5d8a5e-3638-4b29-a49c-3ef488b372f5\" (UID: \"ff5d8a5e-3638-4b29-a49c-3ef488b372f5\") " Nov 03 19:49:59.651816 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.651782 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff5d8a5e-3638-4b29-a49c-3ef488b372f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff5d8a5e-3638-4b29-a49c-3ef488b372f5-maas-api-openshift-ca\") pod \"ff5d8a5e-3638-4b29-a49c-3ef488b372f5\" (UID: \"ff5d8a5e-3638-4b29-a49c-3ef488b372f5\") " Nov 03 19:49:59.652530 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.652501 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff5d8a5e-3638-4b29-a49c-3ef488b372f5-maas-api-openshift-ca" pod "ff5d8a5e-3638-4b29-a49c-3ef488b372f5" (UID: "ff5d8a5e-3638-4b29-a49c-3ef488b372f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:49:59.672026 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.660795 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff5d8a5e-3638-4b29-a49c-3ef488b372f5-kube-api-access-dzjjn" pod "ff5d8a5e-3638-4b29-a49c-3ef488b372f5" (UID: "ff5d8a5e-3638-4b29-a49c-3ef488b372f5"). InnerVolumeSpecName "kube-api-access-dzjjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:49:59.672026 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.662820 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff5d8a5e-3638-4b29-a49c-3ef488b372f5-maas-api-xks-ca" pod "ff5d8a5e-3638-4b29-a49c-3ef488b372f5" (UID: "ff5d8a5e-3638-4b29-a49c-3ef488b372f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:49:59.679073 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.679043 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w"] Nov 03 19:49:59.753604 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.753526 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.753604 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.753562 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.753604 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.753581 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.753834 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.753638 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.753834 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.753710 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.753834 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.753805 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.753963 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.753840 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.753963 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.753861 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bdv6\" (UniqueName: \"kubernetes.io/projected/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-kube-api-access-6bdv6\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.754069 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.753983 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.754168 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.754151 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff5d8a5e-3638-4b29-a49c-3ef488b372f5-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:59.754230 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.754174 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff5d8a5e-3638-4b29-a49c-3ef488b372f5-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:59.754230 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.754193 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzjjn\" (UniqueName: \"kubernetes.io/projected/ff5d8a5e-3638-4b29-a49c-3ef488b372f5-kube-api-access-dzjjn\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:49:59.854775 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.854748 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.854955 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.854791 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.854955 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.854806 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bdv6\" (UniqueName: \"kubernetes.io/projected/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-kube-api-access-6bdv6\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.854955 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.854841 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.854955 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.854941 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.855196 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.854973 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.855196 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.855002 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.855196 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.855027 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.855196 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.855056 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.855396 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.855205 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.855396 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.855270 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.855396 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.855346 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.855520 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.855432 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.855814 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.855791 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.857396 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.857368 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.857512 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.857496 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.868304 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.868281 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.873713 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.873695 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bdv6\" (UniqueName: \"kubernetes.io/projected/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-kube-api-access-6bdv6\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:49:59.981309 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:49:59.981273 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:50:00.268977 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:00.268950 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w"] Nov 03 19:50:00.270407 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:50:00.270371 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef4417c_3d04_4d65_a5bc_6ec1af6f09bb.slice/crio-701216e74fc88586037ac5f2aabf13be12a48852e8dc32da7bea458c18dba20f WatchSource:0}: Error finding container 701216e74fc88586037ac5f2aabf13be12a48852e8dc32da7bea458c18dba20f: Status 404 returned error can't find the container with id 701216e74fc88586037ac5f2aabf13be12a48852e8dc32da7bea458c18dba20f Nov 03 19:50:00.273367 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:00.273315 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:50:00.273478 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:00.273431 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:50:00.273540 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:00.273479 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:50:00.455265 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:00.455238 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6991b942-jnrzn" Nov 03 19:50:00.455265 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:00.455243 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6991b942-jnrzn" event={"ID":"ff5d8a5e-3638-4b29-a49c-3ef488b372f5","Type":"ContainerDied","Data":"16103de4d717ff3619a302d94f1be8ae4e08b01f22787dcf162833f66b7958c9"} Nov 03 19:50:00.455477 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:00.455295 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16103de4d717ff3619a302d94f1be8ae4e08b01f22787dcf162833f66b7958c9" Nov 03 19:50:00.457167 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:00.457143 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" event={"ID":"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb","Type":"ContainerStarted","Data":"c56ee037a40c322b89e375002cf24e08b31d3e1b30e1b41b0f4819b338d40ec8"} Nov 03 19:50:00.457325 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:00.457188 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" event={"ID":"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb","Type":"ContainerStarted","Data":"701216e74fc88586037ac5f2aabf13be12a48852e8dc32da7bea458c18dba20f"} Nov 03 19:50:00.484393 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:00.484351 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" podStartSLOduration=1.484338962 podStartE2EDuration="1.484338962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:00.48111882 +0000 UTC m=+794.738916077" watchObservedRunningTime="2026-11-03 19:50:00.484338962 +0000 UTC m=+794.742136219" Nov 03 19:50:00.982850 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:00.982812 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:50:01.984619 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:01.984580 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 19:50:01.985033 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:01.984647 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" podUID="cef4417c-3d04-4d65-a5bc-6ec1af6f09bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 19:50:02.982685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:02.981992 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 19:50:02.982685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:02.982066 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" podUID="cef4417c-3d04-4d65-a5bc-6ec1af6f09bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 19:50:03.985096 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:03.985044 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 19:50:03.985617 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:03.985146 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" podUID="cef4417c-3d04-4d65-a5bc-6ec1af6f09bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 19:50:04.983058 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:04.983013 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 19:50:04.983236 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:04.983087 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" podUID="cef4417c-3d04-4d65-a5bc-6ec1af6f09bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 19:50:05.983073 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:05.983034 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 19:50:05.983571 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:05.983103 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" podUID="cef4417c-3d04-4d65-a5bc-6ec1af6f09bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 19:50:06.347569 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:06.347537 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c158645-v77cp"] Nov 03 19:50:06.352972 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:06.352953 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c158645-v77cp" Nov 03 19:50:06.359310 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:06.359286 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:50:06.360457 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:06.360436 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 19:50:06.365957 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:06.365928 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c158645-v77cp"] Nov 03 19:50:06.446606 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:06.446567 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rdsj\" (UniqueName: \"kubernetes.io/projected/2d7e0838-dcc9-4758-8490-79e1a1d86c24-kube-api-access-5rdsj\") pod \"maas-api-revoke-sub-2c158645-v77cp\" (UID: \"2d7e0838-dcc9-4758-8490-79e1a1d86c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c158645-v77cp" Nov 03 19:50:06.446764 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:06.446628 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d7e0838-dcc9-4758-8490-79e1a1d86c24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c158645-v77cp\" (UID: \"2d7e0838-dcc9-4758-8490-79e1a1d86c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c158645-v77cp" Nov 03 19:50:06.446822 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:06.446805 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d7e0838-dcc9-4758-8490-79e1a1d86c24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c158645-v77cp\" (UID: \"2d7e0838-dcc9-4758-8490-79e1a1d86c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c158645-v77cp" Nov 03 19:50:06.547498 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:06.547458 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d7e0838-dcc9-4758-8490-79e1a1d86c24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c158645-v77cp\" (UID: \"2d7e0838-dcc9-4758-8490-79e1a1d86c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c158645-v77cp" Nov 03 19:50:06.547776 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:06.547520 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rdsj\" (UniqueName: \"kubernetes.io/projected/2d7e0838-dcc9-4758-8490-79e1a1d86c24-kube-api-access-5rdsj\") pod \"maas-api-revoke-sub-2c158645-v77cp\" (UID: \"2d7e0838-dcc9-4758-8490-79e1a1d86c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c158645-v77cp" Nov 03 19:50:06.547776 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:06.547573 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d7e0838-dcc9-4758-8490-79e1a1d86c24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c158645-v77cp\" (UID: \"2d7e0838-dcc9-4758-8490-79e1a1d86c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c158645-v77cp" Nov 03 19:50:06.548420 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:06.548388 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d7e0838-dcc9-4758-8490-79e1a1d86c24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c158645-v77cp\" (UID: \"2d7e0838-dcc9-4758-8490-79e1a1d86c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c158645-v77cp" Nov 03 19:50:06.550384 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:06.550351 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d7e0838-dcc9-4758-8490-79e1a1d86c24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c158645-v77cp\" (UID: \"2d7e0838-dcc9-4758-8490-79e1a1d86c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c158645-v77cp" Nov 03 19:50:06.560128 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:06.560104 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rdsj\" (UniqueName: \"kubernetes.io/projected/2d7e0838-dcc9-4758-8490-79e1a1d86c24-kube-api-access-5rdsj\") pod \"maas-api-revoke-sub-2c158645-v77cp\" (UID: \"2d7e0838-dcc9-4758-8490-79e1a1d86c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c158645-v77cp" Nov 03 19:50:06.673524 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:06.673432 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c158645-v77cp" Nov 03 19:50:07.010025 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:07.009938 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:50:07.010519 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:07.010331 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:50:07.011407 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:07.011384 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 19:50:07.074017 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:07.073991 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c158645-v77cp"] Nov 03 19:50:07.074885 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:50:07.074860 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7e0838_dcc9_4758_8490_79e1a1d86c24.slice/crio-0ad3924dc7fabb2179337b43c342464042991298df9b97e66834db4ef76b259a WatchSource:0}: Error finding container 0ad3924dc7fabb2179337b43c342464042991298df9b97e66834db4ef76b259a: Status 404 returned error can't find the container with id 0ad3924dc7fabb2179337b43c342464042991298df9b97e66834db4ef76b259a Nov 03 19:50:07.537801 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:07.537694 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c158645-v77cp" event={"ID":"2d7e0838-dcc9-4758-8490-79e1a1d86c24","Type":"ContainerStarted","Data":"6987f41867483c4fdb7b5390e026fd3dd85ada0efb09df08d7fb089b25952eb2"} Nov 03 19:50:07.537801 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:07.537768 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c158645-v77cp" event={"ID":"2d7e0838-dcc9-4758-8490-79e1a1d86c24","Type":"ContainerStarted","Data":"0ad3924dc7fabb2179337b43c342464042991298df9b97e66834db4ef76b259a"} Nov 03 19:50:08.546410 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:08.546365 5059 generic.go:356] "Generic (PLEG): container finished" podID="2d7e0838-dcc9-4758-8490-79e1a1d86c24" containerID="6987f41867483c4fdb7b5390e026fd3dd85ada0efb09df08d7fb089b25952eb2" exitCode=0 Nov 03 19:50:08.546955 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:08.546444 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c158645-v77cp" event={"ID":"2d7e0838-dcc9-4758-8490-79e1a1d86c24","Type":"ContainerDied","Data":"6987f41867483c4fdb7b5390e026fd3dd85ada0efb09df08d7fb089b25952eb2"} Nov 03 19:50:09.297682 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.297211 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk"] Nov 03 19:50:09.303701 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.303675 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.329390 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.329354 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk"] Nov 03 19:50:09.406191 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.406158 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4946e3aa-a961-415a-b71d-95164d5fbca3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.407119 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.407095 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.407332 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.407313 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wltn5\" (UniqueName: \"kubernetes.io/projected/4946e3aa-a961-415a-b71d-95164d5fbca3-kube-api-access-wltn5\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.407501 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.407484 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.407824 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.407805 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.408182 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.408162 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.408360 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.408342 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.408528 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.408512 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.408710 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.408692 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.510309 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.510271 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.510504 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.510325 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wltn5\" (UniqueName: \"kubernetes.io/projected/4946e3aa-a961-415a-b71d-95164d5fbca3-kube-api-access-wltn5\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.510504 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.510360 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.510635 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.510516 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.510635 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.510570 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.510635 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.510605 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.510865 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.510646 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.510865 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.510697 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.510865 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.510734 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4946e3aa-a961-415a-b71d-95164d5fbca3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.512149 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.511693 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4946e3aa-a961-415a-b71d-95164d5fbca3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.512149 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.511754 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.512149 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.511978 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.512386 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.512232 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.513010 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.512969 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.515851 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.515805 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.516466 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.516421 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.530499 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.530478 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.531058 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.531037 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wltn5\" (UniqueName: \"kubernetes.io/projected/4946e3aa-a961-415a-b71d-95164d5fbca3-kube-api-access-wltn5\") pod \"maas-default-gateway-openshift-default-855f9574cf-7krsk\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.627889 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.627853 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:09.767036 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.766956 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c158645-v77cp" Nov 03 19:50:09.835199 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.835163 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d7e0838-dcc9-4758-8490-79e1a1d86c24-kube-api-access-5rdsj\" (UniqueName: \"kubernetes.io/projected/2d7e0838-dcc9-4758-8490-79e1a1d86c24-kube-api-access-5rdsj\") pod \"2d7e0838-dcc9-4758-8490-79e1a1d86c24\" (UID: \"2d7e0838-dcc9-4758-8490-79e1a1d86c24\") " Nov 03 19:50:09.835901 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.835878 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d7e0838-dcc9-4758-8490-79e1a1d86c24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d7e0838-dcc9-4758-8490-79e1a1d86c24-maas-api-openshift-ca\") pod \"2d7e0838-dcc9-4758-8490-79e1a1d86c24\" (UID: \"2d7e0838-dcc9-4758-8490-79e1a1d86c24\") " Nov 03 19:50:09.836303 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.836273 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d7e0838-dcc9-4758-8490-79e1a1d86c24-maas-api-openshift-ca" pod "2d7e0838-dcc9-4758-8490-79e1a1d86c24" (UID: "2d7e0838-dcc9-4758-8490-79e1a1d86c24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:50:09.836426 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.836405 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d7e0838-dcc9-4758-8490-79e1a1d86c24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d7e0838-dcc9-4758-8490-79e1a1d86c24-maas-api-xks-ca\") pod \"2d7e0838-dcc9-4758-8490-79e1a1d86c24\" (UID: \"2d7e0838-dcc9-4758-8490-79e1a1d86c24\") " Nov 03 19:50:09.837518 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.837495 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d7e0838-dcc9-4758-8490-79e1a1d86c24-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:09.837955 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.837940 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7e0838-dcc9-4758-8490-79e1a1d86c24-kube-api-access-5rdsj" pod "2d7e0838-dcc9-4758-8490-79e1a1d86c24" (UID: "2d7e0838-dcc9-4758-8490-79e1a1d86c24"). InnerVolumeSpecName "kube-api-access-5rdsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:50:09.839275 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.839246 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d7e0838-dcc9-4758-8490-79e1a1d86c24-maas-api-xks-ca" pod "2d7e0838-dcc9-4758-8490-79e1a1d86c24" (UID: "2d7e0838-dcc9-4758-8490-79e1a1d86c24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:50:09.939187 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.939157 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rdsj\" (UniqueName: \"kubernetes.io/projected/2d7e0838-dcc9-4758-8490-79e1a1d86c24-kube-api-access-5rdsj\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:09.939187 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.939184 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d7e0838-dcc9-4758-8490-79e1a1d86c24-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:09.988156 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.987073 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk"] Nov 03 19:50:09.990283 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:50:09.990239 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4946e3aa_a961_415a_b71d_95164d5fbca3.slice/crio-3ced3533cde36871565d3f5b342e68e1183b5249dab2a5f6aecb26a3d24f2a80 WatchSource:0}: Error finding container 3ced3533cde36871565d3f5b342e68e1183b5249dab2a5f6aecb26a3d24f2a80: Status 404 returned error can't find the container with id 3ced3533cde36871565d3f5b342e68e1183b5249dab2a5f6aecb26a3d24f2a80 Nov 03 19:50:09.993060 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.993027 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:50:09.993158 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.993101 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:50:09.993158 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:09.993142 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:50:10.564034 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:10.563979 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" event={"ID":"4946e3aa-a961-415a-b71d-95164d5fbca3","Type":"ContainerStarted","Data":"2ce7833679c135af351a934eeaaf542087c31098688f1a7b779e25f9a2bf4bc1"} Nov 03 19:50:10.564209 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:10.564082 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" event={"ID":"4946e3aa-a961-415a-b71d-95164d5fbca3","Type":"ContainerStarted","Data":"3ced3533cde36871565d3f5b342e68e1183b5249dab2a5f6aecb26a3d24f2a80"} Nov 03 19:50:10.566749 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:10.566713 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c158645-v77cp" event={"ID":"2d7e0838-dcc9-4758-8490-79e1a1d86c24","Type":"ContainerDied","Data":"0ad3924dc7fabb2179337b43c342464042991298df9b97e66834db4ef76b259a"} Nov 03 19:50:10.566987 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:10.566947 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ad3924dc7fabb2179337b43c342464042991298df9b97e66834db4ef76b259a" Nov 03 19:50:10.567099 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:10.566732 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c158645-v77cp" Nov 03 19:50:10.628541 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:10.628500 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:10.650374 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:50:10.649690 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:50:10.813804 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:10.813693 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" podStartSLOduration=1.813649157 podStartE2EDuration="1.813649157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:10.598534216 +0000 UTC m=+804.856331476" watchObservedRunningTime="2026-11-03 19:50:10.813649157 +0000 UTC m=+805.071446413" Nov 03 19:50:11.630133 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:11.630098 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7krsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 19:50:11.630530 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:11.630154 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" podUID="4946e3aa-a961-415a-b71d-95164d5fbca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 19:50:12.629431 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:12.629392 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7krsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 19:50:12.629631 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:12.629447 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" podUID="4946e3aa-a961-415a-b71d-95164d5fbca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 19:50:13.289013 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:13.288972 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b644db666-cldhm"] Nov 03 19:50:13.629421 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:13.629391 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7krsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 19:50:13.629589 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:13.629439 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" podUID="4946e3aa-a961-415a-b71d-95164d5fbca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 19:50:13.914984 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:13.914904 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz"] Nov 03 19:50:13.915512 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:13.915492 5059 cpu_manager.go:414] "containerMap: removing container" podUID="2d7e0838-dcc9-4758-8490-79e1a1d86c24" containerName="revoke-keys" Nov 03 19:50:13.915612 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:13.915514 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d7e0838-dcc9-4758-8490-79e1a1d86c24" containerName="revoke-keys" Nov 03 19:50:13.915712 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:13.915698 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d7e0838-dcc9-4758-8490-79e1a1d86c24" containerName="revoke-keys" Nov 03 19:50:13.919386 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:13.919366 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:13.928913 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:13.928889 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"e2e-conflict-b-a0f38071-kserve-self-signed-certs\"" Nov 03 19:50:13.930376 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:13.930354 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz"] Nov 03 19:50:14.008552 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:14.008523 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-dshm\") pod \"e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz\" (UID: \"812d93af-9b5f-42e4-bb78-da7622071a16\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:14.008739 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:14.008576 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/812d93af-9b5f-42e4-bb78-da7622071a16-tls-certs\") pod \"e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz\" (UID: \"812d93af-9b5f-42e4-bb78-da7622071a16\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:14.008792 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:14.008748 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-tmp-dir\") pod \"e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz\" (UID: \"812d93af-9b5f-42e4-bb78-da7622071a16\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:14.008792 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:14.008784 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-home\") pod \"e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz\" (UID: \"812d93af-9b5f-42e4-bb78-da7622071a16\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:14.008902 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:14.008888 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-model-cache\") pod \"e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz\" (UID: \"812d93af-9b5f-42e4-bb78-da7622071a16\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:14.008943 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:14.008917 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l82fr\" (UniqueName: \"kubernetes.io/projected/812d93af-9b5f-42e4-bb78-da7622071a16-kube-api-access-l82fr\") pod \"e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz\" (UID: \"812d93af-9b5f-42e4-bb78-da7622071a16\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:14.111100 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:14.111069 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-tmp-dir\") pod \"e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz\" (UID: \"812d93af-9b5f-42e4-bb78-da7622071a16\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:14.111317 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:14.111305 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-home\") pod \"e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz\" (UID: \"812d93af-9b5f-42e4-bb78-da7622071a16\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:14.111428 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:14.111419 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-model-cache\") pod \"e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz\" (UID: \"812d93af-9b5f-42e4-bb78-da7622071a16\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:14.111501 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:14.111492 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l82fr\" (UniqueName: \"kubernetes.io/projected/812d93af-9b5f-42e4-bb78-da7622071a16-kube-api-access-l82fr\") pod \"e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz\" (UID: \"812d93af-9b5f-42e4-bb78-da7622071a16\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:14.111583 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:14.111575 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-dshm\") pod \"e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz\" (UID: \"812d93af-9b5f-42e4-bb78-da7622071a16\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:14.111690 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:14.111679 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/812d93af-9b5f-42e4-bb78-da7622071a16-tls-certs\") pod \"e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz\" (UID: \"812d93af-9b5f-42e4-bb78-da7622071a16\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:14.112485 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:14.112459 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-tmp-dir\") pod \"e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz\" (UID: \"812d93af-9b5f-42e4-bb78-da7622071a16\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:14.112726 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:14.112523 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-home\") pod \"e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz\" (UID: \"812d93af-9b5f-42e4-bb78-da7622071a16\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:14.113076 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:14.113060 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-model-cache\") pod \"e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz\" (UID: \"812d93af-9b5f-42e4-bb78-da7622071a16\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:14.114365 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:14.114349 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/812d93af-9b5f-42e4-bb78-da7622071a16-tls-certs\") pod \"e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz\" (UID: \"812d93af-9b5f-42e4-bb78-da7622071a16\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:14.115001 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:14.114977 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-dshm\") pod \"e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz\" (UID: \"812d93af-9b5f-42e4-bb78-da7622071a16\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:14.124623 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:14.124596 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l82fr\" (UniqueName: \"kubernetes.io/projected/812d93af-9b5f-42e4-bb78-da7622071a16-kube-api-access-l82fr\") pod \"e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz\" (UID: \"812d93af-9b5f-42e4-bb78-da7622071a16\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:14.234837 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:14.234756 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:14.543743 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:14.543707 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz"] Nov 03 19:50:14.544450 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:50:14.544415 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod812d93af_9b5f_42e4_bb78_da7622071a16.slice/crio-a9efad5101b64364a116212ad206639774db7d740170356c8716d589beb38b7a WatchSource:0}: Error finding container a9efad5101b64364a116212ad206639774db7d740170356c8716d589beb38b7a: Status 404 returned error can't find the container with id a9efad5101b64364a116212ad206639774db7d740170356c8716d589beb38b7a Nov 03 19:50:14.607454 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:14.607425 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" event={"ID":"812d93af-9b5f-42e4-bb78-da7622071a16","Type":"ContainerStarted","Data":"a9efad5101b64364a116212ad206639774db7d740170356c8716d589beb38b7a"} Nov 03 19:50:14.629005 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:14.628982 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7krsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 19:50:14.629091 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:14.629024 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" podUID="4946e3aa-a961-415a-b71d-95164d5fbca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 19:50:15.618176 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:15.618116 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" event={"ID":"812d93af-9b5f-42e4-bb78-da7622071a16","Type":"ContainerStarted","Data":"1e31e24e213896e89ffc43948dfb753e88d54e9365a739ab53e0c5c95bb6498d"} Nov 03 19:50:15.629201 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:15.629173 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7krsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 19:50:15.629332 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:15.629218 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" podUID="4946e3aa-a961-415a-b71d-95164d5fbca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 19:50:15.640928 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:15.640882 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" podStartSLOduration=2.6408683440000003 podStartE2EDuration="2.640868344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:15.63893519 +0000 UTC m=+809.896732447" watchObservedRunningTime="2026-11-03 19:50:15.640868344 +0000 UTC m=+809.898665601" Nov 03 19:50:16.235092 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:16.235060 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:16.628370 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:16.628337 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7krsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 19:50:16.628917 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:16.628382 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" podUID="4946e3aa-a961-415a-b71d-95164d5fbca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 19:50:17.629472 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:17.629438 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7krsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 19:50:17.629877 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:17.629494 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" podUID="4946e3aa-a961-415a-b71d-95164d5fbca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 19:50:18.289535 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.289444 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" podUID="4a0419a6-8280-42c3-be8b-12c18824688e" containerName="payload-processing" containerID="cri-o://d6cb71bb0883731596fa89d9915fca7ef38cecc343f69a539b42bebfbf8a7ad3" gracePeriod=25 Nov 03 19:50:18.330823 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:50:18.330803 5059 logging.go:55] [core] [Channel #362 SubChannel #363]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.87:50052", ServerName: "10.132.0.87:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.87:50052: connect: connection refused" Nov 03 19:50:18.577600 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.577576 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" Nov 03 19:50:18.644236 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.644200 5059 generic.go:356] "Generic (PLEG): container finished" podID="4a0419a6-8280-42c3-be8b-12c18824688e" containerID="d6cb71bb0883731596fa89d9915fca7ef38cecc343f69a539b42bebfbf8a7ad3" exitCode=0 Nov 03 19:50:18.644652 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.644249 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" Nov 03 19:50:18.644652 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.644244 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" event={"ID":"4a0419a6-8280-42c3-be8b-12c18824688e","Type":"ContainerDied","Data":"d6cb71bb0883731596fa89d9915fca7ef38cecc343f69a539b42bebfbf8a7ad3"} Nov 03 19:50:18.644652 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.644461 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" event={"ID":"4a0419a6-8280-42c3-be8b-12c18824688e","Type":"ContainerDied","Data":"fa556354ef374468b57a8ce3ff62f535dbf4c7af5977f55453ab9add162f34c9"} Nov 03 19:50:18.644652 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.644495 5059 scope.go:122] "RemoveContainer" containerID="d6cb71bb0883731596fa89d9915fca7ef38cecc343f69a539b42bebfbf8a7ad3" Nov 03 19:50:18.659068 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.659033 5059 scope.go:122] "RemoveContainer" containerID="d6cb71bb0883731596fa89d9915fca7ef38cecc343f69a539b42bebfbf8a7ad3" Nov 03 19:50:18.659381 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:50:18.659355 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6cb71bb0883731596fa89d9915fca7ef38cecc343f69a539b42bebfbf8a7ad3\": container with ID starting with d6cb71bb0883731596fa89d9915fca7ef38cecc343f69a539b42bebfbf8a7ad3 not found: ID does not exist" containerID="d6cb71bb0883731596fa89d9915fca7ef38cecc343f69a539b42bebfbf8a7ad3" Nov 03 19:50:18.659479 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.659390 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6cb71bb0883731596fa89d9915fca7ef38cecc343f69a539b42bebfbf8a7ad3"} err="failed to get container status \"d6cb71bb0883731596fa89d9915fca7ef38cecc343f69a539b42bebfbf8a7ad3\": rpc error: code = NotFound desc = could not find container \"d6cb71bb0883731596fa89d9915fca7ef38cecc343f69a539b42bebfbf8a7ad3\": container with ID starting with d6cb71bb0883731596fa89d9915fca7ef38cecc343f69a539b42bebfbf8a7ad3 not found: ID does not exist" Nov 03 19:50:18.670822 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.670790 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a0419a6-8280-42c3-be8b-12c18824688e-kube-api-access-xpxhl\" (UniqueName: \"kubernetes.io/projected/4a0419a6-8280-42c3-be8b-12c18824688e-kube-api-access-xpxhl\") pod \"4a0419a6-8280-42c3-be8b-12c18824688e\" (UID: \"4a0419a6-8280-42c3-be8b-12c18824688e\") " Nov 03 19:50:18.670973 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.670947 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a0419a6-8280-42c3-be8b-12c18824688e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0419a6-8280-42c3-be8b-12c18824688e-plugins-config-volume\") pod \"4a0419a6-8280-42c3-be8b-12c18824688e\" (UID: \"4a0419a6-8280-42c3-be8b-12c18824688e\") " Nov 03 19:50:18.671090 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.671022 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a0419a6-8280-42c3-be8b-12c18824688e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a0419a6-8280-42c3-be8b-12c18824688e-routing-overlay-volume\") pod \"4a0419a6-8280-42c3-be8b-12c18824688e\" (UID: \"4a0419a6-8280-42c3-be8b-12c18824688e\") " Nov 03 19:50:18.671090 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.671074 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a0419a6-8280-42c3-be8b-12c18824688e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a0419a6-8280-42c3-be8b-12c18824688e-provider-credentials\") pod \"4a0419a6-8280-42c3-be8b-12c18824688e\" (UID: \"4a0419a6-8280-42c3-be8b-12c18824688e\") " Nov 03 19:50:18.672179 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.672151 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a0419a6-8280-42c3-be8b-12c18824688e-routing-overlay-volume" pod "4a0419a6-8280-42c3-be8b-12c18824688e" (UID: "4a0419a6-8280-42c3-be8b-12c18824688e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:50:18.672281 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.672138 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a0419a6-8280-42c3-be8b-12c18824688e-plugins-config-volume" pod "4a0419a6-8280-42c3-be8b-12c18824688e" (UID: "4a0419a6-8280-42c3-be8b-12c18824688e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:50:18.694759 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.692859 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0419a6-8280-42c3-be8b-12c18824688e-kube-api-access-xpxhl" pod "4a0419a6-8280-42c3-be8b-12c18824688e" (UID: "4a0419a6-8280-42c3-be8b-12c18824688e"). InnerVolumeSpecName "kube-api-access-xpxhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:50:18.694759 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.692902 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0419a6-8280-42c3-be8b-12c18824688e-provider-credentials" pod "4a0419a6-8280-42c3-be8b-12c18824688e" (UID: "4a0419a6-8280-42c3-be8b-12c18824688e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:50:18.778804 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.775519 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0419a6-8280-42c3-be8b-12c18824688e-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:18.778804 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.775559 5059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a0419a6-8280-42c3-be8b-12c18824688e-routing-overlay-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:18.778804 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.775576 5059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a0419a6-8280-42c3-be8b-12c18824688e-provider-credentials\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:18.778804 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.775593 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpxhl\" (UniqueName: \"kubernetes.io/projected/4a0419a6-8280-42c3-be8b-12c18824688e-kube-api-access-xpxhl\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:18.778804 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.775599 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:18.778804 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.776412 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:18.781710 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.781169 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 19:50:18.972989 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.972162 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b644db666-cldhm"] Nov 03 19:50:18.976860 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:18.976829 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b644db666-cldhm"] Nov 03 19:50:19.268307 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.268161 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh"] Nov 03 19:50:19.268983 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.268953 5059 cpu_manager.go:414] "containerMap: removing container" podUID="4a0419a6-8280-42c3-be8b-12c18824688e" containerName="payload-processing" Nov 03 19:50:19.268983 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.268982 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a0419a6-8280-42c3-be8b-12c18824688e" containerName="payload-processing" Nov 03 19:50:19.269183 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.269171 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a0419a6-8280-42c3-be8b-12c18824688e" containerName="payload-processing" Nov 03 19:50:19.274400 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.274374 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.283613 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.283488 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh"] Nov 03 19:50:19.331461 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.331423 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6b644db666-cldhm" podUID="4a0419a6-8280-42c3-be8b-12c18824688e" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.87:50052\" within 1s: context deadline exceeded" Nov 03 19:50:19.389213 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.389178 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmg2l\" (UniqueName: \"kubernetes.io/projected/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-kube-api-access-cmg2l\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.389398 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.389296 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.389398 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.389360 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.389398 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.389390 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.389577 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.389426 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.389577 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.389550 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.389701 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.389608 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.389701 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.389655 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.389794 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.389701 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.491274 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.491236 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.491274 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.491277 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.491509 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.491346 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmg2l\" (UniqueName: \"kubernetes.io/projected/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-kube-api-access-cmg2l\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.491509 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.491421 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.491509 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.491452 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.491509 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.491483 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.491767 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.491525 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.491767 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.491613 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.491767 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.491614 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.491767 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.491723 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.491973 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.491767 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.491973 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.491847 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.491973 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.491943 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.492295 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.492271 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.493689 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.493631 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.493940 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.493923 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.504866 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.504689 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmg2l\" (UniqueName: \"kubernetes.io/projected/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-kube-api-access-cmg2l\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.505225 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.505203 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.593743 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.593715 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:19.857656 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.857587 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a0419a6-8280-42c3-be8b-12c18824688e" path="/var/lib/kubelet/pods/4a0419a6-8280-42c3-be8b-12c18824688e/volumes" Nov 03 19:50:19.881962 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.881933 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh"] Nov 03 19:50:19.882960 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:50:19.882924 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a8b7ef6_cd1f_4b59_82a4_34a1097aebb5.slice/crio-fe913e89e3800deb12d201df21ed470aee93e4a175ee4dd15cffe54289881a09 WatchSource:0}: Error finding container fe913e89e3800deb12d201df21ed470aee93e4a175ee4dd15cffe54289881a09: Status 404 returned error can't find the container with id fe913e89e3800deb12d201df21ed470aee93e4a175ee4dd15cffe54289881a09 Nov 03 19:50:19.885441 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.885406 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:50:19.885552 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.885481 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:50:19.885552 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:19.885528 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:50:20.662924 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:20.662880 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" event={"ID":"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5","Type":"ContainerStarted","Data":"b2abe7e147b8cbdfcc2c9422f064572e7cb631adeb4ac0fe474279b47dc73326"} Nov 03 19:50:20.663116 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:20.662977 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" event={"ID":"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5","Type":"ContainerStarted","Data":"fe913e89e3800deb12d201df21ed470aee93e4a175ee4dd15cffe54289881a09"} Nov 03 19:50:20.688746 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:20.688684 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" podStartSLOduration=1.688655785 podStartE2EDuration="1.688655785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:20.685556224 +0000 UTC m=+814.943353494" watchObservedRunningTime="2026-11-03 19:50:20.688655785 +0000 UTC m=+814.946453042" Nov 03 19:50:21.594257 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.594207 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:21.596441 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.596412 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 19:50:21.596588 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.596469 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" podUID="7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 19:50:21.753131 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.753100 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4"] Nov 03 19:50:21.757475 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.757456 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:50:21.784740 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.776937 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"kube-root-ca.crt\"" Nov 03 19:50:21.790000 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.789970 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"default-dockercfg-4rsdq\"" Nov 03 19:50:21.790235 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.790146 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"distinct-w2-2b5703-kserve-self-signed-certs\"" Nov 03 19:50:21.791959 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.791933 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4"] Nov 03 19:50:21.824677 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.824639 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-tmp-dir\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-qhvz4\" (UID: \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:50:21.824796 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.824699 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a57d851-12b3-4535-b12c-b7d2c579d0fb-tls-certs\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-qhvz4\" (UID: \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:50:21.824796 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.824753 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-dshm\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-qhvz4\" (UID: \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:50:21.824894 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.824855 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-model-cache\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-qhvz4\" (UID: \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:50:21.824996 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.824974 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nn9g\" (UniqueName: \"kubernetes.io/projected/1a57d851-12b3-4535-b12c-b7d2c579d0fb-kube-api-access-9nn9g\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-qhvz4\" (UID: \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:50:21.825064 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.825044 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-home\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-qhvz4\" (UID: \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:50:21.882733 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.881904 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"openshift-service-ca.crt\"" Nov 03 19:50:21.926686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.925884 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-tmp-dir\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-qhvz4\" (UID: \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:50:21.926686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.925953 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a57d851-12b3-4535-b12c-b7d2c579d0fb-tls-certs\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-qhvz4\" (UID: \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:50:21.926686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.926005 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-dshm\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-qhvz4\" (UID: \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:50:21.926686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.926036 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-model-cache\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-qhvz4\" (UID: \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:50:21.926686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.926085 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nn9g\" (UniqueName: \"kubernetes.io/projected/1a57d851-12b3-4535-b12c-b7d2c579d0fb-kube-api-access-9nn9g\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-qhvz4\" (UID: \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:50:21.926686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.926153 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-home\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-qhvz4\" (UID: \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:50:21.938695 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.937731 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-home\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-qhvz4\" (UID: \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:50:21.938695 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.938525 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-tmp-dir\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-qhvz4\" (UID: \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:50:21.942685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.939201 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-model-cache\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-qhvz4\" (UID: \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:50:21.964680 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.957218 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nn9g\" (UniqueName: \"kubernetes.io/projected/1a57d851-12b3-4535-b12c-b7d2c579d0fb-kube-api-access-9nn9g\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-qhvz4\" (UID: \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:50:21.980429 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.980360 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a57d851-12b3-4535-b12c-b7d2c579d0fb-tls-certs\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-qhvz4\" (UID: \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:50:21.998686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:21.997068 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-dshm\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-qhvz4\" (UID: \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:50:22.107816 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:22.107777 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:50:22.507960 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:22.507053 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4"] Nov 03 19:50:22.597393 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:22.596990 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 19:50:22.597393 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:22.597045 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" podUID="7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 19:50:22.718682 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:22.713524 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" event={"ID":"1a57d851-12b3-4535-b12c-b7d2c579d0fb","Type":"ContainerStarted","Data":"b8d82ca6d1d4fd46ae40cb0b5685d96dc2c2f3703f2b90b33d10896376edb8cc"} Nov 03 19:50:22.718682 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:22.713567 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" event={"ID":"1a57d851-12b3-4535-b12c-b7d2c579d0fb","Type":"ContainerStarted","Data":"0ecf7d084b4aa7d821db382650cbb5f46133e201ecbfd77d88b048f1a09a26c3"} Nov 03 19:50:22.735642 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:22.735576 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podStartSLOduration=1.735558757 podStartE2EDuration="1.735558757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:22.735242425 +0000 UTC m=+816.993039684" watchObservedRunningTime="2026-11-03 19:50:22.735558757 +0000 UTC m=+816.993356016" Nov 03 19:50:23.114695 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:23.108758 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:50:23.594858 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:23.594826 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 19:50:23.595003 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:23.594879 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" podUID="7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 19:50:24.235499 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:24.235469 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:24.248598 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:24.248573 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:24.594739 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:24.594699 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 19:50:24.594891 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:24.594765 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" podUID="7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 19:50:24.735552 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:24.735523 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:25.595186 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:25.595148 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 19:50:25.595583 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:25.595208 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" podUID="7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 19:50:26.594475 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:26.594440 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 19:50:26.594651 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:26.594490 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" podUID="7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 19:50:27.631420 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:27.631389 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:27.632030 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:27.631882 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:27.632840 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:27.632817 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 19:50:29.768593 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:29.768559 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz"] Nov 03 19:50:32.114686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:32.113687 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:50:32.139011 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:32.138984 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:50:32.880481 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:32.880451 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:50:34.087309 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.087275 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl"] Nov 03 19:50:34.091810 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.091787 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.106376 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.106339 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl"] Nov 03 19:50:34.149958 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.149928 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.150083 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.149969 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.150130 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.150079 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjjmx\" (UniqueName: \"kubernetes.io/projected/3a73f1d8-301f-40db-91e8-b4963410e1a0-kube-api-access-gjjmx\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.150130 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.150122 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.150199 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.150163 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.150272 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.150259 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.150311 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.150299 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.150345 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.150326 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.150407 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.150387 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a73f1d8-301f-40db-91e8-b4963410e1a0-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.251028 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.250992 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.251213 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.251071 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.251213 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.251123 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.251213 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.251163 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.251213 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.251198 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a73f1d8-301f-40db-91e8-b4963410e1a0-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.251459 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.251236 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.251459 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.251287 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.251459 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.251346 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjjmx\" (UniqueName: \"kubernetes.io/projected/3a73f1d8-301f-40db-91e8-b4963410e1a0-kube-api-access-gjjmx\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.251459 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.251403 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.251678 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.251608 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.251807 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.251781 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.251982 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.251955 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.252085 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.252057 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.252085 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.252074 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a73f1d8-301f-40db-91e8-b4963410e1a0-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.253589 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.253568 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.253902 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.253884 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.267905 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.267879 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.267990 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.267948 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjjmx\" (UniqueName: \"kubernetes.io/projected/3a73f1d8-301f-40db-91e8-b4963410e1a0-kube-api-access-gjjmx\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.335147 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.335121 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg"] Nov 03 19:50:34.340628 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.340579 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.352258 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.352231 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.352552 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.352282 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.352552 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.352338 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.352552 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.352398 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.352552 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.352433 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrp8r\" (UniqueName: \"kubernetes.io/projected/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-kube-api-access-jrp8r\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.352552 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.352503 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.352925 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.352586 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.352925 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.352614 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.352925 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.352716 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.363591 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.363567 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg"] Nov 03 19:50:34.410246 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.410222 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:34.453992 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.453964 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.454126 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.454005 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.454126 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.454024 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.454126 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.454063 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.454301 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.454175 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.454301 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.454197 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.454301 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.454227 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.454301 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.454244 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.454496 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.454439 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrp8r\" (UniqueName: \"kubernetes.io/projected/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-kube-api-access-jrp8r\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.454496 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.454459 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.454950 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.454926 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.455041 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.455026 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.455117 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.455033 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.455305 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.455284 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.456786 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.456768 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.457051 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.457016 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.468131 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.467933 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.468131 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.468068 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrp8r\" (UniqueName: \"kubernetes.io/projected/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-kube-api-access-jrp8r\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.656071 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.655993 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:34.706488 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.706456 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl"] Nov 03 19:50:34.707592 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:50:34.707565 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a73f1d8_301f_40db_91e8_b4963410e1a0.slice/crio-d6c40070166be85674df1b5ba4d1477a146efb5f13c6855289fc790faa6b519b WatchSource:0}: Error finding container d6c40070166be85674df1b5ba4d1477a146efb5f13c6855289fc790faa6b519b: Status 404 returned error can't find the container with id d6c40070166be85674df1b5ba4d1477a146efb5f13c6855289fc790faa6b519b Nov 03 19:50:34.715046 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.715005 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:50:34.715119 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.715101 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:50:34.715176 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.715159 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:50:34.869016 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.868121 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" event={"ID":"3a73f1d8-301f-40db-91e8-b4963410e1a0","Type":"ContainerStarted","Data":"c44d5229a7ea4579d97216bf3a68c0945f698e93fa10deb42a260172c495bada"} Nov 03 19:50:34.869016 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.868169 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" event={"ID":"3a73f1d8-301f-40db-91e8-b4963410e1a0","Type":"ContainerStarted","Data":"d6c40070166be85674df1b5ba4d1477a146efb5f13c6855289fc790faa6b519b"} Nov 03 19:50:34.897506 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.897423 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" podStartSLOduration=0.897410589 podStartE2EDuration="897.410589ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:34.893958795 +0000 UTC m=+829.151756052" watchObservedRunningTime="2026-11-03 19:50:34.897410589 +0000 UTC m=+829.155207846" Nov 03 19:50:34.957002 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.956967 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg"] Nov 03 19:50:34.959942 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:50:34.959916 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5fe53c_7085_4890_99ad_0c4e9381a2a3.slice/crio-3ded4288ac515f1b40cb1a2d33f09c25df46585a69d8eabd0f1d5dee0fb583d7 WatchSource:0}: Error finding container 3ded4288ac515f1b40cb1a2d33f09c25df46585a69d8eabd0f1d5dee0fb583d7: Status 404 returned error can't find the container with id 3ded4288ac515f1b40cb1a2d33f09c25df46585a69d8eabd0f1d5dee0fb583d7 Nov 03 19:50:34.962478 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.962447 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:50:34.962568 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.962513 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:50:34.962568 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:34.962546 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:50:35.410609 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:35.410574 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:35.412182 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:35.412159 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:50:35.412239 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:35.412209 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" podUID="3a73f1d8-301f-40db-91e8-b4963410e1a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 19:50:35.916509 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:35.916464 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" event={"ID":"8f5fe53c-7085-4890-99ad-0c4e9381a2a3","Type":"ContainerStarted","Data":"d1293ad477191c8e2709b91c327804bb9bf71ec3889c9546e956d8ceb782cc26"} Nov 03 19:50:35.916732 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:35.916548 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" event={"ID":"8f5fe53c-7085-4890-99ad-0c4e9381a2a3","Type":"ContainerStarted","Data":"3ded4288ac515f1b40cb1a2d33f09c25df46585a69d8eabd0f1d5dee0fb583d7"} Nov 03 19:50:35.942154 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:35.942104 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" podStartSLOduration=1.942089861 podStartE2EDuration="1.942089861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:35.939998292 +0000 UTC m=+830.197795549" watchObservedRunningTime="2026-11-03 19:50:35.942089861 +0000 UTC m=+830.199887117" Nov 03 19:50:36.411234 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:36.411185 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:50:36.411735 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:36.411253 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" podUID="3a73f1d8-301f-40db-91e8-b4963410e1a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 19:50:36.657058 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:36.657024 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:36.676905 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:36.676805 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:50:36.677196 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:36.677164 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" podUID="8f5fe53c-7085-4890-99ad-0c4e9381a2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 19:50:37.411481 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:37.411439 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:50:37.411864 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:37.411501 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" podUID="3a73f1d8-301f-40db-91e8-b4963410e1a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 19:50:37.656721 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:37.656684 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:50:37.656894 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:37.656758 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" podUID="8f5fe53c-7085-4890-99ad-0c4e9381a2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 19:50:38.410783 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:38.410736 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:50:38.410960 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:38.410821 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" podUID="3a73f1d8-301f-40db-91e8-b4963410e1a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 19:50:38.656825 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:38.656796 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:50:38.657268 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:38.657243 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" podUID="8f5fe53c-7085-4890-99ad-0c4e9381a2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 19:50:39.411259 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:39.411180 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:50:39.411259 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:39.411236 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" podUID="3a73f1d8-301f-40db-91e8-b4963410e1a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 19:50:39.657039 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:39.657002 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:50:39.657544 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:39.657053 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" podUID="8f5fe53c-7085-4890-99ad-0c4e9381a2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 19:50:40.412153 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:40.412116 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:50:40.412461 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:40.412430 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" podUID="3a73f1d8-301f-40db-91e8-b4963410e1a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 19:50:40.629329 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:50:40.629295 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:50:40.656870 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:40.656839 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:50:40.657100 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:40.657076 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" podUID="8f5fe53c-7085-4890-99ad-0c4e9381a2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 19:50:41.411616 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:41.411573 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:50:41.411809 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:41.411640 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" podUID="3a73f1d8-301f-40db-91e8-b4963410e1a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 19:50:41.735010 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:41.734917 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:41.735520 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:41.735500 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:41.736357 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:41.736332 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 19:50:43.265270 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:43.265236 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:43.266889 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:43.266858 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:43.267792 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:43.267766 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:50:44.789581 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:44.789541 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" podUID="812d93af-9b5f-42e4-bb78-da7622071a16" containerName="main" containerID="cri-o://1e31e24e213896e89ffc43948dfb753e88d54e9365a739ab53e0c5c95bb6498d" gracePeriod=45 Nov 03 19:50:44.997880 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:44.997828 5059 generic.go:356] "Generic (PLEG): container finished" podID="812d93af-9b5f-42e4-bb78-da7622071a16" containerID="1e31e24e213896e89ffc43948dfb753e88d54e9365a739ab53e0c5c95bb6498d" exitCode=0 Nov 03 19:50:44.998763 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:44.998734 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" event={"ID":"812d93af-9b5f-42e4-bb78-da7622071a16","Type":"ContainerDied","Data":"1e31e24e213896e89ffc43948dfb753e88d54e9365a739ab53e0c5c95bb6498d"} Nov 03 19:50:45.076934 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:45.076910 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:45.127908 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:45.127862 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-model-cache\") pod \"812d93af-9b5f-42e4-bb78-da7622071a16\" (UID: \"812d93af-9b5f-42e4-bb78-da7622071a16\") " Nov 03 19:50:45.128080 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:45.127926 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-dshm\" (UniqueName: \"kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-dshm\") pod \"812d93af-9b5f-42e4-bb78-da7622071a16\" (UID: \"812d93af-9b5f-42e4-bb78-da7622071a16\") " Nov 03 19:50:45.128080 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:45.128012 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-home\" (UniqueName: \"kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-home\") pod \"812d93af-9b5f-42e4-bb78-da7622071a16\" (UID: \"812d93af-9b5f-42e4-bb78-da7622071a16\") " Nov 03 19:50:45.128218 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:45.128072 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/812d93af-9b5f-42e4-bb78-da7622071a16-tls-certs\" (UniqueName: \"kubernetes.io/secret/812d93af-9b5f-42e4-bb78-da7622071a16-tls-certs\") pod \"812d93af-9b5f-42e4-bb78-da7622071a16\" (UID: \"812d93af-9b5f-42e4-bb78-da7622071a16\") " Nov 03 19:50:45.128218 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:45.128156 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-model-cache" pod "812d93af-9b5f-42e4-bb78-da7622071a16" (UID: "812d93af-9b5f-42e4-bb78-da7622071a16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:50:45.128335 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:45.128227 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-home" pod "812d93af-9b5f-42e4-bb78-da7622071a16" (UID: "812d93af-9b5f-42e4-bb78-da7622071a16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:50:45.128624 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:45.128601 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-tmp-dir\") pod \"812d93af-9b5f-42e4-bb78-da7622071a16\" (UID: \"812d93af-9b5f-42e4-bb78-da7622071a16\") " Nov 03 19:50:45.128900 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:45.128866 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-tmp-dir" pod "812d93af-9b5f-42e4-bb78-da7622071a16" (UID: "812d93af-9b5f-42e4-bb78-da7622071a16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:50:45.129039 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:45.129023 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/812d93af-9b5f-42e4-bb78-da7622071a16-kube-api-access-l82fr\" (UniqueName: \"kubernetes.io/projected/812d93af-9b5f-42e4-bb78-da7622071a16-kube-api-access-l82fr\") pod \"812d93af-9b5f-42e4-bb78-da7622071a16\" (UID: \"812d93af-9b5f-42e4-bb78-da7622071a16\") " Nov 03 19:50:45.129962 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:45.129940 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:45.130132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:45.130087 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:45.130251 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:45.130235 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:45.130963 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:45.130939 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-dshm" pod "812d93af-9b5f-42e4-bb78-da7622071a16" (UID: "812d93af-9b5f-42e4-bb78-da7622071a16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:50:45.131062 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:45.131012 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/812d93af-9b5f-42e4-bb78-da7622071a16-tls-certs" pod "812d93af-9b5f-42e4-bb78-da7622071a16" (UID: "812d93af-9b5f-42e4-bb78-da7622071a16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:50:45.131467 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:45.131447 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/812d93af-9b5f-42e4-bb78-da7622071a16-kube-api-access-l82fr" pod "812d93af-9b5f-42e4-bb78-da7622071a16" (UID: "812d93af-9b5f-42e4-bb78-da7622071a16"). InnerVolumeSpecName "kube-api-access-l82fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:50:45.231293 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:45.231260 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l82fr\" (UniqueName: \"kubernetes.io/projected/812d93af-9b5f-42e4-bb78-da7622071a16-kube-api-access-l82fr\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:45.231293 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:45.231290 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/812d93af-9b5f-42e4-bb78-da7622071a16-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:45.231293 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:45.231300 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/812d93af-9b5f-42e4-bb78-da7622071a16-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:50:46.011385 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:46.011346 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" event={"ID":"812d93af-9b5f-42e4-bb78-da7622071a16","Type":"ContainerDied","Data":"a9efad5101b64364a116212ad206639774db7d740170356c8716d589beb38b7a"} Nov 03 19:50:46.011912 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:46.011411 5059 scope.go:122] "RemoveContainer" containerID="1e31e24e213896e89ffc43948dfb753e88d54e9365a739ab53e0c5c95bb6498d" Nov 03 19:50:46.012105 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:46.012078 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz" Nov 03 19:50:46.039715 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:46.038620 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz"] Nov 03 19:50:46.043077 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:46.043055 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/e2e-conflict-b-a0f38071-kserve-f655cb547-tzlqz"] Nov 03 19:50:47.859676 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:47.859640 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="812d93af-9b5f-42e4-bb78-da7622071a16" path="/var/lib/kubelet/pods/812d93af-9b5f-42e4-bb78-da7622071a16/volumes" Nov 03 19:50:49.085704 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.085654 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c"] Nov 03 19:50:49.086414 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.086391 5059 cpu_manager.go:414] "containerMap: removing container" podUID="812d93af-9b5f-42e4-bb78-da7622071a16" containerName="main" Nov 03 19:50:49.086414 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.086413 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="812d93af-9b5f-42e4-bb78-da7622071a16" containerName="main" Nov 03 19:50:49.086592 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.086530 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="812d93af-9b5f-42e4-bb78-da7622071a16" containerName="main" Nov 03 19:50:49.091786 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.091768 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.106280 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.106255 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c"] Nov 03 19:50:49.125179 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.125138 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.125384 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.125192 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e53359d-3f88-4a9c-b50a-050984844949-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.125384 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.125274 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.125384 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.125305 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.125384 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.125337 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e53359d-3f88-4a9c-b50a-050984844949-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.125573 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.125386 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e53359d-3f88-4a9c-b50a-050984844949-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.125573 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.125434 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd9vp\" (UniqueName: \"kubernetes.io/projected/0e53359d-3f88-4a9c-b50a-050984844949-kube-api-access-kd9vp\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.125573 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.125522 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.126174 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.125581 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.129576 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.129548 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9"] Nov 03 19:50:49.134574 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.134552 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.152126 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.152089 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9"] Nov 03 19:50:49.226439 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.226404 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.226618 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.226453 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.226618 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.226511 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.226757 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.226645 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.226757 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.226705 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.226757 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.226727 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e53359d-3f88-4a9c-b50a-050984844949-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.226757 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.226757 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.226973 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.226780 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.226973 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.226818 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.226973 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.226956 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e53359d-3f88-4a9c-b50a-050984844949-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.227109 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.226976 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.227109 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.226983 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.227109 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.226988 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.227109 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.227034 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.227285 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.227104 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhmgb\" (UniqueName: \"kubernetes.io/projected/63fb9260-c216-4e29-99c9-95dd0d4e4c69-kube-api-access-qhmgb\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.227285 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.227138 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e53359d-3f88-4a9c-b50a-050984844949-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.227285 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.227164 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.227285 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.227228 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.227285 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.227273 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd9vp\" (UniqueName: \"kubernetes.io/projected/0e53359d-3f88-4a9c-b50a-050984844949-kube-api-access-kd9vp\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.227548 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.227310 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.227548 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.227333 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.227548 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.227350 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.227853 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.227833 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e53359d-3f88-4a9c-b50a-050984844949-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.229335 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.229316 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.229428 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.229353 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e53359d-3f88-4a9c-b50a-050984844949-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.239210 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.239189 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e53359d-3f88-4a9c-b50a-050984844949-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.239423 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.239407 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd9vp\" (UniqueName: \"kubernetes.io/projected/0e53359d-3f88-4a9c-b50a-050984844949-kube-api-access-kd9vp\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.328802 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.328776 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.328924 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.328819 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.328924 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.328858 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.328924 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.328875 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.329082 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.328923 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.329082 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.328986 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.329082 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.329053 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.329242 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.329142 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.329301 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.329259 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.329301 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.329261 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhmgb\" (UniqueName: \"kubernetes.io/projected/63fb9260-c216-4e29-99c9-95dd0d4e4c69-kube-api-access-qhmgb\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.329401 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.329350 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.329498 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.329473 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.329618 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.329600 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.329618 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.329606 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.331278 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.331259 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.331378 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.331290 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.341051 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.340988 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.341234 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.341214 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhmgb\" (UniqueName: \"kubernetes.io/projected/63fb9260-c216-4e29-99c9-95dd0d4e4c69-kube-api-access-qhmgb\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.407689 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.406880 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:49.464178 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.463908 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:49.819737 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.819705 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c"] Nov 03 19:50:49.827094 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:50:49.827061 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e53359d_3f88_4a9c_b50a_050984844949.slice/crio-7830b1eea2e42b5af9da0c974f0b976a19dbf7c8be5f25f7d8e50807e7ff5470 WatchSource:0}: Error finding container 7830b1eea2e42b5af9da0c974f0b976a19dbf7c8be5f25f7d8e50807e7ff5470: Status 404 returned error can't find the container with id 7830b1eea2e42b5af9da0c974f0b976a19dbf7c8be5f25f7d8e50807e7ff5470 Nov 03 19:50:49.829709 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.829642 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:50:49.829805 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.829743 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:50:49.829805 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.829776 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:50:49.842039 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.842014 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9"] Nov 03 19:50:49.842945 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:50:49.842921 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63fb9260_c216_4e29_99c9_95dd0d4e4c69.slice/crio-7abed323dd86db25ce994b774fd90800d0bc1b8deb9422fbb88d78811a4c23c6 WatchSource:0}: Error finding container 7abed323dd86db25ce994b774fd90800d0bc1b8deb9422fbb88d78811a4c23c6: Status 404 returned error can't find the container with id 7abed323dd86db25ce994b774fd90800d0bc1b8deb9422fbb88d78811a4c23c6 Nov 03 19:50:49.848959 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.848933 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:50:49.849033 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.848992 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:50:49.849033 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:49.849023 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:50:50.041448 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:50.041403 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" event={"ID":"63fb9260-c216-4e29-99c9-95dd0d4e4c69","Type":"ContainerStarted","Data":"832030defdd41fcfdac7fdd33897d9aafc05fd3cb9f6ae3535e2570a1aab65ed"} Nov 03 19:50:50.041635 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:50.041471 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" event={"ID":"63fb9260-c216-4e29-99c9-95dd0d4e4c69","Type":"ContainerStarted","Data":"7abed323dd86db25ce994b774fd90800d0bc1b8deb9422fbb88d78811a4c23c6"} Nov 03 19:50:50.043567 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:50.043539 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" event={"ID":"0e53359d-3f88-4a9c-b50a-050984844949","Type":"ContainerStarted","Data":"61d4bffc78086245d3366e6ac7434f3e64b57bca64e4d3b1bfa2cc08dc31fded"} Nov 03 19:50:50.043713 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:50.043585 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" event={"ID":"0e53359d-3f88-4a9c-b50a-050984844949","Type":"ContainerStarted","Data":"7830b1eea2e42b5af9da0c974f0b976a19dbf7c8be5f25f7d8e50807e7ff5470"} Nov 03 19:50:50.066516 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:50.066458 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" podStartSLOduration=1.066440472 podStartE2EDuration="1.066440472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:50.065390147 +0000 UTC m=+844.323187404" watchObservedRunningTime="2026-11-03 19:50:50.066440472 +0000 UTC m=+844.324237731" Nov 03 19:50:50.091598 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:50.091537 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" podStartSLOduration=1.091519641 podStartE2EDuration="1.091519641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:50.088244118 +0000 UTC m=+844.346041374" watchObservedRunningTime="2026-11-03 19:50:50.091519641 +0000 UTC m=+844.349316898" Nov 03 19:50:50.407879 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:50.407838 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:50.409697 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:50.409645 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:50:50.409845 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:50.409719 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" podUID="0e53359d-3f88-4a9c-b50a-050984844949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 19:50:50.473065 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:50.473028 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:50.482393 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:50.482354 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 19:50:50.482684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:50.482636 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" podUID="63fb9260-c216-4e29-99c9-95dd0d4e4c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 19:50:51.408593 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:51.408083 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:50:51.408593 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:51.408143 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" podUID="0e53359d-3f88-4a9c-b50a-050984844949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 19:50:51.469685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:51.466024 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 19:50:51.469685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:51.466079 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" podUID="63fb9260-c216-4e29-99c9-95dd0d4e4c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 19:50:52.408722 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:52.408295 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:50:52.408722 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:52.408365 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" podUID="0e53359d-3f88-4a9c-b50a-050984844949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 19:50:52.466687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:52.466053 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 19:50:52.466687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:52.466110 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" podUID="63fb9260-c216-4e29-99c9-95dd0d4e4c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 19:50:53.408630 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:53.408106 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:50:53.408630 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:53.408170 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" podUID="0e53359d-3f88-4a9c-b50a-050984844949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 19:50:53.473146 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:53.473104 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 19:50:53.473436 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:53.473402 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" podUID="63fb9260-c216-4e29-99c9-95dd0d4e4c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 19:50:54.411990 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:54.411916 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:50:54.412577 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:54.412543 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" podUID="0e53359d-3f88-4a9c-b50a-050984844949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 19:50:54.468687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:54.466096 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 19:50:54.468687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:54.466159 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" podUID="63fb9260-c216-4e29-99c9-95dd0d4e4c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 19:50:55.407757 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:55.407715 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:50:55.407931 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:55.407783 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" podUID="0e53359d-3f88-4a9c-b50a-050984844949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 19:50:55.468166 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:55.468131 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 19:50:55.468741 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:55.468713 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" podUID="63fb9260-c216-4e29-99c9-95dd0d4e4c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 19:50:56.408351 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:56.408309 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:50:56.408550 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:56.408381 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" podUID="0e53359d-3f88-4a9c-b50a-050984844949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 19:50:56.465597 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:56.465557 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 19:50:56.465760 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:56.465617 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" podUID="63fb9260-c216-4e29-99c9-95dd0d4e4c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 19:50:57.412243 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:57.412213 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:57.412688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:57.412671 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:57.413411 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:57.413393 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:50:57.471365 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:57.471329 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:57.472205 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:57.472180 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:50:57.473013 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:50:57.472991 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 19:51:03.428517 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.428482 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc"] Nov 03 19:51:03.432827 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.432808 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:03.438325 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.438303 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 19:51:03.441229 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.441205 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 19:51:03.441352 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.441300 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 19:51:03.441549 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.441424 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-nh6bx\"" Nov 03 19:51:03.453129 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.453090 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc"] Nov 03 19:51:03.567824 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.567783 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf9vc\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:03.567998 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.567843 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf9vc\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:03.567998 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.567935 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf9vc\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:03.567998 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.567984 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf9vc\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:03.568111 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.568086 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf9vc\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:03.568148 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.568130 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf9vc\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:03.568220 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.568202 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7pbn\" (UniqueName: \"kubernetes.io/projected/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-kube-api-access-f7pbn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf9vc\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:03.669342 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.669308 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf9vc\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:03.669537 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.669385 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf9vc\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:03.669537 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.669445 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf9vc\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:03.669537 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.669516 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf9vc\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:03.669761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.669554 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf9vc\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:03.669761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.669593 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7pbn\" (UniqueName: \"kubernetes.io/projected/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-kube-api-access-f7pbn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf9vc\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:03.669761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.669643 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf9vc\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:03.670347 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.670315 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf9vc\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:03.671984 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.671953 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf9vc\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:03.672090 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.672034 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf9vc\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:03.672174 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.672149 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf9vc\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:03.672215 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.672153 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf9vc\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:03.681831 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.681748 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf9vc\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:03.682203 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.682178 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7pbn\" (UniqueName: \"kubernetes.io/projected/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-kube-api-access-f7pbn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf9vc\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:03.753429 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:03.753396 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:04.064288 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:04.064257 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc"] Nov 03 19:51:04.064859 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:51:04.064834 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677a5153_5fe3_4624_bf06_3c8dfc6bc72d.slice/crio-b393f72f0eb96740f96ac2b965ee7b282d6d33d6db11a8993b42ebddaee9a7fc WatchSource:0}: Error finding container b393f72f0eb96740f96ac2b965ee7b282d6d33d6db11a8993b42ebddaee9a7fc: Status 404 returned error can't find the container with id b393f72f0eb96740f96ac2b965ee7b282d6d33d6db11a8993b42ebddaee9a7fc Nov 03 19:51:04.252410 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:04.252372 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" event={"ID":"677a5153-5fe3-4624-bf06-3c8dfc6bc72d","Type":"ContainerStarted","Data":"b393f72f0eb96740f96ac2b965ee7b282d6d33d6db11a8993b42ebddaee9a7fc"} Nov 03 19:51:07.710104 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:07.710059 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:51:07.764714 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:07.763757 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:51:07.864358 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:07.864309 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 19:51:10.608733 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:51:10.608640 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:51:10.946499 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:10.946462 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:51:10.946616 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:10.946524 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:51:11.612910 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:11.612870 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" event={"ID":"677a5153-5fe3-4624-bf06-3c8dfc6bc72d","Type":"ContainerStarted","Data":"daf533566a4042ce35f672f5974749c01a13dc2086255c8268966b3679ba35ea"} Nov 03 19:51:11.613387 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:11.612960 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:11.646402 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:11.646350 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" podStartSLOduration=1.768208155 podStartE2EDuration="8.646335217s" totalImagesPullingTime="6.878127062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:11.640367145 +0000 UTC m=+865.898164400" watchObservedRunningTime="2026-11-03 19:51:11.646335217 +0000 UTC m=+865.904132473" Nov 03 19:51:12.628069 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:12.628042 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 19:51:15.618062 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:15.618030 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d9b2be9-fx69j"] Nov 03 19:51:15.622166 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:15.622149 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9b2be9-fx69j" Nov 03 19:51:15.628010 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:15.627985 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:51:15.628128 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:15.628074 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 19:51:15.631797 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:15.631777 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d9b2be9-fx69j"] Nov 03 19:51:15.678736 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:15.678702 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0dd6c49-3d40-40e7-b793-8a6880d8d04d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d9b2be9-fx69j\" (UID: \"c0dd6c49-3d40-40e7-b793-8a6880d8d04d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9b2be9-fx69j" Nov 03 19:51:15.678915 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:15.678881 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpsml\" (UniqueName: \"kubernetes.io/projected/c0dd6c49-3d40-40e7-b793-8a6880d8d04d-kube-api-access-rpsml\") pod \"maas-api-revoke-sub-5d9b2be9-fx69j\" (UID: \"c0dd6c49-3d40-40e7-b793-8a6880d8d04d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9b2be9-fx69j" Nov 03 19:51:15.679053 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:15.679027 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0dd6c49-3d40-40e7-b793-8a6880d8d04d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d9b2be9-fx69j\" (UID: \"c0dd6c49-3d40-40e7-b793-8a6880d8d04d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9b2be9-fx69j" Nov 03 19:51:15.780593 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:15.780563 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpsml\" (UniqueName: \"kubernetes.io/projected/c0dd6c49-3d40-40e7-b793-8a6880d8d04d-kube-api-access-rpsml\") pod \"maas-api-revoke-sub-5d9b2be9-fx69j\" (UID: \"c0dd6c49-3d40-40e7-b793-8a6880d8d04d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9b2be9-fx69j" Nov 03 19:51:15.780755 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:15.780630 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0dd6c49-3d40-40e7-b793-8a6880d8d04d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d9b2be9-fx69j\" (UID: \"c0dd6c49-3d40-40e7-b793-8a6880d8d04d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9b2be9-fx69j" Nov 03 19:51:15.780755 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:15.780729 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0dd6c49-3d40-40e7-b793-8a6880d8d04d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d9b2be9-fx69j\" (UID: \"c0dd6c49-3d40-40e7-b793-8a6880d8d04d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9b2be9-fx69j" Nov 03 19:51:15.781365 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:15.781343 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0dd6c49-3d40-40e7-b793-8a6880d8d04d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d9b2be9-fx69j\" (UID: \"c0dd6c49-3d40-40e7-b793-8a6880d8d04d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9b2be9-fx69j" Nov 03 19:51:15.782934 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:15.782912 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0dd6c49-3d40-40e7-b793-8a6880d8d04d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d9b2be9-fx69j\" (UID: \"c0dd6c49-3d40-40e7-b793-8a6880d8d04d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9b2be9-fx69j" Nov 03 19:51:15.792599 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:15.792576 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpsml\" (UniqueName: \"kubernetes.io/projected/c0dd6c49-3d40-40e7-b793-8a6880d8d04d-kube-api-access-rpsml\") pod \"maas-api-revoke-sub-5d9b2be9-fx69j\" (UID: \"c0dd6c49-3d40-40e7-b793-8a6880d8d04d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9b2be9-fx69j" Nov 03 19:51:15.938376 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:15.938284 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9b2be9-fx69j" Nov 03 19:51:16.229083 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:16.229048 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d9b2be9-fx69j"] Nov 03 19:51:16.231630 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:51:16.231593 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0dd6c49_3d40_40e7_b793_8a6880d8d04d.slice/crio-7afec9a01922c997c4281153e24cd66823373f59ce6f486964f89ace9070e71a WatchSource:0}: Error finding container 7afec9a01922c997c4281153e24cd66823373f59ce6f486964f89ace9070e71a: Status 404 returned error can't find the container with id 7afec9a01922c997c4281153e24cd66823373f59ce6f486964f89ace9070e71a Nov 03 19:51:16.664504 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:16.664462 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9b2be9-fx69j" event={"ID":"c0dd6c49-3d40-40e7-b793-8a6880d8d04d","Type":"ContainerStarted","Data":"3193a5482642c97d5980c5ec9f05bbabd5cc1028d3adeed48033b3eeeecbf24c"} Nov 03 19:51:16.665031 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:16.664558 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9b2be9-fx69j" event={"ID":"c0dd6c49-3d40-40e7-b793-8a6880d8d04d","Type":"ContainerStarted","Data":"7afec9a01922c997c4281153e24cd66823373f59ce6f486964f89ace9070e71a"} Nov 03 19:51:16.713830 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:16.713761 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9b2be9-fx69j" podStartSLOduration=1.713729252 podStartE2EDuration="1.713729252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:16.712929652 +0000 UTC m=+870.970726912" watchObservedRunningTime="2026-11-03 19:51:16.713729252 +0000 UTC m=+870.971526518" Nov 03 19:51:17.756462 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:17.756431 5059 generic.go:356] "Generic (PLEG): container finished" podID="c0dd6c49-3d40-40e7-b793-8a6880d8d04d" containerID="3193a5482642c97d5980c5ec9f05bbabd5cc1028d3adeed48033b3eeeecbf24c" exitCode=0 Nov 03 19:51:17.757016 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:17.756990 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9b2be9-fx69j" event={"ID":"c0dd6c49-3d40-40e7-b793-8a6880d8d04d","Type":"ContainerDied","Data":"3193a5482642c97d5980c5ec9f05bbabd5cc1028d3adeed48033b3eeeecbf24c"} Nov 03 19:51:18.967565 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:18.967535 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9b2be9-fx69j" Nov 03 19:51:19.013842 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:19.013808 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0dd6c49-3d40-40e7-b793-8a6880d8d04d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0dd6c49-3d40-40e7-b793-8a6880d8d04d-maas-api-openshift-ca\") pod \"c0dd6c49-3d40-40e7-b793-8a6880d8d04d\" (UID: \"c0dd6c49-3d40-40e7-b793-8a6880d8d04d\") " Nov 03 19:51:19.014245 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:19.014217 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0dd6c49-3d40-40e7-b793-8a6880d8d04d-maas-api-openshift-ca" pod "c0dd6c49-3d40-40e7-b793-8a6880d8d04d" (UID: "c0dd6c49-3d40-40e7-b793-8a6880d8d04d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:19.115107 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:19.115075 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0dd6c49-3d40-40e7-b793-8a6880d8d04d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0dd6c49-3d40-40e7-b793-8a6880d8d04d-maas-api-xks-ca\") pod \"c0dd6c49-3d40-40e7-b793-8a6880d8d04d\" (UID: \"c0dd6c49-3d40-40e7-b793-8a6880d8d04d\") " Nov 03 19:51:19.115242 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:19.115184 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0dd6c49-3d40-40e7-b793-8a6880d8d04d-kube-api-access-rpsml\" (UniqueName: \"kubernetes.io/projected/c0dd6c49-3d40-40e7-b793-8a6880d8d04d-kube-api-access-rpsml\") pod \"c0dd6c49-3d40-40e7-b793-8a6880d8d04d\" (UID: \"c0dd6c49-3d40-40e7-b793-8a6880d8d04d\") " Nov 03 19:51:19.115581 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:19.115563 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0dd6c49-3d40-40e7-b793-8a6880d8d04d-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:19.117223 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:19.117196 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0dd6c49-3d40-40e7-b793-8a6880d8d04d-maas-api-xks-ca" pod "c0dd6c49-3d40-40e7-b793-8a6880d8d04d" (UID: "c0dd6c49-3d40-40e7-b793-8a6880d8d04d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:51:19.117314 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:19.117226 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0dd6c49-3d40-40e7-b793-8a6880d8d04d-kube-api-access-rpsml" pod "c0dd6c49-3d40-40e7-b793-8a6880d8d04d" (UID: "c0dd6c49-3d40-40e7-b793-8a6880d8d04d"). InnerVolumeSpecName "kube-api-access-rpsml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:19.216997 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:19.216971 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0dd6c49-3d40-40e7-b793-8a6880d8d04d-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:19.216997 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:19.216997 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpsml\" (UniqueName: \"kubernetes.io/projected/c0dd6c49-3d40-40e7-b793-8a6880d8d04d-kube-api-access-rpsml\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:19.774737 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:19.774709 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9b2be9-fx69j" Nov 03 19:51:19.774916 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:19.774770 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d9b2be9-fx69j" event={"ID":"c0dd6c49-3d40-40e7-b793-8a6880d8d04d","Type":"ContainerDied","Data":"7afec9a01922c997c4281153e24cd66823373f59ce6f486964f89ace9070e71a"} Nov 03 19:51:19.774916 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:19.774816 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7afec9a01922c997c4281153e24cd66823373f59ce6f486964f89ace9070e71a" Nov 03 19:51:27.642213 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:27.642170 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ee5e119-rzrcp"] Nov 03 19:51:27.643041 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:27.643019 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c0dd6c49-3d40-40e7-b793-8a6880d8d04d" containerName="revoke-keys" Nov 03 19:51:27.643190 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:27.643177 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0dd6c49-3d40-40e7-b793-8a6880d8d04d" containerName="revoke-keys" Nov 03 19:51:27.643495 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:27.643474 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0dd6c49-3d40-40e7-b793-8a6880d8d04d" containerName="revoke-keys" Nov 03 19:51:27.648763 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:27.648737 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee5e119-rzrcp" Nov 03 19:51:27.667143 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:27.656977 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:51:27.667143 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:27.658874 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 19:51:27.667143 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:27.659479 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ee5e119-rzrcp"] Nov 03 19:51:27.828745 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:27.828709 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmbmc\" (UniqueName: \"kubernetes.io/projected/5112d575-7ebb-4eb4-ac0e-21db52f8b9cb-kube-api-access-jmbmc\") pod \"maas-api-revoke-sub-7ee5e119-rzrcp\" (UID: \"5112d575-7ebb-4eb4-ac0e-21db52f8b9cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee5e119-rzrcp" Nov 03 19:51:27.828916 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:27.828755 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5112d575-7ebb-4eb4-ac0e-21db52f8b9cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ee5e119-rzrcp\" (UID: \"5112d575-7ebb-4eb4-ac0e-21db52f8b9cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee5e119-rzrcp" Nov 03 19:51:27.829032 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:27.829011 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5112d575-7ebb-4eb4-ac0e-21db52f8b9cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ee5e119-rzrcp\" (UID: \"5112d575-7ebb-4eb4-ac0e-21db52f8b9cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee5e119-rzrcp" Nov 03 19:51:27.930419 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:27.930344 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5112d575-7ebb-4eb4-ac0e-21db52f8b9cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ee5e119-rzrcp\" (UID: \"5112d575-7ebb-4eb4-ac0e-21db52f8b9cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee5e119-rzrcp" Nov 03 19:51:27.930562 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:27.930433 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmbmc\" (UniqueName: \"kubernetes.io/projected/5112d575-7ebb-4eb4-ac0e-21db52f8b9cb-kube-api-access-jmbmc\") pod \"maas-api-revoke-sub-7ee5e119-rzrcp\" (UID: \"5112d575-7ebb-4eb4-ac0e-21db52f8b9cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee5e119-rzrcp" Nov 03 19:51:27.930562 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:27.930464 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5112d575-7ebb-4eb4-ac0e-21db52f8b9cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ee5e119-rzrcp\" (UID: \"5112d575-7ebb-4eb4-ac0e-21db52f8b9cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee5e119-rzrcp" Nov 03 19:51:27.931127 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:27.931103 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5112d575-7ebb-4eb4-ac0e-21db52f8b9cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ee5e119-rzrcp\" (UID: \"5112d575-7ebb-4eb4-ac0e-21db52f8b9cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee5e119-rzrcp" Nov 03 19:51:27.932577 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:27.932549 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5112d575-7ebb-4eb4-ac0e-21db52f8b9cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ee5e119-rzrcp\" (UID: \"5112d575-7ebb-4eb4-ac0e-21db52f8b9cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee5e119-rzrcp" Nov 03 19:51:27.943776 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:27.943752 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmbmc\" (UniqueName: \"kubernetes.io/projected/5112d575-7ebb-4eb4-ac0e-21db52f8b9cb-kube-api-access-jmbmc\") pod \"maas-api-revoke-sub-7ee5e119-rzrcp\" (UID: \"5112d575-7ebb-4eb4-ac0e-21db52f8b9cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee5e119-rzrcp" Nov 03 19:51:27.982356 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:27.982337 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee5e119-rzrcp" Nov 03 19:51:28.260350 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:28.260322 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ee5e119-rzrcp"] Nov 03 19:51:28.260517 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:51:28.260495 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5112d575_7ebb_4eb4_ac0e_21db52f8b9cb.slice/crio-54e228ee5c93e93cf70c5490c6959ae45302c4000b308742160e4838eeb00da6 WatchSource:0}: Error finding container 54e228ee5c93e93cf70c5490c6959ae45302c4000b308742160e4838eeb00da6: Status 404 returned error can't find the container with id 54e228ee5c93e93cf70c5490c6959ae45302c4000b308742160e4838eeb00da6 Nov 03 19:51:28.897405 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:28.897306 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee5e119-rzrcp" event={"ID":"5112d575-7ebb-4eb4-ac0e-21db52f8b9cb","Type":"ContainerStarted","Data":"1c176221b6f0afbbee88fb63e5a4e27c9d86780001543e9c5430676a90f7b0bb"} Nov 03 19:51:28.897405 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:28.897372 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee5e119-rzrcp" event={"ID":"5112d575-7ebb-4eb4-ac0e-21db52f8b9cb","Type":"ContainerStarted","Data":"54e228ee5c93e93cf70c5490c6959ae45302c4000b308742160e4838eeb00da6"} Nov 03 19:51:28.916565 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:28.916518 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee5e119-rzrcp" podStartSLOduration=1.916506127 podStartE2EDuration="1.916506127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:28.915621541 +0000 UTC m=+883.173418798" watchObservedRunningTime="2026-11-03 19:51:28.916506127 +0000 UTC m=+883.174303383" Nov 03 19:51:29.910859 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:29.910823 5059 generic.go:356] "Generic (PLEG): container finished" podID="5112d575-7ebb-4eb4-ac0e-21db52f8b9cb" containerID="1c176221b6f0afbbee88fb63e5a4e27c9d86780001543e9c5430676a90f7b0bb" exitCode=0 Nov 03 19:51:29.911349 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:29.910888 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee5e119-rzrcp" event={"ID":"5112d575-7ebb-4eb4-ac0e-21db52f8b9cb","Type":"ContainerDied","Data":"1c176221b6f0afbbee88fb63e5a4e27c9d86780001543e9c5430676a90f7b0bb"} Nov 03 19:51:30.111343 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:30.111310 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0b708d2-hzzcx"] Nov 03 19:51:30.115488 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:30.115465 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0b708d2-hzzcx" Nov 03 19:51:30.126146 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:30.126122 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0b708d2-hzzcx"] Nov 03 19:51:30.164165 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:30.164077 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0b708d2-hzzcx\" (UID: \"cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0b708d2-hzzcx" Nov 03 19:51:30.164165 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:30.164125 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ggww\" (UniqueName: \"kubernetes.io/projected/cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2-kube-api-access-5ggww\") pod \"maas-api-revoke-sub-b0b708d2-hzzcx\" (UID: \"cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0b708d2-hzzcx" Nov 03 19:51:30.164426 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:30.164401 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0b708d2-hzzcx\" (UID: \"cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0b708d2-hzzcx" Nov 03 19:51:30.265326 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:30.265280 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0b708d2-hzzcx\" (UID: \"cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0b708d2-hzzcx" Nov 03 19:51:30.265524 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:30.265392 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0b708d2-hzzcx\" (UID: \"cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0b708d2-hzzcx" Nov 03 19:51:30.265524 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:30.265418 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ggww\" (UniqueName: \"kubernetes.io/projected/cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2-kube-api-access-5ggww\") pod \"maas-api-revoke-sub-b0b708d2-hzzcx\" (UID: \"cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0b708d2-hzzcx" Nov 03 19:51:30.266026 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:30.266003 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0b708d2-hzzcx\" (UID: \"cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0b708d2-hzzcx" Nov 03 19:51:30.267628 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:30.267598 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0b708d2-hzzcx\" (UID: \"cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0b708d2-hzzcx" Nov 03 19:51:30.279532 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:30.279508 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ggww\" (UniqueName: \"kubernetes.io/projected/cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2-kube-api-access-5ggww\") pod \"maas-api-revoke-sub-b0b708d2-hzzcx\" (UID: \"cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0b708d2-hzzcx" Nov 03 19:51:30.431809 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:30.431723 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0b708d2-hzzcx" Nov 03 19:51:30.714088 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:30.714059 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0b708d2-hzzcx"] Nov 03 19:51:30.714757 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:51:30.714726 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdedd4b2_78d7_4a99_be4e_3ebfbbc224b2.slice/crio-00ae101ec37521db338188df3f2be5e504001a5fd33210b596b8f90b0ad2e6f5 WatchSource:0}: Error finding container 00ae101ec37521db338188df3f2be5e504001a5fd33210b596b8f90b0ad2e6f5: Status 404 returned error can't find the container with id 00ae101ec37521db338188df3f2be5e504001a5fd33210b596b8f90b0ad2e6f5 Nov 03 19:51:30.918891 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:30.918855 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0b708d2-hzzcx" event={"ID":"cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2","Type":"ContainerStarted","Data":"00ae101ec37521db338188df3f2be5e504001a5fd33210b596b8f90b0ad2e6f5"} Nov 03 19:51:31.053675 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:31.053641 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee5e119-rzrcp" Nov 03 19:51:31.075990 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:31.075963 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5112d575-7ebb-4eb4-ac0e-21db52f8b9cb-kube-api-access-jmbmc\" (UniqueName: \"kubernetes.io/projected/5112d575-7ebb-4eb4-ac0e-21db52f8b9cb-kube-api-access-jmbmc\") pod \"5112d575-7ebb-4eb4-ac0e-21db52f8b9cb\" (UID: \"5112d575-7ebb-4eb4-ac0e-21db52f8b9cb\") " Nov 03 19:51:31.076124 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:31.076103 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5112d575-7ebb-4eb4-ac0e-21db52f8b9cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5112d575-7ebb-4eb4-ac0e-21db52f8b9cb-maas-api-xks-ca\") pod \"5112d575-7ebb-4eb4-ac0e-21db52f8b9cb\" (UID: \"5112d575-7ebb-4eb4-ac0e-21db52f8b9cb\") " Nov 03 19:51:31.076444 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:31.076423 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5112d575-7ebb-4eb4-ac0e-21db52f8b9cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5112d575-7ebb-4eb4-ac0e-21db52f8b9cb-maas-api-openshift-ca\") pod \"5112d575-7ebb-4eb4-ac0e-21db52f8b9cb\" (UID: \"5112d575-7ebb-4eb4-ac0e-21db52f8b9cb\") " Nov 03 19:51:31.076907 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:31.076880 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5112d575-7ebb-4eb4-ac0e-21db52f8b9cb-maas-api-openshift-ca" pod "5112d575-7ebb-4eb4-ac0e-21db52f8b9cb" (UID: "5112d575-7ebb-4eb4-ac0e-21db52f8b9cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:31.077412 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:31.077386 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5112d575-7ebb-4eb4-ac0e-21db52f8b9cb-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:31.078540 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:31.078294 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5112d575-7ebb-4eb4-ac0e-21db52f8b9cb-kube-api-access-jmbmc" pod "5112d575-7ebb-4eb4-ac0e-21db52f8b9cb" (UID: "5112d575-7ebb-4eb4-ac0e-21db52f8b9cb"). InnerVolumeSpecName "kube-api-access-jmbmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:31.080012 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:31.079987 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5112d575-7ebb-4eb4-ac0e-21db52f8b9cb-maas-api-xks-ca" pod "5112d575-7ebb-4eb4-ac0e-21db52f8b9cb" (UID: "5112d575-7ebb-4eb4-ac0e-21db52f8b9cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:51:31.179024 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:31.178995 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmbmc\" (UniqueName: \"kubernetes.io/projected/5112d575-7ebb-4eb4-ac0e-21db52f8b9cb-kube-api-access-jmbmc\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:31.179024 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:31.179021 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5112d575-7ebb-4eb4-ac0e-21db52f8b9cb-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:31.927832 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:31.927735 5059 generic.go:356] "Generic (PLEG): container finished" podID="cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2" containerID="979ea1ebd80c99181c9faabff7e0386a72fd68c1ee79d4ad8c04c774a6c1e808" exitCode=0 Nov 03 19:51:31.928314 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:31.927827 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0b708d2-hzzcx" event={"ID":"cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2","Type":"ContainerDied","Data":"979ea1ebd80c99181c9faabff7e0386a72fd68c1ee79d4ad8c04c774a6c1e808"} Nov 03 19:51:31.929436 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:31.929415 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee5e119-rzrcp" Nov 03 19:51:31.929544 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:31.929436 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee5e119-rzrcp" event={"ID":"5112d575-7ebb-4eb4-ac0e-21db52f8b9cb","Type":"ContainerDied","Data":"54e228ee5c93e93cf70c5490c6959ae45302c4000b308742160e4838eeb00da6"} Nov 03 19:51:31.929544 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:31.929480 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54e228ee5c93e93cf70c5490c6959ae45302c4000b308742160e4838eeb00da6" Nov 03 19:51:33.544430 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:33.544395 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0b708d2-hzzcx" Nov 03 19:51:33.764397 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:33.764232 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2-maas-api-openshift-ca\") pod \"cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2\" (UID: \"cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2\") " Nov 03 19:51:33.764397 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:33.764308 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2-kube-api-access-5ggww\" (UniqueName: \"kubernetes.io/projected/cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2-kube-api-access-5ggww\") pod \"cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2\" (UID: \"cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2\") " Nov 03 19:51:33.764397 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:33.764359 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2-maas-api-xks-ca\") pod \"cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2\" (UID: \"cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2\") " Nov 03 19:51:33.767534 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:33.767490 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2-maas-api-xks-ca" pod "cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2" (UID: "cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:51:33.767835 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:33.767812 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2-maas-api-openshift-ca" pod "cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2" (UID: "cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:33.770168 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:33.770140 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2-kube-api-access-5ggww" pod "cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2" (UID: "cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2"). InnerVolumeSpecName "kube-api-access-5ggww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:33.868693 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:33.868019 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:33.868693 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:33.868055 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ggww\" (UniqueName: \"kubernetes.io/projected/cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2-kube-api-access-5ggww\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:33.868693 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:33.868072 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:33.970687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:33.969828 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0b708d2-hzzcx" event={"ID":"cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2","Type":"ContainerDied","Data":"00ae101ec37521db338188df3f2be5e504001a5fd33210b596b8f90b0ad2e6f5"} Nov 03 19:51:33.970687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:33.969891 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00ae101ec37521db338188df3f2be5e504001a5fd33210b596b8f90b0ad2e6f5" Nov 03 19:51:33.970687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:33.969966 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0b708d2-hzzcx" Nov 03 19:51:40.605101 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:51:40.605060 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:51:41.798865 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:41.798828 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b79c04b-4tggl"] Nov 03 19:51:41.799422 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:41.799409 5059 cpu_manager.go:414] "containerMap: removing container" podUID="cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2" containerName="revoke-keys" Nov 03 19:51:41.799470 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:41.799423 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2" containerName="revoke-keys" Nov 03 19:51:41.799508 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:41.799469 5059 cpu_manager.go:414] "containerMap: removing container" podUID="5112d575-7ebb-4eb4-ac0e-21db52f8b9cb" containerName="revoke-keys" Nov 03 19:51:41.799508 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:41.799474 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5112d575-7ebb-4eb4-ac0e-21db52f8b9cb" containerName="revoke-keys" Nov 03 19:51:41.799575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:41.799564 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2" containerName="revoke-keys" Nov 03 19:51:41.799610 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:41.799576 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5112d575-7ebb-4eb4-ac0e-21db52f8b9cb" containerName="revoke-keys" Nov 03 19:51:41.808560 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:41.808538 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b79c04b-4tggl" Nov 03 19:51:41.811919 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:41.811894 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b79c04b-4tggl"] Nov 03 19:51:41.812359 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:41.812339 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:51:41.813573 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:41.813551 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 19:51:41.841896 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:41.841870 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b018bb-eb68-478d-9752-f3d07ed2cacf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b79c04b-4tggl\" (UID: \"50b018bb-eb68-478d-9752-f3d07ed2cacf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b79c04b-4tggl" Nov 03 19:51:41.842009 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:41.841941 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b018bb-eb68-478d-9752-f3d07ed2cacf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b79c04b-4tggl\" (UID: \"50b018bb-eb68-478d-9752-f3d07ed2cacf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b79c04b-4tggl" Nov 03 19:51:41.842082 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:41.842059 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk5h6\" (UniqueName: \"kubernetes.io/projected/50b018bb-eb68-478d-9752-f3d07ed2cacf-kube-api-access-bk5h6\") pod \"maas-api-revoke-sub-7b79c04b-4tggl\" (UID: \"50b018bb-eb68-478d-9752-f3d07ed2cacf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b79c04b-4tggl" Nov 03 19:51:41.943608 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:41.943570 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b018bb-eb68-478d-9752-f3d07ed2cacf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b79c04b-4tggl\" (UID: \"50b018bb-eb68-478d-9752-f3d07ed2cacf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b79c04b-4tggl" Nov 03 19:51:41.943850 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:41.943725 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b018bb-eb68-478d-9752-f3d07ed2cacf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b79c04b-4tggl\" (UID: \"50b018bb-eb68-478d-9752-f3d07ed2cacf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b79c04b-4tggl" Nov 03 19:51:41.944188 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:41.944160 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk5h6\" (UniqueName: \"kubernetes.io/projected/50b018bb-eb68-478d-9752-f3d07ed2cacf-kube-api-access-bk5h6\") pod \"maas-api-revoke-sub-7b79c04b-4tggl\" (UID: \"50b018bb-eb68-478d-9752-f3d07ed2cacf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b79c04b-4tggl" Nov 03 19:51:41.944409 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:41.944385 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b018bb-eb68-478d-9752-f3d07ed2cacf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b79c04b-4tggl\" (UID: \"50b018bb-eb68-478d-9752-f3d07ed2cacf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b79c04b-4tggl" Nov 03 19:51:41.946177 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:41.946152 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b018bb-eb68-478d-9752-f3d07ed2cacf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b79c04b-4tggl\" (UID: \"50b018bb-eb68-478d-9752-f3d07ed2cacf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b79c04b-4tggl" Nov 03 19:51:41.957115 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:41.957093 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk5h6\" (UniqueName: \"kubernetes.io/projected/50b018bb-eb68-478d-9752-f3d07ed2cacf-kube-api-access-bk5h6\") pod \"maas-api-revoke-sub-7b79c04b-4tggl\" (UID: \"50b018bb-eb68-478d-9752-f3d07ed2cacf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b79c04b-4tggl" Nov 03 19:51:42.126326 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:42.126299 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b79c04b-4tggl" Nov 03 19:51:42.412856 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:42.412823 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b79c04b-4tggl"] Nov 03 19:51:42.412979 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:51:42.412937 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b018bb_eb68_478d_9752_f3d07ed2cacf.slice/crio-7bf504d67982987386b12e3b7cb499113a90bb1e9b0a7684060a5c0537368592 WatchSource:0}: Error finding container 7bf504d67982987386b12e3b7cb499113a90bb1e9b0a7684060a5c0537368592: Status 404 returned error can't find the container with id 7bf504d67982987386b12e3b7cb499113a90bb1e9b0a7684060a5c0537368592 Nov 03 19:51:42.784774 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:42.784750 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276879a2-wwfnr"] Nov 03 19:51:42.789108 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:42.789091 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276879a2-wwfnr" Nov 03 19:51:42.799367 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:42.799346 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276879a2-wwfnr"] Nov 03 19:51:42.860623 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:42.860589 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/944b8276-6b88-4dc7-8a0d-af8f8b879a39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-276879a2-wwfnr\" (UID: \"944b8276-6b88-4dc7-8a0d-af8f8b879a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276879a2-wwfnr" Nov 03 19:51:42.860771 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:42.860752 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6dpd\" (UniqueName: \"kubernetes.io/projected/944b8276-6b88-4dc7-8a0d-af8f8b879a39-kube-api-access-w6dpd\") pod \"maas-api-revoke-sub-276879a2-wwfnr\" (UID: \"944b8276-6b88-4dc7-8a0d-af8f8b879a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276879a2-wwfnr" Nov 03 19:51:42.860922 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:42.860904 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/944b8276-6b88-4dc7-8a0d-af8f8b879a39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-276879a2-wwfnr\" (UID: \"944b8276-6b88-4dc7-8a0d-af8f8b879a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276879a2-wwfnr" Nov 03 19:51:42.962489 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:42.962460 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6dpd\" (UniqueName: \"kubernetes.io/projected/944b8276-6b88-4dc7-8a0d-af8f8b879a39-kube-api-access-w6dpd\") pod \"maas-api-revoke-sub-276879a2-wwfnr\" (UID: \"944b8276-6b88-4dc7-8a0d-af8f8b879a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276879a2-wwfnr" Nov 03 19:51:42.962703 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:42.962684 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/944b8276-6b88-4dc7-8a0d-af8f8b879a39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-276879a2-wwfnr\" (UID: \"944b8276-6b88-4dc7-8a0d-af8f8b879a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276879a2-wwfnr" Nov 03 19:51:42.962813 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:42.962796 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/944b8276-6b88-4dc7-8a0d-af8f8b879a39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-276879a2-wwfnr\" (UID: \"944b8276-6b88-4dc7-8a0d-af8f8b879a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276879a2-wwfnr" Nov 03 19:51:42.963681 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:42.963640 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/944b8276-6b88-4dc7-8a0d-af8f8b879a39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-276879a2-wwfnr\" (UID: \"944b8276-6b88-4dc7-8a0d-af8f8b879a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276879a2-wwfnr" Nov 03 19:51:42.965044 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:42.965025 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/944b8276-6b88-4dc7-8a0d-af8f8b879a39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-276879a2-wwfnr\" (UID: \"944b8276-6b88-4dc7-8a0d-af8f8b879a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276879a2-wwfnr" Nov 03 19:51:42.975174 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:42.975154 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6dpd\" (UniqueName: \"kubernetes.io/projected/944b8276-6b88-4dc7-8a0d-af8f8b879a39-kube-api-access-w6dpd\") pod \"maas-api-revoke-sub-276879a2-wwfnr\" (UID: \"944b8276-6b88-4dc7-8a0d-af8f8b879a39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276879a2-wwfnr" Nov 03 19:51:43.076909 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:43.076881 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b79c04b-4tggl" event={"ID":"50b018bb-eb68-478d-9752-f3d07ed2cacf","Type":"ContainerStarted","Data":"b3f40ef8220aec992a9f7e77fe2204e98bdbdab1b0e917ceaaa55d905170dcdb"} Nov 03 19:51:43.077028 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:43.076934 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b79c04b-4tggl" event={"ID":"50b018bb-eb68-478d-9752-f3d07ed2cacf","Type":"ContainerStarted","Data":"7bf504d67982987386b12e3b7cb499113a90bb1e9b0a7684060a5c0537368592"} Nov 03 19:51:43.099169 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:43.099125 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b79c04b-4tggl" podStartSLOduration=2.099112141 podStartE2EDuration="2.099112141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:43.097184974 +0000 UTC m=+897.354982239" watchObservedRunningTime="2026-11-03 19:51:43.099112141 +0000 UTC m=+897.356909397" Nov 03 19:51:43.104461 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:43.104433 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276879a2-wwfnr" Nov 03 19:51:43.404875 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:51:43.404824 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944b8276_6b88_4dc7_8a0d_af8f8b879a39.slice/crio-bd0c959b5ec31e6fec2817af0eef5fd73412c121c99e2c797281fe0e60a778b8 WatchSource:0}: Error finding container bd0c959b5ec31e6fec2817af0eef5fd73412c121c99e2c797281fe0e60a778b8: Status 404 returned error can't find the container with id bd0c959b5ec31e6fec2817af0eef5fd73412c121c99e2c797281fe0e60a778b8 Nov 03 19:51:43.405015 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:43.404937 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276879a2-wwfnr"] Nov 03 19:51:44.085394 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:44.085362 5059 generic.go:356] "Generic (PLEG): container finished" podID="50b018bb-eb68-478d-9752-f3d07ed2cacf" containerID="b3f40ef8220aec992a9f7e77fe2204e98bdbdab1b0e917ceaaa55d905170dcdb" exitCode=0 Nov 03 19:51:44.085893 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:44.085409 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b79c04b-4tggl" event={"ID":"50b018bb-eb68-478d-9752-f3d07ed2cacf","Type":"ContainerDied","Data":"b3f40ef8220aec992a9f7e77fe2204e98bdbdab1b0e917ceaaa55d905170dcdb"} Nov 03 19:51:44.087076 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:44.087053 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276879a2-wwfnr" event={"ID":"944b8276-6b88-4dc7-8a0d-af8f8b879a39","Type":"ContainerStarted","Data":"1ee3205fcc0f585d0889816f5ed739e5b42e54c51656376496c468865a5ec4f9"} Nov 03 19:51:44.087230 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:44.087098 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276879a2-wwfnr" event={"ID":"944b8276-6b88-4dc7-8a0d-af8f8b879a39","Type":"ContainerStarted","Data":"bd0c959b5ec31e6fec2817af0eef5fd73412c121c99e2c797281fe0e60a778b8"} Nov 03 19:51:44.122921 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:44.122863 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276879a2-wwfnr" podStartSLOduration=2.122844768 podStartE2EDuration="2.122844768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:44.120530498 +0000 UTC m=+898.378327756" watchObservedRunningTime="2026-11-03 19:51:44.122844768 +0000 UTC m=+898.380642027" Nov 03 19:51:45.095241 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:45.095207 5059 generic.go:356] "Generic (PLEG): container finished" podID="944b8276-6b88-4dc7-8a0d-af8f8b879a39" containerID="1ee3205fcc0f585d0889816f5ed739e5b42e54c51656376496c468865a5ec4f9" exitCode=0 Nov 03 19:51:45.095649 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:45.095290 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276879a2-wwfnr" event={"ID":"944b8276-6b88-4dc7-8a0d-af8f8b879a39","Type":"ContainerDied","Data":"1ee3205fcc0f585d0889816f5ed739e5b42e54c51656376496c468865a5ec4f9"} Nov 03 19:51:45.262473 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:45.262447 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b79c04b-4tggl" Nov 03 19:51:45.401354 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:45.401272 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50b018bb-eb68-478d-9752-f3d07ed2cacf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b018bb-eb68-478d-9752-f3d07ed2cacf-maas-api-xks-ca\") pod \"50b018bb-eb68-478d-9752-f3d07ed2cacf\" (UID: \"50b018bb-eb68-478d-9752-f3d07ed2cacf\") " Nov 03 19:51:45.401354 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:45.401311 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b018bb-eb68-478d-9752-f3d07ed2cacf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b018bb-eb68-478d-9752-f3d07ed2cacf-maas-api-openshift-ca\") pod \"50b018bb-eb68-478d-9752-f3d07ed2cacf\" (UID: \"50b018bb-eb68-478d-9752-f3d07ed2cacf\") " Nov 03 19:51:45.401565 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:45.401397 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b018bb-eb68-478d-9752-f3d07ed2cacf-kube-api-access-bk5h6\" (UniqueName: \"kubernetes.io/projected/50b018bb-eb68-478d-9752-f3d07ed2cacf-kube-api-access-bk5h6\") pod \"50b018bb-eb68-478d-9752-f3d07ed2cacf\" (UID: \"50b018bb-eb68-478d-9752-f3d07ed2cacf\") " Nov 03 19:51:45.401741 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:45.401710 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b018bb-eb68-478d-9752-f3d07ed2cacf-maas-api-openshift-ca" pod "50b018bb-eb68-478d-9752-f3d07ed2cacf" (UID: "50b018bb-eb68-478d-9752-f3d07ed2cacf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:45.401978 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:45.401960 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b018bb-eb68-478d-9752-f3d07ed2cacf-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:45.403443 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:45.403418 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50b018bb-eb68-478d-9752-f3d07ed2cacf-maas-api-xks-ca" pod "50b018bb-eb68-478d-9752-f3d07ed2cacf" (UID: "50b018bb-eb68-478d-9752-f3d07ed2cacf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:51:45.403541 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:45.403487 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b018bb-eb68-478d-9752-f3d07ed2cacf-kube-api-access-bk5h6" pod "50b018bb-eb68-478d-9752-f3d07ed2cacf" (UID: "50b018bb-eb68-478d-9752-f3d07ed2cacf"). InnerVolumeSpecName "kube-api-access-bk5h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:45.503575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:45.503542 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk5h6\" (UniqueName: \"kubernetes.io/projected/50b018bb-eb68-478d-9752-f3d07ed2cacf-kube-api-access-bk5h6\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:45.503575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:45.503574 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b018bb-eb68-478d-9752-f3d07ed2cacf-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:46.013984 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:46.013956 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 19:51:46.030773 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:46.030748 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 19:51:46.102559 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:46.102533 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b79c04b-4tggl" Nov 03 19:51:46.102910 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:46.102571 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b79c04b-4tggl" event={"ID":"50b018bb-eb68-478d-9752-f3d07ed2cacf","Type":"ContainerDied","Data":"7bf504d67982987386b12e3b7cb499113a90bb1e9b0a7684060a5c0537368592"} Nov 03 19:51:46.102910 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:46.102614 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bf504d67982987386b12e3b7cb499113a90bb1e9b0a7684060a5c0537368592" Nov 03 19:51:46.249067 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:46.249044 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276879a2-wwfnr" Nov 03 19:51:46.313125 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:46.313092 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/944b8276-6b88-4dc7-8a0d-af8f8b879a39-kube-api-access-w6dpd\" (UniqueName: \"kubernetes.io/projected/944b8276-6b88-4dc7-8a0d-af8f8b879a39-kube-api-access-w6dpd\") pod \"944b8276-6b88-4dc7-8a0d-af8f8b879a39\" (UID: \"944b8276-6b88-4dc7-8a0d-af8f8b879a39\") " Nov 03 19:51:46.313386 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:46.313188 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/944b8276-6b88-4dc7-8a0d-af8f8b879a39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/944b8276-6b88-4dc7-8a0d-af8f8b879a39-maas-api-openshift-ca\") pod \"944b8276-6b88-4dc7-8a0d-af8f8b879a39\" (UID: \"944b8276-6b88-4dc7-8a0d-af8f8b879a39\") " Nov 03 19:51:46.313474 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:46.313381 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/944b8276-6b88-4dc7-8a0d-af8f8b879a39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/944b8276-6b88-4dc7-8a0d-af8f8b879a39-maas-api-xks-ca\") pod \"944b8276-6b88-4dc7-8a0d-af8f8b879a39\" (UID: \"944b8276-6b88-4dc7-8a0d-af8f8b879a39\") " Nov 03 19:51:46.313571 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:46.313548 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/944b8276-6b88-4dc7-8a0d-af8f8b879a39-maas-api-openshift-ca" pod "944b8276-6b88-4dc7-8a0d-af8f8b879a39" (UID: "944b8276-6b88-4dc7-8a0d-af8f8b879a39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:46.313837 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:46.313796 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/944b8276-6b88-4dc7-8a0d-af8f8b879a39-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:46.316243 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:46.316216 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944b8276-6b88-4dc7-8a0d-af8f8b879a39-kube-api-access-w6dpd" pod "944b8276-6b88-4dc7-8a0d-af8f8b879a39" (UID: "944b8276-6b88-4dc7-8a0d-af8f8b879a39"). InnerVolumeSpecName "kube-api-access-w6dpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:46.334152 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:46.334115 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/944b8276-6b88-4dc7-8a0d-af8f8b879a39-maas-api-xks-ca" pod "944b8276-6b88-4dc7-8a0d-af8f8b879a39" (UID: "944b8276-6b88-4dc7-8a0d-af8f8b879a39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:51:46.414986 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:46.414914 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/944b8276-6b88-4dc7-8a0d-af8f8b879a39-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:46.414986 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:46.414944 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6dpd\" (UniqueName: \"kubernetes.io/projected/944b8276-6b88-4dc7-8a0d-af8f8b879a39-kube-api-access-w6dpd\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:47.111362 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:47.111332 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276879a2-wwfnr" Nov 03 19:51:47.111362 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:47.111349 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276879a2-wwfnr" event={"ID":"944b8276-6b88-4dc7-8a0d-af8f8b879a39","Type":"ContainerDied","Data":"bd0c959b5ec31e6fec2817af0eef5fd73412c121c99e2c797281fe0e60a778b8"} Nov 03 19:51:47.111827 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:47.111401 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd0c959b5ec31e6fec2817af0eef5fd73412c121c99e2c797281fe0e60a778b8" Nov 03 19:51:49.848825 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:49.847745 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:51:49.866233 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:49.864747 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:51:50.695239 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:50.695189 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:51:50.880232 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:50.880183 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 19:51:50.881452 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:50.881426 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:51:50.920790 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:50.920747 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:51:50.930814 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:50.930769 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:51:50.949636 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:50.939363 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:51:50.949890 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:50.949875 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:51:51.220197 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:51.220114 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:51:52.290295 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:52.290263 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27e1f945-kmtg6"] Nov 03 19:51:52.290872 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:52.290856 5059 cpu_manager.go:414] "containerMap: removing container" podUID="50b018bb-eb68-478d-9752-f3d07ed2cacf" containerName="revoke-keys" Nov 03 19:51:52.290872 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:52.290873 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b018bb-eb68-478d-9752-f3d07ed2cacf" containerName="revoke-keys" Nov 03 19:51:52.290970 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:52.290887 5059 cpu_manager.go:414] "containerMap: removing container" podUID="944b8276-6b88-4dc7-8a0d-af8f8b879a39" containerName="revoke-keys" Nov 03 19:51:52.290970 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:52.290894 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="944b8276-6b88-4dc7-8a0d-af8f8b879a39" containerName="revoke-keys" Nov 03 19:51:52.291037 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:52.291005 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b018bb-eb68-478d-9752-f3d07ed2cacf" containerName="revoke-keys" Nov 03 19:51:52.291037 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:52.291015 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="944b8276-6b88-4dc7-8a0d-af8f8b879a39" containerName="revoke-keys" Nov 03 19:51:52.295519 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:52.295499 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e1f945-kmtg6" Nov 03 19:51:52.301228 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:52.301204 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:51:52.302547 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:52.302527 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 19:51:52.304413 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:52.304390 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27e1f945-kmtg6"] Nov 03 19:51:52.379925 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:52.379887 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da70ae49-8ec7-4a56-bb42-89310d7fffa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27e1f945-kmtg6\" (UID: \"da70ae49-8ec7-4a56-bb42-89310d7fffa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e1f945-kmtg6" Nov 03 19:51:52.380087 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:52.380021 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h8wm\" (UniqueName: \"kubernetes.io/projected/da70ae49-8ec7-4a56-bb42-89310d7fffa3-kube-api-access-4h8wm\") pod \"maas-api-revoke-sub-27e1f945-kmtg6\" (UID: \"da70ae49-8ec7-4a56-bb42-89310d7fffa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e1f945-kmtg6" Nov 03 19:51:52.380191 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:52.380172 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da70ae49-8ec7-4a56-bb42-89310d7fffa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27e1f945-kmtg6\" (UID: \"da70ae49-8ec7-4a56-bb42-89310d7fffa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e1f945-kmtg6" Nov 03 19:51:52.481415 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:52.481382 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da70ae49-8ec7-4a56-bb42-89310d7fffa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27e1f945-kmtg6\" (UID: \"da70ae49-8ec7-4a56-bb42-89310d7fffa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e1f945-kmtg6" Nov 03 19:51:52.481574 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:52.481487 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da70ae49-8ec7-4a56-bb42-89310d7fffa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27e1f945-kmtg6\" (UID: \"da70ae49-8ec7-4a56-bb42-89310d7fffa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e1f945-kmtg6" Nov 03 19:51:52.481626 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:52.481592 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h8wm\" (UniqueName: \"kubernetes.io/projected/da70ae49-8ec7-4a56-bb42-89310d7fffa3-kube-api-access-4h8wm\") pod \"maas-api-revoke-sub-27e1f945-kmtg6\" (UID: \"da70ae49-8ec7-4a56-bb42-89310d7fffa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e1f945-kmtg6" Nov 03 19:51:52.482156 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:52.482130 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da70ae49-8ec7-4a56-bb42-89310d7fffa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27e1f945-kmtg6\" (UID: \"da70ae49-8ec7-4a56-bb42-89310d7fffa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e1f945-kmtg6" Nov 03 19:51:52.483849 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:52.483826 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da70ae49-8ec7-4a56-bb42-89310d7fffa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27e1f945-kmtg6\" (UID: \"da70ae49-8ec7-4a56-bb42-89310d7fffa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e1f945-kmtg6" Nov 03 19:51:52.494600 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:52.494570 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h8wm\" (UniqueName: \"kubernetes.io/projected/da70ae49-8ec7-4a56-bb42-89310d7fffa3-kube-api-access-4h8wm\") pod \"maas-api-revoke-sub-27e1f945-kmtg6\" (UID: \"da70ae49-8ec7-4a56-bb42-89310d7fffa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e1f945-kmtg6" Nov 03 19:51:52.621352 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:52.621319 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e1f945-kmtg6" Nov 03 19:51:52.913460 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:51:52.913431 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda70ae49_8ec7_4a56_bb42_89310d7fffa3.slice/crio-e9bc7ee3a784a860e078872fdae2f6a672bda3e38d965931db9a76e4571d0dc6 WatchSource:0}: Error finding container e9bc7ee3a784a860e078872fdae2f6a672bda3e38d965931db9a76e4571d0dc6: Status 404 returned error can't find the container with id e9bc7ee3a784a860e078872fdae2f6a672bda3e38d965931db9a76e4571d0dc6 Nov 03 19:51:52.913716 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:52.913692 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27e1f945-kmtg6"] Nov 03 19:51:53.319719 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:53.319654 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e1f945-kmtg6" event={"ID":"da70ae49-8ec7-4a56-bb42-89310d7fffa3","Type":"ContainerStarted","Data":"557d36a75a0b862d875b6ef517e2887667b563f9858880dacb18ba7d5eb26388"} Nov 03 19:51:53.320089 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:53.319736 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e1f945-kmtg6" event={"ID":"da70ae49-8ec7-4a56-bb42-89310d7fffa3","Type":"ContainerStarted","Data":"e9bc7ee3a784a860e078872fdae2f6a672bda3e38d965931db9a76e4571d0dc6"} Nov 03 19:51:53.345289 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:53.345230 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e1f945-kmtg6" podStartSLOduration=1.345211459 podStartE2EDuration="1.345211459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:53.340888225 +0000 UTC m=+907.598685482" watchObservedRunningTime="2026-11-03 19:51:53.345211459 +0000 UTC m=+907.603008718" Nov 03 19:51:54.328248 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:54.328216 5059 generic.go:356] "Generic (PLEG): container finished" podID="da70ae49-8ec7-4a56-bb42-89310d7fffa3" containerID="557d36a75a0b862d875b6ef517e2887667b563f9858880dacb18ba7d5eb26388" exitCode=0 Nov 03 19:51:54.328646 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:54.328291 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e1f945-kmtg6" event={"ID":"da70ae49-8ec7-4a56-bb42-89310d7fffa3","Type":"ContainerDied","Data":"557d36a75a0b862d875b6ef517e2887667b563f9858880dacb18ba7d5eb26388"} Nov 03 19:51:55.364693 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:55.362451 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a17c84c4-j5bgn"] Nov 03 19:51:55.368066 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:55.367960 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17c84c4-j5bgn" Nov 03 19:51:55.378852 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:55.378822 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a17c84c4-j5bgn"] Nov 03 19:51:55.525180 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:55.525140 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62bf914d-4b66-4d52-ab52-db449030e0a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a17c84c4-j5bgn\" (UID: \"62bf914d-4b66-4d52-ab52-db449030e0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17c84c4-j5bgn" Nov 03 19:51:55.525313 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:55.525229 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn5vj\" (UniqueName: \"kubernetes.io/projected/62bf914d-4b66-4d52-ab52-db449030e0a6-kube-api-access-nn5vj\") pod \"maas-api-revoke-sub-a17c84c4-j5bgn\" (UID: \"62bf914d-4b66-4d52-ab52-db449030e0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17c84c4-j5bgn" Nov 03 19:51:55.525392 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:55.525367 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62bf914d-4b66-4d52-ab52-db449030e0a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a17c84c4-j5bgn\" (UID: \"62bf914d-4b66-4d52-ab52-db449030e0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17c84c4-j5bgn" Nov 03 19:51:55.535999 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:55.535973 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e1f945-kmtg6" Nov 03 19:51:55.626175 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:55.626093 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da70ae49-8ec7-4a56-bb42-89310d7fffa3-kube-api-access-4h8wm\" (UniqueName: \"kubernetes.io/projected/da70ae49-8ec7-4a56-bb42-89310d7fffa3-kube-api-access-4h8wm\") pod \"da70ae49-8ec7-4a56-bb42-89310d7fffa3\" (UID: \"da70ae49-8ec7-4a56-bb42-89310d7fffa3\") " Nov 03 19:51:55.626331 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:55.626222 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da70ae49-8ec7-4a56-bb42-89310d7fffa3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da70ae49-8ec7-4a56-bb42-89310d7fffa3-maas-api-xks-ca\") pod \"da70ae49-8ec7-4a56-bb42-89310d7fffa3\" (UID: \"da70ae49-8ec7-4a56-bb42-89310d7fffa3\") " Nov 03 19:51:55.626381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:55.626347 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da70ae49-8ec7-4a56-bb42-89310d7fffa3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da70ae49-8ec7-4a56-bb42-89310d7fffa3-maas-api-openshift-ca\") pod \"da70ae49-8ec7-4a56-bb42-89310d7fffa3\" (UID: \"da70ae49-8ec7-4a56-bb42-89310d7fffa3\") " Nov 03 19:51:55.626593 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:55.626567 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn5vj\" (UniqueName: \"kubernetes.io/projected/62bf914d-4b66-4d52-ab52-db449030e0a6-kube-api-access-nn5vj\") pod \"maas-api-revoke-sub-a17c84c4-j5bgn\" (UID: \"62bf914d-4b66-4d52-ab52-db449030e0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17c84c4-j5bgn" Nov 03 19:51:55.626715 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:55.626656 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62bf914d-4b66-4d52-ab52-db449030e0a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a17c84c4-j5bgn\" (UID: \"62bf914d-4b66-4d52-ab52-db449030e0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17c84c4-j5bgn" Nov 03 19:51:55.626780 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:55.626715 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da70ae49-8ec7-4a56-bb42-89310d7fffa3-maas-api-openshift-ca" pod "da70ae49-8ec7-4a56-bb42-89310d7fffa3" (UID: "da70ae49-8ec7-4a56-bb42-89310d7fffa3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:55.626918 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:55.626898 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62bf914d-4b66-4d52-ab52-db449030e0a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a17c84c4-j5bgn\" (UID: \"62bf914d-4b66-4d52-ab52-db449030e0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17c84c4-j5bgn" Nov 03 19:51:55.627012 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:55.626985 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da70ae49-8ec7-4a56-bb42-89310d7fffa3-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:55.627291 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:55.627272 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62bf914d-4b66-4d52-ab52-db449030e0a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a17c84c4-j5bgn\" (UID: \"62bf914d-4b66-4d52-ab52-db449030e0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17c84c4-j5bgn" Nov 03 19:51:55.628498 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:55.628463 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da70ae49-8ec7-4a56-bb42-89310d7fffa3-kube-api-access-4h8wm" pod "da70ae49-8ec7-4a56-bb42-89310d7fffa3" (UID: "da70ae49-8ec7-4a56-bb42-89310d7fffa3"). InnerVolumeSpecName "kube-api-access-4h8wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:55.628787 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:55.628759 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da70ae49-8ec7-4a56-bb42-89310d7fffa3-maas-api-xks-ca" pod "da70ae49-8ec7-4a56-bb42-89310d7fffa3" (UID: "da70ae49-8ec7-4a56-bb42-89310d7fffa3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:51:55.629197 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:55.629174 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62bf914d-4b66-4d52-ab52-db449030e0a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a17c84c4-j5bgn\" (UID: \"62bf914d-4b66-4d52-ab52-db449030e0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17c84c4-j5bgn" Nov 03 19:51:55.639313 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:55.639289 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn5vj\" (UniqueName: \"kubernetes.io/projected/62bf914d-4b66-4d52-ab52-db449030e0a6-kube-api-access-nn5vj\") pod \"maas-api-revoke-sub-a17c84c4-j5bgn\" (UID: \"62bf914d-4b66-4d52-ab52-db449030e0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17c84c4-j5bgn" Nov 03 19:51:55.728009 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:55.727983 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h8wm\" (UniqueName: \"kubernetes.io/projected/da70ae49-8ec7-4a56-bb42-89310d7fffa3-kube-api-access-4h8wm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:55.728124 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:55.728012 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da70ae49-8ec7-4a56-bb42-89310d7fffa3-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:55.732583 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:55.732565 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17c84c4-j5bgn" Nov 03 19:51:56.022804 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:51:56.022744 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62bf914d_4b66_4d52_ab52_db449030e0a6.slice/crio-5757bc75de39188bc337e7bd96b2f4e512b6efde34470252542cca2df16e7a35 WatchSource:0}: Error finding container 5757bc75de39188bc337e7bd96b2f4e512b6efde34470252542cca2df16e7a35: Status 404 returned error can't find the container with id 5757bc75de39188bc337e7bd96b2f4e512b6efde34470252542cca2df16e7a35 Nov 03 19:51:56.023262 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:56.023229 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a17c84c4-j5bgn"] Nov 03 19:51:56.029099 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:56.029077 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56391128-m7wnt"] Nov 03 19:51:56.029697 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:56.029683 5059 cpu_manager.go:414] "containerMap: removing container" podUID="da70ae49-8ec7-4a56-bb42-89310d7fffa3" containerName="revoke-keys" Nov 03 19:51:56.029743 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:56.029699 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da70ae49-8ec7-4a56-bb42-89310d7fffa3" containerName="revoke-keys" Nov 03 19:51:56.029807 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:56.029798 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="da70ae49-8ec7-4a56-bb42-89310d7fffa3" containerName="revoke-keys" Nov 03 19:51:56.034481 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:56.034464 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56391128-m7wnt" Nov 03 19:51:56.040873 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:56.040854 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56391128-m7wnt"] Nov 03 19:51:56.133693 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:56.133623 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b2e1bfd-d3e6-4005-bc08-47391001faf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56391128-m7wnt\" (UID: \"5b2e1bfd-d3e6-4005-bc08-47391001faf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56391128-m7wnt" Nov 03 19:51:56.133898 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:56.133872 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cqqm\" (UniqueName: \"kubernetes.io/projected/5b2e1bfd-d3e6-4005-bc08-47391001faf7-kube-api-access-6cqqm\") pod \"maas-api-revoke-sub-56391128-m7wnt\" (UID: \"5b2e1bfd-d3e6-4005-bc08-47391001faf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56391128-m7wnt" Nov 03 19:51:56.133978 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:56.133960 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b2e1bfd-d3e6-4005-bc08-47391001faf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56391128-m7wnt\" (UID: \"5b2e1bfd-d3e6-4005-bc08-47391001faf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56391128-m7wnt" Nov 03 19:51:56.234698 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:56.234654 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cqqm\" (UniqueName: \"kubernetes.io/projected/5b2e1bfd-d3e6-4005-bc08-47391001faf7-kube-api-access-6cqqm\") pod \"maas-api-revoke-sub-56391128-m7wnt\" (UID: \"5b2e1bfd-d3e6-4005-bc08-47391001faf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56391128-m7wnt" Nov 03 19:51:56.234871 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:56.234758 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b2e1bfd-d3e6-4005-bc08-47391001faf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56391128-m7wnt\" (UID: \"5b2e1bfd-d3e6-4005-bc08-47391001faf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56391128-m7wnt" Nov 03 19:51:56.234871 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:56.234799 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b2e1bfd-d3e6-4005-bc08-47391001faf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56391128-m7wnt\" (UID: \"5b2e1bfd-d3e6-4005-bc08-47391001faf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56391128-m7wnt" Nov 03 19:51:56.235556 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:56.235535 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b2e1bfd-d3e6-4005-bc08-47391001faf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56391128-m7wnt\" (UID: \"5b2e1bfd-d3e6-4005-bc08-47391001faf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56391128-m7wnt" Nov 03 19:51:56.259022 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:56.258997 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b2e1bfd-d3e6-4005-bc08-47391001faf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56391128-m7wnt\" (UID: \"5b2e1bfd-d3e6-4005-bc08-47391001faf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56391128-m7wnt" Nov 03 19:51:56.259227 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:56.259209 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cqqm\" (UniqueName: \"kubernetes.io/projected/5b2e1bfd-d3e6-4005-bc08-47391001faf7-kube-api-access-6cqqm\") pod \"maas-api-revoke-sub-56391128-m7wnt\" (UID: \"5b2e1bfd-d3e6-4005-bc08-47391001faf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56391128-m7wnt" Nov 03 19:51:56.341935 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:56.341905 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e1f945-kmtg6" Nov 03 19:51:56.342086 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:56.341940 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e1f945-kmtg6" event={"ID":"da70ae49-8ec7-4a56-bb42-89310d7fffa3","Type":"ContainerDied","Data":"e9bc7ee3a784a860e078872fdae2f6a672bda3e38d965931db9a76e4571d0dc6"} Nov 03 19:51:56.342086 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:56.341994 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9bc7ee3a784a860e078872fdae2f6a672bda3e38d965931db9a76e4571d0dc6" Nov 03 19:51:56.343202 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:56.343180 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17c84c4-j5bgn" event={"ID":"62bf914d-4b66-4d52-ab52-db449030e0a6","Type":"ContainerStarted","Data":"5757bc75de39188bc337e7bd96b2f4e512b6efde34470252542cca2df16e7a35"} Nov 03 19:51:56.355491 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:56.355469 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56391128-m7wnt" Nov 03 19:51:56.642822 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:56.642791 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56391128-m7wnt"] Nov 03 19:51:56.643672 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:51:56.643625 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2e1bfd_d3e6_4005_bc08_47391001faf7.slice/crio-057141bbcc850e5c15fba6857d392dd9fa81524ac2b42c24a36065ec125659b0 WatchSource:0}: Error finding container 057141bbcc850e5c15fba6857d392dd9fa81524ac2b42c24a36065ec125659b0: Status 404 returned error can't find the container with id 057141bbcc850e5c15fba6857d392dd9fa81524ac2b42c24a36065ec125659b0 Nov 03 19:51:57.351904 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:57.351871 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56391128-m7wnt" event={"ID":"5b2e1bfd-d3e6-4005-bc08-47391001faf7","Type":"ContainerStarted","Data":"3319d5ea3dcafc5635eeae15e137d718aa7e14cc6f1ebbb044c276b51e2330d6"} Nov 03 19:51:57.352082 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:57.351923 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56391128-m7wnt" event={"ID":"5b2e1bfd-d3e6-4005-bc08-47391001faf7","Type":"ContainerStarted","Data":"057141bbcc850e5c15fba6857d392dd9fa81524ac2b42c24a36065ec125659b0"} Nov 03 19:51:57.353417 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:57.353389 5059 generic.go:356] "Generic (PLEG): container finished" podID="62bf914d-4b66-4d52-ab52-db449030e0a6" containerID="2da6adaacd3f5eebd686a6a724ad7e81b471cb64db18684a94146f78687ff606" exitCode=0 Nov 03 19:51:57.353520 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:57.353467 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17c84c4-j5bgn" event={"ID":"62bf914d-4b66-4d52-ab52-db449030e0a6","Type":"ContainerDied","Data":"2da6adaacd3f5eebd686a6a724ad7e81b471cb64db18684a94146f78687ff606"} Nov 03 19:51:57.372332 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:57.372291 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56391128-m7wnt" podStartSLOduration=1.37227598 podStartE2EDuration="1.37227598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:57.369943708 +0000 UTC m=+911.627740964" watchObservedRunningTime="2026-11-03 19:51:57.37227598 +0000 UTC m=+911.630073260" Nov 03 19:51:58.362655 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:58.362620 5059 generic.go:356] "Generic (PLEG): container finished" podID="5b2e1bfd-d3e6-4005-bc08-47391001faf7" containerID="3319d5ea3dcafc5635eeae15e137d718aa7e14cc6f1ebbb044c276b51e2330d6" exitCode=0 Nov 03 19:51:58.363125 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:58.362709 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56391128-m7wnt" event={"ID":"5b2e1bfd-d3e6-4005-bc08-47391001faf7","Type":"ContainerDied","Data":"3319d5ea3dcafc5635eeae15e137d718aa7e14cc6f1ebbb044c276b51e2330d6"} Nov 03 19:51:58.548405 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:58.548377 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17c84c4-j5bgn" Nov 03 19:51:58.665072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:58.664994 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62bf914d-4b66-4d52-ab52-db449030e0a6-kube-api-access-nn5vj\" (UniqueName: \"kubernetes.io/projected/62bf914d-4b66-4d52-ab52-db449030e0a6-kube-api-access-nn5vj\") pod \"62bf914d-4b66-4d52-ab52-db449030e0a6\" (UID: \"62bf914d-4b66-4d52-ab52-db449030e0a6\") " Nov 03 19:51:58.665224 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:58.665081 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62bf914d-4b66-4d52-ab52-db449030e0a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62bf914d-4b66-4d52-ab52-db449030e0a6-maas-api-xks-ca\") pod \"62bf914d-4b66-4d52-ab52-db449030e0a6\" (UID: \"62bf914d-4b66-4d52-ab52-db449030e0a6\") " Nov 03 19:51:58.665224 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:58.665156 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62bf914d-4b66-4d52-ab52-db449030e0a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62bf914d-4b66-4d52-ab52-db449030e0a6-maas-api-openshift-ca\") pod \"62bf914d-4b66-4d52-ab52-db449030e0a6\" (UID: \"62bf914d-4b66-4d52-ab52-db449030e0a6\") " Nov 03 19:51:58.665486 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:58.665463 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62bf914d-4b66-4d52-ab52-db449030e0a6-maas-api-openshift-ca" pod "62bf914d-4b66-4d52-ab52-db449030e0a6" (UID: "62bf914d-4b66-4d52-ab52-db449030e0a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:58.667134 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:58.667110 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62bf914d-4b66-4d52-ab52-db449030e0a6-maas-api-xks-ca" pod "62bf914d-4b66-4d52-ab52-db449030e0a6" (UID: "62bf914d-4b66-4d52-ab52-db449030e0a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:51:58.667234 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:58.667187 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62bf914d-4b66-4d52-ab52-db449030e0a6-kube-api-access-nn5vj" pod "62bf914d-4b66-4d52-ab52-db449030e0a6" (UID: "62bf914d-4b66-4d52-ab52-db449030e0a6"). InnerVolumeSpecName "kube-api-access-nn5vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:58.767098 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:58.767066 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62bf914d-4b66-4d52-ab52-db449030e0a6-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:58.767098 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:58.767096 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62bf914d-4b66-4d52-ab52-db449030e0a6-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:58.767098 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:58.767107 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn5vj\" (UniqueName: \"kubernetes.io/projected/62bf914d-4b66-4d52-ab52-db449030e0a6-kube-api-access-nn5vj\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:59.368972 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.368938 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h"] Nov 03 19:51:59.369553 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.369538 5059 cpu_manager.go:414] "containerMap: removing container" podUID="62bf914d-4b66-4d52-ab52-db449030e0a6" containerName="revoke-keys" Nov 03 19:51:59.369610 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.369554 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62bf914d-4b66-4d52-ab52-db449030e0a6" containerName="revoke-keys" Nov 03 19:51:59.369725 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.369706 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="62bf914d-4b66-4d52-ab52-db449030e0a6" containerName="revoke-keys" Nov 03 19:51:59.372380 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.372355 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17c84c4-j5bgn" Nov 03 19:51:59.374192 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.373700 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17c84c4-j5bgn" event={"ID":"62bf914d-4b66-4d52-ab52-db449030e0a6","Type":"ContainerDied","Data":"5757bc75de39188bc337e7bd96b2f4e512b6efde34470252542cca2df16e7a35"} Nov 03 19:51:59.374192 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.373740 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5757bc75de39188bc337e7bd96b2f4e512b6efde34470252542cca2df16e7a35" Nov 03 19:51:59.374192 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.373807 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.390178 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.390127 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h"] Nov 03 19:51:59.476269 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.476238 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc57130b-c950-49f2-8a98-22c47bd98288-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.476405 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.476278 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.476405 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.476299 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc57130b-c950-49f2-8a98-22c47bd98288-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.476482 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.476416 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc57130b-c950-49f2-8a98-22c47bd98288-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.476530 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.476514 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.476568 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.476536 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.476568 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.476552 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9shs\" (UniqueName: \"kubernetes.io/projected/fc57130b-c950-49f2-8a98-22c47bd98288-kube-api-access-p9shs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.476733 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.476720 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.476780 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.476769 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.552313 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.552289 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56391128-m7wnt" Nov 03 19:51:59.578560 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.578531 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.578699 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.578620 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.578755 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.578718 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc57130b-c950-49f2-8a98-22c47bd98288-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.578755 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.578747 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.578840 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.578778 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc57130b-c950-49f2-8a98-22c47bd98288-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.578840 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.578827 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc57130b-c950-49f2-8a98-22c47bd98288-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.578941 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.578872 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.578941 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.578884 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.578941 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.578894 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.578941 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.578918 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9shs\" (UniqueName: \"kubernetes.io/projected/fc57130b-c950-49f2-8a98-22c47bd98288-kube-api-access-p9shs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.579145 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.578959 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.579145 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.579096 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.579265 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.579246 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.579583 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.579556 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc57130b-c950-49f2-8a98-22c47bd98288-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.581087 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.581063 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.581525 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.581496 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc57130b-c950-49f2-8a98-22c47bd98288-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.591951 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.591926 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc57130b-c950-49f2-8a98-22c47bd98288-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.595356 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.595332 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9shs\" (UniqueName: \"kubernetes.io/projected/fc57130b-c950-49f2-8a98-22c47bd98288-kube-api-access-p9shs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.679998 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.679934 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b2e1bfd-d3e6-4005-bc08-47391001faf7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b2e1bfd-d3e6-4005-bc08-47391001faf7-maas-api-openshift-ca\") pod \"5b2e1bfd-d3e6-4005-bc08-47391001faf7\" (UID: \"5b2e1bfd-d3e6-4005-bc08-47391001faf7\") " Nov 03 19:51:59.680127 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.680029 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b2e1bfd-d3e6-4005-bc08-47391001faf7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b2e1bfd-d3e6-4005-bc08-47391001faf7-maas-api-xks-ca\") pod \"5b2e1bfd-d3e6-4005-bc08-47391001faf7\" (UID: \"5b2e1bfd-d3e6-4005-bc08-47391001faf7\") " Nov 03 19:51:59.680196 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.680157 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b2e1bfd-d3e6-4005-bc08-47391001faf7-kube-api-access-6cqqm\" (UniqueName: \"kubernetes.io/projected/5b2e1bfd-d3e6-4005-bc08-47391001faf7-kube-api-access-6cqqm\") pod \"5b2e1bfd-d3e6-4005-bc08-47391001faf7\" (UID: \"5b2e1bfd-d3e6-4005-bc08-47391001faf7\") " Nov 03 19:51:59.680269 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.680248 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b2e1bfd-d3e6-4005-bc08-47391001faf7-maas-api-openshift-ca" pod "5b2e1bfd-d3e6-4005-bc08-47391001faf7" (UID: "5b2e1bfd-d3e6-4005-bc08-47391001faf7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:51:59.680518 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.680501 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b2e1bfd-d3e6-4005-bc08-47391001faf7-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:59.681988 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.681968 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b2e1bfd-d3e6-4005-bc08-47391001faf7-maas-api-xks-ca" pod "5b2e1bfd-d3e6-4005-bc08-47391001faf7" (UID: "5b2e1bfd-d3e6-4005-bc08-47391001faf7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:51:59.682079 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.682064 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b2e1bfd-d3e6-4005-bc08-47391001faf7-kube-api-access-6cqqm" pod "5b2e1bfd-d3e6-4005-bc08-47391001faf7" (UID: "5b2e1bfd-d3e6-4005-bc08-47391001faf7"). InnerVolumeSpecName "kube-api-access-6cqqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:51:59.702013 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.701989 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:51:59.781913 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.781880 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cqqm\" (UniqueName: \"kubernetes.io/projected/5b2e1bfd-d3e6-4005-bc08-47391001faf7-kube-api-access-6cqqm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:51:59.782029 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.781920 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b2e1bfd-d3e6-4005-bc08-47391001faf7-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:00.002704 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.994284 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h"] Nov 03 19:52:00.002704 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.998152 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:52:00.002704 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.998249 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:52:00.002704 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:51:59.998290 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:52:00.382324 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:00.382291 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56391128-m7wnt" Nov 03 19:52:00.382324 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:00.382318 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56391128-m7wnt" event={"ID":"5b2e1bfd-d3e6-4005-bc08-47391001faf7","Type":"ContainerDied","Data":"057141bbcc850e5c15fba6857d392dd9fa81524ac2b42c24a36065ec125659b0"} Nov 03 19:52:00.382838 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:00.382371 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="057141bbcc850e5c15fba6857d392dd9fa81524ac2b42c24a36065ec125659b0" Nov 03 19:52:00.387991 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:00.387963 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" event={"ID":"fc57130b-c950-49f2-8a98-22c47bd98288","Type":"ContainerStarted","Data":"7598d2be586d16550aff9c80d082d04770669cb98c756f3811527543a1ac8d0f"} Nov 03 19:52:00.388165 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:00.388010 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" event={"ID":"fc57130b-c950-49f2-8a98-22c47bd98288","Type":"ContainerStarted","Data":"55e7a86a0d53f29a6fa9de295bc44317de3712a5ba1053f7449d5d580aac5ed8"} Nov 03 19:52:00.418073 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:00.418030 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" podStartSLOduration=1.418017468 podStartE2EDuration="1.418017468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:00.413537607 +0000 UTC m=+914.671334863" watchObservedRunningTime="2026-11-03 19:52:00.418017468 +0000 UTC m=+914.675814725" Nov 03 19:52:00.702364 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:00.702290 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:52:01.704577 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:01.704533 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 19:52:01.705054 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:01.704602 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" podUID="fc57130b-c950-49f2-8a98-22c47bd98288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 19:52:02.702738 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:02.702702 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 19:52:02.702914 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:02.702752 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" podUID="fc57130b-c950-49f2-8a98-22c47bd98288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 19:52:03.703499 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:03.703466 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 19:52:03.703872 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:03.703524 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" podUID="fc57130b-c950-49f2-8a98-22c47bd98288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 19:52:04.702838 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:04.702441 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 19:52:04.702838 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:04.702499 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" podUID="fc57130b-c950-49f2-8a98-22c47bd98288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 19:52:05.712687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:05.706061 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 19:52:05.712687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:05.706170 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" podUID="fc57130b-c950-49f2-8a98-22c47bd98288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 19:52:06.690315 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:06.687753 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 19:52:06.711118 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:06.710751 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:06.721717 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:06.717463 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 19:52:06.721717 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:06.717524 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" podUID="fc57130b-c950-49f2-8a98-22c47bd98288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 19:52:06.847691 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:06.846739 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 19:52:06.990779 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:06.959231 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f49ab41a-npc7f"] Nov 03 19:52:06.990779 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:06.960202 5059 cpu_manager.go:414] "containerMap: removing container" podUID="5b2e1bfd-d3e6-4005-bc08-47391001faf7" containerName="revoke-keys" Nov 03 19:52:06.990779 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:06.960224 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b2e1bfd-d3e6-4005-bc08-47391001faf7" containerName="revoke-keys" Nov 03 19:52:06.990779 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:06.960398 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b2e1bfd-d3e6-4005-bc08-47391001faf7" containerName="revoke-keys" Nov 03 19:52:06.990779 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:06.964404 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49ab41a-npc7f" Nov 03 19:52:06.990779 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:06.964529 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f49ab41a-npc7f"] Nov 03 19:52:06.992354 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:06.992327 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:52:07.117687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:06.994000 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 19:52:07.186688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:07.012502 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpp84\" (UniqueName: \"kubernetes.io/projected/6b6308dc-ce77-434c-b414-55561e6d3192-kube-api-access-cpp84\") pod \"maas-api-revoke-sub-f49ab41a-npc7f\" (UID: \"6b6308dc-ce77-434c-b414-55561e6d3192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49ab41a-npc7f" Nov 03 19:52:07.193690 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:07.187015 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6308dc-ce77-434c-b414-55561e6d3192-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f49ab41a-npc7f\" (UID: \"6b6308dc-ce77-434c-b414-55561e6d3192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49ab41a-npc7f" Nov 03 19:52:07.194010 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:07.193981 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6308dc-ce77-434c-b414-55561e6d3192-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f49ab41a-npc7f\" (UID: \"6b6308dc-ce77-434c-b414-55561e6d3192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49ab41a-npc7f" Nov 03 19:52:07.297034 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:07.296935 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpp84\" (UniqueName: \"kubernetes.io/projected/6b6308dc-ce77-434c-b414-55561e6d3192-kube-api-access-cpp84\") pod \"maas-api-revoke-sub-f49ab41a-npc7f\" (UID: \"6b6308dc-ce77-434c-b414-55561e6d3192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49ab41a-npc7f" Nov 03 19:52:07.302551 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:07.302420 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6308dc-ce77-434c-b414-55561e6d3192-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f49ab41a-npc7f\" (UID: \"6b6308dc-ce77-434c-b414-55561e6d3192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49ab41a-npc7f" Nov 03 19:52:07.302982 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:07.302960 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6308dc-ce77-434c-b414-55561e6d3192-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f49ab41a-npc7f\" (UID: \"6b6308dc-ce77-434c-b414-55561e6d3192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49ab41a-npc7f" Nov 03 19:52:07.315730 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:07.304282 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6308dc-ce77-434c-b414-55561e6d3192-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f49ab41a-npc7f\" (UID: \"6b6308dc-ce77-434c-b414-55561e6d3192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49ab41a-npc7f" Nov 03 19:52:07.328690 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:07.322114 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6308dc-ce77-434c-b414-55561e6d3192-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f49ab41a-npc7f\" (UID: \"6b6308dc-ce77-434c-b414-55561e6d3192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49ab41a-npc7f" Nov 03 19:52:07.386691 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:07.378383 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpp84\" (UniqueName: \"kubernetes.io/projected/6b6308dc-ce77-434c-b414-55561e6d3192-kube-api-access-cpp84\") pod \"maas-api-revoke-sub-f49ab41a-npc7f\" (UID: \"6b6308dc-ce77-434c-b414-55561e6d3192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49ab41a-npc7f" Nov 03 19:52:07.482686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:07.481903 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49ab41a-npc7f" Nov 03 19:52:07.770728 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:07.745587 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:07.785740 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:07.772025 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 19:52:07.922770 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:07.873480 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:07.922770 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:07.906817 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 19:52:07.922770 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:07.921506 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" podUID="fc57130b-c950-49f2-8a98-22c47bd98288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 19:52:07.984711 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:07.959293 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:52:08.010991 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:07.960772 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 19:52:08.030691 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:07.967542 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:52:08.050688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:08.045953 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" podUID="40b88c7d-08e9-4203-9ebd-2c9f69b884dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:08.239745 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:08.230832 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:08.324160 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:08.324120 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:52:08.484708 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:08.476167 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" podUID="cef4417c-3d04-4d65-a5bc-6ec1af6f09bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:08.484708 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:08.369432 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:52:08.484708 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:08.476282 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" podUID="c2d80995-5f1c-46db-8d71-a130cb92e279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:08.759265 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:08.759218 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 19:52:08.803023 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:08.802992 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 19:52:08.824727 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:08.824678 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" podUID="fc57130b-c950-49f2-8a98-22c47bd98288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 19:52:08.868897 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:08.868867 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:52:08.869481 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:08.869464 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:52:08.942732 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:08.942694 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:08.962782 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:08.943370 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:09.012971 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:09.012870 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:09.061687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:09.018556 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:52:09.258717 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:09.258425 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:09.264363 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:09.259048 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:52:09.440695 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:09.438509 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:52:09.568734 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:09.567729 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:52:09.568734 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:09.567831 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:52:09.843738 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:09.837405 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 19:52:09.843738 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:09.837469 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" podUID="fc57130b-c950-49f2-8a98-22c47bd98288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 19:52:09.958311 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:09.955793 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:10.141909 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:10.141871 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49ab41a-npc7f" event={"ID":"6b6308dc-ce77-434c-b414-55561e6d3192","Type":"ContainerStarted","Data":"93d234152b20fa22113f2ad5ca61ba612f3d7dbff01b9d0c15a14717e7b59779"} Nov 03 19:52:10.142156 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:10.142140 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f49ab41a-npc7f"] Nov 03 19:52:10.381850 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:10.381755 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:10.443685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:10.438691 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:10.528684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:10.523990 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:52:10.637778 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:10.636553 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:52:10.722262 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:52:10.722228 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:52:10.812780 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:10.812748 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:52:10.813225 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:10.813204 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:52:10.814375 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:10.814348 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 19:52:10.915494 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:10.915398 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05c4e799-gwzk8"] Nov 03 19:52:10.922286 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:10.921020 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c4e799-gwzk8" Nov 03 19:52:10.932065 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:10.932013 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05c4e799-gwzk8"] Nov 03 19:52:11.029769 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:11.029735 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhkjh\" (UniqueName: \"kubernetes.io/projected/96eb415b-e41d-45a3-ab0b-a52329ee3622-kube-api-access-jhkjh\") pod \"maas-api-revoke-sub-05c4e799-gwzk8\" (UID: \"96eb415b-e41d-45a3-ab0b-a52329ee3622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c4e799-gwzk8" Nov 03 19:52:11.029954 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:11.029849 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96eb415b-e41d-45a3-ab0b-a52329ee3622-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05c4e799-gwzk8\" (UID: \"96eb415b-e41d-45a3-ab0b-a52329ee3622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c4e799-gwzk8" Nov 03 19:52:11.030042 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:11.030020 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96eb415b-e41d-45a3-ab0b-a52329ee3622-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05c4e799-gwzk8\" (UID: \"96eb415b-e41d-45a3-ab0b-a52329ee3622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c4e799-gwzk8" Nov 03 19:52:11.131093 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:11.130947 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96eb415b-e41d-45a3-ab0b-a52329ee3622-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05c4e799-gwzk8\" (UID: \"96eb415b-e41d-45a3-ab0b-a52329ee3622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c4e799-gwzk8" Nov 03 19:52:11.131093 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:11.131091 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96eb415b-e41d-45a3-ab0b-a52329ee3622-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05c4e799-gwzk8\" (UID: \"96eb415b-e41d-45a3-ab0b-a52329ee3622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c4e799-gwzk8" Nov 03 19:52:11.131683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:11.131224 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhkjh\" (UniqueName: \"kubernetes.io/projected/96eb415b-e41d-45a3-ab0b-a52329ee3622-kube-api-access-jhkjh\") pod \"maas-api-revoke-sub-05c4e799-gwzk8\" (UID: \"96eb415b-e41d-45a3-ab0b-a52329ee3622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c4e799-gwzk8" Nov 03 19:52:11.132846 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:11.132551 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96eb415b-e41d-45a3-ab0b-a52329ee3622-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05c4e799-gwzk8\" (UID: \"96eb415b-e41d-45a3-ab0b-a52329ee3622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c4e799-gwzk8" Nov 03 19:52:11.135024 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:11.135000 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96eb415b-e41d-45a3-ab0b-a52329ee3622-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05c4e799-gwzk8\" (UID: \"96eb415b-e41d-45a3-ab0b-a52329ee3622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c4e799-gwzk8" Nov 03 19:52:11.153577 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:11.153544 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhkjh\" (UniqueName: \"kubernetes.io/projected/96eb415b-e41d-45a3-ab0b-a52329ee3622-kube-api-access-jhkjh\") pod \"maas-api-revoke-sub-05c4e799-gwzk8\" (UID: \"96eb415b-e41d-45a3-ab0b-a52329ee3622\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c4e799-gwzk8" Nov 03 19:52:11.155347 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:11.155315 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49ab41a-npc7f" event={"ID":"6b6308dc-ce77-434c-b414-55561e6d3192","Type":"ContainerStarted","Data":"b584bffd1d14ca3ade450f93059b444a13328671b77f2598cb2af2a34a34d9e3"} Nov 03 19:52:11.188739 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:11.184861 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49ab41a-npc7f" podStartSLOduration=5.184841271 podStartE2EDuration="5.184841271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:11.178902809 +0000 UTC m=+925.436700068" watchObservedRunningTime="2026-11-03 19:52:11.184841271 +0000 UTC m=+925.442638530" Nov 03 19:52:11.250371 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:11.250337 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c4e799-gwzk8" Nov 03 19:52:11.544924 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:11.544893 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05c4e799-gwzk8"] Nov 03 19:52:11.545089 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:52:11.545066 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96eb415b_e41d_45a3_ab0b_a52329ee3622.slice/crio-249fd46b1d562b31b6a89d892ecc04c17e2cde35d30a8bb0392072ce8fcda429 WatchSource:0}: Error finding container 249fd46b1d562b31b6a89d892ecc04c17e2cde35d30a8bb0392072ce8fcda429: Status 404 returned error can't find the container with id 249fd46b1d562b31b6a89d892ecc04c17e2cde35d30a8bb0392072ce8fcda429 Nov 03 19:52:12.170236 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:12.170133 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c4e799-gwzk8" event={"ID":"96eb415b-e41d-45a3-ab0b-a52329ee3622","Type":"ContainerStarted","Data":"99fd9fcecfbd449cf96d2c83c139b6ca85b554e622be3e8403a5c1b6000c9911"} Nov 03 19:52:12.170236 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:12.170219 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c4e799-gwzk8" event={"ID":"96eb415b-e41d-45a3-ab0b-a52329ee3622","Type":"ContainerStarted","Data":"249fd46b1d562b31b6a89d892ecc04c17e2cde35d30a8bb0392072ce8fcda429"} Nov 03 19:52:12.172133 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:12.171970 5059 generic.go:356] "Generic (PLEG): container finished" podID="6b6308dc-ce77-434c-b414-55561e6d3192" containerID="b584bffd1d14ca3ade450f93059b444a13328671b77f2598cb2af2a34a34d9e3" exitCode=0 Nov 03 19:52:12.172133 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:12.172048 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49ab41a-npc7f" event={"ID":"6b6308dc-ce77-434c-b414-55561e6d3192","Type":"ContainerDied","Data":"b584bffd1d14ca3ade450f93059b444a13328671b77f2598cb2af2a34a34d9e3"} Nov 03 19:52:12.196010 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:12.195956 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c4e799-gwzk8" podStartSLOduration=2.195942798 podStartE2EDuration="2.195942798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:12.193574942 +0000 UTC m=+926.451372197" watchObservedRunningTime="2026-11-03 19:52:12.195942798 +0000 UTC m=+926.453740054" Nov 03 19:52:13.181912 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:13.181880 5059 generic.go:356] "Generic (PLEG): container finished" podID="96eb415b-e41d-45a3-ab0b-a52329ee3622" containerID="99fd9fcecfbd449cf96d2c83c139b6ca85b554e622be3e8403a5c1b6000c9911" exitCode=0 Nov 03 19:52:13.182366 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:13.181963 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c4e799-gwzk8" event={"ID":"96eb415b-e41d-45a3-ab0b-a52329ee3622","Type":"ContainerDied","Data":"99fd9fcecfbd449cf96d2c83c139b6ca85b554e622be3e8403a5c1b6000c9911"} Nov 03 19:52:13.332325 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:13.332302 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49ab41a-npc7f" Nov 03 19:52:13.372800 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:13.372767 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b6308dc-ce77-434c-b414-55561e6d3192-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6308dc-ce77-434c-b414-55561e6d3192-maas-api-openshift-ca\") pod \"6b6308dc-ce77-434c-b414-55561e6d3192\" (UID: \"6b6308dc-ce77-434c-b414-55561e6d3192\") " Nov 03 19:52:13.372942 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:13.372826 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b6308dc-ce77-434c-b414-55561e6d3192-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6308dc-ce77-434c-b414-55561e6d3192-maas-api-xks-ca\") pod \"6b6308dc-ce77-434c-b414-55561e6d3192\" (UID: \"6b6308dc-ce77-434c-b414-55561e6d3192\") " Nov 03 19:52:13.373006 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:13.372938 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b6308dc-ce77-434c-b414-55561e6d3192-kube-api-access-cpp84\" (UniqueName: \"kubernetes.io/projected/6b6308dc-ce77-434c-b414-55561e6d3192-kube-api-access-cpp84\") pod \"6b6308dc-ce77-434c-b414-55561e6d3192\" (UID: \"6b6308dc-ce77-434c-b414-55561e6d3192\") " Nov 03 19:52:13.373175 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:13.373147 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6308dc-ce77-434c-b414-55561e6d3192-maas-api-openshift-ca" pod "6b6308dc-ce77-434c-b414-55561e6d3192" (UID: "6b6308dc-ce77-434c-b414-55561e6d3192"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:13.373611 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:13.373587 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6308dc-ce77-434c-b414-55561e6d3192-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:13.375004 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:13.374977 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6308dc-ce77-434c-b414-55561e6d3192-kube-api-access-cpp84" pod "6b6308dc-ce77-434c-b414-55561e6d3192" (UID: "6b6308dc-ce77-434c-b414-55561e6d3192"). InnerVolumeSpecName "kube-api-access-cpp84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:13.375105 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:13.375066 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b6308dc-ce77-434c-b414-55561e6d3192-maas-api-xks-ca" pod "6b6308dc-ce77-434c-b414-55561e6d3192" (UID: "6b6308dc-ce77-434c-b414-55561e6d3192"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:13.474975 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:13.474912 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6308dc-ce77-434c-b414-55561e6d3192-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:13.474975 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:13.474935 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpp84\" (UniqueName: \"kubernetes.io/projected/6b6308dc-ce77-434c-b414-55561e6d3192-kube-api-access-cpp84\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:14.192094 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:14.192006 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49ab41a-npc7f" Nov 03 19:52:14.192094 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:14.192026 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49ab41a-npc7f" event={"ID":"6b6308dc-ce77-434c-b414-55561e6d3192","Type":"ContainerDied","Data":"93d234152b20fa22113f2ad5ca61ba612f3d7dbff01b9d0c15a14717e7b59779"} Nov 03 19:52:14.192094 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:14.192084 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93d234152b20fa22113f2ad5ca61ba612f3d7dbff01b9d0c15a14717e7b59779" Nov 03 19:52:14.350830 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:14.350800 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c4e799-gwzk8" Nov 03 19:52:14.387289 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:14.387254 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96eb415b-e41d-45a3-ab0b-a52329ee3622-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96eb415b-e41d-45a3-ab0b-a52329ee3622-maas-api-xks-ca\") pod \"96eb415b-e41d-45a3-ab0b-a52329ee3622\" (UID: \"96eb415b-e41d-45a3-ab0b-a52329ee3622\") " Nov 03 19:52:14.387474 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:14.387323 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96eb415b-e41d-45a3-ab0b-a52329ee3622-kube-api-access-jhkjh\" (UniqueName: \"kubernetes.io/projected/96eb415b-e41d-45a3-ab0b-a52329ee3622-kube-api-access-jhkjh\") pod \"96eb415b-e41d-45a3-ab0b-a52329ee3622\" (UID: \"96eb415b-e41d-45a3-ab0b-a52329ee3622\") " Nov 03 19:52:14.387541 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:14.387520 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96eb415b-e41d-45a3-ab0b-a52329ee3622-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96eb415b-e41d-45a3-ab0b-a52329ee3622-maas-api-openshift-ca\") pod \"96eb415b-e41d-45a3-ab0b-a52329ee3622\" (UID: \"96eb415b-e41d-45a3-ab0b-a52329ee3622\") " Nov 03 19:52:14.388077 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:14.388039 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96eb415b-e41d-45a3-ab0b-a52329ee3622-maas-api-openshift-ca" pod "96eb415b-e41d-45a3-ab0b-a52329ee3622" (UID: "96eb415b-e41d-45a3-ab0b-a52329ee3622"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:14.389763 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:14.389729 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96eb415b-e41d-45a3-ab0b-a52329ee3622-kube-api-access-jhkjh" pod "96eb415b-e41d-45a3-ab0b-a52329ee3622" (UID: "96eb415b-e41d-45a3-ab0b-a52329ee3622"). InnerVolumeSpecName "kube-api-access-jhkjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:14.390164 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:14.390136 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96eb415b-e41d-45a3-ab0b-a52329ee3622-maas-api-xks-ca" pod "96eb415b-e41d-45a3-ab0b-a52329ee3622" (UID: "96eb415b-e41d-45a3-ab0b-a52329ee3622"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:14.489063 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:14.488984 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96eb415b-e41d-45a3-ab0b-a52329ee3622-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:14.489063 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:14.489014 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96eb415b-e41d-45a3-ab0b-a52329ee3622-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:14.489063 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:14.489025 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhkjh\" (UniqueName: \"kubernetes.io/projected/96eb415b-e41d-45a3-ab0b-a52329ee3622-kube-api-access-jhkjh\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:15.203131 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.203093 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c4e799-gwzk8" event={"ID":"96eb415b-e41d-45a3-ab0b-a52329ee3622","Type":"ContainerDied","Data":"249fd46b1d562b31b6a89d892ecc04c17e2cde35d30a8bb0392072ce8fcda429"} Nov 03 19:52:15.203544 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.203150 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="249fd46b1d562b31b6a89d892ecc04c17e2cde35d30a8bb0392072ce8fcda429" Nov 03 19:52:15.203544 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.203104 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05c4e799-gwzk8" Nov 03 19:52:15.222608 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.222581 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc342650-bsnk9"] Nov 03 19:52:15.223224 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.223210 5059 cpu_manager.go:414] "containerMap: removing container" podUID="6b6308dc-ce77-434c-b414-55561e6d3192" containerName="revoke-keys" Nov 03 19:52:15.223267 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.223225 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b6308dc-ce77-434c-b414-55561e6d3192" containerName="revoke-keys" Nov 03 19:52:15.223267 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.223236 5059 cpu_manager.go:414] "containerMap: removing container" podUID="96eb415b-e41d-45a3-ab0b-a52329ee3622" containerName="revoke-keys" Nov 03 19:52:15.223267 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.223241 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96eb415b-e41d-45a3-ab0b-a52329ee3622" containerName="revoke-keys" Nov 03 19:52:15.223357 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.223342 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b6308dc-ce77-434c-b414-55561e6d3192" containerName="revoke-keys" Nov 03 19:52:15.223357 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.223351 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="96eb415b-e41d-45a3-ab0b-a52329ee3622" containerName="revoke-keys" Nov 03 19:52:15.227037 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.227018 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc342650-bsnk9" Nov 03 19:52:15.231218 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.231188 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:52:15.231345 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.231306 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 19:52:15.237562 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.237540 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc342650-bsnk9"] Nov 03 19:52:15.300803 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.300771 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg7dk\" (UniqueName: \"kubernetes.io/projected/f1893940-65b3-420f-9d5a-730248788311-kube-api-access-wg7dk\") pod \"maas-api-revoke-sub-cc342650-bsnk9\" (UID: \"f1893940-65b3-420f-9d5a-730248788311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc342650-bsnk9" Nov 03 19:52:15.300961 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.300856 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1893940-65b3-420f-9d5a-730248788311-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc342650-bsnk9\" (UID: \"f1893940-65b3-420f-9d5a-730248788311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc342650-bsnk9" Nov 03 19:52:15.300961 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.300923 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1893940-65b3-420f-9d5a-730248788311-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc342650-bsnk9\" (UID: \"f1893940-65b3-420f-9d5a-730248788311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc342650-bsnk9" Nov 03 19:52:15.402499 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.402471 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg7dk\" (UniqueName: \"kubernetes.io/projected/f1893940-65b3-420f-9d5a-730248788311-kube-api-access-wg7dk\") pod \"maas-api-revoke-sub-cc342650-bsnk9\" (UID: \"f1893940-65b3-420f-9d5a-730248788311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc342650-bsnk9" Nov 03 19:52:15.402652 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.402541 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1893940-65b3-420f-9d5a-730248788311-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc342650-bsnk9\" (UID: \"f1893940-65b3-420f-9d5a-730248788311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc342650-bsnk9" Nov 03 19:52:15.402652 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.402565 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1893940-65b3-420f-9d5a-730248788311-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc342650-bsnk9\" (UID: \"f1893940-65b3-420f-9d5a-730248788311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc342650-bsnk9" Nov 03 19:52:15.403183 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.403164 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1893940-65b3-420f-9d5a-730248788311-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc342650-bsnk9\" (UID: \"f1893940-65b3-420f-9d5a-730248788311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc342650-bsnk9" Nov 03 19:52:15.405003 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.404982 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1893940-65b3-420f-9d5a-730248788311-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc342650-bsnk9\" (UID: \"f1893940-65b3-420f-9d5a-730248788311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc342650-bsnk9" Nov 03 19:52:15.417188 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.417166 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg7dk\" (UniqueName: \"kubernetes.io/projected/f1893940-65b3-420f-9d5a-730248788311-kube-api-access-wg7dk\") pod \"maas-api-revoke-sub-cc342650-bsnk9\" (UID: \"f1893940-65b3-420f-9d5a-730248788311\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc342650-bsnk9" Nov 03 19:52:15.544311 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.544242 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc342650-bsnk9" Nov 03 19:52:15.830627 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:52:15.830591 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1893940_65b3_420f_9d5a_730248788311.slice/crio-53e01ea579061e01922c35bcc05cebc0f4b501347ea4fa6fa669cd22816d43bf WatchSource:0}: Error finding container 53e01ea579061e01922c35bcc05cebc0f4b501347ea4fa6fa669cd22816d43bf: Status 404 returned error can't find the container with id 53e01ea579061e01922c35bcc05cebc0f4b501347ea4fa6fa669cd22816d43bf Nov 03 19:52:15.830874 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:15.830850 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc342650-bsnk9"] Nov 03 19:52:16.211574 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:16.211541 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc342650-bsnk9" event={"ID":"f1893940-65b3-420f-9d5a-730248788311","Type":"ContainerStarted","Data":"13779577fa40a32b08441e2d1eb1c8db93189edcd27b739284273db8de4cd358"} Nov 03 19:52:16.211959 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:16.211602 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc342650-bsnk9" event={"ID":"f1893940-65b3-420f-9d5a-730248788311","Type":"ContainerStarted","Data":"53e01ea579061e01922c35bcc05cebc0f4b501347ea4fa6fa669cd22816d43bf"} Nov 03 19:52:16.234170 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:16.234124 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc342650-bsnk9" podStartSLOduration=1.234108986 podStartE2EDuration="1.234108986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:16.2337191 +0000 UTC m=+930.491516358" watchObservedRunningTime="2026-11-03 19:52:16.234108986 +0000 UTC m=+930.491906242" Nov 03 19:52:17.219915 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:17.219878 5059 generic.go:356] "Generic (PLEG): container finished" podID="f1893940-65b3-420f-9d5a-730248788311" containerID="13779577fa40a32b08441e2d1eb1c8db93189edcd27b739284273db8de4cd358" exitCode=0 Nov 03 19:52:17.220302 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:17.219967 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc342650-bsnk9" event={"ID":"f1893940-65b3-420f-9d5a-730248788311","Type":"ContainerDied","Data":"13779577fa40a32b08441e2d1eb1c8db93189edcd27b739284273db8de4cd358"} Nov 03 19:52:18.426333 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:18.426299 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc342650-bsnk9" Nov 03 19:52:18.537809 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:18.537727 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1893940-65b3-420f-9d5a-730248788311-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1893940-65b3-420f-9d5a-730248788311-maas-api-xks-ca\") pod \"f1893940-65b3-420f-9d5a-730248788311\" (UID: \"f1893940-65b3-420f-9d5a-730248788311\") " Nov 03 19:52:18.537809 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:18.537784 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1893940-65b3-420f-9d5a-730248788311-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1893940-65b3-420f-9d5a-730248788311-maas-api-openshift-ca\") pod \"f1893940-65b3-420f-9d5a-730248788311\" (UID: \"f1893940-65b3-420f-9d5a-730248788311\") " Nov 03 19:52:18.537991 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:18.537834 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1893940-65b3-420f-9d5a-730248788311-kube-api-access-wg7dk\" (UniqueName: \"kubernetes.io/projected/f1893940-65b3-420f-9d5a-730248788311-kube-api-access-wg7dk\") pod \"f1893940-65b3-420f-9d5a-730248788311\" (UID: \"f1893940-65b3-420f-9d5a-730248788311\") " Nov 03 19:52:18.538164 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:18.538135 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1893940-65b3-420f-9d5a-730248788311-maas-api-openshift-ca" pod "f1893940-65b3-420f-9d5a-730248788311" (UID: "f1893940-65b3-420f-9d5a-730248788311"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:18.538409 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:18.538391 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1893940-65b3-420f-9d5a-730248788311-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:18.539882 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:18.539860 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1893940-65b3-420f-9d5a-730248788311-kube-api-access-wg7dk" pod "f1893940-65b3-420f-9d5a-730248788311" (UID: "f1893940-65b3-420f-9d5a-730248788311"). InnerVolumeSpecName "kube-api-access-wg7dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:18.539933 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:18.539901 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1893940-65b3-420f-9d5a-730248788311-maas-api-xks-ca" pod "f1893940-65b3-420f-9d5a-730248788311" (UID: "f1893940-65b3-420f-9d5a-730248788311"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:18.639747 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:18.639707 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1893940-65b3-420f-9d5a-730248788311-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:18.639747 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:18.639740 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg7dk\" (UniqueName: \"kubernetes.io/projected/f1893940-65b3-420f-9d5a-730248788311-kube-api-access-wg7dk\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:19.236595 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:19.236563 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc342650-bsnk9" Nov 03 19:52:19.236774 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:19.236602 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc342650-bsnk9" event={"ID":"f1893940-65b3-420f-9d5a-730248788311","Type":"ContainerDied","Data":"53e01ea579061e01922c35bcc05cebc0f4b501347ea4fa6fa669cd22816d43bf"} Nov 03 19:52:19.236774 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:19.236649 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53e01ea579061e01922c35bcc05cebc0f4b501347ea4fa6fa669cd22816d43bf" Nov 03 19:52:21.445840 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:21.444840 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:21.540701 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:21.540085 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 03 19:52:21.567798 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:21.567746 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:21.569160 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:21.569125 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 19:52:21.809837 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:21.809734 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded" Nov 03 19:52:22.448035 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:22.445748 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 19:52:22.448035 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:22.446200 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:52:22.592086 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:22.592040 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 19:52:22.609692 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:22.599749 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:22.687689 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:22.600227 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 03 19:52:22.687689 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:22.610007 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:52:22.687689 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:22.648899 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:52:22.687689 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:22.648959 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:52:23.270642 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:23.269376 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:23.301715 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:23.300971 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": context deadline exceeded" Nov 03 19:52:23.451637 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:23.374785 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:52:23.451637 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:23.374850 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" podUID="40b88c7d-08e9-4203-9ebd-2c9f69b884dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:23.451637 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:23.374924 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:52:23.451637 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:23.374945 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" podUID="c2d80995-5f1c-46db-8d71-a130cb92e279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:23.451637 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:23.450041 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:23.485838 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:23.460182 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:52:23.485838 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:23.460241 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" podUID="cef4417c-3d04-4d65-a5bc-6ec1af6f09bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:23.604832 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:23.604781 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:23.697826 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:23.694422 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 19:52:23.709506 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:23.708778 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 03 19:52:23.964773 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:23.942289 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:24.164190 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:24.146367 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:52:24.279469 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:24.278945 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:52:24.287212 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:24.287187 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:52:24.288077 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:24.288056 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:52:27.585353 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:27.585315 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff81e09f-vlzbb"] Nov 03 19:52:27.586003 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:27.585986 5059 cpu_manager.go:414] "containerMap: removing container" podUID="f1893940-65b3-420f-9d5a-730248788311" containerName="revoke-keys" Nov 03 19:52:27.586049 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:27.586005 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1893940-65b3-420f-9d5a-730248788311" containerName="revoke-keys" Nov 03 19:52:27.586105 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:27.586096 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1893940-65b3-420f-9d5a-730248788311" containerName="revoke-keys" Nov 03 19:52:27.589849 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:27.589834 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff81e09f-vlzbb" Nov 03 19:52:27.594793 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:27.594766 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:52:27.594966 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:27.594821 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 19:52:27.599308 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:27.598770 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff81e09f-vlzbb"] Nov 03 19:52:27.645767 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:27.645734 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fadac4c8-e38d-43e5-89b7-b4460521b6f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff81e09f-vlzbb\" (UID: \"fadac4c8-e38d-43e5-89b7-b4460521b6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff81e09f-vlzbb" Nov 03 19:52:27.645992 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:27.645909 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fadac4c8-e38d-43e5-89b7-b4460521b6f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff81e09f-vlzbb\" (UID: \"fadac4c8-e38d-43e5-89b7-b4460521b6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff81e09f-vlzbb" Nov 03 19:52:27.646071 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:27.646046 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htn74\" (UniqueName: \"kubernetes.io/projected/fadac4c8-e38d-43e5-89b7-b4460521b6f1-kube-api-access-htn74\") pod \"maas-api-revoke-sub-ff81e09f-vlzbb\" (UID: \"fadac4c8-e38d-43e5-89b7-b4460521b6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff81e09f-vlzbb" Nov 03 19:52:27.747178 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:27.747153 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fadac4c8-e38d-43e5-89b7-b4460521b6f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff81e09f-vlzbb\" (UID: \"fadac4c8-e38d-43e5-89b7-b4460521b6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff81e09f-vlzbb" Nov 03 19:52:27.747307 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:27.747203 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htn74\" (UniqueName: \"kubernetes.io/projected/fadac4c8-e38d-43e5-89b7-b4460521b6f1-kube-api-access-htn74\") pod \"maas-api-revoke-sub-ff81e09f-vlzbb\" (UID: \"fadac4c8-e38d-43e5-89b7-b4460521b6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff81e09f-vlzbb" Nov 03 19:52:27.747307 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:27.747285 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fadac4c8-e38d-43e5-89b7-b4460521b6f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff81e09f-vlzbb\" (UID: \"fadac4c8-e38d-43e5-89b7-b4460521b6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff81e09f-vlzbb" Nov 03 19:52:27.747865 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:27.747843 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fadac4c8-e38d-43e5-89b7-b4460521b6f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff81e09f-vlzbb\" (UID: \"fadac4c8-e38d-43e5-89b7-b4460521b6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff81e09f-vlzbb" Nov 03 19:52:27.749496 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:27.749477 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fadac4c8-e38d-43e5-89b7-b4460521b6f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff81e09f-vlzbb\" (UID: \"fadac4c8-e38d-43e5-89b7-b4460521b6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff81e09f-vlzbb" Nov 03 19:52:27.759767 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:27.759746 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htn74\" (UniqueName: \"kubernetes.io/projected/fadac4c8-e38d-43e5-89b7-b4460521b6f1-kube-api-access-htn74\") pod \"maas-api-revoke-sub-ff81e09f-vlzbb\" (UID: \"fadac4c8-e38d-43e5-89b7-b4460521b6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff81e09f-vlzbb" Nov 03 19:52:27.905377 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:27.905311 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff81e09f-vlzbb" Nov 03 19:52:28.190209 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:52:28.190169 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfadac4c8_e38d_43e5_89b7_b4460521b6f1.slice/crio-767cc4a86031bde5ec1f3b250c6c27cce74a6563dc895d2e8952c5de1913db73 WatchSource:0}: Error finding container 767cc4a86031bde5ec1f3b250c6c27cce74a6563dc895d2e8952c5de1913db73: Status 404 returned error can't find the container with id 767cc4a86031bde5ec1f3b250c6c27cce74a6563dc895d2e8952c5de1913db73 Nov 03 19:52:28.191012 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:28.190985 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff81e09f-vlzbb"] Nov 03 19:52:28.558002 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:28.557967 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff81e09f-vlzbb" event={"ID":"fadac4c8-e38d-43e5-89b7-b4460521b6f1","Type":"ContainerStarted","Data":"767cc4a86031bde5ec1f3b250c6c27cce74a6563dc895d2e8952c5de1913db73"} Nov 03 19:52:29.566929 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:29.566839 5059 generic.go:356] "Generic (PLEG): container finished" podID="fadac4c8-e38d-43e5-89b7-b4460521b6f1" containerID="0d325d1d6cb0e466855f3633d6c42e87c402787f80961edd9a8b1e39361399e8" exitCode=0 Nov 03 19:52:29.566929 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:29.566895 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff81e09f-vlzbb" event={"ID":"fadac4c8-e38d-43e5-89b7-b4460521b6f1","Type":"ContainerDied","Data":"0d325d1d6cb0e466855f3633d6c42e87c402787f80961edd9a8b1e39361399e8"} Nov 03 19:52:30.775828 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:30.775794 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff81e09f-vlzbb" Nov 03 19:52:30.884782 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:30.884711 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fadac4c8-e38d-43e5-89b7-b4460521b6f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fadac4c8-e38d-43e5-89b7-b4460521b6f1-maas-api-openshift-ca\") pod \"fadac4c8-e38d-43e5-89b7-b4460521b6f1\" (UID: \"fadac4c8-e38d-43e5-89b7-b4460521b6f1\") " Nov 03 19:52:30.884782 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:30.884775 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fadac4c8-e38d-43e5-89b7-b4460521b6f1-kube-api-access-htn74\" (UniqueName: \"kubernetes.io/projected/fadac4c8-e38d-43e5-89b7-b4460521b6f1-kube-api-access-htn74\") pod \"fadac4c8-e38d-43e5-89b7-b4460521b6f1\" (UID: \"fadac4c8-e38d-43e5-89b7-b4460521b6f1\") " Nov 03 19:52:30.884954 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:30.884857 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fadac4c8-e38d-43e5-89b7-b4460521b6f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fadac4c8-e38d-43e5-89b7-b4460521b6f1-maas-api-xks-ca\") pod \"fadac4c8-e38d-43e5-89b7-b4460521b6f1\" (UID: \"fadac4c8-e38d-43e5-89b7-b4460521b6f1\") " Nov 03 19:52:30.885143 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:30.885116 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fadac4c8-e38d-43e5-89b7-b4460521b6f1-maas-api-openshift-ca" pod "fadac4c8-e38d-43e5-89b7-b4460521b6f1" (UID: "fadac4c8-e38d-43e5-89b7-b4460521b6f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:30.885403 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:30.885385 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fadac4c8-e38d-43e5-89b7-b4460521b6f1-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:30.886886 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:30.886858 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fadac4c8-e38d-43e5-89b7-b4460521b6f1-maas-api-xks-ca" pod "fadac4c8-e38d-43e5-89b7-b4460521b6f1" (UID: "fadac4c8-e38d-43e5-89b7-b4460521b6f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:30.886984 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:30.886923 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fadac4c8-e38d-43e5-89b7-b4460521b6f1-kube-api-access-htn74" pod "fadac4c8-e38d-43e5-89b7-b4460521b6f1" (UID: "fadac4c8-e38d-43e5-89b7-b4460521b6f1"). InnerVolumeSpecName "kube-api-access-htn74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:30.986333 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:30.986306 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htn74\" (UniqueName: \"kubernetes.io/projected/fadac4c8-e38d-43e5-89b7-b4460521b6f1-kube-api-access-htn74\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:30.986333 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:30.986333 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fadac4c8-e38d-43e5-89b7-b4460521b6f1-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:31.014814 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:31.014790 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-464e0cfc-xghmv"] Nov 03 19:52:31.015393 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:31.015380 5059 cpu_manager.go:414] "containerMap: removing container" podUID="fadac4c8-e38d-43e5-89b7-b4460521b6f1" containerName="revoke-keys" Nov 03 19:52:31.015443 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:31.015394 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fadac4c8-e38d-43e5-89b7-b4460521b6f1" containerName="revoke-keys" Nov 03 19:52:31.015507 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:31.015498 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="fadac4c8-e38d-43e5-89b7-b4460521b6f1" containerName="revoke-keys" Nov 03 19:52:31.019007 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:31.018992 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-464e0cfc-xghmv" Nov 03 19:52:31.028753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:31.028727 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-464e0cfc-xghmv"] Nov 03 19:52:31.087905 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:31.087876 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21882329-e6fc-4657-a4da-036fee584872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-464e0cfc-xghmv\" (UID: \"21882329-e6fc-4657-a4da-036fee584872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-464e0cfc-xghmv" Nov 03 19:52:31.088088 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:31.088063 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65fqz\" (UniqueName: \"kubernetes.io/projected/21882329-e6fc-4657-a4da-036fee584872-kube-api-access-65fqz\") pod \"maas-api-revoke-sub-464e0cfc-xghmv\" (UID: \"21882329-e6fc-4657-a4da-036fee584872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-464e0cfc-xghmv" Nov 03 19:52:31.088157 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:31.088103 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21882329-e6fc-4657-a4da-036fee584872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-464e0cfc-xghmv\" (UID: \"21882329-e6fc-4657-a4da-036fee584872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-464e0cfc-xghmv" Nov 03 19:52:31.189599 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:31.189508 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21882329-e6fc-4657-a4da-036fee584872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-464e0cfc-xghmv\" (UID: \"21882329-e6fc-4657-a4da-036fee584872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-464e0cfc-xghmv" Nov 03 19:52:31.189785 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:31.189647 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65fqz\" (UniqueName: \"kubernetes.io/projected/21882329-e6fc-4657-a4da-036fee584872-kube-api-access-65fqz\") pod \"maas-api-revoke-sub-464e0cfc-xghmv\" (UID: \"21882329-e6fc-4657-a4da-036fee584872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-464e0cfc-xghmv" Nov 03 19:52:31.189785 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:31.189717 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21882329-e6fc-4657-a4da-036fee584872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-464e0cfc-xghmv\" (UID: \"21882329-e6fc-4657-a4da-036fee584872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-464e0cfc-xghmv" Nov 03 19:52:31.190378 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:31.190350 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21882329-e6fc-4657-a4da-036fee584872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-464e0cfc-xghmv\" (UID: \"21882329-e6fc-4657-a4da-036fee584872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-464e0cfc-xghmv" Nov 03 19:52:31.191962 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:31.191944 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21882329-e6fc-4657-a4da-036fee584872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-464e0cfc-xghmv\" (UID: \"21882329-e6fc-4657-a4da-036fee584872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-464e0cfc-xghmv" Nov 03 19:52:31.201823 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:31.201801 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65fqz\" (UniqueName: \"kubernetes.io/projected/21882329-e6fc-4657-a4da-036fee584872-kube-api-access-65fqz\") pod \"maas-api-revoke-sub-464e0cfc-xghmv\" (UID: \"21882329-e6fc-4657-a4da-036fee584872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-464e0cfc-xghmv" Nov 03 19:52:31.335907 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:31.335467 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-464e0cfc-xghmv" Nov 03 19:52:31.584224 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:31.584194 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff81e09f-vlzbb" Nov 03 19:52:31.584388 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:31.584235 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff81e09f-vlzbb" event={"ID":"fadac4c8-e38d-43e5-89b7-b4460521b6f1","Type":"ContainerDied","Data":"767cc4a86031bde5ec1f3b250c6c27cce74a6563dc895d2e8952c5de1913db73"} Nov 03 19:52:31.584388 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:31.584282 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="767cc4a86031bde5ec1f3b250c6c27cce74a6563dc895d2e8952c5de1913db73" Nov 03 19:52:31.634560 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:31.634531 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-464e0cfc-xghmv"] Nov 03 19:52:31.634786 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:52:31.634765 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21882329_e6fc_4657_a4da_036fee584872.slice/crio-fde2bcf8d2f259021c6821a5ca924ff073d4738f9c1818b8cf138a713dbda7ea WatchSource:0}: Error finding container fde2bcf8d2f259021c6821a5ca924ff073d4738f9c1818b8cf138a713dbda7ea: Status 404 returned error can't find the container with id fde2bcf8d2f259021c6821a5ca924ff073d4738f9c1818b8cf138a713dbda7ea Nov 03 19:52:32.592598 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:32.592571 5059 generic.go:356] "Generic (PLEG): container finished" podID="21882329-e6fc-4657-a4da-036fee584872" containerID="91c16f32baf87d5cbf46c52ad29a472d3bcba41d8f42bd22e8fd5e8960113c9a" exitCode=0 Nov 03 19:52:32.592949 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:32.592683 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-464e0cfc-xghmv" event={"ID":"21882329-e6fc-4657-a4da-036fee584872","Type":"ContainerDied","Data":"91c16f32baf87d5cbf46c52ad29a472d3bcba41d8f42bd22e8fd5e8960113c9a"} Nov 03 19:52:32.592949 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:32.592744 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-464e0cfc-xghmv" event={"ID":"21882329-e6fc-4657-a4da-036fee584872","Type":"ContainerStarted","Data":"fde2bcf8d2f259021c6821a5ca924ff073d4738f9c1818b8cf138a713dbda7ea"} Nov 03 19:52:33.776330 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:33.776303 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-464e0cfc-xghmv" Nov 03 19:52:33.925783 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:33.925708 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21882329-e6fc-4657-a4da-036fee584872-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21882329-e6fc-4657-a4da-036fee584872-maas-api-xks-ca\") pod \"21882329-e6fc-4657-a4da-036fee584872\" (UID: \"21882329-e6fc-4657-a4da-036fee584872\") " Nov 03 19:52:33.925783 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:33.925765 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21882329-e6fc-4657-a4da-036fee584872-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21882329-e6fc-4657-a4da-036fee584872-maas-api-openshift-ca\") pod \"21882329-e6fc-4657-a4da-036fee584872\" (UID: \"21882329-e6fc-4657-a4da-036fee584872\") " Nov 03 19:52:33.925963 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:33.925803 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21882329-e6fc-4657-a4da-036fee584872-kube-api-access-65fqz\" (UniqueName: \"kubernetes.io/projected/21882329-e6fc-4657-a4da-036fee584872-kube-api-access-65fqz\") pod \"21882329-e6fc-4657-a4da-036fee584872\" (UID: \"21882329-e6fc-4657-a4da-036fee584872\") " Nov 03 19:52:33.926173 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:33.926141 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21882329-e6fc-4657-a4da-036fee584872-maas-api-openshift-ca" pod "21882329-e6fc-4657-a4da-036fee584872" (UID: "21882329-e6fc-4657-a4da-036fee584872"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:33.926459 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:33.926435 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21882329-e6fc-4657-a4da-036fee584872-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:33.927878 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:33.927853 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21882329-e6fc-4657-a4da-036fee584872-maas-api-xks-ca" pod "21882329-e6fc-4657-a4da-036fee584872" (UID: "21882329-e6fc-4657-a4da-036fee584872"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:33.927950 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:33.927930 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21882329-e6fc-4657-a4da-036fee584872-kube-api-access-65fqz" pod "21882329-e6fc-4657-a4da-036fee584872" (UID: "21882329-e6fc-4657-a4da-036fee584872"). InnerVolumeSpecName "kube-api-access-65fqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:34.027280 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:34.027239 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21882329-e6fc-4657-a4da-036fee584872-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:34.027280 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:34.027281 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65fqz\" (UniqueName: \"kubernetes.io/projected/21882329-e6fc-4657-a4da-036fee584872-kube-api-access-65fqz\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:34.610097 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:34.610067 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-464e0cfc-xghmv" Nov 03 19:52:34.610296 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:34.610061 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-464e0cfc-xghmv" event={"ID":"21882329-e6fc-4657-a4da-036fee584872","Type":"ContainerDied","Data":"fde2bcf8d2f259021c6821a5ca924ff073d4738f9c1818b8cf138a713dbda7ea"} Nov 03 19:52:34.610296 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:34.610190 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fde2bcf8d2f259021c6821a5ca924ff073d4738f9c1818b8cf138a713dbda7ea" Nov 03 19:52:39.618838 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:39.618758 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 19:52:39.703791 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:39.703741 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:39.721816 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:39.721761 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:39.950871 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:39.950765 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:40.319351 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:40.319299 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:40.622896 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:40.621730 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 19:52:40.635685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:40.623785 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:52:40.764822 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:40.764769 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:40.765894 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:40.765865 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 19:52:40.767276 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:40.767256 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:52:40.767492 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:40.767477 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:52:40.962980 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:40.962875 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:40.984686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:40.967276 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:52:41.426864 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:41.426819 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 19:52:41.452875 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:41.452833 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:52:41.568687 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:52:41.512793 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:52:41.847684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:41.846890 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 03 19:52:41.847684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:41.847282 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 19:52:42.024682 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:41.999417 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:42.037980 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.037935 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 19:52:42.380087 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.380044 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:42.433684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.428938 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:42.609154 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.609119 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:52:42.609443 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.609422 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:52:42.653247 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.609698 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:52:42.662893 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.662867 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:52:42.712384 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.712356 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d9da1c-cslfr"] Nov 03 19:52:42.713482 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.713455 5059 cpu_manager.go:414] "containerMap: removing container" podUID="21882329-e6fc-4657-a4da-036fee584872" containerName="revoke-keys" Nov 03 19:52:42.713627 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.713612 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21882329-e6fc-4657-a4da-036fee584872" containerName="revoke-keys" Nov 03 19:52:42.713912 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.713895 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="21882329-e6fc-4657-a4da-036fee584872" containerName="revoke-keys" Nov 03 19:52:42.728116 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.728098 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d9da1c-cslfr" Nov 03 19:52:42.739714 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.739693 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d9da1c-cslfr"] Nov 03 19:52:42.740191 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.740176 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:52:42.743244 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.743083 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:52:42.743244 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.743213 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 19:52:42.748088 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.748059 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/588f1533-b01d-4fd9-a022-afe753fa6a52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01d9da1c-cslfr\" (UID: \"588f1533-b01d-4fd9-a022-afe753fa6a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d9da1c-cslfr" Nov 03 19:52:42.748303 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.748283 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/588f1533-b01d-4fd9-a022-afe753fa6a52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01d9da1c-cslfr\" (UID: \"588f1533-b01d-4fd9-a022-afe753fa6a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d9da1c-cslfr" Nov 03 19:52:42.748818 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.748791 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs7sl\" (UniqueName: \"kubernetes.io/projected/588f1533-b01d-4fd9-a022-afe753fa6a52-kube-api-access-vs7sl\") pod \"maas-api-revoke-sub-01d9da1c-cslfr\" (UID: \"588f1533-b01d-4fd9-a022-afe753fa6a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d9da1c-cslfr" Nov 03 19:52:42.850407 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.850369 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs7sl\" (UniqueName: \"kubernetes.io/projected/588f1533-b01d-4fd9-a022-afe753fa6a52-kube-api-access-vs7sl\") pod \"maas-api-revoke-sub-01d9da1c-cslfr\" (UID: \"588f1533-b01d-4fd9-a022-afe753fa6a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d9da1c-cslfr" Nov 03 19:52:42.851188 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.851158 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/588f1533-b01d-4fd9-a022-afe753fa6a52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01d9da1c-cslfr\" (UID: \"588f1533-b01d-4fd9-a022-afe753fa6a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d9da1c-cslfr" Nov 03 19:52:42.851396 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.851375 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/588f1533-b01d-4fd9-a022-afe753fa6a52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01d9da1c-cslfr\" (UID: \"588f1533-b01d-4fd9-a022-afe753fa6a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d9da1c-cslfr" Nov 03 19:52:42.851989 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.851964 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/588f1533-b01d-4fd9-a022-afe753fa6a52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01d9da1c-cslfr\" (UID: \"588f1533-b01d-4fd9-a022-afe753fa6a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d9da1c-cslfr" Nov 03 19:52:42.868182 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.867058 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/588f1533-b01d-4fd9-a022-afe753fa6a52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01d9da1c-cslfr\" (UID: \"588f1533-b01d-4fd9-a022-afe753fa6a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d9da1c-cslfr" Nov 03 19:52:42.933742 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:42.927107 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs7sl\" (UniqueName: \"kubernetes.io/projected/588f1533-b01d-4fd9-a022-afe753fa6a52-kube-api-access-vs7sl\") pod \"maas-api-revoke-sub-01d9da1c-cslfr\" (UID: \"588f1533-b01d-4fd9-a022-afe753fa6a52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d9da1c-cslfr" Nov 03 19:52:43.082243 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:43.082206 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d9da1c-cslfr" Nov 03 19:52:43.369183 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:43.369151 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d9da1c-cslfr"] Nov 03 19:52:43.370699 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:52:43.370638 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod588f1533_b01d_4fd9_a022_afe753fa6a52.slice/crio-8e852ff51abec44fc200683dc78267e1e338ec174e46871e0ebcde031d5398de WatchSource:0}: Error finding container 8e852ff51abec44fc200683dc78267e1e338ec174e46871e0ebcde031d5398de: Status 404 returned error can't find the container with id 8e852ff51abec44fc200683dc78267e1e338ec174e46871e0ebcde031d5398de Nov 03 19:52:44.381205 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.381172 5059 generic.go:356] "Generic (PLEG): container finished" podID="588f1533-b01d-4fd9-a022-afe753fa6a52" containerID="f82cdb32968c387a7fe51d66808be881c555ea2cb20f58f591b2aa51f7e27a51" exitCode=0 Nov 03 19:52:44.381588 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.381232 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d9da1c-cslfr" event={"ID":"588f1533-b01d-4fd9-a022-afe753fa6a52","Type":"ContainerDied","Data":"f82cdb32968c387a7fe51d66808be881c555ea2cb20f58f591b2aa51f7e27a51"} Nov 03 19:52:44.381588 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.381286 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d9da1c-cslfr" event={"ID":"588f1533-b01d-4fd9-a022-afe753fa6a52","Type":"ContainerStarted","Data":"8e852ff51abec44fc200683dc78267e1e338ec174e46871e0ebcde031d5398de"} Nov 03 19:52:44.418330 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.418293 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fe573bf-wc7sh"] Nov 03 19:52:44.423268 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.423246 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe573bf-wc7sh" Nov 03 19:52:44.437884 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.436860 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fe573bf-wc7sh"] Nov 03 19:52:44.471195 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.471160 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs"] Nov 03 19:52:44.473396 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.473367 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d4f80e-7c28-44c4-9f87-4d690d6a5e35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fe573bf-wc7sh\" (UID: \"b6d4f80e-7c28-44c4-9f87-4d690d6a5e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe573bf-wc7sh" Nov 03 19:52:44.473544 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.473494 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d4f80e-7c28-44c4-9f87-4d690d6a5e35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fe573bf-wc7sh\" (UID: \"b6d4f80e-7c28-44c4-9f87-4d690d6a5e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe573bf-wc7sh" Nov 03 19:52:44.473614 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.473597 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6w5s\" (UniqueName: \"kubernetes.io/projected/b6d4f80e-7c28-44c4-9f87-4d690d6a5e35-kube-api-access-x6w5s\") pod \"maas-api-revoke-sub-9fe573bf-wc7sh\" (UID: \"b6d4f80e-7c28-44c4-9f87-4d690d6a5e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe573bf-wc7sh" Nov 03 19:52:44.477027 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.477006 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.494583 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.494559 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs"] Nov 03 19:52:44.575147 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.575112 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.575312 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.575156 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.575312 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.575174 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5992c515-22ba-4191-b5f6-8181c094340c-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.575312 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.575261 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db29t\" (UniqueName: \"kubernetes.io/projected/5992c515-22ba-4191-b5f6-8181c094340c-kube-api-access-db29t\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.575312 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.575298 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d4f80e-7c28-44c4-9f87-4d690d6a5e35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fe573bf-wc7sh\" (UID: \"b6d4f80e-7c28-44c4-9f87-4d690d6a5e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe573bf-wc7sh" Nov 03 19:52:44.575463 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.575313 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5992c515-22ba-4191-b5f6-8181c094340c-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.575498 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.575455 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.575536 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.575501 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d4f80e-7c28-44c4-9f87-4d690d6a5e35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fe573bf-wc7sh\" (UID: \"b6d4f80e-7c28-44c4-9f87-4d690d6a5e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe573bf-wc7sh" Nov 03 19:52:44.575614 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.575593 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.575670 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.575631 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6w5s\" (UniqueName: \"kubernetes.io/projected/b6d4f80e-7c28-44c4-9f87-4d690d6a5e35-kube-api-access-x6w5s\") pod \"maas-api-revoke-sub-9fe573bf-wc7sh\" (UID: \"b6d4f80e-7c28-44c4-9f87-4d690d6a5e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe573bf-wc7sh" Nov 03 19:52:44.575670 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.575656 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.575844 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.575823 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5992c515-22ba-4191-b5f6-8181c094340c-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.576083 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.576065 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d4f80e-7c28-44c4-9f87-4d690d6a5e35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fe573bf-wc7sh\" (UID: \"b6d4f80e-7c28-44c4-9f87-4d690d6a5e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe573bf-wc7sh" Nov 03 19:52:44.577531 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.577514 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d4f80e-7c28-44c4-9f87-4d690d6a5e35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fe573bf-wc7sh\" (UID: \"b6d4f80e-7c28-44c4-9f87-4d690d6a5e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe573bf-wc7sh" Nov 03 19:52:44.592643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.592618 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6w5s\" (UniqueName: \"kubernetes.io/projected/b6d4f80e-7c28-44c4-9f87-4d690d6a5e35-kube-api-access-x6w5s\") pod \"maas-api-revoke-sub-9fe573bf-wc7sh\" (UID: \"b6d4f80e-7c28-44c4-9f87-4d690d6a5e35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe573bf-wc7sh" Nov 03 19:52:44.677503 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.677473 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.677712 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.677519 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.677712 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.677539 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5992c515-22ba-4191-b5f6-8181c094340c-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.677712 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.677563 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db29t\" (UniqueName: \"kubernetes.io/projected/5992c515-22ba-4191-b5f6-8181c094340c-kube-api-access-db29t\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.677712 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.677604 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5992c515-22ba-4191-b5f6-8181c094340c-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.677712 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.677648 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.678000 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.677784 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.678000 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.677837 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.678000 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.677866 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5992c515-22ba-4191-b5f6-8181c094340c-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.678157 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.678003 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.678157 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.678111 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.678274 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.678197 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.678331 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.678293 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.678570 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.678553 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5992c515-22ba-4191-b5f6-8181c094340c-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.680398 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.680373 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.680583 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.680562 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5992c515-22ba-4191-b5f6-8181c094340c-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.692771 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.692739 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db29t\" (UniqueName: \"kubernetes.io/projected/5992c515-22ba-4191-b5f6-8181c094340c-kube-api-access-db29t\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.694491 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.694469 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5992c515-22ba-4191-b5f6-8181c094340c-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.740581 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.740554 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe573bf-wc7sh" Nov 03 19:52:44.798219 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.798176 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:44.848452 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.847737 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr"] Nov 03 19:52:44.855352 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.855328 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.871049 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.869040 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr"] Nov 03 19:52:44.891559 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.881001 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.891559 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.881167 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc4bk\" (UniqueName: \"kubernetes.io/projected/9d117f5b-5d47-488d-9d42-0ee87596813b-kube-api-access-fc4bk\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.891559 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.881253 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.891559 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.881281 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.891559 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.881355 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.891559 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.881442 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.891559 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.881470 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d117f5b-5d47-488d-9d42-0ee87596813b-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.891559 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.881611 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.891559 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.882083 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.983635 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.983600 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.983827 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.983650 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.983827 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.983722 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.983827 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.983749 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d117f5b-5d47-488d-9d42-0ee87596813b-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.983827 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.983784 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.984072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.983891 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.984072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.983959 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.984072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.984046 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc4bk\" (UniqueName: \"kubernetes.io/projected/9d117f5b-5d47-488d-9d42-0ee87596813b-kube-api-access-fc4bk\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.984225 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.984077 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.984225 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.984130 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.984333 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.984265 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.984389 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.984353 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.984449 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.984426 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.985056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.985032 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d117f5b-5d47-488d-9d42-0ee87596813b-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.986555 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.986538 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.986639 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.986597 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.996743 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.996652 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:44.997334 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:44.996915 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc4bk\" (UniqueName: \"kubernetes.io/projected/9d117f5b-5d47-488d-9d42-0ee87596813b-kube-api-access-fc4bk\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:45.031970 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:52:45.031943 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d4f80e_7c28_44c4_9f87_4d690d6a5e35.slice/crio-a99db0edda6910cd89101cfd3dc99d0338a9e0e2c9c729dcecfd73b7c1142159 WatchSource:0}: Error finding container a99db0edda6910cd89101cfd3dc99d0338a9e0e2c9c729dcecfd73b7c1142159: Status 404 returned error can't find the container with id a99db0edda6910cd89101cfd3dc99d0338a9e0e2c9c729dcecfd73b7c1142159 Nov 03 19:52:45.032841 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.032813 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fe573bf-wc7sh"] Nov 03 19:52:45.131513 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.131486 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs"] Nov 03 19:52:45.131616 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:52:45.131589 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5992c515_22ba_4191_b5f6_8181c094340c.slice/crio-ff98738e80a507ddeec44b24ad2a5cc42928be4ad60b38c4d107ac52a6785c37 WatchSource:0}: Error finding container ff98738e80a507ddeec44b24ad2a5cc42928be4ad60b38c4d107ac52a6785c37: Status 404 returned error can't find the container with id ff98738e80a507ddeec44b24ad2a5cc42928be4ad60b38c4d107ac52a6785c37 Nov 03 19:52:45.134174 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.134141 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:52:45.134274 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.134215 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:52:45.134274 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.134247 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:52:45.230991 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.230956 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:45.391377 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.391298 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" event={"ID":"5992c515-22ba-4191-b5f6-8181c094340c","Type":"ContainerStarted","Data":"6624d12d15032ac6bbfd8aebd5287d8b324d73fb942cc55f988eead98a46a289"} Nov 03 19:52:45.391377 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.391356 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" event={"ID":"5992c515-22ba-4191-b5f6-8181c094340c","Type":"ContainerStarted","Data":"ff98738e80a507ddeec44b24ad2a5cc42928be4ad60b38c4d107ac52a6785c37"} Nov 03 19:52:45.392721 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.392693 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe573bf-wc7sh" event={"ID":"b6d4f80e-7c28-44c4-9f87-4d690d6a5e35","Type":"ContainerStarted","Data":"a99db0edda6910cd89101cfd3dc99d0338a9e0e2c9c729dcecfd73b7c1142159"} Nov 03 19:52:45.420774 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.420716 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" podStartSLOduration=1.420696801 podStartE2EDuration="1.420696801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:45.416764716 +0000 UTC m=+959.674561973" watchObservedRunningTime="2026-11-03 19:52:45.420696801 +0000 UTC m=+959.678494060" Nov 03 19:52:45.528456 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:52:45.527422 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d117f5b_5d47_488d_9d42_0ee87596813b.slice/crio-aa3f8b5aa191e8dc15d26636a56b15c767e55d2621b24a40a08e8b2056e39567 WatchSource:0}: Error finding container aa3f8b5aa191e8dc15d26636a56b15c767e55d2621b24a40a08e8b2056e39567: Status 404 returned error can't find the container with id aa3f8b5aa191e8dc15d26636a56b15c767e55d2621b24a40a08e8b2056e39567 Nov 03 19:52:45.528456 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.527841 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr"] Nov 03 19:52:45.530786 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.530744 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:52:45.530885 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.530825 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:52:45.530885 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.530868 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:52:45.556613 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.556585 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d9da1c-cslfr" Nov 03 19:52:45.591216 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.591178 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/588f1533-b01d-4fd9-a022-afe753fa6a52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/588f1533-b01d-4fd9-a022-afe753fa6a52-maas-api-xks-ca\") pod \"588f1533-b01d-4fd9-a022-afe753fa6a52\" (UID: \"588f1533-b01d-4fd9-a022-afe753fa6a52\") " Nov 03 19:52:45.591341 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.591278 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/588f1533-b01d-4fd9-a022-afe753fa6a52-kube-api-access-vs7sl\" (UniqueName: \"kubernetes.io/projected/588f1533-b01d-4fd9-a022-afe753fa6a52-kube-api-access-vs7sl\") pod \"588f1533-b01d-4fd9-a022-afe753fa6a52\" (UID: \"588f1533-b01d-4fd9-a022-afe753fa6a52\") " Nov 03 19:52:45.592187 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.591502 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/588f1533-b01d-4fd9-a022-afe753fa6a52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/588f1533-b01d-4fd9-a022-afe753fa6a52-maas-api-openshift-ca\") pod \"588f1533-b01d-4fd9-a022-afe753fa6a52\" (UID: \"588f1533-b01d-4fd9-a022-afe753fa6a52\") " Nov 03 19:52:45.592187 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.591978 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/588f1533-b01d-4fd9-a022-afe753fa6a52-maas-api-openshift-ca" pod "588f1533-b01d-4fd9-a022-afe753fa6a52" (UID: "588f1533-b01d-4fd9-a022-afe753fa6a52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:45.592187 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.592157 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/588f1533-b01d-4fd9-a022-afe753fa6a52-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:45.593606 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.593579 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/588f1533-b01d-4fd9-a022-afe753fa6a52-kube-api-access-vs7sl" pod "588f1533-b01d-4fd9-a022-afe753fa6a52" (UID: "588f1533-b01d-4fd9-a022-afe753fa6a52"). InnerVolumeSpecName "kube-api-access-vs7sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:45.594536 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.594514 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/588f1533-b01d-4fd9-a022-afe753fa6a52-maas-api-xks-ca" pod "588f1533-b01d-4fd9-a022-afe753fa6a52" (UID: "588f1533-b01d-4fd9-a022-afe753fa6a52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:45.693682 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.693539 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/588f1533-b01d-4fd9-a022-afe753fa6a52-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:45.693682 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.693582 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs7sl\" (UniqueName: \"kubernetes.io/projected/588f1533-b01d-4fd9-a022-afe753fa6a52-kube-api-access-vs7sl\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:45.798302 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:45.798268 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:46.104251 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.104213 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ff859ce-w7znl"] Nov 03 19:52:46.116995 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.106717 5059 cpu_manager.go:414] "containerMap: removing container" podUID="588f1533-b01d-4fd9-a022-afe753fa6a52" containerName="revoke-keys" Nov 03 19:52:46.116995 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.106742 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="588f1533-b01d-4fd9-a022-afe753fa6a52" containerName="revoke-keys" Nov 03 19:52:46.116995 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.106884 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="588f1533-b01d-4fd9-a022-afe753fa6a52" containerName="revoke-keys" Nov 03 19:52:46.116995 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.111518 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ff859ce-w7znl"] Nov 03 19:52:46.116995 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.111630 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff859ce-w7znl" Nov 03 19:52:46.204693 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.202427 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89c4dbf6-cf42-43b4-95cb-48676a3a4869-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ff859ce-w7znl\" (UID: \"89c4dbf6-cf42-43b4-95cb-48676a3a4869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff859ce-w7znl" Nov 03 19:52:46.204693 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.202502 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bn6w\" (UniqueName: \"kubernetes.io/projected/89c4dbf6-cf42-43b4-95cb-48676a3a4869-kube-api-access-4bn6w\") pod \"maas-api-revoke-sub-9ff859ce-w7znl\" (UID: \"89c4dbf6-cf42-43b4-95cb-48676a3a4869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff859ce-w7znl" Nov 03 19:52:46.204693 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.202542 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89c4dbf6-cf42-43b4-95cb-48676a3a4869-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ff859ce-w7znl\" (UID: \"89c4dbf6-cf42-43b4-95cb-48676a3a4869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff859ce-w7znl" Nov 03 19:52:46.306983 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.306284 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89c4dbf6-cf42-43b4-95cb-48676a3a4869-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ff859ce-w7znl\" (UID: \"89c4dbf6-cf42-43b4-95cb-48676a3a4869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff859ce-w7znl" Nov 03 19:52:46.306983 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.306354 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bn6w\" (UniqueName: \"kubernetes.io/projected/89c4dbf6-cf42-43b4-95cb-48676a3a4869-kube-api-access-4bn6w\") pod \"maas-api-revoke-sub-9ff859ce-w7znl\" (UID: \"89c4dbf6-cf42-43b4-95cb-48676a3a4869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff859ce-w7znl" Nov 03 19:52:46.306983 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.306389 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89c4dbf6-cf42-43b4-95cb-48676a3a4869-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ff859ce-w7znl\" (UID: \"89c4dbf6-cf42-43b4-95cb-48676a3a4869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff859ce-w7znl" Nov 03 19:52:46.308117 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.308092 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89c4dbf6-cf42-43b4-95cb-48676a3a4869-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ff859ce-w7znl\" (UID: \"89c4dbf6-cf42-43b4-95cb-48676a3a4869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff859ce-w7znl" Nov 03 19:52:46.313709 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.312004 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89c4dbf6-cf42-43b4-95cb-48676a3a4869-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ff859ce-w7znl\" (UID: \"89c4dbf6-cf42-43b4-95cb-48676a3a4869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff859ce-w7znl" Nov 03 19:52:46.321764 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.320793 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bn6w\" (UniqueName: \"kubernetes.io/projected/89c4dbf6-cf42-43b4-95cb-48676a3a4869-kube-api-access-4bn6w\") pod \"maas-api-revoke-sub-9ff859ce-w7znl\" (UID: \"89c4dbf6-cf42-43b4-95cb-48676a3a4869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff859ce-w7znl" Nov 03 19:52:46.419107 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.419063 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d9da1c-cslfr" event={"ID":"588f1533-b01d-4fd9-a022-afe753fa6a52","Type":"ContainerDied","Data":"8e852ff51abec44fc200683dc78267e1e338ec174e46871e0ebcde031d5398de"} Nov 03 19:52:46.419107 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.419095 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d9da1c-cslfr" Nov 03 19:52:46.419632 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.419135 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e852ff51abec44fc200683dc78267e1e338ec174e46871e0ebcde031d5398de" Nov 03 19:52:46.421500 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.421470 5059 generic.go:356] "Generic (PLEG): container finished" podID="b6d4f80e-7c28-44c4-9f87-4d690d6a5e35" containerID="34d4fc10c1c858abdc76cfe27baafae85d7e2920066716b6747e0a3ea01e8c5c" exitCode=0 Nov 03 19:52:46.421627 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.421599 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe573bf-wc7sh" event={"ID":"b6d4f80e-7c28-44c4-9f87-4d690d6a5e35","Type":"ContainerDied","Data":"34d4fc10c1c858abdc76cfe27baafae85d7e2920066716b6747e0a3ea01e8c5c"} Nov 03 19:52:46.423916 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.423886 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" event={"ID":"9d117f5b-5d47-488d-9d42-0ee87596813b","Type":"ContainerStarted","Data":"c547d1b656a3bb125078038eede3d512f20261edfee3ed4dd5d1bdb89c29881e"} Nov 03 19:52:46.424117 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.423934 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" event={"ID":"9d117f5b-5d47-488d-9d42-0ee87596813b","Type":"ContainerStarted","Data":"aa3f8b5aa191e8dc15d26636a56b15c767e55d2621b24a40a08e8b2056e39567"} Nov 03 19:52:46.489965 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.488158 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" podStartSLOduration=2.488139792 podStartE2EDuration="2.488139792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:46.487767927 +0000 UTC m=+960.745565185" watchObservedRunningTime="2026-11-03 19:52:46.488139792 +0000 UTC m=+960.745937050" Nov 03 19:52:46.512689 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.509305 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff859ce-w7znl" Nov 03 19:52:46.810740 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.803508 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 19:52:46.810740 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.803623 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" podUID="5992c515-22ba-4191-b5f6-8181c094340c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 19:52:46.853473 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:46.853447 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ff859ce-w7znl"] Nov 03 19:52:47.231848 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:47.231760 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:47.233108 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:47.233081 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:52:47.233216 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:47.233132 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" podUID="9d117f5b-5d47-488d-9d42-0ee87596813b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 19:52:47.434410 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:47.434376 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff859ce-w7znl" event={"ID":"89c4dbf6-cf42-43b4-95cb-48676a3a4869","Type":"ContainerStarted","Data":"7dff1f9587b4e2eaf860585edcee5f798f89c74d723d658603fc095d3d74f9ac"} Nov 03 19:52:47.434910 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:47.434432 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff859ce-w7znl" event={"ID":"89c4dbf6-cf42-43b4-95cb-48676a3a4869","Type":"ContainerStarted","Data":"7987613a9a9bfc99047792c1cf20fbda025123154ed2954d5c25017cb0614fd0"} Nov 03 19:52:47.482462 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:47.482337 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff859ce-w7znl" podStartSLOduration=1.4823151540000001 podStartE2EDuration="1.482315154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:47.453788185 +0000 UTC m=+961.711585465" watchObservedRunningTime="2026-11-03 19:52:47.482315154 +0000 UTC m=+961.740112412" Nov 03 19:52:47.778223 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:47.777849 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe573bf-wc7sh" Nov 03 19:52:47.806615 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:47.806185 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 19:52:47.806615 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:47.806259 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" podUID="5992c515-22ba-4191-b5f6-8181c094340c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 19:52:47.839461 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:47.839433 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6d4f80e-7c28-44c4-9f87-4d690d6a5e35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d4f80e-7c28-44c4-9f87-4d690d6a5e35-maas-api-xks-ca\") pod \"b6d4f80e-7c28-44c4-9f87-4d690d6a5e35\" (UID: \"b6d4f80e-7c28-44c4-9f87-4d690d6a5e35\") " Nov 03 19:52:47.839716 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:47.839702 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6d4f80e-7c28-44c4-9f87-4d690d6a5e35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d4f80e-7c28-44c4-9f87-4d690d6a5e35-maas-api-openshift-ca\") pod \"b6d4f80e-7c28-44c4-9f87-4d690d6a5e35\" (UID: \"b6d4f80e-7c28-44c4-9f87-4d690d6a5e35\") " Nov 03 19:52:47.839861 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:47.839848 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d4f80e-7c28-44c4-9f87-4d690d6a5e35-kube-api-access-x6w5s\" (UniqueName: \"kubernetes.io/projected/b6d4f80e-7c28-44c4-9f87-4d690d6a5e35-kube-api-access-x6w5s\") pod \"b6d4f80e-7c28-44c4-9f87-4d690d6a5e35\" (UID: \"b6d4f80e-7c28-44c4-9f87-4d690d6a5e35\") " Nov 03 19:52:47.840086 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:47.840062 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d4f80e-7c28-44c4-9f87-4d690d6a5e35-maas-api-openshift-ca" pod "b6d4f80e-7c28-44c4-9f87-4d690d6a5e35" (UID: "b6d4f80e-7c28-44c4-9f87-4d690d6a5e35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:47.840370 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:47.840356 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d4f80e-7c28-44c4-9f87-4d690d6a5e35-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:47.842258 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:47.842238 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6d4f80e-7c28-44c4-9f87-4d690d6a5e35-maas-api-xks-ca" pod "b6d4f80e-7c28-44c4-9f87-4d690d6a5e35" (UID: "b6d4f80e-7c28-44c4-9f87-4d690d6a5e35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:47.842310 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:47.842252 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d4f80e-7c28-44c4-9f87-4d690d6a5e35-kube-api-access-x6w5s" pod "b6d4f80e-7c28-44c4-9f87-4d690d6a5e35" (UID: "b6d4f80e-7c28-44c4-9f87-4d690d6a5e35"). InnerVolumeSpecName "kube-api-access-x6w5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:47.942554 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:47.942517 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d4f80e-7c28-44c4-9f87-4d690d6a5e35-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:47.942554 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:47.942549 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6w5s\" (UniqueName: \"kubernetes.io/projected/b6d4f80e-7c28-44c4-9f87-4d690d6a5e35-kube-api-access-x6w5s\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:48.231749 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:48.231715 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:52:48.231911 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:48.231768 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" podUID="9d117f5b-5d47-488d-9d42-0ee87596813b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 19:52:48.452443 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:48.452273 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe573bf-wc7sh" Nov 03 19:52:48.452443 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:48.452300 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe573bf-wc7sh" event={"ID":"b6d4f80e-7c28-44c4-9f87-4d690d6a5e35","Type":"ContainerDied","Data":"a99db0edda6910cd89101cfd3dc99d0338a9e0e2c9c729dcecfd73b7c1142159"} Nov 03 19:52:48.452443 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:48.452394 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a99db0edda6910cd89101cfd3dc99d0338a9e0e2c9c729dcecfd73b7c1142159" Nov 03 19:52:48.454752 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:48.454725 5059 generic.go:356] "Generic (PLEG): container finished" podID="89c4dbf6-cf42-43b4-95cb-48676a3a4869" containerID="7dff1f9587b4e2eaf860585edcee5f798f89c74d723d658603fc095d3d74f9ac" exitCode=0 Nov 03 19:52:48.455819 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:48.455798 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff859ce-w7znl" event={"ID":"89c4dbf6-cf42-43b4-95cb-48676a3a4869","Type":"ContainerDied","Data":"7dff1f9587b4e2eaf860585edcee5f798f89c74d723d658603fc095d3d74f9ac"} Nov 03 19:52:48.648049 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:48.648017 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d722f54a-ftbll"] Nov 03 19:52:48.648956 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:48.648940 5059 cpu_manager.go:414] "containerMap: removing container" podUID="b6d4f80e-7c28-44c4-9f87-4d690d6a5e35" containerName="revoke-keys" Nov 03 19:52:48.649001 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:48.648959 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d4f80e-7c28-44c4-9f87-4d690d6a5e35" containerName="revoke-keys" Nov 03 19:52:48.649106 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:48.649092 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d4f80e-7c28-44c4-9f87-4d690d6a5e35" containerName="revoke-keys" Nov 03 19:52:48.652936 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:48.652915 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d722f54a-ftbll" Nov 03 19:52:48.670685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:48.670054 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d722f54a-ftbll"] Nov 03 19:52:48.754145 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:48.754063 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8clc7\" (UniqueName: \"kubernetes.io/projected/c55f0fac-4308-423f-bc06-994606b40c14-kube-api-access-8clc7\") pod \"maas-api-revoke-sub-d722f54a-ftbll\" (UID: \"c55f0fac-4308-423f-bc06-994606b40c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d722f54a-ftbll" Nov 03 19:52:48.754279 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:48.754240 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55f0fac-4308-423f-bc06-994606b40c14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d722f54a-ftbll\" (UID: \"c55f0fac-4308-423f-bc06-994606b40c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d722f54a-ftbll" Nov 03 19:52:48.754334 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:48.754317 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55f0fac-4308-423f-bc06-994606b40c14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d722f54a-ftbll\" (UID: \"c55f0fac-4308-423f-bc06-994606b40c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d722f54a-ftbll" Nov 03 19:52:48.812686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:48.812298 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 19:52:48.812686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:48.812358 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" podUID="5992c515-22ba-4191-b5f6-8181c094340c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 19:52:48.855898 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:48.855864 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55f0fac-4308-423f-bc06-994606b40c14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d722f54a-ftbll\" (UID: \"c55f0fac-4308-423f-bc06-994606b40c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d722f54a-ftbll" Nov 03 19:52:48.856031 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:48.855933 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55f0fac-4308-423f-bc06-994606b40c14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d722f54a-ftbll\" (UID: \"c55f0fac-4308-423f-bc06-994606b40c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d722f54a-ftbll" Nov 03 19:52:48.856085 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:48.856052 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8clc7\" (UniqueName: \"kubernetes.io/projected/c55f0fac-4308-423f-bc06-994606b40c14-kube-api-access-8clc7\") pod \"maas-api-revoke-sub-d722f54a-ftbll\" (UID: \"c55f0fac-4308-423f-bc06-994606b40c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d722f54a-ftbll" Nov 03 19:52:48.857241 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:48.857215 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55f0fac-4308-423f-bc06-994606b40c14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d722f54a-ftbll\" (UID: \"c55f0fac-4308-423f-bc06-994606b40c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d722f54a-ftbll" Nov 03 19:52:48.888691 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:48.873073 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55f0fac-4308-423f-bc06-994606b40c14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d722f54a-ftbll\" (UID: \"c55f0fac-4308-423f-bc06-994606b40c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d722f54a-ftbll" Nov 03 19:52:48.904690 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:48.902543 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8clc7\" (UniqueName: \"kubernetes.io/projected/c55f0fac-4308-423f-bc06-994606b40c14-kube-api-access-8clc7\") pod \"maas-api-revoke-sub-d722f54a-ftbll\" (UID: \"c55f0fac-4308-423f-bc06-994606b40c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d722f54a-ftbll" Nov 03 19:52:49.020579 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:49.020497 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d722f54a-ftbll" Nov 03 19:52:49.231770 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:49.231704 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:52:49.231770 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:49.231774 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" podUID="9d117f5b-5d47-488d-9d42-0ee87596813b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 19:52:49.377712 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:49.358064 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d722f54a-ftbll"] Nov 03 19:52:49.518609 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:49.518559 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d722f54a-ftbll" event={"ID":"c55f0fac-4308-423f-bc06-994606b40c14","Type":"ContainerStarted","Data":"d57909fc88a0e4c070d963be3ae39ba5e0f8c77c56970292270ef8ba11119dec"} Nov 03 19:52:49.819214 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:49.819181 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 19:52:49.819487 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:49.819455 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" podUID="5992c515-22ba-4191-b5f6-8181c094340c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 19:52:49.839267 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:49.839246 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff859ce-w7znl" Nov 03 19:52:49.925582 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:49.925368 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89c4dbf6-cf42-43b4-95cb-48676a3a4869-kube-api-access-4bn6w\" (UniqueName: \"kubernetes.io/projected/89c4dbf6-cf42-43b4-95cb-48676a3a4869-kube-api-access-4bn6w\") pod \"89c4dbf6-cf42-43b4-95cb-48676a3a4869\" (UID: \"89c4dbf6-cf42-43b4-95cb-48676a3a4869\") " Nov 03 19:52:49.925582 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:49.925479 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89c4dbf6-cf42-43b4-95cb-48676a3a4869-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89c4dbf6-cf42-43b4-95cb-48676a3a4869-maas-api-openshift-ca\") pod \"89c4dbf6-cf42-43b4-95cb-48676a3a4869\" (UID: \"89c4dbf6-cf42-43b4-95cb-48676a3a4869\") " Nov 03 19:52:49.925582 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:49.925526 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89c4dbf6-cf42-43b4-95cb-48676a3a4869-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89c4dbf6-cf42-43b4-95cb-48676a3a4869-maas-api-xks-ca\") pod \"89c4dbf6-cf42-43b4-95cb-48676a3a4869\" (UID: \"89c4dbf6-cf42-43b4-95cb-48676a3a4869\") " Nov 03 19:52:49.927290 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:49.927266 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c4dbf6-cf42-43b4-95cb-48676a3a4869-maas-api-openshift-ca" pod "89c4dbf6-cf42-43b4-95cb-48676a3a4869" (UID: "89c4dbf6-cf42-43b4-95cb-48676a3a4869"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:49.929027 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:49.929008 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89c4dbf6-cf42-43b4-95cb-48676a3a4869-maas-api-xks-ca" pod "89c4dbf6-cf42-43b4-95cb-48676a3a4869" (UID: "89c4dbf6-cf42-43b4-95cb-48676a3a4869"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:49.935721 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:49.935684 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c4dbf6-cf42-43b4-95cb-48676a3a4869-kube-api-access-4bn6w" pod "89c4dbf6-cf42-43b4-95cb-48676a3a4869" (UID: "89c4dbf6-cf42-43b4-95cb-48676a3a4869"). InnerVolumeSpecName "kube-api-access-4bn6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:50.027163 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:50.027121 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bn6w\" (UniqueName: \"kubernetes.io/projected/89c4dbf6-cf42-43b4-95cb-48676a3a4869-kube-api-access-4bn6w\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:50.027163 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:50.027164 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89c4dbf6-cf42-43b4-95cb-48676a3a4869-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:50.027361 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:50.027186 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89c4dbf6-cf42-43b4-95cb-48676a3a4869-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:50.231883 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:50.231792 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:52:50.231883 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:50.231860 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" podUID="9d117f5b-5d47-488d-9d42-0ee87596813b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 19:52:50.533739 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:50.533706 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff859ce-w7znl" event={"ID":"89c4dbf6-cf42-43b4-95cb-48676a3a4869","Type":"ContainerDied","Data":"7987613a9a9bfc99047792c1cf20fbda025123154ed2954d5c25017cb0614fd0"} Nov 03 19:52:50.534156 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:50.533759 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7987613a9a9bfc99047792c1cf20fbda025123154ed2954d5c25017cb0614fd0" Nov 03 19:52:50.534156 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:50.533761 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ff859ce-w7znl" Nov 03 19:52:50.536799 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:50.536766 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d722f54a-ftbll" event={"ID":"c55f0fac-4308-423f-bc06-994606b40c14","Type":"ContainerStarted","Data":"25d3556b8415448f4f453e3b13f7abdb1e9b1baea12dab448e0e72865ddbbef7"} Nov 03 19:52:50.810881 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:50.810788 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 19:52:50.811151 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:50.811114 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" podUID="5992c515-22ba-4191-b5f6-8181c094340c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 19:52:51.232146 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:51.232111 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:52:51.232301 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:51.232172 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" podUID="9d117f5b-5d47-488d-9d42-0ee87596813b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 19:52:51.548369 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:51.548287 5059 generic.go:356] "Generic (PLEG): container finished" podID="c55f0fac-4308-423f-bc06-994606b40c14" containerID="25d3556b8415448f4f453e3b13f7abdb1e9b1baea12dab448e0e72865ddbbef7" exitCode=0 Nov 03 19:52:51.548369 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:51.548324 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d722f54a-ftbll" event={"ID":"c55f0fac-4308-423f-bc06-994606b40c14","Type":"ContainerDied","Data":"25d3556b8415448f4f453e3b13f7abdb1e9b1baea12dab448e0e72865ddbbef7"} Nov 03 19:52:51.798731 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:51.798637 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 19:52:51.798731 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:51.798711 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" podUID="5992c515-22ba-4191-b5f6-8181c094340c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 19:52:51.803433 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:51.803402 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d722f54a-ftbll" Nov 03 19:52:51.965085 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:51.965048 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c55f0fac-4308-423f-bc06-994606b40c14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55f0fac-4308-423f-bc06-994606b40c14-maas-api-xks-ca\") pod \"c55f0fac-4308-423f-bc06-994606b40c14\" (UID: \"c55f0fac-4308-423f-bc06-994606b40c14\") " Nov 03 19:52:51.965354 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:51.965335 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c55f0fac-4308-423f-bc06-994606b40c14-kube-api-access-8clc7\" (UniqueName: \"kubernetes.io/projected/c55f0fac-4308-423f-bc06-994606b40c14-kube-api-access-8clc7\") pod \"c55f0fac-4308-423f-bc06-994606b40c14\" (UID: \"c55f0fac-4308-423f-bc06-994606b40c14\") " Nov 03 19:52:51.965436 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:51.965421 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c55f0fac-4308-423f-bc06-994606b40c14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55f0fac-4308-423f-bc06-994606b40c14-maas-api-openshift-ca\") pod \"c55f0fac-4308-423f-bc06-994606b40c14\" (UID: \"c55f0fac-4308-423f-bc06-994606b40c14\") " Nov 03 19:52:51.965882 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:51.965848 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c55f0fac-4308-423f-bc06-994606b40c14-maas-api-openshift-ca" pod "c55f0fac-4308-423f-bc06-994606b40c14" (UID: "c55f0fac-4308-423f-bc06-994606b40c14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:51.966124 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:51.966100 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55f0fac-4308-423f-bc06-994606b40c14-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:51.967626 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:51.967595 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c55f0fac-4308-423f-bc06-994606b40c14-kube-api-access-8clc7" pod "c55f0fac-4308-423f-bc06-994606b40c14" (UID: "c55f0fac-4308-423f-bc06-994606b40c14"). InnerVolumeSpecName "kube-api-access-8clc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:51.967894 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:51.967789 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c55f0fac-4308-423f-bc06-994606b40c14-maas-api-xks-ca" pod "c55f0fac-4308-423f-bc06-994606b40c14" (UID: "c55f0fac-4308-423f-bc06-994606b40c14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:52.067285 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:52.067207 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8clc7\" (UniqueName: \"kubernetes.io/projected/c55f0fac-4308-423f-bc06-994606b40c14-kube-api-access-8clc7\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:52.067285 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:52.067239 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55f0fac-4308-423f-bc06-994606b40c14-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:52.231540 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:52.231498 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:52:52.231725 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:52.231605 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" podUID="9d117f5b-5d47-488d-9d42-0ee87596813b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 19:52:52.563784 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:52.561228 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d722f54a-ftbll" Nov 03 19:52:52.563784 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:52.561234 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d722f54a-ftbll" event={"ID":"c55f0fac-4308-423f-bc06-994606b40c14","Type":"ContainerDied","Data":"d57909fc88a0e4c070d963be3ae39ba5e0f8c77c56970292270ef8ba11119dec"} Nov 03 19:52:52.563784 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:52.561327 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d57909fc88a0e4c070d963be3ae39ba5e0f8c77c56970292270ef8ba11119dec" Nov 03 19:52:52.892805 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:52.892776 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:52.893908 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:52.893884 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:52.894630 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:52.894608 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 19:52:54.018686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:54.018183 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:54.020186 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:54.020163 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:54.029935 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:54.029906 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:52:54.857688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:54.857298 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:55.019954 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:55.019857 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:55.100784 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:55.100741 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:55.436815 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:55.436768 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:55.446771 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:55.446725 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 19:52:56.282685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.281953 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2"] Nov 03 19:52:56.283201 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.283016 5059 cpu_manager.go:414] "containerMap: removing container" podUID="89c4dbf6-cf42-43b4-95cb-48676a3a4869" containerName="revoke-keys" Nov 03 19:52:56.283201 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.283039 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89c4dbf6-cf42-43b4-95cb-48676a3a4869" containerName="revoke-keys" Nov 03 19:52:56.283201 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.283055 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c55f0fac-4308-423f-bc06-994606b40c14" containerName="revoke-keys" Nov 03 19:52:56.283201 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.283064 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c55f0fac-4308-423f-bc06-994606b40c14" containerName="revoke-keys" Nov 03 19:52:56.283441 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.283263 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="89c4dbf6-cf42-43b4-95cb-48676a3a4869" containerName="revoke-keys" Nov 03 19:52:56.283441 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.283280 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c55f0fac-4308-423f-bc06-994606b40c14" containerName="revoke-keys" Nov 03 19:52:56.413688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.409932 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2"] Nov 03 19:52:56.413688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.410059 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.449688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.446546 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d8d238bf-openshift-default-dockercfg-dkj4k\"" Nov 03 19:52:56.555575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.554924 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-workload-certs\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.555575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.554970 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-podinfo\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.555575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.555000 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-envoy\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.555575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.555029 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sndh\" (UniqueName: \"kubernetes.io/projected/070f2e92-dc4a-479a-a8dc-724e4cc4a050-kube-api-access-2sndh\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.555575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.555061 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-token\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.555575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.555112 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-workload-socket\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.555575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.555151 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-credential-socket\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.555575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.555223 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-data\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.555575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.555287 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istiod-ca-cert\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.676192 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.670256 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-workload-certs\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.677081 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.677057 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-podinfo\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.677472 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.677455 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-envoy\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.677580 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.677567 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sndh\" (UniqueName: \"kubernetes.io/projected/070f2e92-dc4a-479a-a8dc-724e4cc4a050-kube-api-access-2sndh\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.677691 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.677677 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-token\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.677818 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.677805 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-workload-socket\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.677924 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.677913 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-credential-socket\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.678064 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.678051 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-data\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.678222 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.678196 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istiod-ca-cert\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.687035 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.687008 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istiod-ca-cert\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.687202 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.677002 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-workload-certs\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.708718 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.690795 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-credential-socket\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.708718 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.691046 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-workload-socket\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.708718 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.691274 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-data\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.731946 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.731170 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-envoy\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.731946 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.731889 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-podinfo\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.812001 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.795505 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sndh\" (UniqueName: \"kubernetes.io/projected/070f2e92-dc4a-479a-a8dc-724e4cc4a050-kube-api-access-2sndh\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.915692 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.853850 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-token\") pod \"e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:56.926724 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:56.926695 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:57.405648 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:57.404634 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2"] Nov 03 19:52:57.408632 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:57.408586 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:52:57.408785 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:57.408698 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:52:57.408785 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:57.408747 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:52:57.936175 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:57.936135 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" event={"ID":"070f2e92-dc4a-479a-a8dc-724e4cc4a050","Type":"ContainerStarted","Data":"4a9f00bb1e92bce9af0673cd4868bc76735066c2845448bfdb9dce793d7271d3"} Nov 03 19:52:57.936349 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:57.936198 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" event={"ID":"070f2e92-dc4a-479a-a8dc-724e4cc4a050","Type":"ContainerStarted","Data":"3c889bc5ebf707d4467868c453c2b228776eb4ac127e251b2c510d23ae7bb9a6"} Nov 03 19:52:57.965940 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:57.965891 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" podStartSLOduration=1.965878416 podStartE2EDuration="1.965878416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:57.962164576 +0000 UTC m=+972.219961835" watchObservedRunningTime="2026-11-03 19:52:57.965878416 +0000 UTC m=+972.223675673" Nov 03 19:52:58.927578 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:58.927546 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:58.932965 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:58.932933 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:58.944247 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:58.944222 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:58.945042 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:58.945022 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:52:59.321010 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:59.320981 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7820f1dc-g629w"] Nov 03 19:52:59.325612 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:59.325588 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7820f1dc-g629w" Nov 03 19:52:59.332215 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:59.332193 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:52:59.332347 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:59.332279 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 19:52:59.334840 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:59.334814 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7820f1dc-g629w"] Nov 03 19:52:59.468984 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:59.468953 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61622474-878f-4bf9-bb70-e8e4ce4e2bf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7820f1dc-g629w\" (UID: \"61622474-878f-4bf9-bb70-e8e4ce4e2bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7820f1dc-g629w" Nov 03 19:52:59.469140 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:59.468997 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61622474-878f-4bf9-bb70-e8e4ce4e2bf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7820f1dc-g629w\" (UID: \"61622474-878f-4bf9-bb70-e8e4ce4e2bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7820f1dc-g629w" Nov 03 19:52:59.469140 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:59.469015 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95sff\" (UniqueName: \"kubernetes.io/projected/61622474-878f-4bf9-bb70-e8e4ce4e2bf0-kube-api-access-95sff\") pod \"maas-api-revoke-sub-7820f1dc-g629w\" (UID: \"61622474-878f-4bf9-bb70-e8e4ce4e2bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7820f1dc-g629w" Nov 03 19:52:59.570467 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:59.570427 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61622474-878f-4bf9-bb70-e8e4ce4e2bf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7820f1dc-g629w\" (UID: \"61622474-878f-4bf9-bb70-e8e4ce4e2bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7820f1dc-g629w" Nov 03 19:52:59.570711 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:59.570480 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61622474-878f-4bf9-bb70-e8e4ce4e2bf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7820f1dc-g629w\" (UID: \"61622474-878f-4bf9-bb70-e8e4ce4e2bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7820f1dc-g629w" Nov 03 19:52:59.570711 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:59.570500 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95sff\" (UniqueName: \"kubernetes.io/projected/61622474-878f-4bf9-bb70-e8e4ce4e2bf0-kube-api-access-95sff\") pod \"maas-api-revoke-sub-7820f1dc-g629w\" (UID: \"61622474-878f-4bf9-bb70-e8e4ce4e2bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7820f1dc-g629w" Nov 03 19:52:59.571260 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:59.571205 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61622474-878f-4bf9-bb70-e8e4ce4e2bf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7820f1dc-g629w\" (UID: \"61622474-878f-4bf9-bb70-e8e4ce4e2bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7820f1dc-g629w" Nov 03 19:52:59.572717 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:59.572693 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61622474-878f-4bf9-bb70-e8e4ce4e2bf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7820f1dc-g629w\" (UID: \"61622474-878f-4bf9-bb70-e8e4ce4e2bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7820f1dc-g629w" Nov 03 19:52:59.582831 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:59.582810 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95sff\" (UniqueName: \"kubernetes.io/projected/61622474-878f-4bf9-bb70-e8e4ce4e2bf0-kube-api-access-95sff\") pod \"maas-api-revoke-sub-7820f1dc-g629w\" (UID: \"61622474-878f-4bf9-bb70-e8e4ce4e2bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7820f1dc-g629w" Nov 03 19:52:59.642209 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:59.642184 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7820f1dc-g629w" Nov 03 19:52:59.929239 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:52:59.929203 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61622474_878f_4bf9_bb70_e8e4ce4e2bf0.slice/crio-386448bf81923ac87be674e08cbf5a9d03549d6d358028b37b68ba337fc17db2 WatchSource:0}: Error finding container 386448bf81923ac87be674e08cbf5a9d03549d6d358028b37b68ba337fc17db2: Status 404 returned error can't find the container with id 386448bf81923ac87be674e08cbf5a9d03549d6d358028b37b68ba337fc17db2 Nov 03 19:52:59.929719 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:59.929697 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7820f1dc-g629w"] Nov 03 19:52:59.959741 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:52:59.959713 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7820f1dc-g629w" event={"ID":"61622474-878f-4bf9-bb70-e8e4ce4e2bf0","Type":"ContainerStarted","Data":"386448bf81923ac87be674e08cbf5a9d03549d6d358028b37b68ba337fc17db2"} Nov 03 19:53:01.035758 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.035724 5059 generic.go:356] "Generic (PLEG): container finished" podID="61622474-878f-4bf9-bb70-e8e4ce4e2bf0" containerID="2eb28b2903edbb43c568a1d84342cee7d2e5444d4a0d07d27b1512860ef218a4" exitCode=0 Nov 03 19:53:01.037139 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.037100 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7820f1dc-g629w" event={"ID":"61622474-878f-4bf9-bb70-e8e4ce4e2bf0","Type":"ContainerDied","Data":"2eb28b2903edbb43c568a1d84342cee7d2e5444d4a0d07d27b1512860ef218a4"} Nov 03 19:53:01.783357 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.783323 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk"] Nov 03 19:53:01.788005 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.787981 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.805232 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.805206 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk"] Nov 03 19:53:01.806812 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.806786 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.806928 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.806887 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnfhk\" (UniqueName: \"kubernetes.io/projected/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-kube-api-access-lnfhk\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.806996 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.806925 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.806996 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.806949 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.806996 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.806974 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.807153 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.807052 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.807215 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.807150 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.807290 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.807268 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.807423 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.807346 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.809908 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.809886 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255"] Nov 03 19:53:01.814156 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.814136 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:01.820680 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.820576 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255"] Nov 03 19:53:01.908069 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.908036 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bp2g\" (UniqueName: \"kubernetes.io/projected/34484729-bde1-4b0f-9758-e934b14f33d0-kube-api-access-4bp2g\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:01.908069 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.908070 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:01.908311 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.908090 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34484729-bde1-4b0f-9758-e934b14f33d0-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:01.908311 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.908136 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:01.908311 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.908174 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.908311 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.908238 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.908311 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.908297 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.908568 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.908373 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:01.908568 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.908423 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.908568 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.908499 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:01.908568 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.908529 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34484729-bde1-4b0f-9758-e934b14f33d0-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:01.908890 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.908591 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.908890 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.908616 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34484729-bde1-4b0f-9758-e934b14f33d0-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:01.908890 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.908655 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.908890 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.908779 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:01.908890 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.908813 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.908890 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.908826 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnfhk\" (UniqueName: \"kubernetes.io/projected/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-kube-api-access-lnfhk\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.909100 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.908954 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.909100 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.908964 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.909100 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.909001 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.909100 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.909025 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.909307 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.909261 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.909628 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.909606 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.910834 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.910817 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.911039 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.911020 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.927586 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.927563 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:01.928888 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:01.928867 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnfhk\" (UniqueName: \"kubernetes.io/projected/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-kube-api-access-lnfhk\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:02.010384 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.010356 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:02.010537 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.010430 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:02.010537 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.010449 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34484729-bde1-4b0f-9758-e934b14f33d0-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:02.010537 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.010479 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34484729-bde1-4b0f-9758-e934b14f33d0-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:02.010758 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.010571 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:02.010758 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.010715 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bp2g\" (UniqueName: \"kubernetes.io/projected/34484729-bde1-4b0f-9758-e934b14f33d0-kube-api-access-4bp2g\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:02.010866 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.010755 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:02.010866 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.010784 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34484729-bde1-4b0f-9758-e934b14f33d0-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:02.010866 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.010815 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:02.011130 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.010977 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:02.011438 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.011396 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34484729-bde1-4b0f-9758-e934b14f33d0-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:02.012103 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.012078 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:02.012688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.012644 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:02.012814 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.012795 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:02.014928 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.014907 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:02.015082 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.015058 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34484729-bde1-4b0f-9758-e934b14f33d0-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:02.027356 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.027338 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34484729-bde1-4b0f-9758-e934b14f33d0-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:02.028010 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.027988 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bp2g\" (UniqueName: \"kubernetes.io/projected/34484729-bde1-4b0f-9758-e934b14f33d0-kube-api-access-4bp2g\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:02.107486 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.107457 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:02.143711 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.138054 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:02.328855 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.328824 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7820f1dc-g629w" Nov 03 19:53:02.441179 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.441145 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk"] Nov 03 19:53:02.447027 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:53:02.447003 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedcfb7ab_b14f_4d5c_b752_5b43a6eee163.slice/crio-05d1f7c6803b28af0b4c887a8576d44944ba6af0d944a9770f2f845a5a03f004 WatchSource:0}: Error finding container 05d1f7c6803b28af0b4c887a8576d44944ba6af0d944a9770f2f845a5a03f004: Status 404 returned error can't find the container with id 05d1f7c6803b28af0b4c887a8576d44944ba6af0d944a9770f2f845a5a03f004 Nov 03 19:53:02.453945 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.453292 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:53:02.453945 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.453388 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:53:02.453945 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.453437 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:53:02.527403 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.527379 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61622474-878f-4bf9-bb70-e8e4ce4e2bf0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61622474-878f-4bf9-bb70-e8e4ce4e2bf0-maas-api-openshift-ca\") pod \"61622474-878f-4bf9-bb70-e8e4ce4e2bf0\" (UID: \"61622474-878f-4bf9-bb70-e8e4ce4e2bf0\") " Nov 03 19:53:02.527537 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.527523 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61622474-878f-4bf9-bb70-e8e4ce4e2bf0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61622474-878f-4bf9-bb70-e8e4ce4e2bf0-maas-api-xks-ca\") pod \"61622474-878f-4bf9-bb70-e8e4ce4e2bf0\" (UID: \"61622474-878f-4bf9-bb70-e8e4ce4e2bf0\") " Nov 03 19:53:02.527639 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.527626 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61622474-878f-4bf9-bb70-e8e4ce4e2bf0-kube-api-access-95sff\" (UniqueName: \"kubernetes.io/projected/61622474-878f-4bf9-bb70-e8e4ce4e2bf0-kube-api-access-95sff\") pod \"61622474-878f-4bf9-bb70-e8e4ce4e2bf0\" (UID: \"61622474-878f-4bf9-bb70-e8e4ce4e2bf0\") " Nov 03 19:53:02.527840 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.527809 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61622474-878f-4bf9-bb70-e8e4ce4e2bf0-maas-api-openshift-ca" pod "61622474-878f-4bf9-bb70-e8e4ce4e2bf0" (UID: "61622474-878f-4bf9-bb70-e8e4ce4e2bf0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:02.528238 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.528220 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61622474-878f-4bf9-bb70-e8e4ce4e2bf0-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:02.530179 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.530153 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61622474-878f-4bf9-bb70-e8e4ce4e2bf0-kube-api-access-95sff" pod "61622474-878f-4bf9-bb70-e8e4ce4e2bf0" (UID: "61622474-878f-4bf9-bb70-e8e4ce4e2bf0"). InnerVolumeSpecName "kube-api-access-95sff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:02.530273 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.530182 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61622474-878f-4bf9-bb70-e8e4ce4e2bf0-maas-api-xks-ca" pod "61622474-878f-4bf9-bb70-e8e4ce4e2bf0" (UID: "61622474-878f-4bf9-bb70-e8e4ce4e2bf0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:53:02.609479 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:53:02.609449 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34484729_bde1_4b0f_9758_e934b14f33d0.slice/crio-667ef8ff7cf2400fbf5fb14999b09d30ef4076fa8d683b04e71a8e6a8629d152 WatchSource:0}: Error finding container 667ef8ff7cf2400fbf5fb14999b09d30ef4076fa8d683b04e71a8e6a8629d152: Status 404 returned error can't find the container with id 667ef8ff7cf2400fbf5fb14999b09d30ef4076fa8d683b04e71a8e6a8629d152 Nov 03 19:53:02.610077 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.610053 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255"] Nov 03 19:53:02.612291 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.612259 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:53:02.612373 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.612320 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:53:02.612373 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.612348 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:53:02.629487 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.629465 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61622474-878f-4bf9-bb70-e8e4ce4e2bf0-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:02.629577 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:02.629492 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95sff\" (UniqueName: \"kubernetes.io/projected/61622474-878f-4bf9-bb70-e8e4ce4e2bf0-kube-api-access-95sff\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:03.101353 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.101314 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7820f1dc-g629w" event={"ID":"61622474-878f-4bf9-bb70-e8e4ce4e2bf0","Type":"ContainerDied","Data":"386448bf81923ac87be674e08cbf5a9d03549d6d358028b37b68ba337fc17db2"} Nov 03 19:53:03.101603 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.101583 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="386448bf81923ac87be674e08cbf5a9d03549d6d358028b37b68ba337fc17db2" Nov 03 19:53:03.105934 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.105909 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7820f1dc-g629w" Nov 03 19:53:03.126339 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.126301 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" event={"ID":"34484729-bde1-4b0f-9758-e934b14f33d0","Type":"ContainerStarted","Data":"c05cf95cff32ad56454940f6ca93b383dfeefc5aaef39852a6353bdb13080488"} Nov 03 19:53:03.126793 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.126369 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" event={"ID":"34484729-bde1-4b0f-9758-e934b14f33d0","Type":"ContainerStarted","Data":"667ef8ff7cf2400fbf5fb14999b09d30ef4076fa8d683b04e71a8e6a8629d152"} Nov 03 19:53:03.130351 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.130272 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" event={"ID":"edcfb7ab-b14f-4d5c-b752-5b43a6eee163","Type":"ContainerStarted","Data":"bab3644b9638d4ac499f2df75a736f3f63d2b12e17964b2ccc440b6a34499ce5"} Nov 03 19:53:03.130554 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.130532 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" event={"ID":"edcfb7ab-b14f-4d5c-b752-5b43a6eee163","Type":"ContainerStarted","Data":"05d1f7c6803b28af0b4c887a8576d44944ba6af0d944a9770f2f845a5a03f004"} Nov 03 19:53:03.140095 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.140035 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:03.185693 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.169850 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 19:53:03.185693 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.169908 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" podUID="34484729-bde1-4b0f-9758-e934b14f33d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 19:53:03.322589 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.282884 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e4f739-np4pk"] Nov 03 19:53:03.322589 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.283860 5059 cpu_manager.go:414] "containerMap: removing container" podUID="61622474-878f-4bf9-bb70-e8e4ce4e2bf0" containerName="revoke-keys" Nov 03 19:53:03.322589 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.283880 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61622474-878f-4bf9-bb70-e8e4ce4e2bf0" containerName="revoke-keys" Nov 03 19:53:03.322589 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.284091 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="61622474-878f-4bf9-bb70-e8e4ce4e2bf0" containerName="revoke-keys" Nov 03 19:53:03.322589 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.297262 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e4f739-np4pk"] Nov 03 19:53:03.322589 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.297389 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e4f739-np4pk" Nov 03 19:53:03.322589 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.302541 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:53:03.322589 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.312838 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 19:53:03.322589 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.321111 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" podStartSLOduration=2.321092041 podStartE2EDuration="2.321092041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:03.16946479 +0000 UTC m=+977.427262120" watchObservedRunningTime="2026-11-03 19:53:03.321092041 +0000 UTC m=+977.578889301" Nov 03 19:53:03.341183 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.341137 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" podStartSLOduration=2.341122562 podStartE2EDuration="2.341122562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:03.282130972 +0000 UTC m=+977.539928233" watchObservedRunningTime="2026-11-03 19:53:03.341122562 +0000 UTC m=+977.598919818" Nov 03 19:53:03.380476 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.380429 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwzxg\" (UniqueName: \"kubernetes.io/projected/29f9fdd0-72c1-46b1-888a-b413f9756773-kube-api-access-mwzxg\") pod \"maas-api-revoke-sub-d2e4f739-np4pk\" (UID: \"29f9fdd0-72c1-46b1-888a-b413f9756773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e4f739-np4pk" Nov 03 19:53:03.382870 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.382838 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29f9fdd0-72c1-46b1-888a-b413f9756773-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2e4f739-np4pk\" (UID: \"29f9fdd0-72c1-46b1-888a-b413f9756773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e4f739-np4pk" Nov 03 19:53:03.385831 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.385808 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29f9fdd0-72c1-46b1-888a-b413f9756773-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2e4f739-np4pk\" (UID: \"29f9fdd0-72c1-46b1-888a-b413f9756773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e4f739-np4pk" Nov 03 19:53:03.490686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.488527 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwzxg\" (UniqueName: \"kubernetes.io/projected/29f9fdd0-72c1-46b1-888a-b413f9756773-kube-api-access-mwzxg\") pod \"maas-api-revoke-sub-d2e4f739-np4pk\" (UID: \"29f9fdd0-72c1-46b1-888a-b413f9756773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e4f739-np4pk" Nov 03 19:53:03.490686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.488610 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29f9fdd0-72c1-46b1-888a-b413f9756773-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2e4f739-np4pk\" (UID: \"29f9fdd0-72c1-46b1-888a-b413f9756773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e4f739-np4pk" Nov 03 19:53:03.490686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.488756 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29f9fdd0-72c1-46b1-888a-b413f9756773-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2e4f739-np4pk\" (UID: \"29f9fdd0-72c1-46b1-888a-b413f9756773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e4f739-np4pk" Nov 03 19:53:03.490686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.490451 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29f9fdd0-72c1-46b1-888a-b413f9756773-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2e4f739-np4pk\" (UID: \"29f9fdd0-72c1-46b1-888a-b413f9756773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e4f739-np4pk" Nov 03 19:53:03.493127 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.493101 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29f9fdd0-72c1-46b1-888a-b413f9756773-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2e4f739-np4pk\" (UID: \"29f9fdd0-72c1-46b1-888a-b413f9756773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e4f739-np4pk" Nov 03 19:53:03.517653 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.517604 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwzxg\" (UniqueName: \"kubernetes.io/projected/29f9fdd0-72c1-46b1-888a-b413f9756773-kube-api-access-mwzxg\") pod \"maas-api-revoke-sub-d2e4f739-np4pk\" (UID: \"29f9fdd0-72c1-46b1-888a-b413f9756773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e4f739-np4pk" Nov 03 19:53:03.644600 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:03.644520 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e4f739-np4pk" Nov 03 19:53:04.040855 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:04.040820 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e4f739-np4pk"] Nov 03 19:53:04.122684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:04.116877 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:04.122684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:04.117012 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 19:53:04.122684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:04.117062 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" podUID="edcfb7ab-b14f-4d5c-b752-5b43a6eee163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 19:53:04.142335 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:04.142300 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 19:53:04.142900 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:04.142871 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" podUID="34484729-bde1-4b0f-9758-e934b14f33d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 19:53:04.174684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:04.166704 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e4f739-np4pk" event={"ID":"29f9fdd0-72c1-46b1-888a-b413f9756773","Type":"ContainerStarted","Data":"4cdc9a42181d44bfb77ee0ac3623be26d422f2def3149b7294b7e01e8c42af17"} Nov 03 19:53:05.112106 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:05.112060 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 19:53:05.112313 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:05.112126 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" podUID="edcfb7ab-b14f-4d5c-b752-5b43a6eee163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 19:53:05.138639 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:05.138601 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 19:53:05.138812 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:05.138647 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" podUID="34484729-bde1-4b0f-9758-e934b14f33d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 19:53:05.221630 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:05.221592 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e4f739-np4pk" event={"ID":"29f9fdd0-72c1-46b1-888a-b413f9756773","Type":"ContainerStarted","Data":"5c3861f203c47baef9469e3007567abfe7482aa19062187054c5a5411e088e7a"} Nov 03 19:53:05.245123 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:05.245056 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e4f739-np4pk" podStartSLOduration=2.24503782 podStartE2EDuration="2.24503782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:05.240906952 +0000 UTC m=+979.498704209" watchObservedRunningTime="2026-11-03 19:53:05.24503782 +0000 UTC m=+979.502835078" Nov 03 19:53:06.108695 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:06.108038 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 19:53:06.108695 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:06.108098 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" podUID="edcfb7ab-b14f-4d5c-b752-5b43a6eee163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 19:53:06.139025 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:06.138995 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 19:53:06.139194 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:06.139041 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" podUID="34484729-bde1-4b0f-9758-e934b14f33d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 19:53:06.231126 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:06.231043 5059 generic.go:356] "Generic (PLEG): container finished" podID="29f9fdd0-72c1-46b1-888a-b413f9756773" containerID="5c3861f203c47baef9469e3007567abfe7482aa19062187054c5a5411e088e7a" exitCode=0 Nov 03 19:53:06.231126 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:06.231088 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e4f739-np4pk" event={"ID":"29f9fdd0-72c1-46b1-888a-b413f9756773","Type":"ContainerDied","Data":"5c3861f203c47baef9469e3007567abfe7482aa19062187054c5a5411e088e7a"} Nov 03 19:53:07.109694 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:07.109161 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 19:53:07.109694 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:07.109232 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" podUID="edcfb7ab-b14f-4d5c-b752-5b43a6eee163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 19:53:07.143684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:07.141847 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 19:53:07.143684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:07.141907 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" podUID="34484729-bde1-4b0f-9758-e934b14f33d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 19:53:07.824327 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:07.824297 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e4f739-np4pk" Nov 03 19:53:07.846743 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:07.846717 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29f9fdd0-72c1-46b1-888a-b413f9756773-kube-api-access-mwzxg\" (UniqueName: \"kubernetes.io/projected/29f9fdd0-72c1-46b1-888a-b413f9756773-kube-api-access-mwzxg\") pod \"29f9fdd0-72c1-46b1-888a-b413f9756773\" (UID: \"29f9fdd0-72c1-46b1-888a-b413f9756773\") " Nov 03 19:53:07.846981 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:07.846958 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29f9fdd0-72c1-46b1-888a-b413f9756773-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29f9fdd0-72c1-46b1-888a-b413f9756773-maas-api-xks-ca\") pod \"29f9fdd0-72c1-46b1-888a-b413f9756773\" (UID: \"29f9fdd0-72c1-46b1-888a-b413f9756773\") " Nov 03 19:53:07.847286 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:07.847266 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29f9fdd0-72c1-46b1-888a-b413f9756773-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29f9fdd0-72c1-46b1-888a-b413f9756773-maas-api-openshift-ca\") pod \"29f9fdd0-72c1-46b1-888a-b413f9756773\" (UID: \"29f9fdd0-72c1-46b1-888a-b413f9756773\") " Nov 03 19:53:07.848219 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:07.848188 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29f9fdd0-72c1-46b1-888a-b413f9756773-maas-api-openshift-ca" pod "29f9fdd0-72c1-46b1-888a-b413f9756773" (UID: "29f9fdd0-72c1-46b1-888a-b413f9756773"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:07.851701 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:07.850009 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29f9fdd0-72c1-46b1-888a-b413f9756773-kube-api-access-mwzxg" pod "29f9fdd0-72c1-46b1-888a-b413f9756773" (UID: "29f9fdd0-72c1-46b1-888a-b413f9756773"). InnerVolumeSpecName "kube-api-access-mwzxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:07.853705 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:07.853682 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29f9fdd0-72c1-46b1-888a-b413f9756773-maas-api-xks-ca" pod "29f9fdd0-72c1-46b1-888a-b413f9756773" (UID: "29f9fdd0-72c1-46b1-888a-b413f9756773"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:53:07.949517 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:07.949462 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwzxg\" (UniqueName: \"kubernetes.io/projected/29f9fdd0-72c1-46b1-888a-b413f9756773-kube-api-access-mwzxg\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:07.949709 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:07.949696 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29f9fdd0-72c1-46b1-888a-b413f9756773-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:07.949787 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:07.949778 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29f9fdd0-72c1-46b1-888a-b413f9756773-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:08.110742 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:08.110036 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 19:53:08.110742 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:08.110100 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" podUID="edcfb7ab-b14f-4d5c-b752-5b43a6eee163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 19:53:08.138581 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:08.138555 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 19:53:08.138799 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:08.138600 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" podUID="34484729-bde1-4b0f-9758-e934b14f33d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 19:53:08.290689 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:08.289252 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e4f739-np4pk" event={"ID":"29f9fdd0-72c1-46b1-888a-b413f9756773","Type":"ContainerDied","Data":"4cdc9a42181d44bfb77ee0ac3623be26d422f2def3149b7294b7e01e8c42af17"} Nov 03 19:53:08.290689 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:08.289317 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cdc9a42181d44bfb77ee0ac3623be26d422f2def3149b7294b7e01e8c42af17" Nov 03 19:53:08.290689 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:08.289387 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e4f739-np4pk" Nov 03 19:53:09.107864 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:09.107833 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 19:53:09.108281 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:09.107884 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" podUID="edcfb7ab-b14f-4d5c-b752-5b43a6eee163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 19:53:09.139415 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:09.139388 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 19:53:09.139557 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:09.139435 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" podUID="34484729-bde1-4b0f-9758-e934b14f33d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 19:53:10.250689 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:10.250623 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:10.251115 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:10.251060 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:10.252337 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:10.252312 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 19:53:10.559463 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:10.559370 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:10.559883 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:10.559860 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:10.560812 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:10.560787 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 19:53:10.605964 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:53:10.605930 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:53:14.440816 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:14.440749 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:14.536708 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:14.535755 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:14.876697 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:14.875750 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:15.038687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:15.032742 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": context deadline exceeded" Nov 03 19:53:15.081844 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:15.081797 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:15.452844 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:15.452793 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:15.490686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:15.479083 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:53:15.919710 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:15.910720 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 19:53:15.919710 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:15.911099 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 19:53:15.949710 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:15.949461 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:16.195751 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:16.127110 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:16.250590 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:16.250555 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:53:16.551848 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:16.551770 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:53:16.558780 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:16.558757 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:53:16.559019 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:16.559004 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 19:53:16.559799 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:16.492758 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:16.930685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:16.915747 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:16.930685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:16.917745 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:16.930685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:16.926774 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:53:17.043912 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:17.042724 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Nov 03 19:53:17.060712 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:17.058420 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:53:17.094356 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:17.094324 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:53:17.175008 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:17.160774 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:17.442903 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:17.442876 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 19:53:17.445184 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:17.445030 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:53:20.494223 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:20.494182 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-361e6db7-k66z9"] Nov 03 19:53:20.495107 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:20.495086 5059 cpu_manager.go:414] "containerMap: removing container" podUID="29f9fdd0-72c1-46b1-888a-b413f9756773" containerName="revoke-keys" Nov 03 19:53:20.495107 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:20.495107 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29f9fdd0-72c1-46b1-888a-b413f9756773" containerName="revoke-keys" Nov 03 19:53:20.495310 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:20.495274 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="29f9fdd0-72c1-46b1-888a-b413f9756773" containerName="revoke-keys" Nov 03 19:53:20.500186 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:20.500164 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361e6db7-k66z9" Nov 03 19:53:20.507932 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:20.507908 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:53:20.509195 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:20.509174 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 19:53:20.510325 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:20.510300 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-361e6db7-k66z9"] Nov 03 19:53:20.598189 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:20.598159 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-361e6db7-k66z9\" (UID: \"c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361e6db7-k66z9" Nov 03 19:53:20.598375 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:20.598343 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5bh4\" (UniqueName: \"kubernetes.io/projected/c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a-kube-api-access-f5bh4\") pod \"maas-api-revoke-sub-361e6db7-k66z9\" (UID: \"c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361e6db7-k66z9" Nov 03 19:53:20.598444 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:20.598376 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-361e6db7-k66z9\" (UID: \"c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361e6db7-k66z9" Nov 03 19:53:20.699327 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:20.699294 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-361e6db7-k66z9\" (UID: \"c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361e6db7-k66z9" Nov 03 19:53:20.699517 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:20.699416 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-361e6db7-k66z9\" (UID: \"c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361e6db7-k66z9" Nov 03 19:53:20.699629 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:20.699614 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5bh4\" (UniqueName: \"kubernetes.io/projected/c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a-kube-api-access-f5bh4\") pod \"maas-api-revoke-sub-361e6db7-k66z9\" (UID: \"c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361e6db7-k66z9" Nov 03 19:53:20.700056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:20.700024 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-361e6db7-k66z9\" (UID: \"c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361e6db7-k66z9" Nov 03 19:53:20.701882 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:20.701859 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-361e6db7-k66z9\" (UID: \"c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361e6db7-k66z9" Nov 03 19:53:20.712643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:20.712618 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5bh4\" (UniqueName: \"kubernetes.io/projected/c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a-kube-api-access-f5bh4\") pod \"maas-api-revoke-sub-361e6db7-k66z9\" (UID: \"c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361e6db7-k66z9" Nov 03 19:53:20.836474 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:20.836446 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361e6db7-k66z9" Nov 03 19:53:21.135598 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:21.135281 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-361e6db7-k66z9"] Nov 03 19:53:21.137698 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:53:21.137651 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53ca540_ed8f_49a9_b9a3_c2ffe2e2013a.slice/crio-44d108ae11913e9f7daf072e57304964ff28fb3bc60cb009b8010fca2db923e9 WatchSource:0}: Error finding container 44d108ae11913e9f7daf072e57304964ff28fb3bc60cb009b8010fca2db923e9: Status 404 returned error can't find the container with id 44d108ae11913e9f7daf072e57304964ff28fb3bc60cb009b8010fca2db923e9 Nov 03 19:53:21.368722 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:21.368691 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361e6db7-k66z9" event={"ID":"c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a","Type":"ContainerStarted","Data":"44d108ae11913e9f7daf072e57304964ff28fb3bc60cb009b8010fca2db923e9"} Nov 03 19:53:22.378321 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:22.378288 5059 generic.go:356] "Generic (PLEG): container finished" podID="c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a" containerID="46b72c3b34b23f6c120c42912125e6cd5c93dffc15b678cbee23afdd51a40c8a" exitCode=0 Nov 03 19:53:22.378735 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:22.378377 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361e6db7-k66z9" event={"ID":"c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a","Type":"ContainerDied","Data":"46b72c3b34b23f6c120c42912125e6cd5c93dffc15b678cbee23afdd51a40c8a"} Nov 03 19:53:23.538370 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:23.538349 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361e6db7-k66z9" Nov 03 19:53:23.658950 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:23.658871 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a-maas-api-openshift-ca\") pod \"c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a\" (UID: \"c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a\") " Nov 03 19:53:23.659104 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:23.658972 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a-kube-api-access-f5bh4\" (UniqueName: \"kubernetes.io/projected/c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a-kube-api-access-f5bh4\") pod \"c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a\" (UID: \"c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a\") " Nov 03 19:53:23.659104 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:23.659058 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a-maas-api-xks-ca\") pod \"c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a\" (UID: \"c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a\") " Nov 03 19:53:23.659275 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:23.659251 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a-maas-api-openshift-ca" pod "c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a" (UID: "c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:23.659709 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:23.659685 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:23.661180 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:23.661159 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a-maas-api-xks-ca" pod "c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a" (UID: "c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:53:23.661285 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:23.661256 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a-kube-api-access-f5bh4" pod "c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a" (UID: "c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a"). InnerVolumeSpecName "kube-api-access-f5bh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:23.763196 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:23.760953 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5bh4\" (UniqueName: \"kubernetes.io/projected/c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a-kube-api-access-f5bh4\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:23.763196 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:23.760999 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:24.396130 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:24.396098 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361e6db7-k66z9" Nov 03 19:53:24.396284 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:24.396139 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361e6db7-k66z9" event={"ID":"c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a","Type":"ContainerDied","Data":"44d108ae11913e9f7daf072e57304964ff28fb3bc60cb009b8010fca2db923e9"} Nov 03 19:53:24.396284 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:24.396217 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44d108ae11913e9f7daf072e57304964ff28fb3bc60cb009b8010fca2db923e9" Nov 03 19:53:25.286831 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:25.286796 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac00d17d-466c6"] Nov 03 19:53:25.287741 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:25.287718 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a" containerName="revoke-keys" Nov 03 19:53:25.287741 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:25.287741 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a" containerName="revoke-keys" Nov 03 19:53:25.287950 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:25.287936 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a" containerName="revoke-keys" Nov 03 19:53:25.291615 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:25.291594 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac00d17d-466c6" Nov 03 19:53:25.296102 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:25.296082 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:53:25.298847 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:25.298827 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 19:53:25.302003 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:25.301767 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac00d17d-466c6"] Nov 03 19:53:25.384687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:25.384632 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b1001f7-8d10-41a9-b180-e42edcdb8faf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac00d17d-466c6\" (UID: \"6b1001f7-8d10-41a9-b180-e42edcdb8faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac00d17d-466c6" Nov 03 19:53:25.384837 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:25.384819 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tw6s\" (UniqueName: \"kubernetes.io/projected/6b1001f7-8d10-41a9-b180-e42edcdb8faf-kube-api-access-4tw6s\") pod \"maas-api-revoke-sub-ac00d17d-466c6\" (UID: \"6b1001f7-8d10-41a9-b180-e42edcdb8faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac00d17d-466c6" Nov 03 19:53:25.384889 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:25.384874 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b1001f7-8d10-41a9-b180-e42edcdb8faf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac00d17d-466c6\" (UID: \"6b1001f7-8d10-41a9-b180-e42edcdb8faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac00d17d-466c6" Nov 03 19:53:25.486138 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:25.486100 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tw6s\" (UniqueName: \"kubernetes.io/projected/6b1001f7-8d10-41a9-b180-e42edcdb8faf-kube-api-access-4tw6s\") pod \"maas-api-revoke-sub-ac00d17d-466c6\" (UID: \"6b1001f7-8d10-41a9-b180-e42edcdb8faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac00d17d-466c6" Nov 03 19:53:25.486331 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:25.486172 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b1001f7-8d10-41a9-b180-e42edcdb8faf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac00d17d-466c6\" (UID: \"6b1001f7-8d10-41a9-b180-e42edcdb8faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac00d17d-466c6" Nov 03 19:53:25.486406 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:25.486341 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b1001f7-8d10-41a9-b180-e42edcdb8faf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac00d17d-466c6\" (UID: \"6b1001f7-8d10-41a9-b180-e42edcdb8faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac00d17d-466c6" Nov 03 19:53:25.487096 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:25.487073 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b1001f7-8d10-41a9-b180-e42edcdb8faf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac00d17d-466c6\" (UID: \"6b1001f7-8d10-41a9-b180-e42edcdb8faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac00d17d-466c6" Nov 03 19:53:25.488974 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:25.488947 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b1001f7-8d10-41a9-b180-e42edcdb8faf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac00d17d-466c6\" (UID: \"6b1001f7-8d10-41a9-b180-e42edcdb8faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac00d17d-466c6" Nov 03 19:53:25.498533 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:25.498509 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tw6s\" (UniqueName: \"kubernetes.io/projected/6b1001f7-8d10-41a9-b180-e42edcdb8faf-kube-api-access-4tw6s\") pod \"maas-api-revoke-sub-ac00d17d-466c6\" (UID: \"6b1001f7-8d10-41a9-b180-e42edcdb8faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac00d17d-466c6" Nov 03 19:53:25.615675 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:25.615628 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac00d17d-466c6" Nov 03 19:53:25.959447 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:53:25.959408 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b1001f7_8d10_41a9_b180_e42edcdb8faf.slice/crio-b569fbe43956d709d43c43be78a870564529df40eb6200b42dcc03cf7c5ffaac WatchSource:0}: Error finding container b569fbe43956d709d43c43be78a870564529df40eb6200b42dcc03cf7c5ffaac: Status 404 returned error can't find the container with id b569fbe43956d709d43c43be78a870564529df40eb6200b42dcc03cf7c5ffaac Nov 03 19:53:25.959955 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:25.959930 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac00d17d-466c6"] Nov 03 19:53:26.414048 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:26.414016 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac00d17d-466c6" event={"ID":"6b1001f7-8d10-41a9-b180-e42edcdb8faf","Type":"ContainerStarted","Data":"ce1f3ec2b07fb6c3130cce90f9ce0da535e7d8528d2cfa3f212189b7edff7789"} Nov 03 19:53:26.414435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:26.414070 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac00d17d-466c6" event={"ID":"6b1001f7-8d10-41a9-b180-e42edcdb8faf","Type":"ContainerStarted","Data":"b569fbe43956d709d43c43be78a870564529df40eb6200b42dcc03cf7c5ffaac"} Nov 03 19:53:26.433906 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:26.433863 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac00d17d-466c6" podStartSLOduration=1.433850193 podStartE2EDuration="1.433850193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:26.431522724 +0000 UTC m=+1000.689319981" watchObservedRunningTime="2026-11-03 19:53:26.433850193 +0000 UTC m=+1000.691647450" Nov 03 19:53:27.424350 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:27.424315 5059 generic.go:356] "Generic (PLEG): container finished" podID="6b1001f7-8d10-41a9-b180-e42edcdb8faf" containerID="ce1f3ec2b07fb6c3130cce90f9ce0da535e7d8528d2cfa3f212189b7edff7789" exitCode=0 Nov 03 19:53:27.424802 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:27.424491 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac00d17d-466c6" event={"ID":"6b1001f7-8d10-41a9-b180-e42edcdb8faf","Type":"ContainerDied","Data":"ce1f3ec2b07fb6c3130cce90f9ce0da535e7d8528d2cfa3f212189b7edff7789"} Nov 03 19:53:29.452746 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:29.448044 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:29.936241 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:29.882111 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:30.009198 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:30.009163 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac00d17d-466c6" Nov 03 19:53:30.101744 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:30.086570 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:30.157656 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:30.131385 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 19:53:30.181223 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:30.181180 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b1001f7-8d10-41a9-b180-e42edcdb8faf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b1001f7-8d10-41a9-b180-e42edcdb8faf-maas-api-xks-ca\") pod \"6b1001f7-8d10-41a9-b180-e42edcdb8faf\" (UID: \"6b1001f7-8d10-41a9-b180-e42edcdb8faf\") " Nov 03 19:53:30.226972 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:30.226873 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b1001f7-8d10-41a9-b180-e42edcdb8faf-kube-api-access-4tw6s\" (UniqueName: \"kubernetes.io/projected/6b1001f7-8d10-41a9-b180-e42edcdb8faf-kube-api-access-4tw6s\") pod \"6b1001f7-8d10-41a9-b180-e42edcdb8faf\" (UID: \"6b1001f7-8d10-41a9-b180-e42edcdb8faf\") " Nov 03 19:53:30.262897 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:30.262864 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b1001f7-8d10-41a9-b180-e42edcdb8faf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b1001f7-8d10-41a9-b180-e42edcdb8faf-maas-api-openshift-ca\") pod \"6b1001f7-8d10-41a9-b180-e42edcdb8faf\" (UID: \"6b1001f7-8d10-41a9-b180-e42edcdb8faf\") " Nov 03 19:53:30.264560 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:30.263768 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b1001f7-8d10-41a9-b180-e42edcdb8faf-maas-api-openshift-ca" pod "6b1001f7-8d10-41a9-b180-e42edcdb8faf" (UID: "6b1001f7-8d10-41a9-b180-e42edcdb8faf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:30.417909 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:30.371867 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b1001f7-8d10-41a9-b180-e42edcdb8faf-kube-api-access-4tw6s" pod "6b1001f7-8d10-41a9-b180-e42edcdb8faf" (UID: "6b1001f7-8d10-41a9-b180-e42edcdb8faf"). InnerVolumeSpecName "kube-api-access-4tw6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:30.419065 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:30.419034 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tw6s\" (UniqueName: \"kubernetes.io/projected/6b1001f7-8d10-41a9-b180-e42edcdb8faf-kube-api-access-4tw6s\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:30.419448 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:30.419424 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b1001f7-8d10-41a9-b180-e42edcdb8faf-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:30.445925 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:30.445888 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b1001f7-8d10-41a9-b180-e42edcdb8faf-maas-api-xks-ca" pod "6b1001f7-8d10-41a9-b180-e42edcdb8faf" (UID: "6b1001f7-8d10-41a9-b180-e42edcdb8faf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:53:30.465340 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:30.465299 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:30.466940 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:30.466916 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:53:30.506838 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:30.506748 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:53:30.507108 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:30.507078 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" podUID="d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:30.573686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:30.547023 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b1001f7-8d10-41a9-b180-e42edcdb8faf-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:30.972456 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:30.972418 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f4349e5-7kcw6"] Nov 03 19:53:31.110708 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:30.991965 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac00d17d-466c6" Nov 03 19:53:31.111208 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:31.111017 5059 cpu_manager.go:414] "containerMap: removing container" podUID="6b1001f7-8d10-41a9-b180-e42edcdb8faf" containerName="revoke-keys" Nov 03 19:53:31.111393 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:31.111375 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b1001f7-8d10-41a9-b180-e42edcdb8faf" containerName="revoke-keys" Nov 03 19:53:31.111874 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:31.111855 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b1001f7-8d10-41a9-b180-e42edcdb8faf" containerName="revoke-keys" Nov 03 19:53:31.122282 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:31.121569 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:31.122603 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:31.122585 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f4349e5-7kcw6"] Nov 03 19:53:31.132760 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:31.132732 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac00d17d-466c6" event={"ID":"6b1001f7-8d10-41a9-b180-e42edcdb8faf","Type":"ContainerDied","Data":"b569fbe43956d709d43c43be78a870564529df40eb6200b42dcc03cf7c5ffaac"} Nov 03 19:53:31.132945 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:31.132929 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b569fbe43956d709d43c43be78a870564529df40eb6200b42dcc03cf7c5ffaac" Nov 03 19:53:31.133101 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:31.133090 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:53:31.138786 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:31.138764 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4349e5-7kcw6" Nov 03 19:53:31.155074 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:31.145612 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:53:31.155074 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:31.153176 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:53:31.155074 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:31.154052 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 19:53:31.223844 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:31.200254 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d3059c-9b84-4630-9e19-e4eef5d458f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f4349e5-7kcw6\" (UID: \"10d3059c-9b84-4630-9e19-e4eef5d458f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4349e5-7kcw6" Nov 03 19:53:31.223844 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:31.200466 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hphx\" (UniqueName: \"kubernetes.io/projected/10d3059c-9b84-4630-9e19-e4eef5d458f1-kube-api-access-7hphx\") pod \"maas-api-revoke-sub-5f4349e5-7kcw6\" (UID: \"10d3059c-9b84-4630-9e19-e4eef5d458f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4349e5-7kcw6" Nov 03 19:53:31.223844 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:31.200541 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d3059c-9b84-4630-9e19-e4eef5d458f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f4349e5-7kcw6\" (UID: \"10d3059c-9b84-4630-9e19-e4eef5d458f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4349e5-7kcw6" Nov 03 19:53:31.316688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:31.305322 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hphx\" (UniqueName: \"kubernetes.io/projected/10d3059c-9b84-4630-9e19-e4eef5d458f1-kube-api-access-7hphx\") pod \"maas-api-revoke-sub-5f4349e5-7kcw6\" (UID: \"10d3059c-9b84-4630-9e19-e4eef5d458f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4349e5-7kcw6" Nov 03 19:53:31.317012 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:31.316986 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d3059c-9b84-4630-9e19-e4eef5d458f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f4349e5-7kcw6\" (UID: \"10d3059c-9b84-4630-9e19-e4eef5d458f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4349e5-7kcw6" Nov 03 19:53:31.317303 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:31.317283 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d3059c-9b84-4630-9e19-e4eef5d458f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f4349e5-7kcw6\" (UID: \"10d3059c-9b84-4630-9e19-e4eef5d458f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4349e5-7kcw6" Nov 03 19:53:31.329180 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:31.329152 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d3059c-9b84-4630-9e19-e4eef5d458f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f4349e5-7kcw6\" (UID: \"10d3059c-9b84-4630-9e19-e4eef5d458f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4349e5-7kcw6" Nov 03 19:53:31.466731 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:31.453232 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hphx\" (UniqueName: \"kubernetes.io/projected/10d3059c-9b84-4630-9e19-e4eef5d458f1-kube-api-access-7hphx\") pod \"maas-api-revoke-sub-5f4349e5-7kcw6\" (UID: \"10d3059c-9b84-4630-9e19-e4eef5d458f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4349e5-7kcw6" Nov 03 19:53:31.466731 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:31.453573 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d3059c-9b84-4630-9e19-e4eef5d458f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f4349e5-7kcw6\" (UID: \"10d3059c-9b84-4630-9e19-e4eef5d458f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4349e5-7kcw6" Nov 03 19:53:31.537955 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:31.537917 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:31.586685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:31.556467 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4349e5-7kcw6" Nov 03 19:53:31.754261 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:31.754230 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:53:31.802709 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:53:31.800706 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b1001f7_8d10_41a9_b180_e42edcdb8faf.slice\": RecentStats: unable to find data in memory cache]" Nov 03 19:53:32.139776 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:32.139751 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f4349e5-7kcw6"] Nov 03 19:53:33.026920 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:33.026884 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4349e5-7kcw6" event={"ID":"10d3059c-9b84-4630-9e19-e4eef5d458f1","Type":"ContainerStarted","Data":"6ff5e1dfb0ee483d176ff2e04b3217dba0c168d58625e1cb163e71d5b8eeadfd"} Nov 03 19:53:33.027388 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:33.026935 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4349e5-7kcw6" event={"ID":"10d3059c-9b84-4630-9e19-e4eef5d458f1","Type":"ContainerStarted","Data":"9f9d82fa658c46de219578acd52b218b4e2174b53df664d9a144551ab6659526"} Nov 03 19:53:33.048761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:33.048719 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4349e5-7kcw6" podStartSLOduration=3.048704891 podStartE2EDuration="3.048704891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:33.04617908 +0000 UTC m=+1007.303976338" watchObservedRunningTime="2026-11-03 19:53:33.048704891 +0000 UTC m=+1007.306502180" Nov 03 19:53:34.038412 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:34.038378 5059 generic.go:356] "Generic (PLEG): container finished" podID="10d3059c-9b84-4630-9e19-e4eef5d458f1" containerID="6ff5e1dfb0ee483d176ff2e04b3217dba0c168d58625e1cb163e71d5b8eeadfd" exitCode=0 Nov 03 19:53:34.038930 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:34.038462 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4349e5-7kcw6" event={"ID":"10d3059c-9b84-4630-9e19-e4eef5d458f1","Type":"ContainerDied","Data":"6ff5e1dfb0ee483d176ff2e04b3217dba0c168d58625e1cb163e71d5b8eeadfd"} Nov 03 19:53:35.366744 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.366709 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4349e5-7kcw6" Nov 03 19:53:35.497589 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.497559 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10d3059c-9b84-4630-9e19-e4eef5d458f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d3059c-9b84-4630-9e19-e4eef5d458f1-maas-api-openshift-ca\") pod \"10d3059c-9b84-4630-9e19-e4eef5d458f1\" (UID: \"10d3059c-9b84-4630-9e19-e4eef5d458f1\") " Nov 03 19:53:35.497724 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.497648 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10d3059c-9b84-4630-9e19-e4eef5d458f1-kube-api-access-7hphx\" (UniqueName: \"kubernetes.io/projected/10d3059c-9b84-4630-9e19-e4eef5d458f1-kube-api-access-7hphx\") pod \"10d3059c-9b84-4630-9e19-e4eef5d458f1\" (UID: \"10d3059c-9b84-4630-9e19-e4eef5d458f1\") " Nov 03 19:53:35.498822 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.498050 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10d3059c-9b84-4630-9e19-e4eef5d458f1-maas-api-openshift-ca" pod "10d3059c-9b84-4630-9e19-e4eef5d458f1" (UID: "10d3059c-9b84-4630-9e19-e4eef5d458f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:35.498822 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.498113 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10d3059c-9b84-4630-9e19-e4eef5d458f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d3059c-9b84-4630-9e19-e4eef5d458f1-maas-api-xks-ca\") pod \"10d3059c-9b84-4630-9e19-e4eef5d458f1\" (UID: \"10d3059c-9b84-4630-9e19-e4eef5d458f1\") " Nov 03 19:53:35.498822 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.498789 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d3059c-9b84-4630-9e19-e4eef5d458f1-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:35.501746 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.501717 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10d3059c-9b84-4630-9e19-e4eef5d458f1-kube-api-access-7hphx" pod "10d3059c-9b84-4630-9e19-e4eef5d458f1" (UID: "10d3059c-9b84-4630-9e19-e4eef5d458f1"). InnerVolumeSpecName "kube-api-access-7hphx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:35.502017 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.501988 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10d3059c-9b84-4630-9e19-e4eef5d458f1-maas-api-xks-ca" pod "10d3059c-9b84-4630-9e19-e4eef5d458f1" (UID: "10d3059c-9b84-4630-9e19-e4eef5d458f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:53:35.506328 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.506303 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e23cb3e8-47thd"] Nov 03 19:53:35.507059 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.507037 5059 cpu_manager.go:414] "containerMap: removing container" podUID="10d3059c-9b84-4630-9e19-e4eef5d458f1" containerName="revoke-keys" Nov 03 19:53:35.507059 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.507055 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10d3059c-9b84-4630-9e19-e4eef5d458f1" containerName="revoke-keys" Nov 03 19:53:35.507218 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.507193 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="10d3059c-9b84-4630-9e19-e4eef5d458f1" containerName="revoke-keys" Nov 03 19:53:35.511009 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.510992 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23cb3e8-47thd" Nov 03 19:53:35.520985 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.520962 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e23cb3e8-47thd"] Nov 03 19:53:35.599768 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.599692 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4cbb\" (UniqueName: \"kubernetes.io/projected/914d427c-bf1c-43a4-836c-168626ee5397-kube-api-access-m4cbb\") pod \"maas-api-revoke-sub-e23cb3e8-47thd\" (UID: \"914d427c-bf1c-43a4-836c-168626ee5397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23cb3e8-47thd" Nov 03 19:53:35.599901 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.599814 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/914d427c-bf1c-43a4-836c-168626ee5397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e23cb3e8-47thd\" (UID: \"914d427c-bf1c-43a4-836c-168626ee5397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23cb3e8-47thd" Nov 03 19:53:35.600060 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.600033 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/914d427c-bf1c-43a4-836c-168626ee5397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e23cb3e8-47thd\" (UID: \"914d427c-bf1c-43a4-836c-168626ee5397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23cb3e8-47thd" Nov 03 19:53:35.600204 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.600189 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hphx\" (UniqueName: \"kubernetes.io/projected/10d3059c-9b84-4630-9e19-e4eef5d458f1-kube-api-access-7hphx\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:35.600243 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.600212 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d3059c-9b84-4630-9e19-e4eef5d458f1-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:35.701575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.701538 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/914d427c-bf1c-43a4-836c-168626ee5397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e23cb3e8-47thd\" (UID: \"914d427c-bf1c-43a4-836c-168626ee5397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23cb3e8-47thd" Nov 03 19:53:35.701766 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.701648 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4cbb\" (UniqueName: \"kubernetes.io/projected/914d427c-bf1c-43a4-836c-168626ee5397-kube-api-access-m4cbb\") pod \"maas-api-revoke-sub-e23cb3e8-47thd\" (UID: \"914d427c-bf1c-43a4-836c-168626ee5397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23cb3e8-47thd" Nov 03 19:53:35.701766 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.701747 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/914d427c-bf1c-43a4-836c-168626ee5397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e23cb3e8-47thd\" (UID: \"914d427c-bf1c-43a4-836c-168626ee5397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23cb3e8-47thd" Nov 03 19:53:35.702427 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.702400 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/914d427c-bf1c-43a4-836c-168626ee5397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e23cb3e8-47thd\" (UID: \"914d427c-bf1c-43a4-836c-168626ee5397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23cb3e8-47thd" Nov 03 19:53:35.704002 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.703979 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/914d427c-bf1c-43a4-836c-168626ee5397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e23cb3e8-47thd\" (UID: \"914d427c-bf1c-43a4-836c-168626ee5397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23cb3e8-47thd" Nov 03 19:53:35.715163 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.715140 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4cbb\" (UniqueName: \"kubernetes.io/projected/914d427c-bf1c-43a4-836c-168626ee5397-kube-api-access-m4cbb\") pod \"maas-api-revoke-sub-e23cb3e8-47thd\" (UID: \"914d427c-bf1c-43a4-836c-168626ee5397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23cb3e8-47thd" Nov 03 19:53:35.860461 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:35.860374 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23cb3e8-47thd" Nov 03 19:53:36.058811 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:36.058785 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4349e5-7kcw6" Nov 03 19:53:36.058976 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:36.058824 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f4349e5-7kcw6" event={"ID":"10d3059c-9b84-4630-9e19-e4eef5d458f1","Type":"ContainerDied","Data":"9f9d82fa658c46de219578acd52b218b4e2174b53df664d9a144551ab6659526"} Nov 03 19:53:36.058976 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:36.058883 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f9d82fa658c46de219578acd52b218b4e2174b53df664d9a144551ab6659526" Nov 03 19:53:36.156093 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:36.156069 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e23cb3e8-47thd"] Nov 03 19:53:36.159503 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:53:36.159472 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914d427c_bf1c_43a4_836c_168626ee5397.slice/crio-09d9d3c94489ff0e5cd2b775733cf27bced470519f08abc57848bf3e4912b826 WatchSource:0}: Error finding container 09d9d3c94489ff0e5cd2b775733cf27bced470519f08abc57848bf3e4912b826: Status 404 returned error can't find the container with id 09d9d3c94489ff0e5cd2b775733cf27bced470519f08abc57848bf3e4912b826 Nov 03 19:53:37.079850 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:37.079797 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23cb3e8-47thd" event={"ID":"914d427c-bf1c-43a4-836c-168626ee5397","Type":"ContainerStarted","Data":"d8cac496f3ba3b85c9a475d4c226c3e86199e7e674f24280b56e8eefa76ea668"} Nov 03 19:53:37.080262 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:37.079936 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23cb3e8-47thd" event={"ID":"914d427c-bf1c-43a4-836c-168626ee5397","Type":"ContainerStarted","Data":"09d9d3c94489ff0e5cd2b775733cf27bced470519f08abc57848bf3e4912b826"} Nov 03 19:53:37.103379 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:37.103325 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23cb3e8-47thd" podStartSLOduration=2.103310054 podStartE2EDuration="2.103310054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:37.098427705 +0000 UTC m=+1011.356224964" watchObservedRunningTime="2026-11-03 19:53:37.103310054 +0000 UTC m=+1011.361107321" Nov 03 19:53:38.089623 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:38.089575 5059 generic.go:356] "Generic (PLEG): container finished" podID="914d427c-bf1c-43a4-836c-168626ee5397" containerID="d8cac496f3ba3b85c9a475d4c226c3e86199e7e674f24280b56e8eefa76ea668" exitCode=0 Nov 03 19:53:38.090082 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:38.089656 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23cb3e8-47thd" event={"ID":"914d427c-bf1c-43a4-836c-168626ee5397","Type":"ContainerDied","Data":"d8cac496f3ba3b85c9a475d4c226c3e86199e7e674f24280b56e8eefa76ea668"} Nov 03 19:53:39.268606 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:39.268580 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23cb3e8-47thd" Nov 03 19:53:39.355241 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:39.355148 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/914d427c-bf1c-43a4-836c-168626ee5397-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/914d427c-bf1c-43a4-836c-168626ee5397-maas-api-openshift-ca\") pod \"914d427c-bf1c-43a4-836c-168626ee5397\" (UID: \"914d427c-bf1c-43a4-836c-168626ee5397\") " Nov 03 19:53:39.355606 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:39.355403 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914d427c-bf1c-43a4-836c-168626ee5397-kube-api-access-m4cbb\" (UniqueName: \"kubernetes.io/projected/914d427c-bf1c-43a4-836c-168626ee5397-kube-api-access-m4cbb\") pod \"914d427c-bf1c-43a4-836c-168626ee5397\" (UID: \"914d427c-bf1c-43a4-836c-168626ee5397\") " Nov 03 19:53:39.355606 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:39.355466 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/914d427c-bf1c-43a4-836c-168626ee5397-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/914d427c-bf1c-43a4-836c-168626ee5397-maas-api-xks-ca\") pod \"914d427c-bf1c-43a4-836c-168626ee5397\" (UID: \"914d427c-bf1c-43a4-836c-168626ee5397\") " Nov 03 19:53:39.356322 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:39.356094 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/914d427c-bf1c-43a4-836c-168626ee5397-maas-api-openshift-ca" pod "914d427c-bf1c-43a4-836c-168626ee5397" (UID: "914d427c-bf1c-43a4-836c-168626ee5397"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:39.356322 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:39.356294 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/914d427c-bf1c-43a4-836c-168626ee5397-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:39.358584 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:39.358556 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914d427c-bf1c-43a4-836c-168626ee5397-kube-api-access-m4cbb" pod "914d427c-bf1c-43a4-836c-168626ee5397" (UID: "914d427c-bf1c-43a4-836c-168626ee5397"). InnerVolumeSpecName "kube-api-access-m4cbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:39.359015 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:39.358997 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/914d427c-bf1c-43a4-836c-168626ee5397-maas-api-xks-ca" pod "914d427c-bf1c-43a4-836c-168626ee5397" (UID: "914d427c-bf1c-43a4-836c-168626ee5397"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:53:39.467304 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:39.467251 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4cbb\" (UniqueName: \"kubernetes.io/projected/914d427c-bf1c-43a4-836c-168626ee5397-kube-api-access-m4cbb\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:39.467304 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:39.467296 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/914d427c-bf1c-43a4-836c-168626ee5397-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:40.109237 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:40.109208 5059 generic.go:356] "Generic (PLEG): container finished" podID="070f2e92-dc4a-479a-a8dc-724e4cc4a050" containerID="4a9f00bb1e92bce9af0673cd4868bc76735066c2845448bfdb9dce793d7271d3" exitCode=0 Nov 03 19:53:40.109449 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:40.109277 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" event={"ID":"070f2e92-dc4a-479a-a8dc-724e4cc4a050","Type":"ContainerDied","Data":"4a9f00bb1e92bce9af0673cd4868bc76735066c2845448bfdb9dce793d7271d3"} Nov 03 19:53:40.109675 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:40.109642 5059 scope.go:122] "RemoveContainer" containerID="4a9f00bb1e92bce9af0673cd4868bc76735066c2845448bfdb9dce793d7271d3" Nov 03 19:53:40.110842 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:40.110818 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23cb3e8-47thd" Nov 03 19:53:40.110931 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:40.110869 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e23cb3e8-47thd" event={"ID":"914d427c-bf1c-43a4-836c-168626ee5397","Type":"ContainerDied","Data":"09d9d3c94489ff0e5cd2b775733cf27bced470519f08abc57848bf3e4912b826"} Nov 03 19:53:40.110931 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:40.110907 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09d9d3c94489ff0e5cd2b775733cf27bced470519f08abc57848bf3e4912b826" Nov 03 19:53:40.111192 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:40.111167 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:53:40.111247 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:40.111234 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:53:40.111296 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:40.111283 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:53:40.624129 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:53:40.624093 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:53:41.120934 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:41.120899 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" event={"ID":"070f2e92-dc4a-479a-a8dc-724e4cc4a050","Type":"ContainerStarted","Data":"836cc9a7ad5f2fb9f7bbdc2d0bbf637ebed2fac8ba1c6f024f6a6d5a432a64f3"} Nov 03 19:53:41.934192 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:41.931827 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:53:41.934192 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:41.931977 5059 patch_prober.go:29] interesting pod/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 19:53:41.934192 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:41.932022 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" podUID="070f2e92-dc4a-479a-a8dc-724e4cc4a050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 19:53:42.928216 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:42.928181 5059 patch_prober.go:29] interesting pod/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 19:53:42.928392 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:42.928229 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" podUID="070f2e92-dc4a-479a-a8dc-724e4cc4a050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 19:53:43.927442 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:43.927409 5059 patch_prober.go:29] interesting pod/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 19:53:43.927842 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:43.927460 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" podUID="070f2e92-dc4a-479a-a8dc-724e4cc4a050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 19:53:43.945136 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:43.945113 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:53:44.927738 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:44.927702 5059 patch_prober.go:29] interesting pod/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 19:53:44.928130 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:44.927756 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" podUID="070f2e92-dc4a-479a-a8dc-724e4cc4a050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 19:53:44.988622 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:44.988587 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv"] Nov 03 19:53:44.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:44.989315 5059 cpu_manager.go:414] "containerMap: removing container" podUID="914d427c-bf1c-43a4-836c-168626ee5397" containerName="revoke-keys" Nov 03 19:53:44.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:44.989335 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="914d427c-bf1c-43a4-836c-168626ee5397" containerName="revoke-keys" Nov 03 19:53:44.989445 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:44.989436 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="914d427c-bf1c-43a4-836c-168626ee5397" containerName="revoke-keys" Nov 03 19:53:44.992271 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:44.992253 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.006250 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.006227 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv"] Nov 03 19:53:45.113657 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.113603 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.113945 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.113915 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.114039 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.114011 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.114106 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.114052 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.114106 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.114078 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.114278 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.114261 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.114458 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.114413 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dswk2\" (UniqueName: \"kubernetes.io/projected/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-kube-api-access-dswk2\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.114653 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.114628 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.114979 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.114960 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.216173 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.216095 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.216173 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.216161 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.216373 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.216322 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.216435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.216385 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.216435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.216412 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.216544 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.216506 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.216544 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.216534 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.216657 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.216547 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dswk2\" (UniqueName: \"kubernetes.io/projected/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-kube-api-access-dswk2\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.216657 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.216584 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.216781 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.216756 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.216831 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.216790 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.217132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.217107 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.217222 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.217146 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.217222 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.217163 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.218803 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.218782 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.218893 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.218859 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.228925 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.228900 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.229017 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.228969 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dswk2\" (UniqueName: \"kubernetes.io/projected/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-kube-api-access-dswk2\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.308419 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.308385 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:45.618589 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.618559 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv"] Nov 03 19:53:45.619209 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:53:45.619179 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced1fbaf_4771_4fab_8df0_a661dffb0d1f.slice/crio-f6b35f6edc6789fb490759be5752bf83f6655e7e34518f17957ddaab18a9c802 WatchSource:0}: Error finding container f6b35f6edc6789fb490759be5752bf83f6655e7e34518f17957ddaab18a9c802: Status 404 returned error can't find the container with id f6b35f6edc6789fb490759be5752bf83f6655e7e34518f17957ddaab18a9c802 Nov 03 19:53:45.621973 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.621932 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:53:45.622059 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.622026 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:53:45.622100 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.622071 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:53:45.928005 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.927427 5059 patch_prober.go:29] interesting pod/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 19:53:45.928005 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:45.927471 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" podUID="070f2e92-dc4a-479a-a8dc-724e4cc4a050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 19:53:46.158205 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:46.158048 5059 scope.go:122] "RemoveContainer" containerID="98a4dc4f78b76af12391a8da0febef1c9605aa92341e584a518f3e67e5f06dd0" Nov 03 19:53:46.169532 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:46.169501 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" event={"ID":"ced1fbaf-4771-4fab-8df0-a661dffb0d1f","Type":"ContainerStarted","Data":"ea3db214c78be67a09c85ab42f65ff26dbb8345e6f3312da74da713db8595587"} Nov 03 19:53:46.169820 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:46.169591 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" event={"ID":"ced1fbaf-4771-4fab-8df0-a661dffb0d1f","Type":"ContainerStarted","Data":"f6b35f6edc6789fb490759be5752bf83f6655e7e34518f17957ddaab18a9c802"} Nov 03 19:53:46.171362 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:46.171340 5059 scope.go:122] "RemoveContainer" containerID="8592dd0724f9bcb2e4cf1bf26a434c67f9bf07e3467400b6041c3941b8890523" Nov 03 19:53:46.183383 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:46.183357 5059 scope.go:122] "RemoveContainer" containerID="a549776ace8f57bb78ee6a6a39e01694e2d70ae5714b8bbbc9ac2905d6aa41e4" Nov 03 19:53:46.212683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:46.212623 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" podStartSLOduration=2.212609831 podStartE2EDuration="2.212609831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:46.207542056 +0000 UTC m=+1020.465339313" watchObservedRunningTime="2026-11-03 19:53:46.212609831 +0000 UTC m=+1020.470407148" Nov 03 19:53:46.309606 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:46.309572 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:46.310988 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:46.310958 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 19:53:46.311119 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:46.311011 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" podUID="ced1fbaf-4771-4fab-8df0-a661dffb0d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 19:53:46.931924 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:46.931892 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:53:46.933188 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:46.933165 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:53:47.309835 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.309749 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 19:53:47.309835 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.309811 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" podUID="ced1fbaf-4771-4fab-8df0-a661dffb0d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 19:53:47.604572 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.604526 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk"] Nov 03 19:53:47.617907 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.617433 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk"] Nov 03 19:53:47.617907 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.617825 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.626194 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.626169 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0a2237-openshift-default-dockercfg-rjprv\"" Nov 03 19:53:47.755569 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.755526 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-istio-envoy\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.756001 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.755971 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b93733cb-f365-418c-893c-e7e8e358fc55-istiod-ca-cert\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.756199 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.756180 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-istio-data\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.756338 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.756320 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-credential-socket\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.756472 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.756456 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-workload-socket\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.756682 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.756649 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-workload-certs\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.756826 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.756809 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b93733cb-f365-418c-893c-e7e8e358fc55-istio-token\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.762992 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.762969 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b93733cb-f365-418c-893c-e7e8e358fc55-istio-podinfo\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.763154 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.763136 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2w95\" (UniqueName: \"kubernetes.io/projected/b93733cb-f365-418c-893c-e7e8e358fc55-kube-api-access-g2w95\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.869769 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.869627 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-istio-data\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.871455 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.871418 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-credential-socket\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.881894 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.881867 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-workload-socket\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.882572 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.882547 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-workload-certs\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.883286 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.883267 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b93733cb-f365-418c-893c-e7e8e358fc55-istio-token\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.883430 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.883416 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b93733cb-f365-418c-893c-e7e8e358fc55-istio-podinfo\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.884272 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.884252 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2w95\" (UniqueName: \"kubernetes.io/projected/b93733cb-f365-418c-893c-e7e8e358fc55-kube-api-access-g2w95\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.884766 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.884734 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-istio-envoy\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.885387 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.885368 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b93733cb-f365-418c-893c-e7e8e358fc55-istiod-ca-cert\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.886389 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.882389 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-workload-socket\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.886531 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.871924 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-credential-socket\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.886639 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.883007 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-workload-certs\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.886752 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.886173 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b93733cb-f365-418c-893c-e7e8e358fc55-istiod-ca-cert\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.886869 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.870355 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-istio-data\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.934510 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.934477 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2w95\" (UniqueName: \"kubernetes.io/projected/b93733cb-f365-418c-893c-e7e8e358fc55-kube-api-access-g2w95\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.946066 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.945996 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-istio-envoy\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.946247 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.946125 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b93733cb-f365-418c-893c-e7e8e358fc55-istio-token\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:47.946939 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:47.946915 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b93733cb-f365-418c-893c-e7e8e358fc55-istio-podinfo\") pod \"e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:48.237360 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:48.237274 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:48.308980 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:48.308945 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 19:53:48.309138 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:48.309006 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" podUID="ced1fbaf-4771-4fab-8df0-a661dffb0d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 19:53:48.551746 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:53:48.551712 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93733cb_f365_418c_893c_e7e8e358fc55.slice/crio-d72484012ecc90d9c89997e183c445621877ae4e6ecb152fafa8f8280d25ba79 WatchSource:0}: Error finding container d72484012ecc90d9c89997e183c445621877ae4e6ecb152fafa8f8280d25ba79: Status 404 returned error can't find the container with id d72484012ecc90d9c89997e183c445621877ae4e6ecb152fafa8f8280d25ba79 Nov 03 19:53:48.552637 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:48.552600 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk"] Nov 03 19:53:48.554526 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:48.554493 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:53:48.554603 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:48.554560 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:53:48.554603 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:48.554588 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:53:49.210732 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:49.210695 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" event={"ID":"b93733cb-f365-418c-893c-e7e8e358fc55","Type":"ContainerStarted","Data":"944da2903cc93746497d5c2e02b4669ae40c03f989ea3a04d91c59c28c2fac1e"} Nov 03 19:53:49.211197 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:49.210762 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" event={"ID":"b93733cb-f365-418c-893c-e7e8e358fc55","Type":"ContainerStarted","Data":"d72484012ecc90d9c89997e183c445621877ae4e6ecb152fafa8f8280d25ba79"} Nov 03 19:53:49.237357 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:49.237332 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:49.239038 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:49.239006 5059 patch_prober.go:29] interesting pod/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 19:53:49.239174 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:49.239061 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" podUID="b93733cb-f365-418c-893c-e7e8e358fc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 19:53:49.241797 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:49.241756 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" podStartSLOduration=2.241743078 podStartE2EDuration="2.241743078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:49.237586585 +0000 UTC m=+1023.495383843" watchObservedRunningTime="2026-11-03 19:53:49.241743078 +0000 UTC m=+1023.499540334" Nov 03 19:53:49.309899 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:49.309857 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 19:53:49.310063 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:49.309922 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" podUID="ced1fbaf-4771-4fab-8df0-a661dffb0d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 19:53:50.317686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:50.317022 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 19:53:50.317686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:50.317083 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" podUID="ced1fbaf-4771-4fab-8df0-a661dffb0d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 19:53:50.349721 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:50.339990 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:51.328693 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:51.316179 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 19:53:51.328693 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:51.316241 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" podUID="ced1fbaf-4771-4fab-8df0-a661dffb0d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 19:53:51.328693 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:51.318064 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:51.328693 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:51.324309 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:53:51.573690 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:51.570754 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:51.815744 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:51.813830 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:51.873197 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:51.873156 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:52.106686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:52.099741 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 19:53:52.155725 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:52.155681 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:52.187784 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:52.187740 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:52.309685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:52.308788 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 19:53:52.309685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:52.308851 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" podUID="ced1fbaf-4771-4fab-8df0-a661dffb0d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 19:53:52.658685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:52.656334 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:52.658685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:52.657022 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:53:53.192475 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:53.168767 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 19:53:53.192475 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:53.184064 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 19:53:53.192475 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:53.185086 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": context deadline exceeded" Nov 03 19:53:53.192475 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:53.185507 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:53:53.192475 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:53.185564 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:53:53.242682 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:53.242416 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:53:53.242682 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:53.242502 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" podUID="cef4417c-3d04-4d65-a5bc-6ec1af6f09bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:53.469765 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:53.469651 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 19:53:53.469765 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:53.469730 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" podUID="ced1fbaf-4771-4fab-8df0-a661dffb0d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 19:53:53.505473 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:53.505441 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 19:53:53.660134 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:53.639368 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 19:53:53.738645 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:53.737809 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 19:53:53.859781 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:53.859737 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck"] Nov 03 19:53:53.868902 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:53.868873 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck" Nov 03 19:53:53.887463 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:53.885987 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck"] Nov 03 19:53:53.921770 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:53.921741 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0a2237\"" Nov 03 19:53:53.923933 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:53.923907 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0a2237-dockercfg-s7746\"" Nov 03 19:53:54.076748 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:54.075687 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf8c2\" (UniqueName: \"kubernetes.io/projected/ee541613-f0de-462c-be0c-8c296bc90d1f-kube-api-access-nf8c2\") pod \"payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck\" (UID: \"ee541613-f0de-462c-be0c-8c296bc90d1f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck" Nov 03 19:53:54.076748 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:54.075844 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee541613-f0de-462c-be0c-8c296bc90d1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck\" (UID: \"ee541613-f0de-462c-be0c-8c296bc90d1f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck" Nov 03 19:53:54.177228 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:54.177189 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf8c2\" (UniqueName: \"kubernetes.io/projected/ee541613-f0de-462c-be0c-8c296bc90d1f-kube-api-access-nf8c2\") pod \"payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck\" (UID: \"ee541613-f0de-462c-be0c-8c296bc90d1f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck" Nov 03 19:53:54.178804 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:54.178773 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee541613-f0de-462c-be0c-8c296bc90d1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck\" (UID: \"ee541613-f0de-462c-be0c-8c296bc90d1f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck" Nov 03 19:53:54.180314 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:54.179968 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee541613-f0de-462c-be0c-8c296bc90d1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck\" (UID: \"ee541613-f0de-462c-be0c-8c296bc90d1f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck" Nov 03 19:53:54.195354 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:54.195325 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf8c2\" (UniqueName: \"kubernetes.io/projected/ee541613-f0de-462c-be0c-8c296bc90d1f-kube-api-access-nf8c2\") pod \"payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck\" (UID: \"ee541613-f0de-462c-be0c-8c296bc90d1f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck" Nov 03 19:53:54.225438 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:54.225395 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck" Nov 03 19:53:54.311784 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:54.311544 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 19:53:54.311784 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:54.311617 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" podUID="ced1fbaf-4771-4fab-8df0-a661dffb0d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 19:53:54.559753 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:53:54.559722 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee541613_f0de_462c_be0c_8c296bc90d1f.slice/crio-0ab04bfcda4e14d1cee4688e0cf0ce26c428a84b10193b4740ca76873c00cfb4 WatchSource:0}: Error finding container 0ab04bfcda4e14d1cee4688e0cf0ce26c428a84b10193b4740ca76873c00cfb4: Status 404 returned error can't find the container with id 0ab04bfcda4e14d1cee4688e0cf0ce26c428a84b10193b4740ca76873c00cfb4 Nov 03 19:53:54.563202 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:54.563147 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck"] Nov 03 19:53:55.564504 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:55.564451 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck" event={"ID":"ee541613-f0de-462c-be0c-8c296bc90d1f","Type":"ContainerStarted","Data":"35d11f9645b92bb2c958f0c84da12424a56021d8d29afe0f47749929a2158628"} Nov 03 19:53:55.565018 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:55.564529 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck" Nov 03 19:53:55.565018 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:55.564558 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck" event={"ID":"ee541613-f0de-462c-be0c-8c296bc90d1f","Type":"ContainerStarted","Data":"0ab04bfcda4e14d1cee4688e0cf0ce26c428a84b10193b4740ca76873c00cfb4"} Nov 03 19:53:55.587552 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:55.587489 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck" podStartSLOduration=2.587475068 podStartE2EDuration="2.587475068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:55.583507896 +0000 UTC m=+1029.841305164" watchObservedRunningTime="2026-11-03 19:53:55.587475068 +0000 UTC m=+1029.845272326" Nov 03 19:53:56.310239 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:56.310192 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:53:56.310454 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:56.310277 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" podUID="ced1fbaf-4771-4fab-8df0-a661dffb0d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:56.703330 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:56.703298 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:56.704287 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:56.704259 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:53:56.705119 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:53:56.705102 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:54:06.581590 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:06.581559 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck" Nov 03 19:54:10.654686 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:54:10.639637 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:54:13.068774 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:13.068733 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52c129de-8l2t4"] Nov 03 19:54:13.076037 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:13.076005 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52c129de-8l2t4"] Nov 03 19:54:13.917683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:13.916687 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9065a339-0b3b-49f5-ba8f-03ac69b21ca4" path="/var/lib/kubelet/pods/9065a339-0b3b-49f5-ba8f-03ac69b21ca4/volumes" Nov 03 19:54:17.822392 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:17.822355 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb"] Nov 03 19:54:17.828312 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:17.828293 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:17.837102 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:17.837074 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb"] Nov 03 19:54:17.845245 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:17.845226 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ad865c-openshift-default-dockercfg-tmsng\"" Nov 03 19:54:17.894604 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:17.894564 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-envoy\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:17.894909 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:17.894736 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-token\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:17.894997 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:17.894888 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istiod-ca-cert\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:17.895049 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:17.895001 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-workload-socket\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:17.895512 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:17.895482 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-data\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:17.895603 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:17.895524 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-podinfo\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:17.895682 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:17.895616 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-credential-socket\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:17.895763 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:17.895743 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-workload-certs\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:17.895914 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:17.895896 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92vxf\" (UniqueName: \"kubernetes.io/projected/c91e48d0-e218-4c9b-98c4-05415c3fc44c-kube-api-access-92vxf\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:18.014624 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.014577 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-credential-socket\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:18.015507 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.015480 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-workload-certs\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:18.025886 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.023037 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92vxf\" (UniqueName: \"kubernetes.io/projected/c91e48d0-e218-4c9b-98c4-05415c3fc44c-kube-api-access-92vxf\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:18.025886 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.023104 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-envoy\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:18.025886 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.023168 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-token\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:18.025886 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.023214 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istiod-ca-cert\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:18.025886 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.023260 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-workload-socket\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:18.025886 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.023380 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-data\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:18.025886 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.023408 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-podinfo\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:18.025886 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.015329 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-credential-socket\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:18.025886 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.015941 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-workload-certs\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:18.030362 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.030336 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-envoy\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:18.038686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.031560 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istiod-ca-cert\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:18.039168 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.039144 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-workload-socket\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:18.039559 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.039540 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-data\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:18.058151 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.058123 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-podinfo\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:18.063198 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.063171 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92vxf\" (UniqueName: \"kubernetes.io/projected/c91e48d0-e218-4c9b-98c4-05415c3fc44c-kube-api-access-92vxf\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:18.089396 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.089319 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-token\") pod \"e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:18.194686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.193981 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:18.547563 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:54:18.547533 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91e48d0_e218_4c9b_98c4_05415c3fc44c.slice/crio-484db02aeafcc47a5dfb0a2e72d5a8b2a0933f2abd515766fb4e3e64c4c50663 WatchSource:0}: Error finding container 484db02aeafcc47a5dfb0a2e72d5a8b2a0933f2abd515766fb4e3e64c4c50663: Status 404 returned error can't find the container with id 484db02aeafcc47a5dfb0a2e72d5a8b2a0933f2abd515766fb4e3e64c4c50663 Nov 03 19:54:18.549467 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.549445 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb"] Nov 03 19:54:18.550002 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.549968 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:54:18.550048 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.550033 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:54:18.550086 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.550063 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:54:18.990687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.962854 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" event={"ID":"c91e48d0-e218-4c9b-98c4-05415c3fc44c","Type":"ContainerStarted","Data":"11c70e32c74df688f8f88ae5d27bb395ce3774f0723e9acdea3562d1fa6f5624"} Nov 03 19:54:18.990687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:18.962925 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" event={"ID":"c91e48d0-e218-4c9b-98c4-05415c3fc44c","Type":"ContainerStarted","Data":"484db02aeafcc47a5dfb0a2e72d5a8b2a0933f2abd515766fb4e3e64c4c50663"} Nov 03 19:54:19.005859 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:19.005796 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" podStartSLOduration=2.005776226 podStartE2EDuration="2.005776226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:19.000200129 +0000 UTC m=+1053.257997446" watchObservedRunningTime="2026-11-03 19:54:19.005776226 +0000 UTC m=+1053.263573484" Nov 03 19:54:19.196140 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:19.196106 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:19.219218 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:19.219184 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:19.987196 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:19.987163 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:19.988814 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:19.988788 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:54:22.359831 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:22.359786 5059 patch_prober.go:29] interesting pod/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:54:22.360379 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:22.359858 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" podUID="b93733cb-f365-418c-893c-e7e8e358fc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:23.531835 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.531801 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd"] Nov 03 19:54:23.538224 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.538192 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.547958 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.543363 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f20fc0-openshift-default-dockercfg-vm5js\"" Nov 03 19:54:23.561083 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.561059 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd"] Nov 03 19:54:23.799800 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.795596 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-credential-socket\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.799800 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.795705 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-token\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.799800 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.795748 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-envoy\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.799800 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.795768 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7ddbfd-b074-4fed-8748-949d6a015621-istiod-ca-cert\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.799800 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.795802 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gf2p\" (UniqueName: \"kubernetes.io/projected/2c7ddbfd-b074-4fed-8748-949d6a015621-kube-api-access-7gf2p\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.799800 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.795907 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-data\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.799800 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.795991 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-podinfo\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.799800 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.796024 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-workload-certs\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.799800 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.796123 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-workload-socket\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.897593 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.897553 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-credential-socket\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.897803 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.897637 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-token\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.897803 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.897709 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-envoy\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.897803 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.897748 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7ddbfd-b074-4fed-8748-949d6a015621-istiod-ca-cert\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.898052 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.898021 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gf2p\" (UniqueName: \"kubernetes.io/projected/2c7ddbfd-b074-4fed-8748-949d6a015621-kube-api-access-7gf2p\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.898388 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.898356 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-data\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.898536 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.898489 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-podinfo\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.898536 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.898526 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-workload-certs\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.898833 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.898637 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-workload-socket\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.898833 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.898681 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7ddbfd-b074-4fed-8748-949d6a015621-istiod-ca-cert\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.898833 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.898097 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-credential-socket\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.900801 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.899149 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-workload-certs\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.900801 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.899162 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-data\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.901001 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.900914 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-workload-socket\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.918701 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.918649 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gf2p\" (UniqueName: \"kubernetes.io/projected/2c7ddbfd-b074-4fed-8748-949d6a015621-kube-api-access-7gf2p\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.918935 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.918911 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-envoy\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.923116 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.921169 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-token\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:23.923404 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:23.923377 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-podinfo\") pod \"e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:24.169815 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:24.169787 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:24.508600 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:24.506710 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd"] Nov 03 19:54:24.541735 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:24.541684 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:54:24.542081 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:24.541789 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:54:24.542081 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:24.541858 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:54:25.059239 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:25.059194 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" event={"ID":"2c7ddbfd-b074-4fed-8748-949d6a015621","Type":"ContainerStarted","Data":"2bf1203d8393d332419f358c61b7ffc2d7ed4b3606d1b21ee77ea9ed47daa251"} Nov 03 19:54:25.060698 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:25.059506 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" event={"ID":"2c7ddbfd-b074-4fed-8748-949d6a015621","Type":"ContainerStarted","Data":"00fccd4546d4f96c7fed60ff12f220e1fec510574cdf0834d1dfa749b0df69b8"} Nov 03 19:54:25.103477 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:25.103422 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" podStartSLOduration=2.10340785 podStartE2EDuration="2.10340785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:25.100746123 +0000 UTC m=+1059.358543377" watchObservedRunningTime="2026-11-03 19:54:25.10340785 +0000 UTC m=+1059.361205106" Nov 03 19:54:25.170227 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:25.170187 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:25.177144 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:25.177112 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:26.068765 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:26.068732 5059 generic.go:356] "Generic (PLEG): container finished" podID="b93733cb-f365-418c-893c-e7e8e358fc55" containerID="944da2903cc93746497d5c2e02b4669ae40c03f989ea3a04d91c59c28c2fac1e" exitCode=0 Nov 03 19:54:26.069279 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:26.068807 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" event={"ID":"b93733cb-f365-418c-893c-e7e8e358fc55","Type":"ContainerDied","Data":"944da2903cc93746497d5c2e02b4669ae40c03f989ea3a04d91c59c28c2fac1e"} Nov 03 19:54:26.069353 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:26.069283 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:26.069462 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:26.069444 5059 scope.go:122] "RemoveContainer" containerID="944da2903cc93746497d5c2e02b4669ae40c03f989ea3a04d91c59c28c2fac1e" Nov 03 19:54:26.070394 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:26.070251 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:54:26.070948 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:26.070919 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:54:26.071011 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:26.070985 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:54:26.071050 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:26.071024 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:54:27.091545 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:27.091504 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" event={"ID":"b93733cb-f365-418c-893c-e7e8e358fc55","Type":"ContainerStarted","Data":"fd91da36314f70ff464b531ebac4e94b229eed55b3513d1dc71d9c3766d4b04b"} Nov 03 19:54:27.238431 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:27.238399 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:54:27.238604 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:27.238564 5059 patch_prober.go:29] interesting pod/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 19:54:27.238654 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:27.238625 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" podUID="b93733cb-f365-418c-893c-e7e8e358fc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 19:54:27.255622 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:27.255555 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2876b2ef-jfzsx"] Nov 03 19:54:27.293988 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:27.293950 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2876b2ef-jfzsx" Nov 03 19:54:27.295254 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:27.295227 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2876b2ef-jfzsx"] Nov 03 19:54:27.298244 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:27.298109 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:54:27.298244 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:27.298207 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 19:54:27.396988 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:27.396956 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2jfr\" (UniqueName: \"kubernetes.io/projected/1e71b43c-8c02-43ea-bb4a-9051fd3d26b6-kube-api-access-p2jfr\") pod \"maas-api-revoke-sub-2876b2ef-jfzsx\" (UID: \"1e71b43c-8c02-43ea-bb4a-9051fd3d26b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2876b2ef-jfzsx" Nov 03 19:54:27.397169 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:27.397002 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e71b43c-8c02-43ea-bb4a-9051fd3d26b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2876b2ef-jfzsx\" (UID: \"1e71b43c-8c02-43ea-bb4a-9051fd3d26b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2876b2ef-jfzsx" Nov 03 19:54:27.397218 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:27.397194 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e71b43c-8c02-43ea-bb4a-9051fd3d26b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2876b2ef-jfzsx\" (UID: \"1e71b43c-8c02-43ea-bb4a-9051fd3d26b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2876b2ef-jfzsx" Nov 03 19:54:27.499412 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:27.498991 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2jfr\" (UniqueName: \"kubernetes.io/projected/1e71b43c-8c02-43ea-bb4a-9051fd3d26b6-kube-api-access-p2jfr\") pod \"maas-api-revoke-sub-2876b2ef-jfzsx\" (UID: \"1e71b43c-8c02-43ea-bb4a-9051fd3d26b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2876b2ef-jfzsx" Nov 03 19:54:27.500022 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:27.499998 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e71b43c-8c02-43ea-bb4a-9051fd3d26b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2876b2ef-jfzsx\" (UID: \"1e71b43c-8c02-43ea-bb4a-9051fd3d26b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2876b2ef-jfzsx" Nov 03 19:54:27.501247 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:27.501223 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e71b43c-8c02-43ea-bb4a-9051fd3d26b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2876b2ef-jfzsx\" (UID: \"1e71b43c-8c02-43ea-bb4a-9051fd3d26b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2876b2ef-jfzsx" Nov 03 19:54:27.501657 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:27.501048 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e71b43c-8c02-43ea-bb4a-9051fd3d26b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2876b2ef-jfzsx\" (UID: \"1e71b43c-8c02-43ea-bb4a-9051fd3d26b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2876b2ef-jfzsx" Nov 03 19:54:27.505621 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:27.505598 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e71b43c-8c02-43ea-bb4a-9051fd3d26b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2876b2ef-jfzsx\" (UID: \"1e71b43c-8c02-43ea-bb4a-9051fd3d26b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2876b2ef-jfzsx" Nov 03 19:54:27.516620 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:27.516562 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2jfr\" (UniqueName: \"kubernetes.io/projected/1e71b43c-8c02-43ea-bb4a-9051fd3d26b6-kube-api-access-p2jfr\") pod \"maas-api-revoke-sub-2876b2ef-jfzsx\" (UID: \"1e71b43c-8c02-43ea-bb4a-9051fd3d26b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2876b2ef-jfzsx" Nov 03 19:54:27.613687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:27.613355 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2876b2ef-jfzsx" Nov 03 19:54:27.928815 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:54:27.928780 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e71b43c_8c02_43ea_bb4a_9051fd3d26b6.slice/crio-9defebec1404e00b15f10392acd18da38ea3d50539d5c0c0ec1974cf827715c6 WatchSource:0}: Error finding container 9defebec1404e00b15f10392acd18da38ea3d50539d5c0c0ec1974cf827715c6: Status 404 returned error can't find the container with id 9defebec1404e00b15f10392acd18da38ea3d50539d5c0c0ec1974cf827715c6 Nov 03 19:54:27.929844 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:27.929816 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2876b2ef-jfzsx"] Nov 03 19:54:28.101477 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:28.101442 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2876b2ef-jfzsx" event={"ID":"1e71b43c-8c02-43ea-bb4a-9051fd3d26b6","Type":"ContainerStarted","Data":"9defebec1404e00b15f10392acd18da38ea3d50539d5c0c0ec1974cf827715c6"} Nov 03 19:54:28.237956 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:28.237925 5059 patch_prober.go:29] interesting pod/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 19:54:28.238109 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:28.237975 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" podUID="b93733cb-f365-418c-893c-e7e8e358fc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 19:54:29.116394 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:29.116355 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2876b2ef-jfzsx" event={"ID":"1e71b43c-8c02-43ea-bb4a-9051fd3d26b6","Type":"ContainerStarted","Data":"6e88b3c669ba1ae0f42319195d2e11824d33d6baf9b5492a19b8a5d6d474ebb8"} Nov 03 19:54:29.139187 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:29.139142 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2876b2ef-jfzsx" podStartSLOduration=2.139128604 podStartE2EDuration="2.139128604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:29.13618248 +0000 UTC m=+1063.393979740" watchObservedRunningTime="2026-11-03 19:54:29.139128604 +0000 UTC m=+1063.396925907" Nov 03 19:54:29.238287 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:29.238253 5059 patch_prober.go:29] interesting pod/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 19:54:29.238473 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:29.238338 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" podUID="b93733cb-f365-418c-893c-e7e8e358fc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 19:54:30.243570 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:30.243522 5059 patch_prober.go:29] interesting pod/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 19:54:30.244279 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:30.244231 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" podUID="b93733cb-f365-418c-893c-e7e8e358fc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 19:54:30.248088 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:30.248040 5059 generic.go:356] "Generic (PLEG): container finished" podID="1e71b43c-8c02-43ea-bb4a-9051fd3d26b6" containerID="6e88b3c669ba1ae0f42319195d2e11824d33d6baf9b5492a19b8a5d6d474ebb8" exitCode=0 Nov 03 19:54:30.248522 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:30.248491 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2876b2ef-jfzsx" event={"ID":"1e71b43c-8c02-43ea-bb4a-9051fd3d26b6","Type":"ContainerDied","Data":"6e88b3c669ba1ae0f42319195d2e11824d33d6baf9b5492a19b8a5d6d474ebb8"} Nov 03 19:54:31.238320 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:31.238285 5059 patch_prober.go:29] interesting pod/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 19:54:31.238494 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:31.238348 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" podUID="b93733cb-f365-418c-893c-e7e8e358fc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 19:54:31.495185 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:31.495121 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2876b2ef-jfzsx" Nov 03 19:54:31.640005 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:31.637557 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e71b43c-8c02-43ea-bb4a-9051fd3d26b6-kube-api-access-p2jfr\" (UniqueName: \"kubernetes.io/projected/1e71b43c-8c02-43ea-bb4a-9051fd3d26b6-kube-api-access-p2jfr\") pod \"1e71b43c-8c02-43ea-bb4a-9051fd3d26b6\" (UID: \"1e71b43c-8c02-43ea-bb4a-9051fd3d26b6\") " Nov 03 19:54:31.640005 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:31.637912 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e71b43c-8c02-43ea-bb4a-9051fd3d26b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e71b43c-8c02-43ea-bb4a-9051fd3d26b6-maas-api-xks-ca\") pod \"1e71b43c-8c02-43ea-bb4a-9051fd3d26b6\" (UID: \"1e71b43c-8c02-43ea-bb4a-9051fd3d26b6\") " Nov 03 19:54:31.640005 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:31.638046 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e71b43c-8c02-43ea-bb4a-9051fd3d26b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e71b43c-8c02-43ea-bb4a-9051fd3d26b6-maas-api-openshift-ca\") pod \"1e71b43c-8c02-43ea-bb4a-9051fd3d26b6\" (UID: \"1e71b43c-8c02-43ea-bb4a-9051fd3d26b6\") " Nov 03 19:54:31.641088 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:31.640789 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e71b43c-8c02-43ea-bb4a-9051fd3d26b6-maas-api-openshift-ca" pod "1e71b43c-8c02-43ea-bb4a-9051fd3d26b6" (UID: "1e71b43c-8c02-43ea-bb4a-9051fd3d26b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:54:31.656745 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:31.646275 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e71b43c-8c02-43ea-bb4a-9051fd3d26b6-kube-api-access-p2jfr" pod "1e71b43c-8c02-43ea-bb4a-9051fd3d26b6" (UID: "1e71b43c-8c02-43ea-bb4a-9051fd3d26b6"). InnerVolumeSpecName "kube-api-access-p2jfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:54:31.676695 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:31.675795 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e71b43c-8c02-43ea-bb4a-9051fd3d26b6-maas-api-xks-ca" pod "1e71b43c-8c02-43ea-bb4a-9051fd3d26b6" (UID: "1e71b43c-8c02-43ea-bb4a-9051fd3d26b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:54:31.740055 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:31.740023 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2jfr\" (UniqueName: \"kubernetes.io/projected/1e71b43c-8c02-43ea-bb4a-9051fd3d26b6-kube-api-access-p2jfr\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:31.740055 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:31.740052 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e71b43c-8c02-43ea-bb4a-9051fd3d26b6-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:31.740055 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:31.740063 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e71b43c-8c02-43ea-bb4a-9051fd3d26b6-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:32.238449 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:32.238354 5059 patch_prober.go:29] interesting pod/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 19:54:32.238449 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:32.238423 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" podUID="b93733cb-f365-418c-893c-e7e8e358fc55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 19:54:32.267712 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:32.267679 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2876b2ef-jfzsx" event={"ID":"1e71b43c-8c02-43ea-bb4a-9051fd3d26b6","Type":"ContainerDied","Data":"9defebec1404e00b15f10392acd18da38ea3d50539d5c0c0ec1974cf827715c6"} Nov 03 19:54:32.267877 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:32.267733 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9defebec1404e00b15f10392acd18da38ea3d50539d5c0c0ec1974cf827715c6" Nov 03 19:54:32.267877 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:32.267692 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2876b2ef-jfzsx" Nov 03 19:54:33.025546 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:33.025507 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-371219cd-lt9kd"] Nov 03 19:54:33.030558 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:33.030528 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-371219cd-lt9kd"] Nov 03 19:54:33.243830 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:33.243797 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:54:33.244295 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:33.244266 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:54:33.245116 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:33.245086 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:54:33.861980 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:33.861947 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36e9fc9c-cc8e-4775-9d55-11969c44792d" path="/var/lib/kubelet/pods/36e9fc9c-cc8e-4775-9d55-11969c44792d/volumes" Nov 03 19:54:34.030017 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:34.029985 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ce079d3-kn6pp"] Nov 03 19:54:34.039486 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:34.039452 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ce079d3-kn6pp"] Nov 03 19:54:35.863449 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:35.863414 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95f72b2d-9363-46aa-a77e-7f1640eb085d" path="/var/lib/kubelet/pods/95f72b2d-9363-46aa-a77e-7f1640eb085d/volumes" Nov 03 19:54:36.026273 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:36.026243 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b50ae9ac-whrv7"] Nov 03 19:54:36.031368 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:36.031346 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b50ae9ac-whrv7"] Nov 03 19:54:37.859109 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:37.859070 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="470843af-fc92-4961-824f-7a862483a18b" path="/var/lib/kubelet/pods/470843af-fc92-4961-824f-7a862483a18b/volumes" Nov 03 19:54:38.084778 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:38.084748 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7240-131870"] Nov 03 19:54:38.086374 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:38.086331 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1e71b43c-8c02-43ea-bb4a-9051fd3d26b6" containerName="revoke-keys" Nov 03 19:54:38.086512 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:38.086375 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e71b43c-8c02-43ea-bb4a-9051fd3d26b6" containerName="revoke-keys" Nov 03 19:54:38.086965 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:38.086945 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e71b43c-8c02-43ea-bb4a-9051fd3d26b6" containerName="revoke-keys" Nov 03 19:54:38.100865 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:38.100836 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-131870" Nov 03 19:54:38.101194 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:38.101170 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-131870"] Nov 03 19:54:38.115046 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:38.105096 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 19:54:38.115046 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:38.106433 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-tz7np\"" Nov 03 19:54:38.115046 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:38.106471 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 19:54:38.202145 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:38.202118 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh5zh\" (UniqueName: \"kubernetes.io/projected/407f3e4e-7cc5-4d32-ab89-4cbafbbf8412-kube-api-access-sh5zh\") pod \"test-curl-7240-131870\" (UID: \"407f3e4e-7cc5-4d32-ab89-4cbafbbf8412\") " pod="opendatahub/test-curl-7240-131870" Nov 03 19:54:38.303687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:38.303625 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh5zh\" (UniqueName: \"kubernetes.io/projected/407f3e4e-7cc5-4d32-ab89-4cbafbbf8412-kube-api-access-sh5zh\") pod \"test-curl-7240-131870\" (UID: \"407f3e4e-7cc5-4d32-ab89-4cbafbbf8412\") " pod="opendatahub/test-curl-7240-131870" Nov 03 19:54:38.316240 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:38.316216 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh5zh\" (UniqueName: \"kubernetes.io/projected/407f3e4e-7cc5-4d32-ab89-4cbafbbf8412-kube-api-access-sh5zh\") pod \"test-curl-7240-131870\" (UID: \"407f3e4e-7cc5-4d32-ab89-4cbafbbf8412\") " pod="opendatahub/test-curl-7240-131870" Nov 03 19:54:38.415893 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:38.415811 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-131870" Nov 03 19:54:38.702530 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:38.702493 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-131870"] Nov 03 19:54:38.705362 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:54:38.705328 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod407f3e4e_7cc5_4d32_ab89_4cbafbbf8412.slice/crio-76b3e3c6c058d478d7de2a6094d26155286784d7179a2642585d84e015bcaaf2 WatchSource:0}: Error finding container 76b3e3c6c058d478d7de2a6094d26155286784d7179a2642585d84e015bcaaf2: Status 404 returned error can't find the container with id 76b3e3c6c058d478d7de2a6094d26155286784d7179a2642585d84e015bcaaf2 Nov 03 19:54:38.707602 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:38.707582 5059 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:54:39.415697 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:39.410054 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-131870" event={"ID":"407f3e4e-7cc5-4d32-ab89-4cbafbbf8412","Type":"ContainerStarted","Data":"76b3e3c6c058d478d7de2a6094d26155286784d7179a2642585d84e015bcaaf2"} Nov 03 19:54:40.619908 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:54:40.619859 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:54:42.450490 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:42.450445 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-131870" event={"ID":"407f3e4e-7cc5-4d32-ab89-4cbafbbf8412","Type":"ContainerStarted","Data":"db70f99cd4cceec0159efb83db1726aaea915475a74a10f29e39b309e690fef8"} Nov 03 19:54:46.029625 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:46.029570 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7240-131870" podStartSLOduration=5.055798947 podStartE2EDuration="8.029556494s" totalImagesPullingTime="2.973757547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:42.488587662 +0000 UTC m=+1076.746384919" watchObservedRunningTime="2026-11-03 19:54:46.029556494 +0000 UTC m=+1080.287353751" Nov 03 19:54:46.033459 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:46.033427 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cdba36c-svnbd"] Nov 03 19:54:46.038495 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:46.038469 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cdba36c-svnbd"] Nov 03 19:54:47.880085 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:47.880043 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a20d160-602b-4513-94a1-159916a46cb8" path="/var/lib/kubelet/pods/6a20d160-602b-4513-94a1-159916a46cb8/volumes" Nov 03 19:54:48.358344 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:48.358290 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck"] Nov 03 19:54:49.890701 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:49.885765 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:49.890701 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:49.886348 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 19:54:49.933844 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:49.931099 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 19:54:49.933844 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:49.931535 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:49.977155 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:49.977118 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vznnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:54:49.977492 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:49.977413 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" podUID="171c7288-3157-4fe0-80ee-94cf5b2ca2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:50.070799 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:50.070758 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7krsk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:54:50.071075 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:50.071046 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" podUID="4946e3aa-a961-415a-b71d-95164d5fbca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:50.072381 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:50.072350 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cnppr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:54:50.072566 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:50.072541 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" podUID="1b1f91b8-973a-4345-acd7-ea3325faf638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:50.323848 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:50.309462 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:53.375808 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:53.364724 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck" podUID="ee541613-f0de-462c-be0c-8c296bc90d1f" containerName="payload-pre-processing" containerID="cri-o://35d11f9645b92bb2c958f0c84da12424a56021d8d29afe0f47749929a2158628" gracePeriod=25 Nov 03 19:54:53.624762 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:53.624723 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7240-131870"] Nov 03 19:54:53.625001 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:53.624972 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7240-131870" podUID="407f3e4e-7cc5-4d32-ab89-4cbafbbf8412" containerName="test-curl-7240-131870" containerID="cri-o://db70f99cd4cceec0159efb83db1726aaea915475a74a10f29e39b309e690fef8" gracePeriod=2 Nov 03 19:54:53.628077 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:54:53.628030 5059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-131870\" is forbidden: User \"system:node:ip-10-0-132-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-83.us-east-2.compute.internal' and this object" podUID="407f3e4e-7cc5-4d32-ab89-4cbafbbf8412" pod="opendatahub/test-curl-7240-131870" Nov 03 19:54:53.633214 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:53.633180 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7240-131870"] Nov 03 19:54:53.965855 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:53.965824 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck" Nov 03 19:54:54.022633 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.022595 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee541613-f0de-462c-be0c-8c296bc90d1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee541613-f0de-462c-be0c-8c296bc90d1f-plugins-config-volume\") pod \"ee541613-f0de-462c-be0c-8c296bc90d1f\" (UID: \"ee541613-f0de-462c-be0c-8c296bc90d1f\") " Nov 03 19:54:54.022843 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.022829 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee541613-f0de-462c-be0c-8c296bc90d1f-kube-api-access-nf8c2\" (UniqueName: \"kubernetes.io/projected/ee541613-f0de-462c-be0c-8c296bc90d1f-kube-api-access-nf8c2\") pod \"ee541613-f0de-462c-be0c-8c296bc90d1f\" (UID: \"ee541613-f0de-462c-be0c-8c296bc90d1f\") " Nov 03 19:54:54.023109 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.023081 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee541613-f0de-462c-be0c-8c296bc90d1f-plugins-config-volume" pod "ee541613-f0de-462c-be0c-8c296bc90d1f" (UID: "ee541613-f0de-462c-be0c-8c296bc90d1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:54:54.023421 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.023400 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee541613-f0de-462c-be0c-8c296bc90d1f-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:54.025538 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.025505 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee541613-f0de-462c-be0c-8c296bc90d1f-kube-api-access-nf8c2" pod "ee541613-f0de-462c-be0c-8c296bc90d1f" (UID: "ee541613-f0de-462c-be0c-8c296bc90d1f"). InnerVolumeSpecName "kube-api-access-nf8c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:54:54.124962 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.124925 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf8c2\" (UniqueName: \"kubernetes.io/projected/ee541613-f0de-462c-be0c-8c296bc90d1f-kube-api-access-nf8c2\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:54.133781 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.133751 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz"] Nov 03 19:54:54.134486 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.134468 5059 cpu_manager.go:414] "containerMap: removing container" podUID="ee541613-f0de-462c-be0c-8c296bc90d1f" containerName="payload-pre-processing" Nov 03 19:54:54.134550 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.134486 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee541613-f0de-462c-be0c-8c296bc90d1f" containerName="payload-pre-processing" Nov 03 19:54:54.134550 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.134515 5059 cpu_manager.go:414] "containerMap: removing container" podUID="407f3e4e-7cc5-4d32-ab89-4cbafbbf8412" containerName="test-curl-7240-131870" Nov 03 19:54:54.134550 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.134521 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="407f3e4e-7cc5-4d32-ab89-4cbafbbf8412" containerName="test-curl-7240-131870" Nov 03 19:54:54.134644 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.134631 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="407f3e4e-7cc5-4d32-ab89-4cbafbbf8412" containerName="test-curl-7240-131870" Nov 03 19:54:54.134690 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.134647 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee541613-f0de-462c-be0c-8c296bc90d1f" containerName="payload-pre-processing" Nov 03 19:54:54.138514 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.138494 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz" Nov 03 19:54:54.142417 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.142392 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:54:54.142963 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.142719 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 19:54:54.146468 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.146444 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz"] Nov 03 19:54:54.226778 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.226689 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c701c3-ed02-4fd8-ad04-430541e88c6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz\" (UID: \"d6c701c3-ed02-4fd8-ad04-430541e88c6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz" Nov 03 19:54:54.227010 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.226991 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c701c3-ed02-4fd8-ad04-430541e88c6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz\" (UID: \"d6c701c3-ed02-4fd8-ad04-430541e88c6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz" Nov 03 19:54:54.227055 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.227030 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njnbp\" (UniqueName: \"kubernetes.io/projected/d6c701c3-ed02-4fd8-ad04-430541e88c6e-kube-api-access-njnbp\") pod \"maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz\" (UID: \"d6c701c3-ed02-4fd8-ad04-430541e88c6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz" Nov 03 19:54:54.317138 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.317105 5059 generic.go:356] "Generic (PLEG): container finished" podID="ee541613-f0de-462c-be0c-8c296bc90d1f" containerID="35d11f9645b92bb2c958f0c84da12424a56021d8d29afe0f47749929a2158628" exitCode=0 Nov 03 19:54:54.317305 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.317208 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck" Nov 03 19:54:54.317353 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.317323 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck" event={"ID":"ee541613-f0de-462c-be0c-8c296bc90d1f","Type":"ContainerDied","Data":"35d11f9645b92bb2c958f0c84da12424a56021d8d29afe0f47749929a2158628"} Nov 03 19:54:54.317392 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.317360 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck" event={"ID":"ee541613-f0de-462c-be0c-8c296bc90d1f","Type":"ContainerDied","Data":"0ab04bfcda4e14d1cee4688e0cf0ce26c428a84b10193b4740ca76873c00cfb4"} Nov 03 19:54:54.317392 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.317381 5059 scope.go:122] "RemoveContainer" containerID="35d11f9645b92bb2c958f0c84da12424a56021d8d29afe0f47749929a2158628" Nov 03 19:54:54.327938 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.327917 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c701c3-ed02-4fd8-ad04-430541e88c6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz\" (UID: \"d6c701c3-ed02-4fd8-ad04-430541e88c6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz" Nov 03 19:54:54.328125 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.328107 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njnbp\" (UniqueName: \"kubernetes.io/projected/d6c701c3-ed02-4fd8-ad04-430541e88c6e-kube-api-access-njnbp\") pod \"maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz\" (UID: \"d6c701c3-ed02-4fd8-ad04-430541e88c6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz" Nov 03 19:54:54.328315 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.328304 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c701c3-ed02-4fd8-ad04-430541e88c6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz\" (UID: \"d6c701c3-ed02-4fd8-ad04-430541e88c6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz" Nov 03 19:54:54.328850 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.328809 5059 scope.go:122] "RemoveContainer" containerID="35d11f9645b92bb2c958f0c84da12424a56021d8d29afe0f47749929a2158628" Nov 03 19:54:54.329000 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.328980 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c701c3-ed02-4fd8-ad04-430541e88c6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz\" (UID: \"d6c701c3-ed02-4fd8-ad04-430541e88c6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz" Nov 03 19:54:54.329156 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:54:54.329131 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35d11f9645b92bb2c958f0c84da12424a56021d8d29afe0f47749929a2158628\": container with ID starting with 35d11f9645b92bb2c958f0c84da12424a56021d8d29afe0f47749929a2158628 not found: ID does not exist" containerID="35d11f9645b92bb2c958f0c84da12424a56021d8d29afe0f47749929a2158628" Nov 03 19:54:54.329204 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.329170 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d11f9645b92bb2c958f0c84da12424a56021d8d29afe0f47749929a2158628"} err="failed to get container status \"35d11f9645b92bb2c958f0c84da12424a56021d8d29afe0f47749929a2158628\": rpc error: code = NotFound desc = could not find container \"35d11f9645b92bb2c958f0c84da12424a56021d8d29afe0f47749929a2158628\": container with ID starting with 35d11f9645b92bb2c958f0c84da12424a56021d8d29afe0f47749929a2158628 not found: ID does not exist" Nov 03 19:54:54.330690 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.330384 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c701c3-ed02-4fd8-ad04-430541e88c6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz\" (UID: \"d6c701c3-ed02-4fd8-ad04-430541e88c6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz" Nov 03 19:54:54.340699 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.340680 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njnbp\" (UniqueName: \"kubernetes.io/projected/d6c701c3-ed02-4fd8-ad04-430541e88c6e-kube-api-access-njnbp\") pod \"maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz\" (UID: \"d6c701c3-ed02-4fd8-ad04-430541e88c6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz" Nov 03 19:54:54.349567 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.349529 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck"] Nov 03 19:54:54.354616 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.354597 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0a2237-867f7ff655-k2kck"] Nov 03 19:54:54.439293 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:54:54.439255 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee541613_f0de_462c_be0c_8c296bc90d1f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee541613_f0de_462c_be0c_8c296bc90d1f.slice/crio-0ab04bfcda4e14d1cee4688e0cf0ce26c428a84b10193b4740ca76873c00cfb4\": RecentStats: unable to find data in memory cache]" Nov 03 19:54:54.456499 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.456476 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz" Nov 03 19:54:54.829039 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:54:54.828984 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c701c3_ed02_4fd8_ad04_430541e88c6e.slice/crio-bdba58f2fb354e7c22a5ac1e7e4f4c78b3f3ffcd77fa715d4ea98b83e00914ac WatchSource:0}: Error finding container bdba58f2fb354e7c22a5ac1e7e4f4c78b3f3ffcd77fa715d4ea98b83e00914ac: Status 404 returned error can't find the container with id bdba58f2fb354e7c22a5ac1e7e4f4c78b3f3ffcd77fa715d4ea98b83e00914ac Nov 03 19:54:54.830748 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:54.830725 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz"] Nov 03 19:54:55.331893 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:55.331853 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz" event={"ID":"d6c701c3-ed02-4fd8-ad04-430541e88c6e","Type":"ContainerStarted","Data":"bdba58f2fb354e7c22a5ac1e7e4f4c78b3f3ffcd77fa715d4ea98b83e00914ac"} Nov 03 19:54:55.824937 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:55.824910 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-131870" Nov 03 19:54:55.860820 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:55.860788 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee541613-f0de-462c-be0c-8c296bc90d1f" path="/var/lib/kubelet/pods/ee541613-f0de-462c-be0c-8c296bc90d1f/volumes" Nov 03 19:54:55.891764 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:55.891698 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/407f3e4e-7cc5-4d32-ab89-4cbafbbf8412-kube-api-access-sh5zh\" (UniqueName: \"kubernetes.io/projected/407f3e4e-7cc5-4d32-ab89-4cbafbbf8412-kube-api-access-sh5zh\") pod \"407f3e4e-7cc5-4d32-ab89-4cbafbbf8412\" (UID: \"407f3e4e-7cc5-4d32-ab89-4cbafbbf8412\") " Nov 03 19:54:55.893834 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:55.893809 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407f3e4e-7cc5-4d32-ab89-4cbafbbf8412-kube-api-access-sh5zh" pod "407f3e4e-7cc5-4d32-ab89-4cbafbbf8412" (UID: "407f3e4e-7cc5-4d32-ab89-4cbafbbf8412"). InnerVolumeSpecName "kube-api-access-sh5zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:54:55.994024 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:55.993983 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh5zh\" (UniqueName: \"kubernetes.io/projected/407f3e4e-7cc5-4d32-ab89-4cbafbbf8412-kube-api-access-sh5zh\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:56.343715 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:56.343672 5059 generic.go:356] "Generic (PLEG): container finished" podID="d6c701c3-ed02-4fd8-ad04-430541e88c6e" containerID="0ea193a090b91d09c90e23c8abd032827496598ffbb839e0505022e1784dd565" exitCode=0 Nov 03 19:54:56.343715 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:56.343691 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz" event={"ID":"d6c701c3-ed02-4fd8-ad04-430541e88c6e","Type":"ContainerDied","Data":"0ea193a090b91d09c90e23c8abd032827496598ffbb839e0505022e1784dd565"} Nov 03 19:54:56.345393 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:56.345366 5059 generic.go:356] "Generic (PLEG): container finished" podID="407f3e4e-7cc5-4d32-ab89-4cbafbbf8412" containerID="db70f99cd4cceec0159efb83db1726aaea915475a74a10f29e39b309e690fef8" exitCode=137 Nov 03 19:54:56.345538 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:56.345399 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-131870" Nov 03 19:54:56.345538 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:56.345465 5059 scope.go:122] "RemoveContainer" containerID="db70f99cd4cceec0159efb83db1726aaea915475a74a10f29e39b309e690fef8" Nov 03 19:54:56.360132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:56.360113 5059 scope.go:122] "RemoveContainer" containerID="db70f99cd4cceec0159efb83db1726aaea915475a74a10f29e39b309e690fef8" Nov 03 19:54:56.360448 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:54:56.360424 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db70f99cd4cceec0159efb83db1726aaea915475a74a10f29e39b309e690fef8\": container with ID starting with db70f99cd4cceec0159efb83db1726aaea915475a74a10f29e39b309e690fef8 not found: ID does not exist" containerID="db70f99cd4cceec0159efb83db1726aaea915475a74a10f29e39b309e690fef8" Nov 03 19:54:56.360537 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:56.360461 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db70f99cd4cceec0159efb83db1726aaea915475a74a10f29e39b309e690fef8"} err="failed to get container status \"db70f99cd4cceec0159efb83db1726aaea915475a74a10f29e39b309e690fef8\": rpc error: code = NotFound desc = could not find container \"db70f99cd4cceec0159efb83db1726aaea915475a74a10f29e39b309e690fef8\": container with ID starting with db70f99cd4cceec0159efb83db1726aaea915475a74a10f29e39b309e690fef8 not found: ID does not exist" Nov 03 19:54:57.135224 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:57.134840 5059 patch_prober.go:29] interesting pod/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:54:57.135224 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:57.134912 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" podUID="2c7ddbfd-b074-4fed-8748-949d6a015621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:57.544563 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:57.543366 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk"] Nov 03 19:54:57.544563 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:57.543611 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" podUID="b93733cb-f365-418c-893c-e7e8e358fc55" containerName="istio-proxy" containerID="cri-o://fd91da36314f70ff464b531ebac4e94b229eed55b3513d1dc71d9c3766d4b04b" gracePeriod=30 Nov 03 19:54:57.846133 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:57.846089 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz" Nov 03 19:54:57.869018 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:57.865862 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="407f3e4e-7cc5-4d32-ab89-4cbafbbf8412" path="/var/lib/kubelet/pods/407f3e4e-7cc5-4d32-ab89-4cbafbbf8412/volumes" Nov 03 19:54:57.964455 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:57.964423 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c701c3-ed02-4fd8-ad04-430541e88c6e-kube-api-access-njnbp\" (UniqueName: \"kubernetes.io/projected/d6c701c3-ed02-4fd8-ad04-430541e88c6e-kube-api-access-njnbp\") pod \"d6c701c3-ed02-4fd8-ad04-430541e88c6e\" (UID: \"d6c701c3-ed02-4fd8-ad04-430541e88c6e\") " Nov 03 19:54:57.964648 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:57.964503 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c701c3-ed02-4fd8-ad04-430541e88c6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c701c3-ed02-4fd8-ad04-430541e88c6e-maas-api-openshift-ca\") pod \"d6c701c3-ed02-4fd8-ad04-430541e88c6e\" (UID: \"d6c701c3-ed02-4fd8-ad04-430541e88c6e\") " Nov 03 19:54:57.964761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:57.964694 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6c701c3-ed02-4fd8-ad04-430541e88c6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c701c3-ed02-4fd8-ad04-430541e88c6e-maas-api-xks-ca\") pod \"d6c701c3-ed02-4fd8-ad04-430541e88c6e\" (UID: \"d6c701c3-ed02-4fd8-ad04-430541e88c6e\") " Nov 03 19:54:57.965030 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:57.964945 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c701c3-ed02-4fd8-ad04-430541e88c6e-maas-api-openshift-ca" pod "d6c701c3-ed02-4fd8-ad04-430541e88c6e" (UID: "d6c701c3-ed02-4fd8-ad04-430541e88c6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:54:57.965846 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:57.965521 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c701c3-ed02-4fd8-ad04-430541e88c6e-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:57.967109 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:57.967082 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c701c3-ed02-4fd8-ad04-430541e88c6e-kube-api-access-njnbp" pod "d6c701c3-ed02-4fd8-ad04-430541e88c6e" (UID: "d6c701c3-ed02-4fd8-ad04-430541e88c6e"). InnerVolumeSpecName "kube-api-access-njnbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:54:57.967822 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:57.967804 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6c701c3-ed02-4fd8-ad04-430541e88c6e-maas-api-xks-ca" pod "d6c701c3-ed02-4fd8-ad04-430541e88c6e" (UID: "d6c701c3-ed02-4fd8-ad04-430541e88c6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:54:58.066848 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:58.066817 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c701c3-ed02-4fd8-ad04-430541e88c6e-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:58.066848 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:58.066842 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njnbp\" (UniqueName: \"kubernetes.io/projected/d6c701c3-ed02-4fd8-ad04-430541e88c6e-kube-api-access-njnbp\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:58.377733 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:58.377701 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz" Nov 03 19:54:58.378139 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:58.377700 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz" event={"ID":"d6c701c3-ed02-4fd8-ad04-430541e88c6e","Type":"ContainerDied","Data":"bdba58f2fb354e7c22a5ac1e7e4f4c78b3f3ffcd77fa715d4ea98b83e00914ac"} Nov 03 19:54:58.378139 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:54:58.377846 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdba58f2fb354e7c22a5ac1e7e4f4c78b3f3ffcd77fa715d4ea98b83e00914ac" Nov 03 19:55:00.155882 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:00.155851 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6991b942-jnrzn"] Nov 03 19:55:00.161693 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:00.161650 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6991b942-jnrzn"] Nov 03 19:55:01.869310 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:01.869279 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff5d8a5e-3638-4b29-a49c-3ef488b372f5" path="/var/lib/kubelet/pods/ff5d8a5e-3638-4b29-a49c-3ef488b372f5/volumes" Nov 03 19:55:02.556773 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:02.548966 5059 generic.go:356] "Generic (PLEG): container finished" podID="c91e48d0-e218-4c9b-98c4-05415c3fc44c" containerID="11c70e32c74df688f8f88ae5d27bb395ce3774f0723e9acdea3562d1fa6f5624" exitCode=0 Nov 03 19:55:02.556773 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:02.549183 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" event={"ID":"c91e48d0-e218-4c9b-98c4-05415c3fc44c","Type":"ContainerDied","Data":"11c70e32c74df688f8f88ae5d27bb395ce3774f0723e9acdea3562d1fa6f5624"} Nov 03 19:55:02.556773 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:02.549881 5059 scope.go:122] "RemoveContainer" containerID="11c70e32c74df688f8f88ae5d27bb395ce3774f0723e9acdea3562d1fa6f5624" Nov 03 19:55:02.615760 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:02.584268 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:55:02.615760 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:02.611120 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:55:02.615760 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:02.611272 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:55:02.653539 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:02.629047 5059 generic.go:356] "Generic (PLEG): container finished" podID="2c7ddbfd-b074-4fed-8748-949d6a015621" containerID="2bf1203d8393d332419f358c61b7ffc2d7ed4b3606d1b21ee77ea9ed47daa251" exitCode=0 Nov 03 19:55:02.653539 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:02.629367 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" event={"ID":"2c7ddbfd-b074-4fed-8748-949d6a015621","Type":"ContainerDied","Data":"2bf1203d8393d332419f358c61b7ffc2d7ed4b3606d1b21ee77ea9ed47daa251"} Nov 03 19:55:02.653539 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:02.645906 5059 scope.go:122] "RemoveContainer" containerID="2bf1203d8393d332419f358c61b7ffc2d7ed4b3606d1b21ee77ea9ed47daa251" Nov 03 19:55:02.674710 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:02.662389 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:55:02.674710 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:02.662513 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:55:02.674710 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:02.662585 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:55:03.004315 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.003958 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:55:03.111986 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.111955 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-workload-certs\") pod \"b93733cb-f365-418c-893c-e7e8e358fc55\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " Nov 03 19:55:03.112151 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.111996 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-credential-socket\") pod \"b93733cb-f365-418c-893c-e7e8e358fc55\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " Nov 03 19:55:03.112151 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.112026 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b93733cb-f365-418c-893c-e7e8e358fc55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b93733cb-f365-418c-893c-e7e8e358fc55-istiod-ca-cert\") pod \"b93733cb-f365-418c-893c-e7e8e358fc55\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " Nov 03 19:55:03.112151 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.112093 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b93733cb-f365-418c-893c-e7e8e358fc55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b93733cb-f365-418c-893c-e7e8e358fc55-istio-podinfo\") pod \"b93733cb-f365-418c-893c-e7e8e358fc55\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " Nov 03 19:55:03.112335 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.112240 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b93733cb-f365-418c-893c-e7e8e358fc55-kube-api-access-g2w95\" (UniqueName: \"kubernetes.io/projected/b93733cb-f365-418c-893c-e7e8e358fc55-kube-api-access-g2w95\") pod \"b93733cb-f365-418c-893c-e7e8e358fc55\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " Nov 03 19:55:03.112335 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.112293 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-workload-certs" pod "b93733cb-f365-418c-893c-e7e8e358fc55" (UID: "b93733cb-f365-418c-893c-e7e8e358fc55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:55:03.112451 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.112323 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-credential-socket" pod "b93733cb-f365-418c-893c-e7e8e358fc55" (UID: "b93733cb-f365-418c-893c-e7e8e358fc55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:55:03.112451 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.112342 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-istio-envoy\") pod \"b93733cb-f365-418c-893c-e7e8e358fc55\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " Nov 03 19:55:03.112451 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.112375 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-istio-data\") pod \"b93733cb-f365-418c-893c-e7e8e358fc55\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " Nov 03 19:55:03.112616 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.112553 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93733cb-f365-418c-893c-e7e8e358fc55-istiod-ca-cert" pod "b93733cb-f365-418c-893c-e7e8e358fc55" (UID: "b93733cb-f365-418c-893c-e7e8e358fc55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:55:03.112616 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.112565 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-workload-socket\") pod \"b93733cb-f365-418c-893c-e7e8e358fc55\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " Nov 03 19:55:03.112616 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.112608 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b93733cb-f365-418c-893c-e7e8e358fc55-istio-token\" (UniqueName: \"kubernetes.io/projected/b93733cb-f365-418c-893c-e7e8e358fc55-istio-token\") pod \"b93733cb-f365-418c-893c-e7e8e358fc55\" (UID: \"b93733cb-f365-418c-893c-e7e8e358fc55\") " Nov 03 19:55:03.112799 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.112735 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-istio-data" pod "b93733cb-f365-418c-893c-e7e8e358fc55" (UID: "b93733cb-f365-418c-893c-e7e8e358fc55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:55:03.113088 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.112922 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-workload-socket" pod "b93733cb-f365-418c-893c-e7e8e358fc55" (UID: "b93733cb-f365-418c-893c-e7e8e358fc55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:55:03.113355 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.113334 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:03.113440 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.113361 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:03.113440 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.113375 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:03.113440 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.113390 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:03.113440 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.113403 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b93733cb-f365-418c-893c-e7e8e358fc55-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:03.115139 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.115102 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93733cb-f365-418c-893c-e7e8e358fc55-kube-api-access-g2w95" pod "b93733cb-f365-418c-893c-e7e8e358fc55" (UID: "b93733cb-f365-418c-893c-e7e8e358fc55"). InnerVolumeSpecName "kube-api-access-g2w95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:55:03.115396 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.115373 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b93733cb-f365-418c-893c-e7e8e358fc55-istio-podinfo" pod "b93733cb-f365-418c-893c-e7e8e358fc55" (UID: "b93733cb-f365-418c-893c-e7e8e358fc55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:55:03.116186 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.116144 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93733cb-f365-418c-893c-e7e8e358fc55-istio-token" pod "b93733cb-f365-418c-893c-e7e8e358fc55" (UID: "b93733cb-f365-418c-893c-e7e8e358fc55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:55:03.116186 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.116145 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-istio-envoy" pod "b93733cb-f365-418c-893c-e7e8e358fc55" (UID: "b93733cb-f365-418c-893c-e7e8e358fc55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:55:03.214954 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.214918 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b93733cb-f365-418c-893c-e7e8e358fc55-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:03.214954 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.214948 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b93733cb-f365-418c-893c-e7e8e358fc55-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:03.214954 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.214957 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b93733cb-f365-418c-893c-e7e8e358fc55-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:03.215176 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.214967 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2w95\" (UniqueName: \"kubernetes.io/projected/b93733cb-f365-418c-893c-e7e8e358fc55-kube-api-access-g2w95\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:03.638867 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.638832 5059 generic.go:356] "Generic (PLEG): container finished" podID="b93733cb-f365-418c-893c-e7e8e358fc55" containerID="fd91da36314f70ff464b531ebac4e94b229eed55b3513d1dc71d9c3766d4b04b" exitCode=0 Nov 03 19:55:03.639072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.638889 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" Nov 03 19:55:03.639072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.638920 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" event={"ID":"b93733cb-f365-418c-893c-e7e8e358fc55","Type":"ContainerDied","Data":"fd91da36314f70ff464b531ebac4e94b229eed55b3513d1dc71d9c3766d4b04b"} Nov 03 19:55:03.639072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.638981 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk" event={"ID":"b93733cb-f365-418c-893c-e7e8e358fc55","Type":"ContainerDied","Data":"d72484012ecc90d9c89997e183c445621877ae4e6ecb152fafa8f8280d25ba79"} Nov 03 19:55:03.639072 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.639010 5059 scope.go:122] "RemoveContainer" containerID="fd91da36314f70ff464b531ebac4e94b229eed55b3513d1dc71d9c3766d4b04b" Nov 03 19:55:03.642350 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.642221 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" event={"ID":"c91e48d0-e218-4c9b-98c4-05415c3fc44c","Type":"ContainerStarted","Data":"f829983f5bd307ef1324f308935afd689f33b2f97a835dc6dd13e6817d4452d0"} Nov 03 19:55:03.645744 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.645716 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" event={"ID":"2c7ddbfd-b074-4fed-8748-949d6a015621","Type":"ContainerStarted","Data":"565fee6343eb3f597f9fcb7f7f81971c2449be2916cc97add489a97f053dffe1"} Nov 03 19:55:03.691603 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.686095 5059 scope.go:122] "RemoveContainer" containerID="944da2903cc93746497d5c2e02b4669ae40c03f989ea3a04d91c59c28c2fac1e" Nov 03 19:55:03.703570 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.703533 5059 scope.go:122] "RemoveContainer" containerID="fd91da36314f70ff464b531ebac4e94b229eed55b3513d1dc71d9c3766d4b04b" Nov 03 19:55:03.703994 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:55:03.703969 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd91da36314f70ff464b531ebac4e94b229eed55b3513d1dc71d9c3766d4b04b\": container with ID starting with fd91da36314f70ff464b531ebac4e94b229eed55b3513d1dc71d9c3766d4b04b not found: ID does not exist" containerID="fd91da36314f70ff464b531ebac4e94b229eed55b3513d1dc71d9c3766d4b04b" Nov 03 19:55:03.704081 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.704006 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd91da36314f70ff464b531ebac4e94b229eed55b3513d1dc71d9c3766d4b04b"} err="failed to get container status \"fd91da36314f70ff464b531ebac4e94b229eed55b3513d1dc71d9c3766d4b04b\": rpc error: code = NotFound desc = could not find container \"fd91da36314f70ff464b531ebac4e94b229eed55b3513d1dc71d9c3766d4b04b\": container with ID starting with fd91da36314f70ff464b531ebac4e94b229eed55b3513d1dc71d9c3766d4b04b not found: ID does not exist" Nov 03 19:55:03.704081 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.704032 5059 scope.go:122] "RemoveContainer" containerID="944da2903cc93746497d5c2e02b4669ae40c03f989ea3a04d91c59c28c2fac1e" Nov 03 19:55:03.704294 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:55:03.704272 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"944da2903cc93746497d5c2e02b4669ae40c03f989ea3a04d91c59c28c2fac1e\": container with ID starting with 944da2903cc93746497d5c2e02b4669ae40c03f989ea3a04d91c59c28c2fac1e not found: ID does not exist" containerID="944da2903cc93746497d5c2e02b4669ae40c03f989ea3a04d91c59c28c2fac1e" Nov 03 19:55:03.704336 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.704306 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944da2903cc93746497d5c2e02b4669ae40c03f989ea3a04d91c59c28c2fac1e"} err="failed to get container status \"944da2903cc93746497d5c2e02b4669ae40c03f989ea3a04d91c59c28c2fac1e\": rpc error: code = NotFound desc = could not find container \"944da2903cc93746497d5c2e02b4669ae40c03f989ea3a04d91c59c28c2fac1e\": container with ID starting with 944da2903cc93746497d5c2e02b4669ae40c03f989ea3a04d91c59c28c2fac1e not found: ID does not exist" Nov 03 19:55:03.733467 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.733431 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk"] Nov 03 19:55:03.739488 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.739457 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0a2237-openshift-default-c9b5bf779-88plk"] Nov 03 19:55:03.865647 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:03.865609 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b93733cb-f365-418c-893c-e7e8e358fc55" path="/var/lib/kubelet/pods/b93733cb-f365-418c-893c-e7e8e358fc55/volumes" Nov 03 19:55:04.177682 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:04.176091 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:55:04.177682 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:04.176218 5059 patch_prober.go:29] interesting pod/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 19:55:04.177682 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:04.176265 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" podUID="2c7ddbfd-b074-4fed-8748-949d6a015621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 19:55:04.208333 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:04.207445 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:55:04.210703 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:04.210428 5059 patch_prober.go:29] interesting pod/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 19:55:04.210703 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:04.210486 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" podUID="c91e48d0-e218-4c9b-98c4-05415c3fc44c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 19:55:04.992686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:04.989835 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:55:05.188687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:05.180076 5059 patch_prober.go:29] interesting pod/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 19:55:05.188687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:05.180151 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" podUID="2c7ddbfd-b074-4fed-8748-949d6a015621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 19:55:05.208230 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:05.208194 5059 patch_prober.go:29] interesting pod/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 19:55:05.208463 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:05.208439 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" podUID="c91e48d0-e218-4c9b-98c4-05415c3fc44c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 19:55:06.217522 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:06.217482 5059 patch_prober.go:29] interesting pod/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 19:55:06.220805 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:06.220759 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" podUID="2c7ddbfd-b074-4fed-8748-949d6a015621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 19:55:06.220932 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:06.219825 5059 patch_prober.go:29] interesting pod/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 19:55:06.220932 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:06.220833 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" podUID="c91e48d0-e218-4c9b-98c4-05415c3fc44c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 19:55:07.207171 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:07.207126 5059 patch_prober.go:29] interesting pod/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 19:55:07.207429 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:07.207396 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" podUID="2c7ddbfd-b074-4fed-8748-949d6a015621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 19:55:07.208344 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:07.208286 5059 patch_prober.go:29] interesting pod/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 19:55:07.208515 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:07.208493 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" podUID="c91e48d0-e218-4c9b-98c4-05415c3fc44c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 19:55:08.173139 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.173056 5059 patch_prober.go:29] interesting pod/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 19:55:08.173681 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.173141 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" podUID="2c7ddbfd-b074-4fed-8748-949d6a015621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 19:55:08.208060 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.208021 5059 patch_prober.go:29] interesting pod/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 19:55:08.208375 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.208348 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" podUID="c91e48d0-e218-4c9b-98c4-05415c3fc44c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 19:55:08.517197 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.517116 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80a16c0d-mnn86"] Nov 03 19:55:08.518128 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.518110 5059 cpu_manager.go:414] "containerMap: removing container" podUID="b93733cb-f365-418c-893c-e7e8e358fc55" containerName="istio-proxy" Nov 03 19:55:08.518190 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.518131 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b93733cb-f365-418c-893c-e7e8e358fc55" containerName="istio-proxy" Nov 03 19:55:08.518190 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.518156 5059 cpu_manager.go:414] "containerMap: removing container" podUID="d6c701c3-ed02-4fd8-ad04-430541e88c6e" containerName="revoke-keys" Nov 03 19:55:08.518190 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.518164 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c701c3-ed02-4fd8-ad04-430541e88c6e" containerName="revoke-keys" Nov 03 19:55:08.518190 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.518185 5059 cpu_manager.go:414] "containerMap: removing container" podUID="b93733cb-f365-418c-893c-e7e8e358fc55" containerName="istio-proxy" Nov 03 19:55:08.518329 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.518193 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b93733cb-f365-418c-893c-e7e8e358fc55" containerName="istio-proxy" Nov 03 19:55:08.518383 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.518372 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b93733cb-f365-418c-893c-e7e8e358fc55" containerName="istio-proxy" Nov 03 19:55:08.518417 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.518390 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c701c3-ed02-4fd8-ad04-430541e88c6e" containerName="revoke-keys" Nov 03 19:55:08.518417 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.518401 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b93733cb-f365-418c-893c-e7e8e358fc55" containerName="istio-proxy" Nov 03 19:55:08.526536 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.526507 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a16c0d-mnn86" Nov 03 19:55:08.530727 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.530700 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:55:08.531273 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.531254 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80a16c0d-mnn86"] Nov 03 19:55:08.532699 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.532121 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 19:55:08.645601 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.645568 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80a16c0d-mnn86\" (UID: \"ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a16c0d-mnn86" Nov 03 19:55:08.645819 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.645638 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvpz5\" (UniqueName: \"kubernetes.io/projected/ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14-kube-api-access-vvpz5\") pod \"maas-api-revoke-sub-80a16c0d-mnn86\" (UID: \"ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a16c0d-mnn86" Nov 03 19:55:08.645819 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.645803 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80a16c0d-mnn86\" (UID: \"ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a16c0d-mnn86" Nov 03 19:55:08.771735 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.759958 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80a16c0d-mnn86\" (UID: \"ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a16c0d-mnn86" Nov 03 19:55:08.771735 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.760085 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80a16c0d-mnn86\" (UID: \"ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a16c0d-mnn86" Nov 03 19:55:08.771735 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.760172 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvpz5\" (UniqueName: \"kubernetes.io/projected/ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14-kube-api-access-vvpz5\") pod \"maas-api-revoke-sub-80a16c0d-mnn86\" (UID: \"ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a16c0d-mnn86" Nov 03 19:55:08.784685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.782555 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80a16c0d-mnn86\" (UID: \"ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a16c0d-mnn86" Nov 03 19:55:08.818684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.816868 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvpz5\" (UniqueName: \"kubernetes.io/projected/ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14-kube-api-access-vvpz5\") pod \"maas-api-revoke-sub-80a16c0d-mnn86\" (UID: \"ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a16c0d-mnn86" Nov 03 19:55:08.837694 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.830132 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80a16c0d-mnn86\" (UID: \"ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a16c0d-mnn86" Nov 03 19:55:08.866694 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:08.843109 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a16c0d-mnn86" Nov 03 19:55:09.182684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:09.182067 5059 patch_prober.go:29] interesting pod/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 19:55:09.182684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:09.182139 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" podUID="2c7ddbfd-b074-4fed-8748-949d6a015621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 19:55:09.224555 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:09.223417 5059 patch_prober.go:29] interesting pod/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 19:55:09.224555 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:09.223474 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" podUID="c91e48d0-e218-4c9b-98c4-05415c3fc44c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 19:55:09.322914 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:09.322789 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80a16c0d-mnn86"] Nov 03 19:55:09.323538 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:55:09.323478 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac84d4c1_fb3c_47f6_9ee0_33b89f83ae14.slice/crio-d7accdadb27fd09328d8616ca1cf38bbb4dd0ad0c7a18fa4c204bccac4f2fe9b WatchSource:0}: Error finding container d7accdadb27fd09328d8616ca1cf38bbb4dd0ad0c7a18fa4c204bccac4f2fe9b: Status 404 returned error can't find the container with id d7accdadb27fd09328d8616ca1cf38bbb4dd0ad0c7a18fa4c204bccac4f2fe9b Nov 03 19:55:09.921994 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:09.921915 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a16c0d-mnn86" event={"ID":"ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14","Type":"ContainerStarted","Data":"d7accdadb27fd09328d8616ca1cf38bbb4dd0ad0c7a18fa4c204bccac4f2fe9b"} Nov 03 19:55:10.030081 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:10.030042 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c158645-v77cp"] Nov 03 19:55:10.034527 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:10.034496 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c158645-v77cp"] Nov 03 19:55:10.198762 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:10.198611 5059 patch_prober.go:29] interesting pod/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 19:55:10.198762 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:10.198721 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" podUID="c91e48d0-e218-4c9b-98c4-05415c3fc44c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 19:55:10.492425 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:10.492344 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:55:10.492792 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:10.492773 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:55:10.493574 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:10.493553 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:55:10.623542 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:55:10.623495 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:55:10.932102 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:10.932070 5059 generic.go:356] "Generic (PLEG): container finished" podID="ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14" containerID="2012fe05eff12aba66893f4a0842bdb7c135cb9a7413da3da728d65095b721ec" exitCode=0 Nov 03 19:55:10.932293 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:10.932153 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a16c0d-mnn86" event={"ID":"ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14","Type":"ContainerDied","Data":"2012fe05eff12aba66893f4a0842bdb7c135cb9a7413da3da728d65095b721ec"} Nov 03 19:55:11.199031 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:11.199002 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:55:11.200082 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:11.200060 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:55:11.860221 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:11.860178 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d7e0838-dcc9-4758-8490-79e1a1d86c24" path="/var/lib/kubelet/pods/2d7e0838-dcc9-4758-8490-79e1a1d86c24/volumes" Nov 03 19:55:12.164270 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:12.164244 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a16c0d-mnn86" Nov 03 19:55:12.187607 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:12.187576 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14-maas-api-xks-ca\") pod \"ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14\" (UID: \"ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14\") " Nov 03 19:55:12.188218 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:12.187807 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14-maas-api-openshift-ca\") pod \"ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14\" (UID: \"ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14\") " Nov 03 19:55:12.188218 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:12.187881 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14-kube-api-access-vvpz5\" (UniqueName: \"kubernetes.io/projected/ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14-kube-api-access-vvpz5\") pod \"ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14\" (UID: \"ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14\") " Nov 03 19:55:12.188800 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:12.188773 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14-maas-api-openshift-ca" pod "ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14" (UID: "ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:55:12.190411 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:12.190386 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14-kube-api-access-vvpz5" pod "ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14" (UID: "ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14"). InnerVolumeSpecName "kube-api-access-vvpz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:55:12.190558 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:12.190536 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14-maas-api-xks-ca" pod "ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14" (UID: "ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:55:12.290263 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:12.290227 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:12.290263 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:12.290255 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:12.290263 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:12.290265 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvpz5\" (UniqueName: \"kubernetes.io/projected/ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14-kube-api-access-vvpz5\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:12.894810 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:12.894779 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb"] Nov 03 19:55:12.953043 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:12.953016 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a16c0d-mnn86" Nov 03 19:55:12.953207 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:12.953056 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a16c0d-mnn86" event={"ID":"ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14","Type":"ContainerDied","Data":"d7accdadb27fd09328d8616ca1cf38bbb4dd0ad0c7a18fa4c204bccac4f2fe9b"} Nov 03 19:55:12.953207 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:12.953108 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7accdadb27fd09328d8616ca1cf38bbb4dd0ad0c7a18fa4c204bccac4f2fe9b" Nov 03 19:55:12.953349 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:12.953328 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" podUID="c91e48d0-e218-4c9b-98c4-05415c3fc44c" containerName="istio-proxy" containerID="cri-o://f829983f5bd307ef1324f308935afd689f33b2f97a835dc6dd13e6817d4452d0" gracePeriod=30 Nov 03 19:55:17.559400 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:17.559361 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd"] Nov 03 19:55:17.559942 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:17.559608 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" podUID="2c7ddbfd-b074-4fed-8748-949d6a015621" containerName="istio-proxy" containerID="cri-o://565fee6343eb3f597f9fcb7f7f81971c2449be2916cc97add489a97f053dffe1" gracePeriod=30 Nov 03 19:55:18.242509 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.242454 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:55:18.304624 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.304578 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-token\" (UniqueName: \"kubernetes.io/projected/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-token\") pod \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " Nov 03 19:55:18.304847 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.304780 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-podinfo\") pod \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " Nov 03 19:55:18.304847 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.304839 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istiod-ca-cert\") pod \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " Nov 03 19:55:18.306336 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.304985 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-workload-certs\") pod \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " Nov 03 19:55:18.306336 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.305020 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-data\") pod \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " Nov 03 19:55:18.306336 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.305049 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-credential-socket\") pod \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " Nov 03 19:55:18.306336 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.305091 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-workload-socket\") pod \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " Nov 03 19:55:18.306336 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.305124 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-envoy\") pod \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " Nov 03 19:55:18.306336 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.305191 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c91e48d0-e218-4c9b-98c4-05415c3fc44c-kube-api-access-92vxf\" (UniqueName: \"kubernetes.io/projected/c91e48d0-e218-4c9b-98c4-05415c3fc44c-kube-api-access-92vxf\") pod \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\" (UID: \"c91e48d0-e218-4c9b-98c4-05415c3fc44c\") " Nov 03 19:55:18.306336 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.305946 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-workload-socket" pod "c91e48d0-e218-4c9b-98c4-05415c3fc44c" (UID: "c91e48d0-e218-4c9b-98c4-05415c3fc44c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:55:18.306336 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.306155 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-workload-certs" pod "c91e48d0-e218-4c9b-98c4-05415c3fc44c" (UID: "c91e48d0-e218-4c9b-98c4-05415c3fc44c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:55:18.306336 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.306257 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-data" pod "c91e48d0-e218-4c9b-98c4-05415c3fc44c" (UID: "c91e48d0-e218-4c9b-98c4-05415c3fc44c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:55:18.306336 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.306310 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-credential-socket" pod "c91e48d0-e218-4c9b-98c4-05415c3fc44c" (UID: "c91e48d0-e218-4c9b-98c4-05415c3fc44c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:55:18.307132 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.307111 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istiod-ca-cert" pod "c91e48d0-e218-4c9b-98c4-05415c3fc44c" (UID: "c91e48d0-e218-4c9b-98c4-05415c3fc44c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:55:18.308326 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.308299 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-token" pod "c91e48d0-e218-4c9b-98c4-05415c3fc44c" (UID: "c91e48d0-e218-4c9b-98c4-05415c3fc44c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:55:18.309022 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.308995 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-podinfo" pod "c91e48d0-e218-4c9b-98c4-05415c3fc44c" (UID: "c91e48d0-e218-4c9b-98c4-05415c3fc44c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:55:18.309175 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.309154 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c91e48d0-e218-4c9b-98c4-05415c3fc44c-kube-api-access-92vxf" pod "c91e48d0-e218-4c9b-98c4-05415c3fc44c" (UID: "c91e48d0-e218-4c9b-98c4-05415c3fc44c"). InnerVolumeSpecName "kube-api-access-92vxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:55:18.319054 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.319025 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-envoy" pod "c91e48d0-e218-4c9b-98c4-05415c3fc44c" (UID: "c91e48d0-e218-4c9b-98c4-05415c3fc44c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:55:18.407394 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.407358 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:18.407394 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.407388 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:18.407394 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.407398 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:18.407607 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.407409 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:18.407607 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.407418 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:18.407607 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.407426 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:18.407607 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.407434 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:18.407607 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.407442 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c91e48d0-e218-4c9b-98c4-05415c3fc44c-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:18.407607 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:18.407450 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92vxf\" (UniqueName: \"kubernetes.io/projected/c91e48d0-e218-4c9b-98c4-05415c3fc44c-kube-api-access-92vxf\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:19.051057 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:19.051024 5059 generic.go:356] "Generic (PLEG): container finished" podID="c91e48d0-e218-4c9b-98c4-05415c3fc44c" containerID="f829983f5bd307ef1324f308935afd689f33b2f97a835dc6dd13e6817d4452d0" exitCode=0 Nov 03 19:55:19.051539 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:19.051074 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" Nov 03 19:55:19.051539 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:19.051107 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" event={"ID":"c91e48d0-e218-4c9b-98c4-05415c3fc44c","Type":"ContainerDied","Data":"f829983f5bd307ef1324f308935afd689f33b2f97a835dc6dd13e6817d4452d0"} Nov 03 19:55:19.051539 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:19.051180 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb" event={"ID":"c91e48d0-e218-4c9b-98c4-05415c3fc44c","Type":"ContainerDied","Data":"484db02aeafcc47a5dfb0a2e72d5a8b2a0933f2abd515766fb4e3e64c4c50663"} Nov 03 19:55:19.051539 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:19.051217 5059 scope.go:122] "RemoveContainer" containerID="f829983f5bd307ef1324f308935afd689f33b2f97a835dc6dd13e6817d4452d0" Nov 03 19:55:19.063721 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:19.063703 5059 scope.go:122] "RemoveContainer" containerID="11c70e32c74df688f8f88ae5d27bb395ce3774f0723e9acdea3562d1fa6f5624" Nov 03 19:55:19.087916 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:19.087888 5059 scope.go:122] "RemoveContainer" containerID="f829983f5bd307ef1324f308935afd689f33b2f97a835dc6dd13e6817d4452d0" Nov 03 19:55:19.088230 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:55:19.088211 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f829983f5bd307ef1324f308935afd689f33b2f97a835dc6dd13e6817d4452d0\": container with ID starting with f829983f5bd307ef1324f308935afd689f33b2f97a835dc6dd13e6817d4452d0 not found: ID does not exist" containerID="f829983f5bd307ef1324f308935afd689f33b2f97a835dc6dd13e6817d4452d0" Nov 03 19:55:19.088314 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:19.088238 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f829983f5bd307ef1324f308935afd689f33b2f97a835dc6dd13e6817d4452d0"} err="failed to get container status \"f829983f5bd307ef1324f308935afd689f33b2f97a835dc6dd13e6817d4452d0\": rpc error: code = NotFound desc = could not find container \"f829983f5bd307ef1324f308935afd689f33b2f97a835dc6dd13e6817d4452d0\": container with ID starting with f829983f5bd307ef1324f308935afd689f33b2f97a835dc6dd13e6817d4452d0 not found: ID does not exist" Nov 03 19:55:19.088314 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:19.088256 5059 scope.go:122] "RemoveContainer" containerID="11c70e32c74df688f8f88ae5d27bb395ce3774f0723e9acdea3562d1fa6f5624" Nov 03 19:55:19.088506 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:55:19.088486 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11c70e32c74df688f8f88ae5d27bb395ce3774f0723e9acdea3562d1fa6f5624\": container with ID starting with 11c70e32c74df688f8f88ae5d27bb395ce3774f0723e9acdea3562d1fa6f5624 not found: ID does not exist" containerID="11c70e32c74df688f8f88ae5d27bb395ce3774f0723e9acdea3562d1fa6f5624" Nov 03 19:55:19.088591 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:19.088515 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11c70e32c74df688f8f88ae5d27bb395ce3774f0723e9acdea3562d1fa6f5624"} err="failed to get container status \"11c70e32c74df688f8f88ae5d27bb395ce3774f0723e9acdea3562d1fa6f5624\": rpc error: code = NotFound desc = could not find container \"11c70e32c74df688f8f88ae5d27bb395ce3774f0723e9acdea3562d1fa6f5624\": container with ID starting with 11c70e32c74df688f8f88ae5d27bb395ce3774f0723e9acdea3562d1fa6f5624 not found: ID does not exist" Nov 03 19:55:19.098391 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:19.098359 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb"] Nov 03 19:55:19.102213 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:19.102182 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ad865c-openshift-default-6585d5cdd9-cgpdb"] Nov 03 19:55:19.573749 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:19.573705 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz"] Nov 03 19:55:19.586933 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:19.586900 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d8d238bf-84rfz"] Nov 03 19:55:19.877316 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:19.877080 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c91e48d0-e218-4c9b-98c4-05415c3fc44c" path="/var/lib/kubelet/pods/c91e48d0-e218-4c9b-98c4-05415c3fc44c/volumes" Nov 03 19:55:19.877951 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:19.877915 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6c701c3-ed02-4fd8-ad04-430541e88c6e" path="/var/lib/kubelet/pods/d6c701c3-ed02-4fd8-ad04-430541e88c6e/volumes" Nov 03 19:55:22.924031 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:22.924001 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:55:23.000978 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.000943 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-data\") pod \"2c7ddbfd-b074-4fed-8748-949d6a015621\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " Nov 03 19:55:23.001166 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.001005 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c7ddbfd-b074-4fed-8748-949d6a015621-kube-api-access-7gf2p\" (UniqueName: \"kubernetes.io/projected/2c7ddbfd-b074-4fed-8748-949d6a015621-kube-api-access-7gf2p\") pod \"2c7ddbfd-b074-4fed-8748-949d6a015621\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " Nov 03 19:55:23.001166 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.001041 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-envoy\") pod \"2c7ddbfd-b074-4fed-8748-949d6a015621\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " Nov 03 19:55:23.001166 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.001084 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-workload-socket\") pod \"2c7ddbfd-b074-4fed-8748-949d6a015621\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " Nov 03 19:55:23.001350 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.001299 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-data" pod "2c7ddbfd-b074-4fed-8748-949d6a015621" (UID: "2c7ddbfd-b074-4fed-8748-949d6a015621"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:55:23.001458 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.001426 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-workload-socket" pod "2c7ddbfd-b074-4fed-8748-949d6a015621" (UID: "2c7ddbfd-b074-4fed-8748-949d6a015621"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:55:23.001616 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.001594 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-podinfo\") pod \"2c7ddbfd-b074-4fed-8748-949d6a015621\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " Nov 03 19:55:23.002125 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.001713 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-credential-socket\") pod \"2c7ddbfd-b074-4fed-8748-949d6a015621\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " Nov 03 19:55:23.002125 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.001964 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-credential-socket" pod "2c7ddbfd-b074-4fed-8748-949d6a015621" (UID: "2c7ddbfd-b074-4fed-8748-949d6a015621"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:55:23.002125 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.002016 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c7ddbfd-b074-4fed-8748-949d6a015621-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7ddbfd-b074-4fed-8748-949d6a015621-istiod-ca-cert\") pod \"2c7ddbfd-b074-4fed-8748-949d6a015621\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " Nov 03 19:55:23.002384 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.002358 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c7ddbfd-b074-4fed-8748-949d6a015621-istiod-ca-cert" pod "2c7ddbfd-b074-4fed-8748-949d6a015621" (UID: "2c7ddbfd-b074-4fed-8748-949d6a015621"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:55:23.002451 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.002432 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-workload-certs\") pod \"2c7ddbfd-b074-4fed-8748-949d6a015621\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " Nov 03 19:55:23.002828 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.002724 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-workload-certs" pod "2c7ddbfd-b074-4fed-8748-949d6a015621" (UID: "2c7ddbfd-b074-4fed-8748-949d6a015621"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:55:23.003037 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.002911 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-token\" (UniqueName: \"kubernetes.io/projected/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-token\") pod \"2c7ddbfd-b074-4fed-8748-949d6a015621\" (UID: \"2c7ddbfd-b074-4fed-8748-949d6a015621\") " Nov 03 19:55:23.004476 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.003887 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-envoy" pod "2c7ddbfd-b074-4fed-8748-949d6a015621" (UID: "2c7ddbfd-b074-4fed-8748-949d6a015621"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:55:23.004476 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.005314 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:23.004476 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.005488 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:23.004476 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.005536 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:23.004476 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.005552 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7ddbfd-b074-4fed-8748-949d6a015621-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:23.004476 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.005568 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:23.004476 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.005582 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:23.004476 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.006002 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c7ddbfd-b074-4fed-8748-949d6a015621-kube-api-access-7gf2p" pod "2c7ddbfd-b074-4fed-8748-949d6a015621" (UID: "2c7ddbfd-b074-4fed-8748-949d6a015621"). InnerVolumeSpecName "kube-api-access-7gf2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:55:23.013116 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.013084 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-token" pod "2c7ddbfd-b074-4fed-8748-949d6a015621" (UID: "2c7ddbfd-b074-4fed-8748-949d6a015621"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:55:23.015504 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.015449 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-podinfo" pod "2c7ddbfd-b074-4fed-8748-949d6a015621" (UID: "2c7ddbfd-b074-4fed-8748-949d6a015621"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:55:23.096242 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.096162 5059 generic.go:356] "Generic (PLEG): container finished" podID="2c7ddbfd-b074-4fed-8748-949d6a015621" containerID="565fee6343eb3f597f9fcb7f7f81971c2449be2916cc97add489a97f053dffe1" exitCode=0 Nov 03 19:55:23.096242 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.096216 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" Nov 03 19:55:23.096440 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.096240 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" event={"ID":"2c7ddbfd-b074-4fed-8748-949d6a015621","Type":"ContainerDied","Data":"565fee6343eb3f597f9fcb7f7f81971c2449be2916cc97add489a97f053dffe1"} Nov 03 19:55:23.096440 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.096304 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd" event={"ID":"2c7ddbfd-b074-4fed-8748-949d6a015621","Type":"ContainerDied","Data":"00fccd4546d4f96c7fed60ff12f220e1fec510574cdf0834d1dfa749b0df69b8"} Nov 03 19:55:23.096440 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.096330 5059 scope.go:122] "RemoveContainer" containerID="565fee6343eb3f597f9fcb7f7f81971c2449be2916cc97add489a97f053dffe1" Nov 03 19:55:23.106572 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.106549 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:23.106572 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.106573 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c7ddbfd-b074-4fed-8748-949d6a015621-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:23.106763 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.106592 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gf2p\" (UniqueName: \"kubernetes.io/projected/2c7ddbfd-b074-4fed-8748-949d6a015621-kube-api-access-7gf2p\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:23.110733 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.110712 5059 scope.go:122] "RemoveContainer" containerID="2bf1203d8393d332419f358c61b7ffc2d7ed4b3606d1b21ee77ea9ed47daa251" Nov 03 19:55:23.124235 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.124211 5059 scope.go:122] "RemoveContainer" containerID="565fee6343eb3f597f9fcb7f7f81971c2449be2916cc97add489a97f053dffe1" Nov 03 19:55:23.129677 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:55:23.125772 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"565fee6343eb3f597f9fcb7f7f81971c2449be2916cc97add489a97f053dffe1\": container with ID starting with 565fee6343eb3f597f9fcb7f7f81971c2449be2916cc97add489a97f053dffe1 not found: ID does not exist" containerID="565fee6343eb3f597f9fcb7f7f81971c2449be2916cc97add489a97f053dffe1" Nov 03 19:55:23.129677 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.125809 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"565fee6343eb3f597f9fcb7f7f81971c2449be2916cc97add489a97f053dffe1"} err="failed to get container status \"565fee6343eb3f597f9fcb7f7f81971c2449be2916cc97add489a97f053dffe1\": rpc error: code = NotFound desc = could not find container \"565fee6343eb3f597f9fcb7f7f81971c2449be2916cc97add489a97f053dffe1\": container with ID starting with 565fee6343eb3f597f9fcb7f7f81971c2449be2916cc97add489a97f053dffe1 not found: ID does not exist" Nov 03 19:55:23.129677 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.125835 5059 scope.go:122] "RemoveContainer" containerID="2bf1203d8393d332419f358c61b7ffc2d7ed4b3606d1b21ee77ea9ed47daa251" Nov 03 19:55:23.130254 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:55:23.130222 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf1203d8393d332419f358c61b7ffc2d7ed4b3606d1b21ee77ea9ed47daa251\": container with ID starting with 2bf1203d8393d332419f358c61b7ffc2d7ed4b3606d1b21ee77ea9ed47daa251 not found: ID does not exist" containerID="2bf1203d8393d332419f358c61b7ffc2d7ed4b3606d1b21ee77ea9ed47daa251" Nov 03 19:55:23.130351 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.130262 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf1203d8393d332419f358c61b7ffc2d7ed4b3606d1b21ee77ea9ed47daa251"} err="failed to get container status \"2bf1203d8393d332419f358c61b7ffc2d7ed4b3606d1b21ee77ea9ed47daa251\": rpc error: code = NotFound desc = could not find container \"2bf1203d8393d332419f358c61b7ffc2d7ed4b3606d1b21ee77ea9ed47daa251\": container with ID starting with 2bf1203d8393d332419f358c61b7ffc2d7ed4b3606d1b21ee77ea9ed47daa251 not found: ID does not exist" Nov 03 19:55:23.134267 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.134212 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd"] Nov 03 19:55:23.140010 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.139983 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f20fc0-openshift-default-5b67979f5c-5zlrd"] Nov 03 19:55:23.863556 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:23.863518 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c7ddbfd-b074-4fed-8748-949d6a015621" path="/var/lib/kubelet/pods/2c7ddbfd-b074-4fed-8748-949d6a015621/volumes" Nov 03 19:55:24.873149 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:24.873118 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a81ac7b8-flpxz"] Nov 03 19:55:24.873783 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:24.873768 5059 cpu_manager.go:414] "containerMap: removing container" podUID="ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14" containerName="revoke-keys" Nov 03 19:55:24.873831 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:24.873784 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14" containerName="revoke-keys" Nov 03 19:55:24.873831 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:24.873805 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c91e48d0-e218-4c9b-98c4-05415c3fc44c" containerName="istio-proxy" Nov 03 19:55:24.873831 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:24.873810 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c91e48d0-e218-4c9b-98c4-05415c3fc44c" containerName="istio-proxy" Nov 03 19:55:24.873831 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:24.873823 5059 cpu_manager.go:414] "containerMap: removing container" podUID="2c7ddbfd-b074-4fed-8748-949d6a015621" containerName="istio-proxy" Nov 03 19:55:24.873831 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:24.873827 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c7ddbfd-b074-4fed-8748-949d6a015621" containerName="istio-proxy" Nov 03 19:55:24.873996 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:24.873836 5059 cpu_manager.go:414] "containerMap: removing container" podUID="2c7ddbfd-b074-4fed-8748-949d6a015621" containerName="istio-proxy" Nov 03 19:55:24.873996 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:24.873841 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c7ddbfd-b074-4fed-8748-949d6a015621" containerName="istio-proxy" Nov 03 19:55:24.873996 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:24.873861 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c91e48d0-e218-4c9b-98c4-05415c3fc44c" containerName="istio-proxy" Nov 03 19:55:24.873996 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:24.873865 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c91e48d0-e218-4c9b-98c4-05415c3fc44c" containerName="istio-proxy" Nov 03 19:55:24.873996 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:24.873966 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c7ddbfd-b074-4fed-8748-949d6a015621" containerName="istio-proxy" Nov 03 19:55:24.873996 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:24.873973 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c7ddbfd-b074-4fed-8748-949d6a015621" containerName="istio-proxy" Nov 03 19:55:24.873996 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:24.873983 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c91e48d0-e218-4c9b-98c4-05415c3fc44c" containerName="istio-proxy" Nov 03 19:55:24.873996 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:24.873992 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14" containerName="revoke-keys" Nov 03 19:55:24.879226 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:24.879209 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81ac7b8-flpxz" Nov 03 19:55:24.886941 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:24.886918 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a81ac7b8-flpxz"] Nov 03 19:55:24.904705 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:24.904632 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:55:24.904843 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:24.904800 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 19:55:25.037801 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:25.037770 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9502112f-4584-47aa-853a-13c2e33a8e99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a81ac7b8-flpxz\" (UID: \"9502112f-4584-47aa-853a-13c2e33a8e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81ac7b8-flpxz" Nov 03 19:55:25.037959 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:25.037931 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9502112f-4584-47aa-853a-13c2e33a8e99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a81ac7b8-flpxz\" (UID: \"9502112f-4584-47aa-853a-13c2e33a8e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81ac7b8-flpxz" Nov 03 19:55:25.038012 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:25.037997 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9td9j\" (UniqueName: \"kubernetes.io/projected/9502112f-4584-47aa-853a-13c2e33a8e99-kube-api-access-9td9j\") pod \"maas-api-revoke-sub-a81ac7b8-flpxz\" (UID: \"9502112f-4584-47aa-853a-13c2e33a8e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81ac7b8-flpxz" Nov 03 19:55:25.139380 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:25.139307 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9502112f-4584-47aa-853a-13c2e33a8e99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a81ac7b8-flpxz\" (UID: \"9502112f-4584-47aa-853a-13c2e33a8e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81ac7b8-flpxz" Nov 03 19:55:25.139380 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:25.139377 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9td9j\" (UniqueName: \"kubernetes.io/projected/9502112f-4584-47aa-853a-13c2e33a8e99-kube-api-access-9td9j\") pod \"maas-api-revoke-sub-a81ac7b8-flpxz\" (UID: \"9502112f-4584-47aa-853a-13c2e33a8e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81ac7b8-flpxz" Nov 03 19:55:25.139590 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:25.139444 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9502112f-4584-47aa-853a-13c2e33a8e99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a81ac7b8-flpxz\" (UID: \"9502112f-4584-47aa-853a-13c2e33a8e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81ac7b8-flpxz" Nov 03 19:55:25.139994 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:25.139969 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9502112f-4584-47aa-853a-13c2e33a8e99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a81ac7b8-flpxz\" (UID: \"9502112f-4584-47aa-853a-13c2e33a8e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81ac7b8-flpxz" Nov 03 19:55:25.141734 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:25.141714 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9502112f-4584-47aa-853a-13c2e33a8e99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a81ac7b8-flpxz\" (UID: \"9502112f-4584-47aa-853a-13c2e33a8e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81ac7b8-flpxz" Nov 03 19:55:25.152287 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:25.152263 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9td9j\" (UniqueName: \"kubernetes.io/projected/9502112f-4584-47aa-853a-13c2e33a8e99-kube-api-access-9td9j\") pod \"maas-api-revoke-sub-a81ac7b8-flpxz\" (UID: \"9502112f-4584-47aa-853a-13c2e33a8e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81ac7b8-flpxz" Nov 03 19:55:25.196519 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:25.196494 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81ac7b8-flpxz" Nov 03 19:55:25.490004 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:55:25.489964 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9502112f_4584_47aa_853a_13c2e33a8e99.slice/crio-2901b94a3ccedadb50c94f868ea3dc34fb057f5c24675f776ea2a7e11a135fa8 WatchSource:0}: Error finding container 2901b94a3ccedadb50c94f868ea3dc34fb057f5c24675f776ea2a7e11a135fa8: Status 404 returned error can't find the container with id 2901b94a3ccedadb50c94f868ea3dc34fb057f5c24675f776ea2a7e11a135fa8 Nov 03 19:55:25.490711 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:25.490685 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a81ac7b8-flpxz"] Nov 03 19:55:26.123099 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:26.123066 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81ac7b8-flpxz" event={"ID":"9502112f-4584-47aa-853a-13c2e33a8e99","Type":"ContainerStarted","Data":"93a88f6eefe518149493f36ad88ef3160f1d400b6ac95b64287b395c5fb27ef4"} Nov 03 19:55:26.123453 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:26.123120 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81ac7b8-flpxz" event={"ID":"9502112f-4584-47aa-853a-13c2e33a8e99","Type":"ContainerStarted","Data":"2901b94a3ccedadb50c94f868ea3dc34fb057f5c24675f776ea2a7e11a135fa8"} Nov 03 19:55:26.147061 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:26.147005 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81ac7b8-flpxz" podStartSLOduration=2.14699085 podStartE2EDuration="2.14699085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:26.145829891 +0000 UTC m=+1120.403627148" watchObservedRunningTime="2026-11-03 19:55:26.14699085 +0000 UTC m=+1120.404788410" Nov 03 19:55:27.133878 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:27.133849 5059 generic.go:356] "Generic (PLEG): container finished" podID="9502112f-4584-47aa-853a-13c2e33a8e99" containerID="93a88f6eefe518149493f36ad88ef3160f1d400b6ac95b64287b395c5fb27ef4" exitCode=0 Nov 03 19:55:27.134228 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:27.133938 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81ac7b8-flpxz" event={"ID":"9502112f-4584-47aa-853a-13c2e33a8e99","Type":"ContainerDied","Data":"93a88f6eefe518149493f36ad88ef3160f1d400b6ac95b64287b395c5fb27ef4"} Nov 03 19:55:28.291602 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:28.291576 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81ac7b8-flpxz" Nov 03 19:55:28.408828 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:28.408718 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9502112f-4584-47aa-853a-13c2e33a8e99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9502112f-4584-47aa-853a-13c2e33a8e99-maas-api-openshift-ca\") pod \"9502112f-4584-47aa-853a-13c2e33a8e99\" (UID: \"9502112f-4584-47aa-853a-13c2e33a8e99\") " Nov 03 19:55:28.416239 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:28.416198 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9502112f-4584-47aa-853a-13c2e33a8e99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9502112f-4584-47aa-853a-13c2e33a8e99-maas-api-xks-ca\") pod \"9502112f-4584-47aa-853a-13c2e33a8e99\" (UID: \"9502112f-4584-47aa-853a-13c2e33a8e99\") " Nov 03 19:55:28.416442 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:28.416319 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9502112f-4584-47aa-853a-13c2e33a8e99-kube-api-access-9td9j\" (UniqueName: \"kubernetes.io/projected/9502112f-4584-47aa-853a-13c2e33a8e99-kube-api-access-9td9j\") pod \"9502112f-4584-47aa-853a-13c2e33a8e99\" (UID: \"9502112f-4584-47aa-853a-13c2e33a8e99\") " Nov 03 19:55:28.417617 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:28.416638 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9502112f-4584-47aa-853a-13c2e33a8e99-maas-api-openshift-ca" pod "9502112f-4584-47aa-853a-13c2e33a8e99" (UID: "9502112f-4584-47aa-853a-13c2e33a8e99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:55:28.417617 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:28.417210 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9502112f-4584-47aa-853a-13c2e33a8e99-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:28.420605 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:28.420578 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9502112f-4584-47aa-853a-13c2e33a8e99-maas-api-xks-ca" pod "9502112f-4584-47aa-853a-13c2e33a8e99" (UID: "9502112f-4584-47aa-853a-13c2e33a8e99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:55:28.423057 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:28.423016 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9502112f-4584-47aa-853a-13c2e33a8e99-kube-api-access-9td9j" pod "9502112f-4584-47aa-853a-13c2e33a8e99" (UID: "9502112f-4584-47aa-853a-13c2e33a8e99"). InnerVolumeSpecName "kube-api-access-9td9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:55:28.518759 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:28.518728 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9502112f-4584-47aa-853a-13c2e33a8e99-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:28.518759 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:28.518754 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9td9j\" (UniqueName: \"kubernetes.io/projected/9502112f-4584-47aa-853a-13c2e33a8e99-kube-api-access-9td9j\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:55:29.154227 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:29.154177 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81ac7b8-flpxz" Nov 03 19:55:29.154227 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:29.154192 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81ac7b8-flpxz" event={"ID":"9502112f-4584-47aa-853a-13c2e33a8e99","Type":"ContainerDied","Data":"2901b94a3ccedadb50c94f868ea3dc34fb057f5c24675f776ea2a7e11a135fa8"} Nov 03 19:55:29.154470 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:29.154250 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2901b94a3ccedadb50c94f868ea3dc34fb057f5c24675f776ea2a7e11a135fa8" Nov 03 19:55:40.623506 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:55:40.623451 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:55:40.628761 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.628729 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln"] Nov 03 19:55:40.629603 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.629582 5059 cpu_manager.go:414] "containerMap: removing container" podUID="9502112f-4584-47aa-853a-13c2e33a8e99" containerName="revoke-keys" Nov 03 19:55:40.629681 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.629604 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9502112f-4584-47aa-853a-13c2e33a8e99" containerName="revoke-keys" Nov 03 19:55:40.629750 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.629739 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9502112f-4584-47aa-853a-13c2e33a8e99" containerName="revoke-keys" Nov 03 19:55:40.629808 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.629759 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c91e48d0-e218-4c9b-98c4-05415c3fc44c" containerName="istio-proxy" Nov 03 19:55:40.633966 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.633945 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.643964 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.641935 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-389e6d-openshift-default-dockercfg-hxvqw\"" Nov 03 19:55:40.662559 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.662536 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln"] Nov 03 19:55:40.678596 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.678566 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-istio-data\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.678744 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.678628 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-workload-socket\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.678829 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.678805 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-workload-certs\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.678884 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.678864 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d6043a-3156-4ca3-992b-fe613a604883-istio-podinfo\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.679058 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.679032 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-credential-socket\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.679199 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.679177 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmmz7\" (UniqueName: \"kubernetes.io/projected/67d6043a-3156-4ca3-992b-fe613a604883-kube-api-access-gmmz7\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.679277 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.679239 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d6043a-3156-4ca3-992b-fe613a604883-istio-token\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.679380 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.679360 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d6043a-3156-4ca3-992b-fe613a604883-istiod-ca-cert\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.679441 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.679397 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-istio-envoy\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.780801 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.780764 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-credential-socket\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.780989 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.780853 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmmz7\" (UniqueName: \"kubernetes.io/projected/67d6043a-3156-4ca3-992b-fe613a604883-kube-api-access-gmmz7\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.780989 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.780903 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d6043a-3156-4ca3-992b-fe613a604883-istio-token\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.780989 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.780940 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d6043a-3156-4ca3-992b-fe613a604883-istiod-ca-cert\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.780989 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.780964 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-istio-envoy\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.781218 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.781065 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-istio-data\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.781218 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.781114 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-workload-socket\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.781218 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.781154 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-workload-certs\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.781218 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.781205 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d6043a-3156-4ca3-992b-fe613a604883-istio-podinfo\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.781394 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.781235 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-credential-socket\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.781535 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.781511 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-workload-certs\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.781604 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.781533 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-workload-socket\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.781604 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.781573 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-istio-data\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.781824 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.781800 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d6043a-3156-4ca3-992b-fe613a604883-istiod-ca-cert\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.783550 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.783528 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-istio-envoy\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.783780 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.783763 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d6043a-3156-4ca3-992b-fe613a604883-istio-podinfo\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.832431 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.832397 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d6043a-3156-4ca3-992b-fe613a604883-istio-token\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.867993 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.867948 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmmz7\" (UniqueName: \"kubernetes.io/projected/67d6043a-3156-4ca3-992b-fe613a604883-kube-api-access-gmmz7\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:40.985909 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.980600 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h"] Nov 03 19:55:40.985909 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.981881 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:41.000020 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:40.999991 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h"] Nov 03 19:55:41.000280 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.000264 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.028521 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.013326 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6cd7f4-openshift-default-dockercfg-ff57j\"" Nov 03 19:55:41.088443 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.088362 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-podinfo\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.088443 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.088415 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-credential-socket\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.088443 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.088445 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwqqw\" (UniqueName: \"kubernetes.io/projected/5dd98cc4-6a97-4b79-90c6-7118b01966c4-kube-api-access-xwqqw\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.088647 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.088529 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-envoy\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.088647 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.088561 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-workload-certs\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.088647 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.088628 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istiod-ca-cert\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.089067 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.088763 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-data\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.089067 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.088833 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-token\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.089067 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.088862 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-workload-socket\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.201852 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.190442 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-token\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.201852 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.190495 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-workload-socket\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.201852 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.190552 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-podinfo\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.201852 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.190574 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-credential-socket\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.201852 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.190596 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwqqw\" (UniqueName: \"kubernetes.io/projected/5dd98cc4-6a97-4b79-90c6-7118b01966c4-kube-api-access-xwqqw\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.201852 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.199577 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-workload-socket\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.201852 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.199935 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-credential-socket\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.203987 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.202364 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-podinfo\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.203987 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.202840 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-envoy\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.203987 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.202892 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-workload-certs\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.203987 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.203011 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istiod-ca-cert\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.203987 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.203211 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-data\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.203987 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.203611 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-data\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.203987 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.203647 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-workload-certs\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.204977 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.204908 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istiod-ca-cert\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.216318 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.208155 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-envoy\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.216318 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.211758 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-token\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.216318 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.214370 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwqqw\" (UniqueName: \"kubernetes.io/projected/5dd98cc4-6a97-4b79-90c6-7118b01966c4-kube-api-access-xwqqw\") pod \"e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.380310 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.380271 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:41.413588 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:55:41.413553 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d6043a_3156_4ca3_992b_fe613a604883.slice/crio-dd83de92c0866e0e87bb639ccdaf4f956756576c85e827fb56c50871bb63d4f5 WatchSource:0}: Error finding container dd83de92c0866e0e87bb639ccdaf4f956756576c85e827fb56c50871bb63d4f5: Status 404 returned error can't find the container with id dd83de92c0866e0e87bb639ccdaf4f956756576c85e827fb56c50871bb63d4f5 Nov 03 19:55:41.415125 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.415097 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln"] Nov 03 19:55:41.416273 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.416238 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:55:41.416388 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.416314 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:55:41.416388 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.416363 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:55:41.687077 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.687051 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h"] Nov 03 19:55:41.687400 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:55:41.687074 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd98cc4_6a97_4b79_90c6_7118b01966c4.slice/crio-749ce945dc8c15080f39655443b04f11afa38a578f35358d39d58a9dd62892ca WatchSource:0}: Error finding container 749ce945dc8c15080f39655443b04f11afa38a578f35358d39d58a9dd62892ca: Status 404 returned error can't find the container with id 749ce945dc8c15080f39655443b04f11afa38a578f35358d39d58a9dd62892ca Nov 03 19:55:41.689719 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.689686 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:55:41.689824 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.689748 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:55:41.689824 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:41.689779 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:55:42.331976 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:42.331933 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" event={"ID":"67d6043a-3156-4ca3-992b-fe613a604883","Type":"ContainerStarted","Data":"1c0a4f71993101224f72fbd33bc2d0305b4c5c1b3891b51eb07284545f0536e7"} Nov 03 19:55:42.332144 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:42.332004 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" event={"ID":"67d6043a-3156-4ca3-992b-fe613a604883","Type":"ContainerStarted","Data":"dd83de92c0866e0e87bb639ccdaf4f956756576c85e827fb56c50871bb63d4f5"} Nov 03 19:55:42.333647 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:42.333625 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" event={"ID":"5dd98cc4-6a97-4b79-90c6-7118b01966c4","Type":"ContainerStarted","Data":"715614d1ed23234dba3aadf5d1940c312a3de8d2840a50fac06793443a431813"} Nov 03 19:55:42.333780 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:42.333683 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" event={"ID":"5dd98cc4-6a97-4b79-90c6-7118b01966c4","Type":"ContainerStarted","Data":"749ce945dc8c15080f39655443b04f11afa38a578f35358d39d58a9dd62892ca"} Nov 03 19:55:42.360572 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:42.360523 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" podStartSLOduration=2.360511498 podStartE2EDuration="2.360511498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:42.355016343 +0000 UTC m=+1136.612813636" watchObservedRunningTime="2026-11-03 19:55:42.360511498 +0000 UTC m=+1136.618308746" Nov 03 19:55:42.381328 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:42.381304 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:42.381988 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:42.381945 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" podStartSLOduration=2.3819342 podStartE2EDuration="2.3819342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:42.378005033 +0000 UTC m=+1136.635802302" watchObservedRunningTime="2026-11-03 19:55:42.3819342 +0000 UTC m=+1136.639731457" Nov 03 19:55:42.387388 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:42.387367 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:43.007696 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:43.002018 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:43.017106 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:43.017073 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:43.341844 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:43.341810 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:43.342034 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:43.341874 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:43.342566 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:43.342544 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:55:43.342687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:43.342594 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:55:46.334460 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:46.323886 5059 scope.go:122] "RemoveContainer" containerID="46c29a995f2d9acb49d9d32e9a972cc00c24a0c0d6178965b216e4a8ffe93a13" Nov 03 19:55:46.460839 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:46.460806 5059 scope.go:122] "RemoveContainer" containerID="9507c865fc128e60fba96e8045e83025c012bb62e9fa3731771847f3a9d32af8" Nov 03 19:55:46.483059 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:46.483029 5059 scope.go:122] "RemoveContainer" containerID="3979c4877483574442f4dd4ba5cd89ba3e1d3def2157691d8be3e10485f5b6d6" Nov 03 19:55:46.493986 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:46.493961 5059 scope.go:122] "RemoveContainer" containerID="4934f20758fe1e6818e6cfdfdd7652937b28b05db9f7f37a3fe0771ca77b07ce" Nov 03 19:55:46.509440 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:46.509411 5059 scope.go:122] "RemoveContainer" containerID="d0e7c9ae0e393242a028e6fa8725fcdce924e125da4eedba4666cfa18ee8bd5f" Nov 03 19:55:54.862250 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:54.858714 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7240-607ad6"] Nov 03 19:55:54.888744 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:54.883863 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-607ad6"] Nov 03 19:55:54.888744 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:54.884905 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-607ad6" Nov 03 19:55:54.893765 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:54.893739 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 19:55:54.902503 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:54.902479 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-tz7np\"" Nov 03 19:55:54.902768 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:54.902752 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 19:55:55.039299 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:55.039259 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dr52\" (UniqueName: \"kubernetes.io/projected/b37eed71-7119-4d9e-bd5d-5450e815690b-kube-api-access-9dr52\") pod \"test-curl-7240-607ad6\" (UID: \"b37eed71-7119-4d9e-bd5d-5450e815690b\") " pod="opendatahub/test-curl-7240-607ad6" Nov 03 19:55:55.140772 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:55.140690 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dr52\" (UniqueName: \"kubernetes.io/projected/b37eed71-7119-4d9e-bd5d-5450e815690b-kube-api-access-9dr52\") pod \"test-curl-7240-607ad6\" (UID: \"b37eed71-7119-4d9e-bd5d-5450e815690b\") " pod="opendatahub/test-curl-7240-607ad6" Nov 03 19:55:55.155069 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:55.155042 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dr52\" (UniqueName: \"kubernetes.io/projected/b37eed71-7119-4d9e-bd5d-5450e815690b-kube-api-access-9dr52\") pod \"test-curl-7240-607ad6\" (UID: \"b37eed71-7119-4d9e-bd5d-5450e815690b\") " pod="opendatahub/test-curl-7240-607ad6" Nov 03 19:55:55.231651 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:55.231624 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-607ad6" Nov 03 19:55:55.514519 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:55.514489 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-607ad6"] Nov 03 19:55:55.515013 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:55:55.514988 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb37eed71_7119_4d9e_bd5d_5450e815690b.slice/crio-0a7318bb68d05d0d8fff4b39c1266769236f2d711c7bb3e6b059d43cefb91721 WatchSource:0}: Error finding container 0a7318bb68d05d0d8fff4b39c1266769236f2d711c7bb3e6b059d43cefb91721: Status 404 returned error can't find the container with id 0a7318bb68d05d0d8fff4b39c1266769236f2d711c7bb3e6b059d43cefb91721 Nov 03 19:55:56.490160 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.490123 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-607ad6" event={"ID":"b37eed71-7119-4d9e-bd5d-5450e815690b","Type":"ContainerStarted","Data":"7580b7039159e79a298ae8c40a702729916239b6a1965a20a018df8d68e265b0"} Nov 03 19:55:56.490640 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.490169 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-607ad6" event={"ID":"b37eed71-7119-4d9e-bd5d-5450e815690b","Type":"ContainerStarted","Data":"0a7318bb68d05d0d8fff4b39c1266769236f2d711c7bb3e6b059d43cefb91721"} Nov 03 19:55:56.520399 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.520357 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk"] Nov 03 19:55:56.525889 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.525855 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.528248 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.528210 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7240-607ad6" podStartSLOduration=2.5281984140000002 podStartE2EDuration="2.528198414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:56.523085287 +0000 UTC m=+1150.780882546" watchObservedRunningTime="2026-11-03 19:55:56.528198414 +0000 UTC m=+1150.785995671" Nov 03 19:55:56.536205 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.536184 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b0f9d6-openshift-default-dockercfg-z67tw\"" Nov 03 19:55:56.543751 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.543728 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk"] Nov 03 19:55:56.559314 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.559281 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-workload-socket\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.559452 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.559433 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljqpx\" (UniqueName: \"kubernetes.io/projected/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-kube-api-access-ljqpx\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.559531 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.559494 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-workload-certs\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.559591 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.559561 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-token\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.559649 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.559588 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-envoy\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.559649 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.559626 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-data\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.559786 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.559655 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-credential-socket\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.559786 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.559703 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istiod-ca-cert\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.559786 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.559731 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-podinfo\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.664686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.662084 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-token\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.664686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.662141 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-envoy\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.664686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.662191 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-data\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.664686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.662247 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-credential-socket\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.717687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.716851 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istiod-ca-cert\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.717687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.716927 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-podinfo\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.717687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.717047 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-workload-socket\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.717687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.717306 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljqpx\" (UniqueName: \"kubernetes.io/projected/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-kube-api-access-ljqpx\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.717687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.717397 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-workload-certs\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.718100 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.717957 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-workload-certs\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.737683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.720289 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-workload-socket\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.737683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.720406 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istiod-ca-cert\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.737683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.721768 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-data\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.737683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.721987 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-credential-socket\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.779742 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.772446 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljqpx\" (UniqueName: \"kubernetes.io/projected/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-kube-api-access-ljqpx\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.817215 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.814941 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-token\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.831100 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.831064 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-podinfo\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:56.846709 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:56.846274 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-envoy\") pod \"e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:57.143264 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:57.143234 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:57.378864 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:57.378822 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7240-607ad6"] Nov 03 19:55:57.394305 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:57.390121 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7240-607ad6"] Nov 03 19:55:57.438693 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:57.438646 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk"] Nov 03 19:55:57.439703 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:55:57.439633 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0611f73_972b_4e4e_be2f_5c536bdd3d9b.slice/crio-df26621d2dd52ddffade9b4a6c263e74de46ac1f264bcf6859dd406b79d90096 WatchSource:0}: Error finding container df26621d2dd52ddffade9b4a6c263e74de46ac1f264bcf6859dd406b79d90096: Status 404 returned error can't find the container with id df26621d2dd52ddffade9b4a6c263e74de46ac1f264bcf6859dd406b79d90096 Nov 03 19:55:57.443242 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:57.443200 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:55:57.443342 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:57.443290 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:55:57.443342 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:57.443336 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:55:57.500253 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:57.500224 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" event={"ID":"e0611f73-972b-4e4e-be2f-5c536bdd3d9b","Type":"ContainerStarted","Data":"df26621d2dd52ddffade9b4a6c263e74de46ac1f264bcf6859dd406b79d90096"} Nov 03 19:55:58.512910 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:58.512869 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" event={"ID":"e0611f73-972b-4e4e-be2f-5c536bdd3d9b","Type":"ContainerStarted","Data":"4c16a3a918556e1de30cf2cf342198f03949b5f77828d2855186ca37de52e432"} Nov 03 19:55:58.513379 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:58.512958 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7240-607ad6" podUID="b37eed71-7119-4d9e-bd5d-5450e815690b" containerName="test-curl-7240-607ad6" containerID="cri-o://7580b7039159e79a298ae8c40a702729916239b6a1965a20a018df8d68e265b0" gracePeriod=2 Nov 03 19:55:58.555367 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:58.555306 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" podStartSLOduration=2.555288734 podStartE2EDuration="2.555288734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:58.550780944 +0000 UTC m=+1152.808578204" watchObservedRunningTime="2026-11-03 19:55:58.555288734 +0000 UTC m=+1152.813085991" Nov 03 19:55:59.143869 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:59.143831 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:59.149693 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:59.149628 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:59.520854 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:59.520778 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:55:59.521711 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:55:59.521689 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:56:00.748746 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:00.748705 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-607ad6" Nov 03 19:56:00.909104 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:00.909078 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37eed71-7119-4d9e-bd5d-5450e815690b-kube-api-access-9dr52\" (UniqueName: \"kubernetes.io/projected/b37eed71-7119-4d9e-bd5d-5450e815690b-kube-api-access-9dr52\") pod \"b37eed71-7119-4d9e-bd5d-5450e815690b\" (UID: \"b37eed71-7119-4d9e-bd5d-5450e815690b\") " Nov 03 19:56:00.911134 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:00.911103 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37eed71-7119-4d9e-bd5d-5450e815690b-kube-api-access-9dr52" pod "b37eed71-7119-4d9e-bd5d-5450e815690b" (UID: "b37eed71-7119-4d9e-bd5d-5450e815690b"). InnerVolumeSpecName "kube-api-access-9dr52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:56:01.011122 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:01.010816 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dr52\" (UniqueName: \"kubernetes.io/projected/b37eed71-7119-4d9e-bd5d-5450e815690b-kube-api-access-9dr52\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:01.540345 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:01.540308 5059 generic.go:356] "Generic (PLEG): container finished" podID="b37eed71-7119-4d9e-bd5d-5450e815690b" containerID="7580b7039159e79a298ae8c40a702729916239b6a1965a20a018df8d68e265b0" exitCode=137 Nov 03 19:56:01.540519 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:01.540348 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-607ad6" Nov 03 19:56:01.540519 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:01.540400 5059 scope.go:122] "RemoveContainer" containerID="7580b7039159e79a298ae8c40a702729916239b6a1965a20a018df8d68e265b0" Nov 03 19:56:01.552002 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:56:01.551974 5059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-607ad6\" is forbidden: User \"system:node:ip-10-0-132-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-83.us-east-2.compute.internal' and this object" podUID="b37eed71-7119-4d9e-bd5d-5450e815690b" pod="opendatahub/test-curl-7240-607ad6" Nov 03 19:56:01.552457 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:01.552441 5059 scope.go:122] "RemoveContainer" containerID="7580b7039159e79a298ae8c40a702729916239b6a1965a20a018df8d68e265b0" Nov 03 19:56:01.552738 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:56:01.552718 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7580b7039159e79a298ae8c40a702729916239b6a1965a20a018df8d68e265b0\": container with ID starting with 7580b7039159e79a298ae8c40a702729916239b6a1965a20a018df8d68e265b0 not found: ID does not exist" containerID="7580b7039159e79a298ae8c40a702729916239b6a1965a20a018df8d68e265b0" Nov 03 19:56:01.552798 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:01.552746 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7580b7039159e79a298ae8c40a702729916239b6a1965a20a018df8d68e265b0"} err="failed to get container status \"7580b7039159e79a298ae8c40a702729916239b6a1965a20a018df8d68e265b0\": rpc error: code = NotFound desc = could not find container \"7580b7039159e79a298ae8c40a702729916239b6a1965a20a018df8d68e265b0\": container with ID starting with 7580b7039159e79a298ae8c40a702729916239b6a1965a20a018df8d68e265b0 not found: ID does not exist" Nov 03 19:56:01.858910 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:01.858875 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b37eed71-7119-4d9e-bd5d-5450e815690b" path="/var/lib/kubelet/pods/b37eed71-7119-4d9e-bd5d-5450e815690b/volumes" Nov 03 19:56:08.733299 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:08.733253 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48"] Nov 03 19:56:08.737615 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:08.734354 5059 cpu_manager.go:414] "containerMap: removing container" podUID="b37eed71-7119-4d9e-bd5d-5450e815690b" containerName="test-curl-7240-607ad6" Nov 03 19:56:08.737615 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:08.734374 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b37eed71-7119-4d9e-bd5d-5450e815690b" containerName="test-curl-7240-607ad6" Nov 03 19:56:08.737615 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:08.734602 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b37eed71-7119-4d9e-bd5d-5450e815690b" containerName="test-curl-7240-607ad6" Nov 03 19:56:08.742057 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:08.742028 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:08.821965 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:08.821917 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b34f6c-openshift-default-dockercfg-xr69s\"" Nov 03 19:56:08.947717 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:08.936475 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-token\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:08.947717 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:08.936680 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-envoy\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:08.947717 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:08.936733 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtt9b\" (UniqueName: \"kubernetes.io/projected/0953b878-511c-4fff-8639-fc5981f3dc3e-kube-api-access-mtt9b\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:08.947717 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:08.936805 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-data\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:08.947717 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:08.937089 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0953b878-511c-4fff-8639-fc5981f3dc3e-istiod-ca-cert\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:08.947717 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:08.937127 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-credential-socket\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:08.947717 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:08.937220 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-workload-certs\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:08.947717 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:08.937455 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-podinfo\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:08.947717 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:08.937524 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-workload-socket\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:09.218685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:09.217920 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-workload-certs\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:09.218685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:09.218007 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-podinfo\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:09.218685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:09.218165 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-workload-socket\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:09.218685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:09.218290 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-token\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:09.218685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:09.218327 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-envoy\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:09.218685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:09.218468 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtt9b\" (UniqueName: \"kubernetes.io/projected/0953b878-511c-4fff-8639-fc5981f3dc3e-kube-api-access-mtt9b\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:09.218685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:09.218559 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-data\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:09.219194 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:09.218786 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0953b878-511c-4fff-8639-fc5981f3dc3e-istiod-ca-cert\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:09.219194 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:09.218842 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-credential-socket\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:09.222299 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:09.222266 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-workload-certs\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:09.223750 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:09.223720 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48"] Nov 03 19:56:09.224876 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:09.224833 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-envoy\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:09.229235 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:09.229205 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-data\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:09.229235 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:09.229231 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-credential-socket\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:09.229804 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:09.229774 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0953b878-511c-4fff-8639-fc5981f3dc3e-istiod-ca-cert\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:09.229902 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:09.229808 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-workload-socket\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:09.231509 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:09.231484 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-podinfo\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:09.237016 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:09.236993 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-token\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:09.238397 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:09.238353 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtt9b\" (UniqueName: \"kubernetes.io/projected/0953b878-511c-4fff-8639-fc5981f3dc3e-kube-api-access-mtt9b\") pod \"e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:09.285712 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:09.278367 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:09.919090 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:09.919054 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48"] Nov 03 19:56:09.922720 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:09.922531 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:56:09.922720 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:09.922633 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:56:09.922872 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:09.922773 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:56:10.777689 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:10.771381 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" event={"ID":"0953b878-511c-4fff-8639-fc5981f3dc3e","Type":"ContainerStarted","Data":"2c61ae1c1de969bad4d008d73483b6a6d68ffde0a711b5bd024bec245bccfa83"} Nov 03 19:56:10.777689 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:10.771468 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" event={"ID":"0953b878-511c-4fff-8639-fc5981f3dc3e","Type":"ContainerStarted","Data":"a25a486651aa478c427899bd31eeda61adfa34318b309043531a5df8b7d4557a"} Nov 03 19:56:10.835683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:10.819986 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" podStartSLOduration=2.819964963 podStartE2EDuration="2.819964963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:10.817222355 +0000 UTC m=+1165.075019624" watchObservedRunningTime="2026-11-03 19:56:10.819964963 +0000 UTC m=+1165.077762220" Nov 03 19:56:10.937910 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:56:10.937864 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:56:11.295685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:11.294716 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:11.295685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:11.294952 5059 patch_prober.go:29] interesting pod/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 19:56:11.295685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:11.295005 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" podUID="0953b878-511c-4fff-8639-fc5981f3dc3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 19:56:12.308948 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:12.308912 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:12.843617 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:12.843561 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:12.844707 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:12.844640 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:13.338803 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:13.338771 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8"] Nov 03 19:56:13.343945 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:13.343921 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8" Nov 03 19:56:13.351982 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:13.351876 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-389e6d\"" Nov 03 19:56:13.351982 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:13.351947 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-389e6d-dockercfg-f29rb\"" Nov 03 19:56:13.356903 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:13.356876 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8"] Nov 03 19:56:13.430141 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:13.430105 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt"] Nov 03 19:56:13.435932 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:13.435904 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt"] Nov 03 19:56:13.436050 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:13.436023 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt" Nov 03 19:56:13.439528 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:13.439498 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mwcn\" (UniqueName: \"kubernetes.io/projected/1f33bc6f-5525-469a-a399-b8d63af713e5-kube-api-access-7mwcn\") pod \"payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8\" (UID: \"1f33bc6f-5525-469a-a399-b8d63af713e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8" Nov 03 19:56:13.440201 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:13.440097 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f33bc6f-5525-469a-a399-b8d63af713e5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8\" (UID: \"1f33bc6f-5525-469a-a399-b8d63af713e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8" Nov 03 19:56:13.557747 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:13.541316 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb0391f6-c81d-40ca-9542-3d8976d0dbb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt\" (UID: \"eb0391f6-c81d-40ca-9542-3d8976d0dbb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt" Nov 03 19:56:13.557747 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:13.541554 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mwcn\" (UniqueName: \"kubernetes.io/projected/1f33bc6f-5525-469a-a399-b8d63af713e5-kube-api-access-7mwcn\") pod \"payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8\" (UID: \"1f33bc6f-5525-469a-a399-b8d63af713e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8" Nov 03 19:56:13.557747 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:13.541755 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdgr4\" (UniqueName: \"kubernetes.io/projected/eb0391f6-c81d-40ca-9542-3d8976d0dbb3-kube-api-access-hdgr4\") pod \"payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt\" (UID: \"eb0391f6-c81d-40ca-9542-3d8976d0dbb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt" Nov 03 19:56:13.557747 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:13.541797 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f33bc6f-5525-469a-a399-b8d63af713e5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8\" (UID: \"1f33bc6f-5525-469a-a399-b8d63af713e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8" Nov 03 19:56:13.585805 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:13.585754 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f33bc6f-5525-469a-a399-b8d63af713e5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8\" (UID: \"1f33bc6f-5525-469a-a399-b8d63af713e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8" Nov 03 19:56:13.654746 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:13.643272 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdgr4\" (UniqueName: \"kubernetes.io/projected/eb0391f6-c81d-40ca-9542-3d8976d0dbb3-kube-api-access-hdgr4\") pod \"payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt\" (UID: \"eb0391f6-c81d-40ca-9542-3d8976d0dbb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt" Nov 03 19:56:13.654746 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:13.643355 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb0391f6-c81d-40ca-9542-3d8976d0dbb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt\" (UID: \"eb0391f6-c81d-40ca-9542-3d8976d0dbb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt" Nov 03 19:56:13.654746 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:13.644604 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb0391f6-c81d-40ca-9542-3d8976d0dbb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt\" (UID: \"eb0391f6-c81d-40ca-9542-3d8976d0dbb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt" Nov 03 19:56:13.837706 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:13.835117 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mwcn\" (UniqueName: \"kubernetes.io/projected/1f33bc6f-5525-469a-a399-b8d63af713e5-kube-api-access-7mwcn\") pod \"payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8\" (UID: \"1f33bc6f-5525-469a-a399-b8d63af713e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8" Nov 03 19:56:13.837706 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:13.836459 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdgr4\" (UniqueName: \"kubernetes.io/projected/eb0391f6-c81d-40ca-9542-3d8976d0dbb3-kube-api-access-hdgr4\") pod \"payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt\" (UID: \"eb0391f6-c81d-40ca-9542-3d8976d0dbb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt" Nov 03 19:56:13.841560 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:13.840523 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt" Nov 03 19:56:14.099721 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:14.079151 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8" Nov 03 19:56:14.820639 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:56:14.820611 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f33bc6f_5525_469a_a399_b8d63af713e5.slice/crio-f83146740c56fb27a67cc7636301086c6858ffe3c98aa49cc44bf43167b67918 WatchSource:0}: Error finding container f83146740c56fb27a67cc7636301086c6858ffe3c98aa49cc44bf43167b67918: Status 404 returned error can't find the container with id f83146740c56fb27a67cc7636301086c6858ffe3c98aa49cc44bf43167b67918 Nov 03 19:56:14.834727 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:14.761650 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt"] Nov 03 19:56:14.840501 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:14.840420 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8"] Nov 03 19:56:14.927726 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:14.927684 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb"] Nov 03 19:56:14.945176 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:14.945137 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8" event={"ID":"1f33bc6f-5525-469a-a399-b8d63af713e5","Type":"ContainerStarted","Data":"f83146740c56fb27a67cc7636301086c6858ffe3c98aa49cc44bf43167b67918"} Nov 03 19:56:14.945358 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:14.945192 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb"] Nov 03 19:56:14.945358 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:14.945223 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt" event={"ID":"eb0391f6-c81d-40ca-9542-3d8976d0dbb3","Type":"ContainerStarted","Data":"cb28695f8016f727bb062544b854b31b4a8b61b19e1051545dcd32c1d6235b63"} Nov 03 19:56:14.945358 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:14.945304 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb" Nov 03 19:56:14.986691 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:14.980124 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6cd7f4\"" Nov 03 19:56:14.986691 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:14.980313 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6cd7f4-dockercfg-fckgr\"" Nov 03 19:56:14.986691 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:14.980359 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc"] Nov 03 19:56:15.083464 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:15.056930 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8mlm\" (UniqueName: \"kubernetes.io/projected/c1f480c4-4dd7-4622-b480-a5059adc4abb-kube-api-access-k8mlm\") pod \"payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb\" (UID: \"c1f480c4-4dd7-4622-b480-a5059adc4abb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb" Nov 03 19:56:15.083464 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:15.057259 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f480c4-4dd7-4622-b480-a5059adc4abb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb\" (UID: \"c1f480c4-4dd7-4622-b480-a5059adc4abb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb" Nov 03 19:56:15.185218 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:15.185113 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f480c4-4dd7-4622-b480-a5059adc4abb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb\" (UID: \"c1f480c4-4dd7-4622-b480-a5059adc4abb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb" Nov 03 19:56:15.185566 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:15.185532 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8mlm\" (UniqueName: \"kubernetes.io/projected/c1f480c4-4dd7-4622-b480-a5059adc4abb-kube-api-access-k8mlm\") pod \"payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb\" (UID: \"c1f480c4-4dd7-4622-b480-a5059adc4abb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb" Nov 03 19:56:15.243724 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:15.243636 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8mlm\" (UniqueName: \"kubernetes.io/projected/c1f480c4-4dd7-4622-b480-a5059adc4abb-kube-api-access-k8mlm\") pod \"payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb\" (UID: \"c1f480c4-4dd7-4622-b480-a5059adc4abb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb" Nov 03 19:56:15.245058 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:15.244943 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f480c4-4dd7-4622-b480-a5059adc4abb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb\" (UID: \"c1f480c4-4dd7-4622-b480-a5059adc4abb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb" Nov 03 19:56:15.291794 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:15.282216 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc" Nov 03 19:56:15.293410 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:15.293375 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc"] Nov 03 19:56:15.369458 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:15.369422 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb" Nov 03 19:56:15.390410 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:15.390370 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e5ddf9-f49d-4c38-bd61-955f3106fd9c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc\" (UID: \"e1e5ddf9-f49d-4c38-bd61-955f3106fd9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc" Nov 03 19:56:15.390647 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:15.390616 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qp65\" (UniqueName: \"kubernetes.io/projected/e1e5ddf9-f49d-4c38-bd61-955f3106fd9c-kube-api-access-4qp65\") pod \"payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc\" (UID: \"e1e5ddf9-f49d-4c38-bd61-955f3106fd9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc" Nov 03 19:56:15.494427 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:15.492901 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qp65\" (UniqueName: \"kubernetes.io/projected/e1e5ddf9-f49d-4c38-bd61-955f3106fd9c-kube-api-access-4qp65\") pod \"payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc\" (UID: \"e1e5ddf9-f49d-4c38-bd61-955f3106fd9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc" Nov 03 19:56:15.494427 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:15.493157 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e5ddf9-f49d-4c38-bd61-955f3106fd9c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc\" (UID: \"e1e5ddf9-f49d-4c38-bd61-955f3106fd9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc" Nov 03 19:56:15.494427 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:15.494101 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e5ddf9-f49d-4c38-bd61-955f3106fd9c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc\" (UID: \"e1e5ddf9-f49d-4c38-bd61-955f3106fd9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc" Nov 03 19:56:15.514598 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:15.513464 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qp65\" (UniqueName: \"kubernetes.io/projected/e1e5ddf9-f49d-4c38-bd61-955f3106fd9c-kube-api-access-4qp65\") pod \"payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc\" (UID: \"e1e5ddf9-f49d-4c38-bd61-955f3106fd9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc" Nov 03 19:56:15.633355 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:15.626708 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc" Nov 03 19:56:15.672437 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:15.670852 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb"] Nov 03 19:56:15.684203 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:56:15.684171 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f480c4_4dd7_4622_b480_a5059adc4abb.slice/crio-7312c4affde3be8ac2b56324e9a61ef88f95f37f3490da366b1a9907781eebba WatchSource:0}: Error finding container 7312c4affde3be8ac2b56324e9a61ef88f95f37f3490da366b1a9907781eebba: Status 404 returned error can't find the container with id 7312c4affde3be8ac2b56324e9a61ef88f95f37f3490da366b1a9907781eebba Nov 03 19:56:15.989520 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:15.989386 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc"] Nov 03 19:56:15.991890 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:15.991804 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb" event={"ID":"c1f480c4-4dd7-4622-b480-a5059adc4abb","Type":"ContainerStarted","Data":"7312c4affde3be8ac2b56324e9a61ef88f95f37f3490da366b1a9907781eebba"} Nov 03 19:56:17.016571 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:17.016528 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc" event={"ID":"e1e5ddf9-f49d-4c38-bd61-955f3106fd9c","Type":"ContainerStarted","Data":"72f1e131db82a9d04fe1a935327831d092eea2ad230aa7f48750717e378a9d49"} Nov 03 19:56:19.030115 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:19.030075 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d9b2be9-fx69j"] Nov 03 19:56:19.038487 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:19.038455 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d9b2be9-fx69j"] Nov 03 19:56:19.860403 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:19.860370 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0dd6c49-3d40-40e7-b793-8a6880d8d04d" path="/var/lib/kubelet/pods/c0dd6c49-3d40-40e7-b793-8a6880d8d04d/volumes" Nov 03 19:56:21.089247 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:21.089198 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt" event={"ID":"eb0391f6-c81d-40ca-9542-3d8976d0dbb3","Type":"ContainerStarted","Data":"19e4b68c2ec255f8b7aff70ee2855886ce031facb0be51bbad492b5b4b3080f2"} Nov 03 19:56:21.089754 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:21.089301 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt" Nov 03 19:56:21.092138 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:21.092074 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc" event={"ID":"e1e5ddf9-f49d-4c38-bd61-955f3106fd9c","Type":"ContainerStarted","Data":"6ce214f2b3f09dd2bdfc2c7aba5baf90166ae4a6a209973ee958956fc00c02d7"} Nov 03 19:56:21.092617 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:21.092587 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc" Nov 03 19:56:21.094241 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:21.094213 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8" event={"ID":"1f33bc6f-5525-469a-a399-b8d63af713e5","Type":"ContainerStarted","Data":"61f32bf28db3a3751a39d2b47c1cc533d07a803d70f1b0f678e78acebefbc84f"} Nov 03 19:56:21.095239 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:21.094902 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8" Nov 03 19:56:21.098266 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:21.098240 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb" event={"ID":"c1f480c4-4dd7-4622-b480-a5059adc4abb","Type":"ContainerStarted","Data":"99b8ec4f73a050506aa44d6d768433fdbccab298d6fa3e50832e4e1c88c7fc3d"} Nov 03 19:56:21.098984 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:21.098916 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb" Nov 03 19:56:21.118782 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:21.118738 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt" podStartSLOduration=1.9173832960000001 podStartE2EDuration="8.118720985s" totalImagesPullingTime="6.201337689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:21.114447328 +0000 UTC m=+1175.372244589" watchObservedRunningTime="2026-11-03 19:56:21.118720985 +0000 UTC m=+1175.376518243" Nov 03 19:56:21.177174 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:21.177116 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb" podStartSLOduration=2.702402855 podStartE2EDuration="7.177099963s" totalImagesPullingTime="4.474697108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:21.137528791 +0000 UTC m=+1175.395326048" watchObservedRunningTime="2026-11-03 19:56:21.177099963 +0000 UTC m=+1175.434897222" Nov 03 19:56:21.207819 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:21.207767 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8" podStartSLOduration=2.719055159 podStartE2EDuration="8.207750814s" totalImagesPullingTime="5.488695655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:21.206961119 +0000 UTC m=+1175.464758375" watchObservedRunningTime="2026-11-03 19:56:21.207750814 +0000 UTC m=+1175.465548075" Nov 03 19:56:21.209411 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:21.209369 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc" podStartSLOduration=3.059006246 podStartE2EDuration="7.209356918s" totalImagesPullingTime="4.150350672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:21.179356559 +0000 UTC m=+1175.437153820" watchObservedRunningTime="2026-11-03 19:56:21.209356918 +0000 UTC m=+1175.467154176" Nov 03 19:56:23.222862 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:23.222785 5059 generic.go:356] "Generic (PLEG): container finished" podID="5dd98cc4-6a97-4b79-90c6-7118b01966c4" containerID="715614d1ed23234dba3aadf5d1940c312a3de8d2840a50fac06793443a431813" exitCode=0 Nov 03 19:56:23.223385 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:23.222987 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" event={"ID":"5dd98cc4-6a97-4b79-90c6-7118b01966c4","Type":"ContainerDied","Data":"715614d1ed23234dba3aadf5d1940c312a3de8d2840a50fac06793443a431813"} Nov 03 19:56:23.224033 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:23.223814 5059 scope.go:122] "RemoveContainer" containerID="715614d1ed23234dba3aadf5d1940c312a3de8d2840a50fac06793443a431813" Nov 03 19:56:23.225766 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:23.225743 5059 generic.go:356] "Generic (PLEG): container finished" podID="67d6043a-3156-4ca3-992b-fe613a604883" containerID="1c0a4f71993101224f72fbd33bc2d0305b4c5c1b3891b51eb07284545f0536e7" exitCode=0 Nov 03 19:56:23.225858 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:23.225827 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:56:23.225922 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:23.225907 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:56:23.225980 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:23.225951 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:56:23.226039 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:23.225988 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" event={"ID":"67d6043a-3156-4ca3-992b-fe613a604883","Type":"ContainerDied","Data":"1c0a4f71993101224f72fbd33bc2d0305b4c5c1b3891b51eb07284545f0536e7"} Nov 03 19:56:23.226554 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:23.226478 5059 scope.go:122] "RemoveContainer" containerID="1c0a4f71993101224f72fbd33bc2d0305b4c5c1b3891b51eb07284545f0536e7" Nov 03 19:56:23.228260 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:23.228227 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:56:23.228343 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:23.228308 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:56:23.228382 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:23.228351 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:56:24.236551 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:24.236516 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" event={"ID":"67d6043a-3156-4ca3-992b-fe613a604883","Type":"ContainerStarted","Data":"7636f8a662a01ab07cddbeab56ed1cec283a0e38f9f59180d20792fd3241868f"} Nov 03 19:56:24.238869 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:24.238842 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" event={"ID":"5dd98cc4-6a97-4b79-90c6-7118b01966c4","Type":"ContainerStarted","Data":"673937df819966ed0cd13bd7341daeeeaeb5458f8d2020750cb7dac184b4e320"} Nov 03 19:56:24.381416 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:24.381385 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:56:24.382440 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:24.382405 5059 patch_prober.go:29] interesting pod/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 19:56:24.382593 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:24.382452 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" podUID="5dd98cc4-6a97-4b79-90c6-7118b01966c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 19:56:24.984686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:24.984138 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:56:24.984686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:24.984332 5059 patch_prober.go:29] interesting pod/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 03 19:56:24.984686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:24.984388 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" podUID="67d6043a-3156-4ca3-992b-fe613a604883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 03 19:56:25.428684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:25.427879 5059 patch_prober.go:29] interesting pod/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 19:56:25.428684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:25.427943 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" podUID="5dd98cc4-6a97-4b79-90c6-7118b01966c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 19:56:25.988080 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:25.988041 5059 patch_prober.go:29] interesting pod/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 03 19:56:25.988364 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:25.988336 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" podUID="67d6043a-3156-4ca3-992b-fe613a604883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 03 19:56:26.388686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:26.388438 5059 patch_prober.go:29] interesting pod/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 19:56:26.388686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:26.388496 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" podUID="5dd98cc4-6a97-4b79-90c6-7118b01966c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 19:56:27.007933 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:27.007886 5059 patch_prober.go:29] interesting pod/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 03 19:56:27.009114 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:27.008518 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" podUID="67d6043a-3156-4ca3-992b-fe613a604883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 03 19:56:27.385169 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:27.385129 5059 patch_prober.go:29] interesting pod/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 19:56:27.408737 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:27.385453 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" podUID="5dd98cc4-6a97-4b79-90c6-7118b01966c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 19:56:27.985685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:27.984089 5059 patch_prober.go:29] interesting pod/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 03 19:56:27.985685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:27.984165 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" podUID="67d6043a-3156-4ca3-992b-fe613a604883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 03 19:56:28.342707 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:28.342385 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:56:28.342707 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:28.342458 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:56:28.382101 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:28.382063 5059 patch_prober.go:29] interesting pod/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 19:56:28.382398 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:28.382367 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" podUID="5dd98cc4-6a97-4b79-90c6-7118b01966c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 19:56:28.986484 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:28.986437 5059 patch_prober.go:29] interesting pod/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 03 19:56:28.994688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:28.987252 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" podUID="67d6043a-3156-4ca3-992b-fe613a604883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 03 19:56:29.388532 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:29.386080 5059 patch_prober.go:29] interesting pod/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 19:56:29.388532 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:29.386147 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" podUID="5dd98cc4-6a97-4b79-90c6-7118b01966c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 19:56:29.983584 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:29.983125 5059 patch_prober.go:29] interesting pod/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 03 19:56:29.983584 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:29.983196 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" podUID="67d6043a-3156-4ca3-992b-fe613a604883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 03 19:56:30.386695 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:30.386075 5059 patch_prober.go:29] interesting pod/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 19:56:30.386695 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:30.386142 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" podUID="5dd98cc4-6a97-4b79-90c6-7118b01966c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 19:56:30.999949 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:30.999914 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:56:31.003369 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:31.003346 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 19:56:31.028044 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:31.028013 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ee5e119-rzrcp"] Nov 03 19:56:31.036635 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:31.036609 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ee5e119-rzrcp"] Nov 03 19:56:31.387784 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:31.387745 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:56:31.389084 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:31.389061 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:56:31.869085 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:31.869045 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5112d575-7ebb-4eb4-ac0e-21db52f8b9cb" path="/var/lib/kubelet/pods/5112d575-7ebb-4eb4-ac0e-21db52f8b9cb/volumes" Nov 03 19:56:32.112770 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:32.112725 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt" Nov 03 19:56:32.113169 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:32.112819 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb" Nov 03 19:56:33.230703 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:33.229427 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc" Nov 03 19:56:33.230703 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:33.229941 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8" Nov 03 19:56:34.026614 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:34.026577 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0b708d2-hzzcx"] Nov 03 19:56:34.031361 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:34.031334 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0b708d2-hzzcx"] Nov 03 19:56:35.865543 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:35.865511 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2" path="/var/lib/kubelet/pods/cdedd4b2-78d7-4a99-be4e-3ebfbbc224b2/volumes" Nov 03 19:56:38.039688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.036363 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9"] Nov 03 19:56:38.053159 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.053118 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.064858 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.064809 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-9b3619-openshift-default-dockercfg-8qsg4\"" Nov 03 19:56:38.084706 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.080644 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9"] Nov 03 19:56:38.112610 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.112571 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gp6r\" (UniqueName: \"kubernetes.io/projected/c149bde6-19ca-4ce5-9685-c1cb12fe1740-kube-api-access-2gp6r\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.112818 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.112681 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-envoy\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.112818 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.112723 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istiod-ca-cert\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.112818 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.112755 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-workload-socket\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.112818 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.112783 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-credential-socket\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.113047 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.112884 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-podinfo\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.113047 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.112938 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-data\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.113151 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.113045 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-workload-certs\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.113151 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.113069 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-token\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.214255 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.214217 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-credential-socket\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.214460 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.214338 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-podinfo\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.214460 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.214393 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-data\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.214588 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.214496 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-workload-certs\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.214588 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.214524 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-token\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.214768 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.214643 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gp6r\" (UniqueName: \"kubernetes.io/projected/c149bde6-19ca-4ce5-9685-c1cb12fe1740-kube-api-access-2gp6r\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.214768 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.214677 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-credential-socket\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.214880 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.214762 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-envoy\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.214880 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.214821 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istiod-ca-cert\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.214880 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.214859 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-workload-socket\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.215304 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.215272 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-workload-socket\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.215633 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.215611 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-workload-certs\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.215633 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.215625 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-data\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.215840 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.215827 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istiod-ca-cert\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.218136 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.218111 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-podinfo\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.222931 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.222905 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-envoy\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.233398 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.233367 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-token\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.233501 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.233450 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gp6r\" (UniqueName: \"kubernetes.io/projected/c149bde6-19ca-4ce5-9685-c1cb12fe1740-kube-api-access-2gp6r\") pod \"e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.388809 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.388765 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:38.695639 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.695193 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9"] Nov 03 19:56:38.695915 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.695881 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:56:38.695995 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.695960 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:56:38.696056 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:38.696031 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:56:39.691176 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:39.691124 5059 generic.go:356] "Generic (PLEG): container finished" podID="e0611f73-972b-4e4e-be2f-5c536bdd3d9b" containerID="4c16a3a918556e1de30cf2cf342198f03949b5f77828d2855186ca37de52e432" exitCode=0 Nov 03 19:56:39.691802 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:39.691228 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" event={"ID":"e0611f73-972b-4e4e-be2f-5c536bdd3d9b","Type":"ContainerDied","Data":"4c16a3a918556e1de30cf2cf342198f03949b5f77828d2855186ca37de52e432"} Nov 03 19:56:39.692640 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:39.692617 5059 scope.go:122] "RemoveContainer" containerID="4c16a3a918556e1de30cf2cf342198f03949b5f77828d2855186ca37de52e432" Nov 03 19:56:39.694909 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:39.694862 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:56:39.695042 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:39.694980 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:56:39.695042 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:39.695032 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:56:39.695851 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:39.695713 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" event={"ID":"c149bde6-19ca-4ce5-9685-c1cb12fe1740","Type":"ContainerStarted","Data":"2f8abc715842645f0a29b43c181cb7842a915e01b85268981ed036ea730585a7"} Nov 03 19:56:39.695851 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:39.695765 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" event={"ID":"c149bde6-19ca-4ce5-9685-c1cb12fe1740","Type":"ContainerStarted","Data":"f5e10f9d5a1709642f9c13d32c9862f5b43c5672d7b3393bacdb58ea416f13e0"} Nov 03 19:56:39.844265 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:39.844201 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" podStartSLOduration=2.844180073 podStartE2EDuration="2.844180073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:39.741498813 +0000 UTC m=+1193.999296071" watchObservedRunningTime="2026-11-03 19:56:39.844180073 +0000 UTC m=+1194.101977331" Nov 03 19:56:40.389477 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:40.389448 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:40.396017 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:40.395993 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:40.679736 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:56:40.679029 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:56:40.750847 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:40.750798 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" event={"ID":"e0611f73-972b-4e4e-be2f-5c536bdd3d9b","Type":"ContainerStarted","Data":"6833b5a85b8a47638eb76a554e161d4167edd6fba7149e00b7506d1c0ae156dd"} Nov 03 19:56:40.751343 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:40.751154 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:40.752180 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:40.752156 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:56:41.143405 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:41.143377 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:56:41.144891 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:41.144865 5059 patch_prober.go:29] interesting pod/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 19:56:41.145031 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:41.144917 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" podUID="e0611f73-972b-4e4e-be2f-5c536bdd3d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 19:56:42.144484 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:42.144444 5059 patch_prober.go:29] interesting pod/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 19:56:42.145014 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:42.144511 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" podUID="e0611f73-972b-4e4e-be2f-5c536bdd3d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 19:56:43.144691 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:43.144640 5059 patch_prober.go:29] interesting pod/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 19:56:43.145089 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:43.144713 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" podUID="e0611f73-972b-4e4e-be2f-5c536bdd3d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 19:56:44.143970 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:44.143916 5059 patch_prober.go:29] interesting pod/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 19:56:44.144182 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:44.143982 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" podUID="e0611f73-972b-4e4e-be2f-5c536bdd3d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 19:56:44.531736 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:44.525058 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:56:45.143557 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:45.143522 5059 patch_prober.go:29] interesting pod/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 19:56:45.143779 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:45.143574 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" podUID="e0611f73-972b-4e4e-be2f-5c536bdd3d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 19:56:46.417599 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:46.417559 5059 patch_prober.go:29] interesting pod/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 19:56:46.418239 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:46.418204 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" podUID="e0611f73-972b-4e4e-be2f-5c536bdd3d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 19:56:46.444997 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:46.444958 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b79c04b-4tggl"] Nov 03 19:56:46.450041 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:46.450014 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b79c04b-4tggl"] Nov 03 19:56:46.539054 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:46.539021 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 19:56:46.543940 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:46.543919 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 19:56:46.614728 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:46.614692 5059 scope.go:122] "RemoveContainer" containerID="6987f41867483c4fdb7b5390e026fd3dd85ada0efb09df08d7fb089b25952eb2" Nov 03 19:56:46.629799 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:46.629759 5059 scope.go:122] "RemoveContainer" containerID="37bf1ae52160ffa8a5696735fe05152485fb49ba63fbc5baad03b3b596af52f2" Nov 03 19:56:47.027175 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:47.027096 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276879a2-wwfnr"] Nov 03 19:56:47.031756 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:47.031734 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276879a2-wwfnr"] Nov 03 19:56:47.144320 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:47.144288 5059 patch_prober.go:29] interesting pod/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 19:56:47.144489 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:47.144351 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" podUID="e0611f73-972b-4e4e-be2f-5c536bdd3d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 19:56:47.863697 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:47.860433 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50b018bb-eb68-478d-9752-f3d07ed2cacf" path="/var/lib/kubelet/pods/50b018bb-eb68-478d-9752-f3d07ed2cacf/volumes" Nov 03 19:56:47.884227 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:47.884189 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="944b8276-6b88-4dc7-8a0d-af8f8b879a39" path="/var/lib/kubelet/pods/944b8276-6b88-4dc7-8a0d-af8f8b879a39/volumes" Nov 03 19:56:48.148426 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:48.148346 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:56:48.149386 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:48.149365 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:56:54.731286 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:54.731241 5059 generic.go:356] "Generic (PLEG): container finished" podID="0953b878-511c-4fff-8639-fc5981f3dc3e" containerID="2c61ae1c1de969bad4d008d73483b6a6d68ffde0a711b5bd024bec245bccfa83" exitCode=0 Nov 03 19:56:54.731780 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:54.731291 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" event={"ID":"0953b878-511c-4fff-8639-fc5981f3dc3e","Type":"ContainerDied","Data":"2c61ae1c1de969bad4d008d73483b6a6d68ffde0a711b5bd024bec245bccfa83"} Nov 03 19:56:54.732023 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:54.732003 5059 scope.go:122] "RemoveContainer" containerID="2c61ae1c1de969bad4d008d73483b6a6d68ffde0a711b5bd024bec245bccfa83" Nov 03 19:56:54.734083 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:54.734054 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:56:54.734204 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:54.734147 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:56:54.734204 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:54.734191 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:56:55.743902 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:55.743813 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" event={"ID":"0953b878-511c-4fff-8639-fc5981f3dc3e","Type":"ContainerStarted","Data":"407d89b3754c60f101a3cd6263fdfb6cee19e8a1ece10a5419fa991ca8c748f6"} Nov 03 19:56:56.036764 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:56.033644 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27e1f945-kmtg6"] Nov 03 19:56:56.039514 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:56.039478 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27e1f945-kmtg6"] Nov 03 19:56:56.291114 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:56.291023 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:56.333684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:56.304632 5059 patch_prober.go:29] interesting pod/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 19:56:56.333684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:56.304747 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" podUID="0953b878-511c-4fff-8639-fc5981f3dc3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 19:56:57.286743 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:57.286702 5059 patch_prober.go:29] interesting pod/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 19:56:57.287174 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:57.286759 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" podUID="0953b878-511c-4fff-8639-fc5981f3dc3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 19:56:57.843521 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:57.843489 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:56:57.885270 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:57.885229 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da70ae49-8ec7-4a56-bb42-89310d7fffa3" path="/var/lib/kubelet/pods/da70ae49-8ec7-4a56-bb42-89310d7fffa3/volumes" Nov 03 19:56:58.290393 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:58.290302 5059 patch_prober.go:29] interesting pod/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 19:56:58.290393 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:58.290375 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" podUID="0953b878-511c-4fff-8639-fc5981f3dc3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 19:56:58.341389 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:58.341352 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2"] Nov 03 19:56:58.341900 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:58.341865 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" podUID="070f2e92-dc4a-479a-a8dc-724e4cc4a050" containerName="istio-proxy" containerID="cri-o://836cc9a7ad5f2fb9f7bbdc2d0bbf637ebed2fac8ba1c6f024f6a6d5a432a64f3" gracePeriod=30 Nov 03 19:56:59.049762 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.049730 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a17c84c4-j5bgn"] Nov 03 19:56:59.094203 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.094136 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a17c84c4-j5bgn"] Nov 03 19:56:59.287263 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.287231 5059 patch_prober.go:29] interesting pod/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 19:56:59.287430 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.287284 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" podUID="0953b878-511c-4fff-8639-fc5981f3dc3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 19:56:59.599587 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.599555 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk"] Nov 03 19:56:59.607492 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.607464 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.616187 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.615721 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk"] Nov 03 19:56:59.616187 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.615910 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6a7dc9-openshift-default-dockercfg-57msl\"" Nov 03 19:56:59.697948 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.697917 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdss4\" (UniqueName: \"kubernetes.io/projected/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-kube-api-access-fdss4\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.698255 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.698228 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-envoy\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.698332 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.698283 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-credential-socket\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.698332 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.698302 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-data\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.698401 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.698346 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-token\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.698401 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.698364 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istiod-ca-cert\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.698474 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.698421 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-workload-certs\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.698474 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.698459 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-workload-socket\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.698537 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.698504 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-podinfo\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.821685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.820711 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdss4\" (UniqueName: \"kubernetes.io/projected/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-kube-api-access-fdss4\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.821685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.820817 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-envoy\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.821685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.820877 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-credential-socket\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.821685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.820905 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-data\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.821685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.820990 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-token\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.821685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.821018 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istiod-ca-cert\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.821685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.821124 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-workload-certs\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.821685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.821196 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-workload-socket\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.821685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.821272 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-podinfo\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.829688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.824076 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-data\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.829688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.825362 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istiod-ca-cert\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.829688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.825901 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-workload-certs\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.829688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.826146 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-workload-socket\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.829688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.826495 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-credential-socket\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.842682 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.836264 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-podinfo\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.859565 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.850323 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-envoy\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.948698 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.948582 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdss4\" (UniqueName: \"kubernetes.io/projected/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-kube-api-access-fdss4\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:56:59.975691 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.955681 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62bf914d-4b66-4d52-ab52-db449030e0a6" path="/var/lib/kubelet/pods/62bf914d-4b66-4d52-ab52-db449030e0a6/volumes" Nov 03 19:56:59.986685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:56:59.958167 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-token\") pod \"e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:57:00.046789 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:00.046727 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56391128-m7wnt"] Nov 03 19:57:00.056684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:00.056529 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56391128-m7wnt"] Nov 03 19:57:00.231886 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:00.231809 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:57:00.287401 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:00.287360 5059 patch_prober.go:29] interesting pod/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 19:57:00.287553 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:00.287424 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" podUID="0953b878-511c-4fff-8639-fc5981f3dc3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 19:57:00.640938 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:00.640905 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk"] Nov 03 19:57:00.644954 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:57:00.644927 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b4c1a1_b3ca_46a5_982f_0781472ccfa6.slice/crio-3ae280fbf39328e52d7a008eafea01787ce9d2f33fe169aa6f52af1aa4697cc6 WatchSource:0}: Error finding container 3ae280fbf39328e52d7a008eafea01787ce9d2f33fe169aa6f52af1aa4697cc6: Status 404 returned error can't find the container with id 3ae280fbf39328e52d7a008eafea01787ce9d2f33fe169aa6f52af1aa4697cc6 Nov 03 19:57:00.650821 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:00.650786 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:57:00.651024 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:00.651004 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:57:00.651140 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:00.651123 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:57:01.111559 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:01.111516 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" event={"ID":"23b4c1a1-b3ca-46a5-982f-0781472ccfa6","Type":"ContainerStarted","Data":"967ad750ff0377e4aa1d61cff611774e468201221e110f8d8fad5e5d28e4706a"} Nov 03 19:57:01.111811 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:01.111587 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" event={"ID":"23b4c1a1-b3ca-46a5-982f-0781472ccfa6","Type":"ContainerStarted","Data":"3ae280fbf39328e52d7a008eafea01787ce9d2f33fe169aa6f52af1aa4697cc6"} Nov 03 19:57:01.146990 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:01.146929 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" podStartSLOduration=2.146910183 podStartE2EDuration="2.146910183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:01.13545692 +0000 UTC m=+1215.393254178" watchObservedRunningTime="2026-11-03 19:57:01.146910183 +0000 UTC m=+1215.404707441" Nov 03 19:57:01.232249 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:01.232210 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:57:01.237435 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:01.237407 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:57:01.290720 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:01.289609 5059 patch_prober.go:29] interesting pod/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 19:57:01.290720 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:01.289692 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" podUID="0953b878-511c-4fff-8639-fc5981f3dc3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 19:57:01.858790 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:01.858759 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b2e1bfd-d3e6-4005-bc08-47391001faf7" path="/var/lib/kubelet/pods/5b2e1bfd-d3e6-4005-bc08-47391001faf7/volumes" Nov 03 19:57:01.934383 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:01.934352 5059 patch_prober.go:29] interesting pod/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:57:01.934532 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:01.934410 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" podUID="070f2e92-dc4a-479a-a8dc-724e4cc4a050" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:57:02.120498 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:02.120418 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:57:02.121384 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:02.121358 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:57:02.291848 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:02.291809 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:57:02.293111 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:02.293084 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:57:03.636524 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.636498 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:57:03.719655 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.719624 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-data\") pod \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " Nov 03 19:57:03.719836 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.719705 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-envoy\") pod \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " Nov 03 19:57:03.719836 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.719752 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-token\" (UniqueName: \"kubernetes.io/projected/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-token\") pod \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " Nov 03 19:57:03.719955 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.719836 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-podinfo\") pod \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " Nov 03 19:57:03.719955 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.719883 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-workload-certs\") pod \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " Nov 03 19:57:03.719955 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.719913 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-workload-socket\") pod \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " Nov 03 19:57:03.720129 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.719973 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-data" pod "070f2e92-dc4a-479a-a8dc-724e4cc4a050" (UID: "070f2e92-dc4a-479a-a8dc-724e4cc4a050"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:03.720129 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.719992 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istiod-ca-cert\") pod \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " Nov 03 19:57:03.720246 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.720149 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/070f2e92-dc4a-479a-a8dc-724e4cc4a050-kube-api-access-2sndh\" (UniqueName: \"kubernetes.io/projected/070f2e92-dc4a-479a-a8dc-724e4cc4a050-kube-api-access-2sndh\") pod \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " Nov 03 19:57:03.720246 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.720185 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-credential-socket\") pod \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\" (UID: \"070f2e92-dc4a-479a-a8dc-724e4cc4a050\") " Nov 03 19:57:03.720354 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.720254 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-workload-certs" pod "070f2e92-dc4a-479a-a8dc-724e4cc4a050" (UID: "070f2e92-dc4a-479a-a8dc-724e4cc4a050"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:03.720354 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.720290 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-workload-socket" pod "070f2e92-dc4a-479a-a8dc-724e4cc4a050" (UID: "070f2e92-dc4a-479a-a8dc-724e4cc4a050"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:03.720591 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.720561 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-credential-socket" pod "070f2e92-dc4a-479a-a8dc-724e4cc4a050" (UID: "070f2e92-dc4a-479a-a8dc-724e4cc4a050"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:03.720703 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.720567 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istiod-ca-cert" pod "070f2e92-dc4a-479a-a8dc-724e4cc4a050" (UID: "070f2e92-dc4a-479a-a8dc-724e4cc4a050"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:57:03.721027 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.721003 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:03.721125 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.721033 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:03.721125 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.721048 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:03.721454 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.721150 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:03.721454 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.721173 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:03.723188 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.723153 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070f2e92-dc4a-479a-a8dc-724e4cc4a050-kube-api-access-2sndh" pod "070f2e92-dc4a-479a-a8dc-724e4cc4a050" (UID: "070f2e92-dc4a-479a-a8dc-724e4cc4a050"). InnerVolumeSpecName "kube-api-access-2sndh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:57:03.723504 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.723475 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-podinfo" pod "070f2e92-dc4a-479a-a8dc-724e4cc4a050" (UID: "070f2e92-dc4a-479a-a8dc-724e4cc4a050"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:57:03.723624 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.723576 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-envoy" pod "070f2e92-dc4a-479a-a8dc-724e4cc4a050" (UID: "070f2e92-dc4a-479a-a8dc-724e4cc4a050"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:03.724441 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.724411 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-token" pod "070f2e92-dc4a-479a-a8dc-724e4cc4a050" (UID: "070f2e92-dc4a-479a-a8dc-724e4cc4a050"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:57:03.822839 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.822796 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:03.822839 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.822835 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:03.822839 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.822844 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070f2e92-dc4a-479a-a8dc-724e4cc4a050-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:03.823049 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:03.822854 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sndh\" (UniqueName: \"kubernetes.io/projected/070f2e92-dc4a-479a-a8dc-724e4cc4a050-kube-api-access-2sndh\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:04.138372 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:04.138291 5059 generic.go:356] "Generic (PLEG): container finished" podID="070f2e92-dc4a-479a-a8dc-724e4cc4a050" containerID="836cc9a7ad5f2fb9f7bbdc2d0bbf637ebed2fac8ba1c6f024f6a6d5a432a64f3" exitCode=0 Nov 03 19:57:04.138372 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:04.138342 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" Nov 03 19:57:04.138682 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:04.138370 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" event={"ID":"070f2e92-dc4a-479a-a8dc-724e4cc4a050","Type":"ContainerDied","Data":"836cc9a7ad5f2fb9f7bbdc2d0bbf637ebed2fac8ba1c6f024f6a6d5a432a64f3"} Nov 03 19:57:04.138682 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:04.138429 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2" event={"ID":"070f2e92-dc4a-479a-a8dc-724e4cc4a050","Type":"ContainerDied","Data":"3c889bc5ebf707d4467868c453c2b228776eb4ac127e251b2c510d23ae7bb9a6"} Nov 03 19:57:04.138682 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:04.138460 5059 scope.go:122] "RemoveContainer" containerID="836cc9a7ad5f2fb9f7bbdc2d0bbf637ebed2fac8ba1c6f024f6a6d5a432a64f3" Nov 03 19:57:04.149973 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:04.149941 5059 scope.go:122] "RemoveContainer" containerID="4a9f00bb1e92bce9af0673cd4868bc76735066c2845448bfdb9dce793d7271d3" Nov 03 19:57:04.163167 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:04.160859 5059 scope.go:122] "RemoveContainer" containerID="836cc9a7ad5f2fb9f7bbdc2d0bbf637ebed2fac8ba1c6f024f6a6d5a432a64f3" Nov 03 19:57:04.163167 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:57:04.161397 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"836cc9a7ad5f2fb9f7bbdc2d0bbf637ebed2fac8ba1c6f024f6a6d5a432a64f3\": container with ID starting with 836cc9a7ad5f2fb9f7bbdc2d0bbf637ebed2fac8ba1c6f024f6a6d5a432a64f3 not found: ID does not exist" containerID="836cc9a7ad5f2fb9f7bbdc2d0bbf637ebed2fac8ba1c6f024f6a6d5a432a64f3" Nov 03 19:57:04.163167 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:04.161421 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836cc9a7ad5f2fb9f7bbdc2d0bbf637ebed2fac8ba1c6f024f6a6d5a432a64f3"} err="failed to get container status \"836cc9a7ad5f2fb9f7bbdc2d0bbf637ebed2fac8ba1c6f024f6a6d5a432a64f3\": rpc error: code = NotFound desc = could not find container \"836cc9a7ad5f2fb9f7bbdc2d0bbf637ebed2fac8ba1c6f024f6a6d5a432a64f3\": container with ID starting with 836cc9a7ad5f2fb9f7bbdc2d0bbf637ebed2fac8ba1c6f024f6a6d5a432a64f3 not found: ID does not exist" Nov 03 19:57:04.163167 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:04.161438 5059 scope.go:122] "RemoveContainer" containerID="4a9f00bb1e92bce9af0673cd4868bc76735066c2845448bfdb9dce793d7271d3" Nov 03 19:57:04.163167 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:57:04.161760 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a9f00bb1e92bce9af0673cd4868bc76735066c2845448bfdb9dce793d7271d3\": container with ID starting with 4a9f00bb1e92bce9af0673cd4868bc76735066c2845448bfdb9dce793d7271d3 not found: ID does not exist" containerID="4a9f00bb1e92bce9af0673cd4868bc76735066c2845448bfdb9dce793d7271d3" Nov 03 19:57:04.163167 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:04.161781 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a9f00bb1e92bce9af0673cd4868bc76735066c2845448bfdb9dce793d7271d3"} err="failed to get container status \"4a9f00bb1e92bce9af0673cd4868bc76735066c2845448bfdb9dce793d7271d3\": rpc error: code = NotFound desc = could not find container \"4a9f00bb1e92bce9af0673cd4868bc76735066c2845448bfdb9dce793d7271d3\": container with ID starting with 4a9f00bb1e92bce9af0673cd4868bc76735066c2845448bfdb9dce793d7271d3 not found: ID does not exist" Nov 03 19:57:04.163167 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:04.162382 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2"] Nov 03 19:57:04.167305 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:04.167285 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d8d238bf-openshift-default-6d47dc76f7-68zn2"] Nov 03 19:57:05.859859 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:05.859828 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="070f2e92-dc4a-479a-a8dc-724e4cc4a050" path="/var/lib/kubelet/pods/070f2e92-dc4a-479a-a8dc-724e4cc4a050/volumes" Nov 03 19:57:10.607014 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:57:10.606970 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:57:14.053890 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:14.053857 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f49ab41a-npc7f"] Nov 03 19:57:14.082426 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:14.075763 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f49ab41a-npc7f"] Nov 03 19:57:15.043063 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:15.043032 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05c4e799-gwzk8"] Nov 03 19:57:15.055118 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:15.055088 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05c4e799-gwzk8"] Nov 03 19:57:15.885330 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:15.885298 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b6308dc-ce77-434c-b414-55561e6d3192" path="/var/lib/kubelet/pods/6b6308dc-ce77-434c-b414-55561e6d3192/volumes" Nov 03 19:57:15.885984 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:15.885964 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96eb415b-e41d-45a3-ab0b-a52329ee3622" path="/var/lib/kubelet/pods/96eb415b-e41d-45a3-ab0b-a52329ee3622/volumes" Nov 03 19:57:19.039028 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:19.038983 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc342650-bsnk9"] Nov 03 19:57:19.051156 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:19.050959 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc342650-bsnk9"] Nov 03 19:57:19.858408 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:19.858376 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1893940-65b3-420f-9d5a-730248788311" path="/var/lib/kubelet/pods/f1893940-65b3-420f-9d5a-730248788311/volumes" Nov 03 19:57:21.974079 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:21.974040 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7"] Nov 03 19:57:21.975022 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:21.974997 5059 cpu_manager.go:414] "containerMap: removing container" podUID="070f2e92-dc4a-479a-a8dc-724e4cc4a050" containerName="istio-proxy" Nov 03 19:57:21.975022 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:21.975022 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="070f2e92-dc4a-479a-a8dc-724e4cc4a050" containerName="istio-proxy" Nov 03 19:57:21.975235 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:21.975220 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="070f2e92-dc4a-479a-a8dc-724e4cc4a050" containerName="istio-proxy" Nov 03 19:57:21.975293 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:21.975251 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="070f2e92-dc4a-479a-a8dc-724e4cc4a050" containerName="istio-proxy" Nov 03 19:57:22.002158 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.002134 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7" Nov 03 19:57:22.008294 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.008270 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7"] Nov 03 19:57:22.013840 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.013820 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3d1db1\"" Nov 03 19:57:22.022921 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.022900 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3d1db1-dockercfg-9m98k\"" Nov 03 19:57:22.276864 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.276753 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hcfw\" (UniqueName: \"kubernetes.io/projected/1463c361-4736-4158-83f1-e31353a338f4-kube-api-access-6hcfw\") pod \"payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7\" (UID: \"1463c361-4736-4158-83f1-e31353a338f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7" Nov 03 19:57:22.277053 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.276909 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1463c361-4736-4158-83f1-e31353a338f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7\" (UID: \"1463c361-4736-4158-83f1-e31353a338f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7" Nov 03 19:57:22.360213 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.357833 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh"] Nov 03 19:57:22.379153 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.377270 5059 cpu_manager.go:414] "containerMap: removing container" podUID="070f2e92-dc4a-479a-a8dc-724e4cc4a050" containerName="istio-proxy" Nov 03 19:57:22.379153 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.377301 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="070f2e92-dc4a-479a-a8dc-724e4cc4a050" containerName="istio-proxy" Nov 03 19:57:22.379153 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.378936 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hcfw\" (UniqueName: \"kubernetes.io/projected/1463c361-4736-4158-83f1-e31353a338f4-kube-api-access-6hcfw\") pod \"payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7\" (UID: \"1463c361-4736-4158-83f1-e31353a338f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7" Nov 03 19:57:22.379153 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.379012 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1463c361-4736-4158-83f1-e31353a338f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7\" (UID: \"1463c361-4736-4158-83f1-e31353a338f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7" Nov 03 19:57:22.380119 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.380068 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1463c361-4736-4158-83f1-e31353a338f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7\" (UID: \"1463c361-4736-4158-83f1-e31353a338f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7" Nov 03 19:57:22.384444 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.384421 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.390723 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.389884 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3d1db1-openshift-default-dockercfg-2ndm2\"" Nov 03 19:57:22.392879 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.392849 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh"] Nov 03 19:57:22.399181 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.399155 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hcfw\" (UniqueName: \"kubernetes.io/projected/1463c361-4736-4158-83f1-e31353a338f4-kube-api-access-6hcfw\") pod \"payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7\" (UID: \"1463c361-4736-4158-83f1-e31353a338f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7" Nov 03 19:57:22.480927 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.480883 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-credential-socket\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.481095 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.480987 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-token\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.481095 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.481023 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istiod-ca-cert\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.481095 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.481054 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-workload-socket\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.481217 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.481118 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-podinfo\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.481217 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.481180 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg9b6\" (UniqueName: \"kubernetes.io/projected/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-kube-api-access-zg9b6\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.481398 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.481233 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-envoy\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.481398 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.481323 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-data\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.481398 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.481355 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-workload-certs\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.582415 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.582379 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-credential-socket\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.582593 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.582494 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-token\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.582593 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.582525 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istiod-ca-cert\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.582593 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.582554 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-workload-socket\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.582794 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.582619 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-podinfo\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.582794 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.582674 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg9b6\" (UniqueName: \"kubernetes.io/projected/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-kube-api-access-zg9b6\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.582794 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.582735 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-envoy\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.582946 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.582815 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-credential-socket\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.582946 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.582840 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-data\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.582946 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.582873 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-workload-certs\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.583348 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.583323 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-workload-socket\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.583528 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.583366 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-workload-certs\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.583528 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.583416 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-data\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.583697 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.583424 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istiod-ca-cert\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.585118 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.585096 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-envoy\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.585204 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.585148 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-podinfo\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.597530 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.597502 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-token\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.598239 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.598214 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg9b6\" (UniqueName: \"kubernetes.io/projected/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-kube-api-access-zg9b6\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:22.638326 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.638300 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7" Nov 03 19:57:22.766684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:22.738988 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:23.230678 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:23.230631 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7"] Nov 03 19:57:23.231754 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:57:23.231713 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1463c361_4736_4158_83f1_e31353a338f4.slice/crio-33756232f4ae1ff9b6a01b9a4309abcb6c5b247aa0716b6d9a84ca990c4ff60e WatchSource:0}: Error finding container 33756232f4ae1ff9b6a01b9a4309abcb6c5b247aa0716b6d9a84ca990c4ff60e: Status 404 returned error can't find the container with id 33756232f4ae1ff9b6a01b9a4309abcb6c5b247aa0716b6d9a84ca990c4ff60e Nov 03 19:57:23.247977 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:57:23.247954 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ae3583_b9fa_4fe3_9621_fe95f254bce7.slice/crio-10723e3dc7ea42b9274094c99ef94f6eecbceaf7f6506e81bfe854a1077de76f WatchSource:0}: Error finding container 10723e3dc7ea42b9274094c99ef94f6eecbceaf7f6506e81bfe854a1077de76f: Status 404 returned error can't find the container with id 10723e3dc7ea42b9274094c99ef94f6eecbceaf7f6506e81bfe854a1077de76f Nov 03 19:57:23.250458 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:23.250425 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:57:23.250579 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:23.250490 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:57:23.250579 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:23.250520 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:57:23.252915 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:23.252888 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh"] Nov 03 19:57:23.504645 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:23.504506 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" event={"ID":"c7ae3583-b9fa-4fe3-9621-fe95f254bce7","Type":"ContainerStarted","Data":"7c09c7574e1818265f6c829187102752ced557d8a23f1942190802c9afb945e6"} Nov 03 19:57:23.504645 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:23.504564 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" event={"ID":"c7ae3583-b9fa-4fe3-9621-fe95f254bce7","Type":"ContainerStarted","Data":"10723e3dc7ea42b9274094c99ef94f6eecbceaf7f6506e81bfe854a1077de76f"} Nov 03 19:57:23.506834 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:23.506807 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7" event={"ID":"1463c361-4736-4158-83f1-e31353a338f4","Type":"ContainerStarted","Data":"da13d2704a055e1c385f688c426a9e0c1775173a7eed85d675fed505bb8d5572"} Nov 03 19:57:23.506970 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:23.506853 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7" event={"ID":"1463c361-4736-4158-83f1-e31353a338f4","Type":"ContainerStarted","Data":"33756232f4ae1ff9b6a01b9a4309abcb6c5b247aa0716b6d9a84ca990c4ff60e"} Nov 03 19:57:23.506970 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:23.506959 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7" Nov 03 19:57:23.534122 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:23.534070 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" podStartSLOduration=1.534051759 podStartE2EDuration="1.534051759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:23.527729882 +0000 UTC m=+1237.785527140" watchObservedRunningTime="2026-11-03 19:57:23.534051759 +0000 UTC m=+1237.791849019" Nov 03 19:57:23.550866 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:23.550818 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7" podStartSLOduration=2.5508050410000003 podStartE2EDuration="2.550805041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:23.5465853 +0000 UTC m=+1237.804382577" watchObservedRunningTime="2026-11-03 19:57:23.550805041 +0000 UTC m=+1237.808602317" Nov 03 19:57:23.822615 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:23.740647 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:24.410037 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:24.409956 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl"] Nov 03 19:57:24.415620 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:24.415593 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl" Nov 03 19:57:24.417851 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:24.417823 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl"] Nov 03 19:57:24.421620 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:24.421591 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9b3619-dockercfg-9699m\"" Nov 03 19:57:24.443254 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:24.443130 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9b3619\"" Nov 03 19:57:24.548838 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:24.548803 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978df79a-3b55-4b2e-8736-8d95fff03873-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl\" (UID: \"978df79a-3b55-4b2e-8736-8d95fff03873\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl" Nov 03 19:57:24.549037 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:24.548900 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2fkv\" (UniqueName: \"kubernetes.io/projected/978df79a-3b55-4b2e-8736-8d95fff03873-kube-api-access-d2fkv\") pod \"payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl\" (UID: \"978df79a-3b55-4b2e-8736-8d95fff03873\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl" Nov 03 19:57:24.705736 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:24.679795 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978df79a-3b55-4b2e-8736-8d95fff03873-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl\" (UID: \"978df79a-3b55-4b2e-8736-8d95fff03873\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl" Nov 03 19:57:24.705736 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:24.679875 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2fkv\" (UniqueName: \"kubernetes.io/projected/978df79a-3b55-4b2e-8736-8d95fff03873-kube-api-access-d2fkv\") pod \"payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl\" (UID: \"978df79a-3b55-4b2e-8736-8d95fff03873\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl" Nov 03 19:57:24.705736 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:24.681236 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978df79a-3b55-4b2e-8736-8d95fff03873-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl\" (UID: \"978df79a-3b55-4b2e-8736-8d95fff03873\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl" Nov 03 19:57:24.719738 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:24.719386 5059 generic.go:356] "Generic (PLEG): container finished" podID="c149bde6-19ca-4ce5-9685-c1cb12fe1740" containerID="2f8abc715842645f0a29b43c181cb7842a915e01b85268981ed036ea730585a7" exitCode=0 Nov 03 19:57:24.720956 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:24.720927 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" event={"ID":"c149bde6-19ca-4ce5-9685-c1cb12fe1740","Type":"ContainerDied","Data":"2f8abc715842645f0a29b43c181cb7842a915e01b85268981ed036ea730585a7"} Nov 03 19:57:24.721898 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:24.721875 5059 scope.go:122] "RemoveContainer" containerID="2f8abc715842645f0a29b43c181cb7842a915e01b85268981ed036ea730585a7" Nov 03 19:57:24.728554 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:24.728516 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:57:24.728795 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:24.728773 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:57:24.728922 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:24.728907 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:57:24.778686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:24.772657 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2fkv\" (UniqueName: \"kubernetes.io/projected/978df79a-3b55-4b2e-8736-8d95fff03873-kube-api-access-d2fkv\") pod \"payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl\" (UID: \"978df79a-3b55-4b2e-8736-8d95fff03873\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl" Nov 03 19:57:24.778686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:24.773406 5059 patch_prober.go:29] interesting pod/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 19:57:24.778686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:24.773478 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" podUID="c7ae3583-b9fa-4fe3-9621-fe95f254bce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 19:57:25.052924 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:25.052903 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl" Nov 03 19:57:25.513747 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:25.499883 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl"] Nov 03 19:57:25.521612 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:57:25.521441 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978df79a_3b55_4b2e_8736_8d95fff03873.slice/crio-55371203a8a704bd153980042d105006bccc27d607e22a079a06a7875f94e612 WatchSource:0}: Error finding container 55371203a8a704bd153980042d105006bccc27d607e22a079a06a7875f94e612: Status 404 returned error can't find the container with id 55371203a8a704bd153980042d105006bccc27d607e22a079a06a7875f94e612 Nov 03 19:57:25.738295 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:25.738139 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl" event={"ID":"978df79a-3b55-4b2e-8736-8d95fff03873","Type":"ContainerStarted","Data":"55371203a8a704bd153980042d105006bccc27d607e22a079a06a7875f94e612"} Nov 03 19:57:25.739496 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:25.739469 5059 patch_prober.go:29] interesting pod/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 19:57:25.739638 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:25.739515 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" podUID="c7ae3583-b9fa-4fe3-9621-fe95f254bce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 19:57:25.741614 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:25.741586 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" event={"ID":"c149bde6-19ca-4ce5-9685-c1cb12fe1740","Type":"ContainerStarted","Data":"2e7e20cce29e971c016746efa172864a6f8aa4433e310137cd128ff94cfaf22a"} Nov 03 19:57:25.750949 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:25.750928 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:57:25.837515 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:25.837485 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk"] Nov 03 19:57:25.837749 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:25.837720 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" podUID="e0611f73-972b-4e4e-be2f-5c536bdd3d9b" containerName="istio-proxy" containerID="cri-o://6833b5a85b8a47638eb76a554e161d4167edd6fba7149e00b7506d1c0ae156dd" gracePeriod=30 Nov 03 19:57:26.392013 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:26.391546 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:57:26.392013 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:26.391690 5059 patch_prober.go:29] interesting pod/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 19:57:26.392013 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:26.391736 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" podUID="c149bde6-19ca-4ce5-9685-c1cb12fe1740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 19:57:26.739591 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:26.739510 5059 patch_prober.go:29] interesting pod/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 19:57:26.739591 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:26.739565 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" podUID="c7ae3583-b9fa-4fe3-9621-fe95f254bce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 19:57:26.750580 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:26.750552 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl" event={"ID":"978df79a-3b55-4b2e-8736-8d95fff03873","Type":"ContainerStarted","Data":"5305028fa3d14c77186070210eed9eeeb4b54feab85b25eae4c54149649b4af7"} Nov 03 19:57:26.750732 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:26.750646 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl" Nov 03 19:57:26.817815 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:26.813208 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl" podStartSLOduration=2.813187433 podStartE2EDuration="2.813187433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:26.809966362 +0000 UTC m=+1241.067763639" watchObservedRunningTime="2026-11-03 19:57:26.813187433 +0000 UTC m=+1241.070984694" Nov 03 19:57:27.400716 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:27.392762 5059 patch_prober.go:29] interesting pod/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 19:57:27.400716 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:27.392828 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" podUID="c149bde6-19ca-4ce5-9685-c1cb12fe1740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 19:57:27.748741 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:27.741061 5059 patch_prober.go:29] interesting pod/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 19:57:27.748741 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:27.741126 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" podUID="c7ae3583-b9fa-4fe3-9621-fe95f254bce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 19:57:28.404689 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:28.404483 5059 patch_prober.go:29] interesting pod/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 19:57:28.404689 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:28.404545 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" podUID="c149bde6-19ca-4ce5-9685-c1cb12fe1740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 19:57:28.739390 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:28.739306 5059 patch_prober.go:29] interesting pod/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 19:57:28.739390 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:28.739365 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" podUID="c7ae3583-b9fa-4fe3-9621-fe95f254bce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 19:57:29.394695 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:29.393393 5059 patch_prober.go:29] interesting pod/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 19:57:29.394695 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:29.393475 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" podUID="c149bde6-19ca-4ce5-9685-c1cb12fe1740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 19:57:29.755764 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:29.750727 5059 patch_prober.go:29] interesting pod/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 19:57:29.755764 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:29.750788 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" podUID="c7ae3583-b9fa-4fe3-9621-fe95f254bce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 19:57:30.392531 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:30.392082 5059 patch_prober.go:29] interesting pod/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 19:57:30.392531 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:30.392150 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" podUID="c149bde6-19ca-4ce5-9685-c1cb12fe1740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 19:57:30.744839 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:30.744756 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:30.745267 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:30.745257 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:30.746147 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:30.746124 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 19:57:31.055487 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.055398 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff81e09f-vlzbb"] Nov 03 19:57:31.112071 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.112033 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff81e09f-vlzbb"] Nov 03 19:57:31.341127 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.341101 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:57:31.392352 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.392188 5059 patch_prober.go:29] interesting pod/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 19:57:31.392352 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.392256 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" podUID="c149bde6-19ca-4ce5-9685-c1cb12fe1740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 19:57:31.421293 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.421255 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-kube-api-access-ljqpx\" (UniqueName: \"kubernetes.io/projected/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-kube-api-access-ljqpx\") pod \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " Nov 03 19:57:31.421467 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.421334 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-workload-certs\") pod \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " Nov 03 19:57:31.421467 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.421357 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-workload-socket\") pod \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " Nov 03 19:57:31.421467 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.421414 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-podinfo\") pod \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " Nov 03 19:57:31.421467 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.421449 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-data\") pod \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " Nov 03 19:57:31.421739 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.421475 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-token\" (UniqueName: \"kubernetes.io/projected/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-token\") pod \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " Nov 03 19:57:31.421739 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.421558 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-envoy\") pod \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " Nov 03 19:57:31.421739 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.421581 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-credential-socket\") pod \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " Nov 03 19:57:31.421739 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.421714 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istiod-ca-cert\") pod \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\" (UID: \"e0611f73-972b-4e4e-be2f-5c536bdd3d9b\") " Nov 03 19:57:31.422428 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.422404 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istiod-ca-cert" pod "e0611f73-972b-4e4e-be2f-5c536bdd3d9b" (UID: "e0611f73-972b-4e4e-be2f-5c536bdd3d9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:57:31.423490 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.423454 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-workload-socket" pod "e0611f73-972b-4e4e-be2f-5c536bdd3d9b" (UID: "e0611f73-972b-4e4e-be2f-5c536bdd3d9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:31.423623 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.423593 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-data" pod "e0611f73-972b-4e4e-be2f-5c536bdd3d9b" (UID: "e0611f73-972b-4e4e-be2f-5c536bdd3d9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:31.424049 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.423837 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-workload-certs" pod "e0611f73-972b-4e4e-be2f-5c536bdd3d9b" (UID: "e0611f73-972b-4e4e-be2f-5c536bdd3d9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:31.424049 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.424015 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-credential-socket" pod "e0611f73-972b-4e4e-be2f-5c536bdd3d9b" (UID: "e0611f73-972b-4e4e-be2f-5c536bdd3d9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:31.424927 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.424893 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-kube-api-access-ljqpx" pod "e0611f73-972b-4e4e-be2f-5c536bdd3d9b" (UID: "e0611f73-972b-4e4e-be2f-5c536bdd3d9b"). InnerVolumeSpecName "kube-api-access-ljqpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:57:31.426109 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.426086 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-token" pod "e0611f73-972b-4e4e-be2f-5c536bdd3d9b" (UID: "e0611f73-972b-4e4e-be2f-5c536bdd3d9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:57:31.426280 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.426237 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-envoy" pod "e0611f73-972b-4e4e-be2f-5c536bdd3d9b" (UID: "e0611f73-972b-4e4e-be2f-5c536bdd3d9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:31.426379 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.426321 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-podinfo" pod "e0611f73-972b-4e4e-be2f-5c536bdd3d9b" (UID: "e0611f73-972b-4e4e-be2f-5c536bdd3d9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:57:31.527784 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.525675 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:31.527784 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.525706 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljqpx\" (UniqueName: \"kubernetes.io/projected/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-kube-api-access-ljqpx\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:31.527784 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.525716 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:31.527784 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.525724 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:31.527784 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.525733 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:31.527784 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.525742 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:31.527784 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.525750 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:31.527784 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.525759 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:31.527784 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.525767 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0611f73-972b-4e4e-be2f-5c536bdd3d9b-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:31.868508 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.868473 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fadac4c8-e38d-43e5-89b7-b4460521b6f1" path="/var/lib/kubelet/pods/fadac4c8-e38d-43e5-89b7-b4460521b6f1/volumes" Nov 03 19:57:31.869071 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.869052 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48"] Nov 03 19:57:31.869257 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.869236 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" podUID="0953b878-511c-4fff-8639-fc5981f3dc3e" containerName="istio-proxy" containerID="cri-o://407d89b3754c60f101a3cd6263fdfb6cee19e8a1ece10a5419fa991ca8c748f6" gracePeriod=30 Nov 03 19:57:31.930129 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.930094 5059 generic.go:356] "Generic (PLEG): container finished" podID="e0611f73-972b-4e4e-be2f-5c536bdd3d9b" containerID="6833b5a85b8a47638eb76a554e161d4167edd6fba7149e00b7506d1c0ae156dd" exitCode=0 Nov 03 19:57:31.930298 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.930155 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" Nov 03 19:57:31.930298 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.930188 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" event={"ID":"e0611f73-972b-4e4e-be2f-5c536bdd3d9b","Type":"ContainerDied","Data":"6833b5a85b8a47638eb76a554e161d4167edd6fba7149e00b7506d1c0ae156dd"} Nov 03 19:57:31.930298 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.930243 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk" event={"ID":"e0611f73-972b-4e4e-be2f-5c536bdd3d9b","Type":"ContainerDied","Data":"df26621d2dd52ddffade9b4a6c263e74de46ac1f264bcf6859dd406b79d90096"} Nov 03 19:57:31.930298 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.930279 5059 scope.go:122] "RemoveContainer" containerID="6833b5a85b8a47638eb76a554e161d4167edd6fba7149e00b7506d1c0ae156dd" Nov 03 19:57:31.955030 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:31.951791 5059 scope.go:122] "RemoveContainer" containerID="4c16a3a918556e1de30cf2cf342198f03949b5f77828d2855186ca37de52e432" Nov 03 19:57:32.077684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:32.053929 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk"] Nov 03 19:57:32.104789 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:32.085822 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b0f9d6-openshift-default-58c6f44bf-9zqkk"] Nov 03 19:57:32.146310 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:32.146279 5059 scope.go:122] "RemoveContainer" containerID="6833b5a85b8a47638eb76a554e161d4167edd6fba7149e00b7506d1c0ae156dd" Nov 03 19:57:32.157754 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:57:32.146759 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6833b5a85b8a47638eb76a554e161d4167edd6fba7149e00b7506d1c0ae156dd\": container with ID starting with 6833b5a85b8a47638eb76a554e161d4167edd6fba7149e00b7506d1c0ae156dd not found: ID does not exist" containerID="6833b5a85b8a47638eb76a554e161d4167edd6fba7149e00b7506d1c0ae156dd" Nov 03 19:57:32.157754 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:32.146802 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6833b5a85b8a47638eb76a554e161d4167edd6fba7149e00b7506d1c0ae156dd"} err="failed to get container status \"6833b5a85b8a47638eb76a554e161d4167edd6fba7149e00b7506d1c0ae156dd\": rpc error: code = NotFound desc = could not find container \"6833b5a85b8a47638eb76a554e161d4167edd6fba7149e00b7506d1c0ae156dd\": container with ID starting with 6833b5a85b8a47638eb76a554e161d4167edd6fba7149e00b7506d1c0ae156dd not found: ID does not exist" Nov 03 19:57:32.157754 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:32.146827 5059 scope.go:122] "RemoveContainer" containerID="4c16a3a918556e1de30cf2cf342198f03949b5f77828d2855186ca37de52e432" Nov 03 19:57:32.157754 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:57:32.147297 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c16a3a918556e1de30cf2cf342198f03949b5f77828d2855186ca37de52e432\": container with ID starting with 4c16a3a918556e1de30cf2cf342198f03949b5f77828d2855186ca37de52e432 not found: ID does not exist" containerID="4c16a3a918556e1de30cf2cf342198f03949b5f77828d2855186ca37de52e432" Nov 03 19:57:32.157754 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:32.147333 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c16a3a918556e1de30cf2cf342198f03949b5f77828d2855186ca37de52e432"} err="failed to get container status \"4c16a3a918556e1de30cf2cf342198f03949b5f77828d2855186ca37de52e432\": rpc error: code = NotFound desc = could not find container \"4c16a3a918556e1de30cf2cf342198f03949b5f77828d2855186ca37de52e432\": container with ID starting with 4c16a3a918556e1de30cf2cf342198f03949b5f77828d2855186ca37de52e432 not found: ID does not exist" Nov 03 19:57:32.295691 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:32.294995 5059 patch_prober.go:29] interesting pod/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:57:32.295691 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:32.295066 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" podUID="0953b878-511c-4fff-8639-fc5981f3dc3e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:57:32.394347 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:32.394269 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:57:32.396007 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:32.395981 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:57:33.875609 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:33.875574 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0611f73-972b-4e4e-be2f-5c536bdd3d9b" path="/var/lib/kubelet/pods/e0611f73-972b-4e4e-be2f-5c536bdd3d9b/volumes" Nov 03 19:57:34.050347 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:34.050302 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-464e0cfc-xghmv"] Nov 03 19:57:34.062095 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:34.062056 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-464e0cfc-xghmv"] Nov 03 19:57:34.723155 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:34.723127 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7" Nov 03 19:57:35.861136 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:35.861100 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21882329-e6fc-4657-a4da-036fee584872" path="/var/lib/kubelet/pods/21882329-e6fc-4657-a4da-036fee584872/volumes" Nov 03 19:57:37.159833 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.159733 5059 generic.go:356] "Generic (PLEG): container finished" podID="0953b878-511c-4fff-8639-fc5981f3dc3e" containerID="407d89b3754c60f101a3cd6263fdfb6cee19e8a1ece10a5419fa991ca8c748f6" exitCode=0 Nov 03 19:57:37.160297 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.159979 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" event={"ID":"0953b878-511c-4fff-8639-fc5981f3dc3e","Type":"ContainerDied","Data":"407d89b3754c60f101a3cd6263fdfb6cee19e8a1ece10a5419fa991ca8c748f6"} Nov 03 19:57:37.160297 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.160027 5059 scope.go:122] "RemoveContainer" containerID="2c61ae1c1de969bad4d008d73483b6a6d68ffde0a711b5bd024bec245bccfa83" Nov 03 19:57:37.232513 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.232488 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:57:37.337077 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.337044 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0953b878-511c-4fff-8639-fc5981f3dc3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0953b878-511c-4fff-8639-fc5981f3dc3e-istiod-ca-cert\") pod \"0953b878-511c-4fff-8639-fc5981f3dc3e\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " Nov 03 19:57:37.337262 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.337107 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-token\" (UniqueName: \"kubernetes.io/projected/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-token\") pod \"0953b878-511c-4fff-8639-fc5981f3dc3e\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " Nov 03 19:57:37.337262 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.337163 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-credential-socket\") pod \"0953b878-511c-4fff-8639-fc5981f3dc3e\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " Nov 03 19:57:37.337262 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.337215 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-envoy\") pod \"0953b878-511c-4fff-8639-fc5981f3dc3e\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " Nov 03 19:57:37.337501 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.337293 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-workload-certs\") pod \"0953b878-511c-4fff-8639-fc5981f3dc3e\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " Nov 03 19:57:37.337501 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.337385 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-workload-socket\") pod \"0953b878-511c-4fff-8639-fc5981f3dc3e\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " Nov 03 19:57:37.337501 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.337431 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0953b878-511c-4fff-8639-fc5981f3dc3e-kube-api-access-mtt9b\" (UniqueName: \"kubernetes.io/projected/0953b878-511c-4fff-8639-fc5981f3dc3e-kube-api-access-mtt9b\") pod \"0953b878-511c-4fff-8639-fc5981f3dc3e\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " Nov 03 19:57:37.337729 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.337520 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-data\") pod \"0953b878-511c-4fff-8639-fc5981f3dc3e\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " Nov 03 19:57:37.337729 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.337574 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0953b878-511c-4fff-8639-fc5981f3dc3e-istiod-ca-cert" pod "0953b878-511c-4fff-8639-fc5981f3dc3e" (UID: "0953b878-511c-4fff-8639-fc5981f3dc3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:57:37.337729 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.337592 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-podinfo\") pod \"0953b878-511c-4fff-8639-fc5981f3dc3e\" (UID: \"0953b878-511c-4fff-8639-fc5981f3dc3e\") " Nov 03 19:57:37.338725 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.338009 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-credential-socket" pod "0953b878-511c-4fff-8639-fc5981f3dc3e" (UID: "0953b878-511c-4fff-8639-fc5981f3dc3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:37.338725 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.338311 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0953b878-511c-4fff-8639-fc5981f3dc3e-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:37.338725 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.338327 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:37.338725 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.338374 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-workload-socket" pod "0953b878-511c-4fff-8639-fc5981f3dc3e" (UID: "0953b878-511c-4fff-8639-fc5981f3dc3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:37.338725 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.338579 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-workload-certs" pod "0953b878-511c-4fff-8639-fc5981f3dc3e" (UID: "0953b878-511c-4fff-8639-fc5981f3dc3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:37.338725 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.338613 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-data" pod "0953b878-511c-4fff-8639-fc5981f3dc3e" (UID: "0953b878-511c-4fff-8639-fc5981f3dc3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:37.340360 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.340308 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-token" pod "0953b878-511c-4fff-8639-fc5981f3dc3e" (UID: "0953b878-511c-4fff-8639-fc5981f3dc3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:57:37.340469 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.340396 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-podinfo" pod "0953b878-511c-4fff-8639-fc5981f3dc3e" (UID: "0953b878-511c-4fff-8639-fc5981f3dc3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:57:37.341135 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.341108 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0953b878-511c-4fff-8639-fc5981f3dc3e-kube-api-access-mtt9b" pod "0953b878-511c-4fff-8639-fc5981f3dc3e" (UID: "0953b878-511c-4fff-8639-fc5981f3dc3e"). InnerVolumeSpecName "kube-api-access-mtt9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:57:37.341248 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.341151 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-envoy" pod "0953b878-511c-4fff-8639-fc5981f3dc3e" (UID: "0953b878-511c-4fff-8639-fc5981f3dc3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:37.439531 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.439464 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:37.439531 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.439492 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:37.439531 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.439502 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:37.439531 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.439529 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtt9b\" (UniqueName: \"kubernetes.io/projected/0953b878-511c-4fff-8639-fc5981f3dc3e-kube-api-access-mtt9b\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:37.439773 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.439538 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:37.439773 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.439557 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:37.439773 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.439565 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0953b878-511c-4fff-8639-fc5981f3dc3e-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:37.843755 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:37.843722 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl" Nov 03 19:57:38.170478 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:38.170384 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" Nov 03 19:57:38.170478 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:38.170407 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48" event={"ID":"0953b878-511c-4fff-8639-fc5981f3dc3e","Type":"ContainerDied","Data":"a25a486651aa478c427899bd31eeda61adfa34318b309043531a5df8b7d4557a"} Nov 03 19:57:38.170478 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:38.170474 5059 scope.go:122] "RemoveContainer" containerID="407d89b3754c60f101a3cd6263fdfb6cee19e8a1ece10a5419fa991ca8c748f6" Nov 03 19:57:38.195448 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:38.195417 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48"] Nov 03 19:57:38.199941 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:38.199915 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b34f6c-openshift-default-595cd9f785-55l48"] Nov 03 19:57:39.860752 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:39.860722 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0953b878-511c-4fff-8639-fc5981f3dc3e" path="/var/lib/kubelet/pods/0953b878-511c-4fff-8639-fc5981f3dc3e/volumes" Nov 03 19:57:40.607078 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:57:40.607046 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:57:46.028501 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.028461 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d9da1c-cslfr"] Nov 03 19:57:46.032634 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.032608 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d9da1c-cslfr"] Nov 03 19:57:46.845015 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.844976 5059 scope.go:122] "RemoveContainer" containerID="979ea1ebd80c99181c9faabff7e0386a72fd68c1ee79d4ad8c04c774a6c1e808" Nov 03 19:57:46.853935 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.853890 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m"] Nov 03 19:57:46.855956 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.855593 5059 cpu_manager.go:414] "containerMap: removing container" podUID="e0611f73-972b-4e4e-be2f-5c536bdd3d9b" containerName="istio-proxy" Nov 03 19:57:46.855956 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.855618 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0611f73-972b-4e4e-be2f-5c536bdd3d9b" containerName="istio-proxy" Nov 03 19:57:46.855956 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.855649 5059 cpu_manager.go:414] "containerMap: removing container" podUID="e0611f73-972b-4e4e-be2f-5c536bdd3d9b" containerName="istio-proxy" Nov 03 19:57:46.855956 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.855674 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0611f73-972b-4e4e-be2f-5c536bdd3d9b" containerName="istio-proxy" Nov 03 19:57:46.855956 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.855697 5059 cpu_manager.go:414] "containerMap: removing container" podUID="0953b878-511c-4fff-8639-fc5981f3dc3e" containerName="istio-proxy" Nov 03 19:57:46.855956 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.855704 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0953b878-511c-4fff-8639-fc5981f3dc3e" containerName="istio-proxy" Nov 03 19:57:46.855956 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.855731 5059 cpu_manager.go:414] "containerMap: removing container" podUID="0953b878-511c-4fff-8639-fc5981f3dc3e" containerName="istio-proxy" Nov 03 19:57:46.855956 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.855739 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0953b878-511c-4fff-8639-fc5981f3dc3e" containerName="istio-proxy" Nov 03 19:57:46.855956 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.855934 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0611f73-972b-4e4e-be2f-5c536bdd3d9b" containerName="istio-proxy" Nov 03 19:57:46.855956 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.855952 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0611f73-972b-4e4e-be2f-5c536bdd3d9b" containerName="istio-proxy" Nov 03 19:57:46.856457 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.855965 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0953b878-511c-4fff-8639-fc5981f3dc3e" containerName="istio-proxy" Nov 03 19:57:46.856457 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.855976 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0953b878-511c-4fff-8639-fc5981f3dc3e" containerName="istio-proxy" Nov 03 19:57:46.914730 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.900129 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:57:46.914730 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.908006 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-389e6d\"/\"openshift-service-ca.crt\"" Nov 03 19:57:46.914730 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.908061 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-389e6d\"/\"kube-root-ca.crt\"" Nov 03 19:57:46.933847 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.933818 5059 scope.go:122] "RemoveContainer" containerID="1ee3205fcc0f585d0889816f5ed739e5b42e54c51656376496c468865a5ec4f9" Nov 03 19:57:46.934129 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.934106 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m"] Nov 03 19:57:46.945195 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.945172 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-389e6d\"/\"default-dockercfg-t4qvx\"" Nov 03 19:57:46.945538 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.945519 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-389e6d\"/\"ipp-route-389e6d-kserve-self-signed-certs\"" Nov 03 19:57:46.950253 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.950231 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f12977e-2949-4de3-8c7b-2e42dbc495e7-tls-certs\") pod \"ipp-route-389e6d-kserve-9fc78d8d5-cfv6m\" (UID: \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\") " pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:57:46.950754 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.950369 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-dshm\") pod \"ipp-route-389e6d-kserve-9fc78d8d5-cfv6m\" (UID: \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\") " pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:57:46.950754 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.950432 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-tmp-dir\") pod \"ipp-route-389e6d-kserve-9fc78d8d5-cfv6m\" (UID: \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\") " pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:57:46.950754 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.950553 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-home\") pod \"ipp-route-389e6d-kserve-9fc78d8d5-cfv6m\" (UID: \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\") " pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:57:46.950920 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.950897 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-model-cache\") pod \"ipp-route-389e6d-kserve-9fc78d8d5-cfv6m\" (UID: \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\") " pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:57:46.950977 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.950961 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bd2c\" (UniqueName: \"kubernetes.io/projected/5f12977e-2949-4de3-8c7b-2e42dbc495e7-kube-api-access-7bd2c\") pod \"ipp-route-389e6d-kserve-9fc78d8d5-cfv6m\" (UID: \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\") " pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:57:46.953002 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.952984 5059 scope.go:122] "RemoveContainer" containerID="b3f40ef8220aec992a9f7e77fe2204e98bdbdab1b0e917ceaaa55d905170dcdb" Nov 03 19:57:46.982152 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.982092 5059 scope.go:122] "RemoveContainer" containerID="1c176221b6f0afbbee88fb63e5a4e27c9d86780001543e9c5430676a90f7b0bb" Nov 03 19:57:46.999897 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:46.999879 5059 scope.go:122] "RemoveContainer" containerID="3193a5482642c97d5980c5ec9f05bbabd5cc1028d3adeed48033b3eeeecbf24c" Nov 03 19:57:47.053758 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:47.053714 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-dshm\") pod \"ipp-route-389e6d-kserve-9fc78d8d5-cfv6m\" (UID: \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\") " pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:57:47.054154 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:47.053779 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-tmp-dir\") pod \"ipp-route-389e6d-kserve-9fc78d8d5-cfv6m\" (UID: \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\") " pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:57:47.054154 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:47.053838 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-home\") pod \"ipp-route-389e6d-kserve-9fc78d8d5-cfv6m\" (UID: \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\") " pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:57:47.054154 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:47.053962 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-model-cache\") pod \"ipp-route-389e6d-kserve-9fc78d8d5-cfv6m\" (UID: \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\") " pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:57:47.054154 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:47.054019 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bd2c\" (UniqueName: \"kubernetes.io/projected/5f12977e-2949-4de3-8c7b-2e42dbc495e7-kube-api-access-7bd2c\") pod \"ipp-route-389e6d-kserve-9fc78d8d5-cfv6m\" (UID: \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\") " pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:57:47.054366 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:47.054246 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f12977e-2949-4de3-8c7b-2e42dbc495e7-tls-certs\") pod \"ipp-route-389e6d-kserve-9fc78d8d5-cfv6m\" (UID: \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\") " pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:57:47.054419 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:47.054384 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-home\") pod \"ipp-route-389e6d-kserve-9fc78d8d5-cfv6m\" (UID: \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\") " pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:57:47.054647 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:47.054625 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-tmp-dir\") pod \"ipp-route-389e6d-kserve-9fc78d8d5-cfv6m\" (UID: \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\") " pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:57:47.054918 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:47.054898 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-model-cache\") pod \"ipp-route-389e6d-kserve-9fc78d8d5-cfv6m\" (UID: \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\") " pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:57:47.056919 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:47.056895 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-dshm\") pod \"ipp-route-389e6d-kserve-9fc78d8d5-cfv6m\" (UID: \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\") " pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:57:47.057592 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:47.057574 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f12977e-2949-4de3-8c7b-2e42dbc495e7-tls-certs\") pod \"ipp-route-389e6d-kserve-9fc78d8d5-cfv6m\" (UID: \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\") " pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:57:47.091200 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:47.091166 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bd2c\" (UniqueName: \"kubernetes.io/projected/5f12977e-2949-4de3-8c7b-2e42dbc495e7-kube-api-access-7bd2c\") pod \"ipp-route-389e6d-kserve-9fc78d8d5-cfv6m\" (UID: \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\") " pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:57:47.238100 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:47.238068 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:57:47.880686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:47.874962 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="588f1533-b01d-4fd9-a022-afe753fa6a52" path="/var/lib/kubelet/pods/588f1533-b01d-4fd9-a022-afe753fa6a52/volumes" Nov 03 19:57:47.989653 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:47.989625 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m"] Nov 03 19:57:47.990627 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:57:47.990602 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f12977e_2949_4de3_8c7b_2e42dbc495e7.slice/crio-49068727121bc447a4e79ad61c8bfc083985965712406baec96f1999754ef2d2 WatchSource:0}: Error finding container 49068727121bc447a4e79ad61c8bfc083985965712406baec96f1999754ef2d2: Status 404 returned error can't find the container with id 49068727121bc447a4e79ad61c8bfc083985965712406baec96f1999754ef2d2 Nov 03 19:57:48.078727 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:48.069370 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fe573bf-wc7sh"] Nov 03 19:57:48.106367 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:48.106282 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fe573bf-wc7sh"] Nov 03 19:57:48.577927 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:48.577893 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" event={"ID":"5f12977e-2949-4de3-8c7b-2e42dbc495e7","Type":"ContainerStarted","Data":"14d62fb25c7489609d927c24fdefb001dac0545d210b73a0e4401b5dbf2e8e08"} Nov 03 19:57:48.578063 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:48.577939 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" event={"ID":"5f12977e-2949-4de3-8c7b-2e42dbc495e7","Type":"ContainerStarted","Data":"49068727121bc447a4e79ad61c8bfc083985965712406baec96f1999754ef2d2"} Nov 03 19:57:48.635049 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:48.635005 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" podStartSLOduration=2.634988162 podStartE2EDuration="2.634988162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:48.632627005 +0000 UTC m=+1262.890424262" watchObservedRunningTime="2026-11-03 19:57:48.634988162 +0000 UTC m=+1262.892785419" Nov 03 19:57:49.238979 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:49.238951 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:57:49.770249 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:49.770208 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl"] Nov 03 19:57:49.770479 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:49.770442 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" podUID="3a73f1d8-301f-40db-91e8-b4963410e1a0" containerName="istio-proxy" containerID="cri-o://c44d5229a7ea4579d97216bf3a68c0945f698e93fa10deb42a260172c495bada" gracePeriod=30 Nov 03 19:57:49.775294 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:49.775237 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c"] Nov 03 19:57:49.775541 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:49.775476 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" podUID="0e53359d-3f88-4a9c-b50a-050984844949" containerName="istio-proxy" containerID="cri-o://61d4bffc78086245d3366e6ac7434f3e64b57bca64e4d3b1bfa2cc08dc31fded" gracePeriod=30 Nov 03 19:57:49.859219 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:49.859181 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6d4f80e-7c28-44c4-9f87-4d690d6a5e35" path="/var/lib/kubelet/pods/b6d4f80e-7c28-44c4-9f87-4d690d6a5e35/volumes" Nov 03 19:57:50.119807 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:50.119743 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ff859ce-w7znl"] Nov 03 19:57:50.191688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:50.191347 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ff859ce-w7znl"] Nov 03 19:57:51.860087 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:51.860055 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89c4dbf6-cf42-43b4-95cb-48676a3a4869" path="/var/lib/kubelet/pods/89c4dbf6-cf42-43b4-95cb-48676a3a4869/volumes" Nov 03 19:57:52.063152 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:52.063119 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d722f54a-ftbll"] Nov 03 19:57:52.072263 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:52.072235 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d722f54a-ftbll"] Nov 03 19:57:52.676721 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:52.676585 5059 generic.go:356] "Generic (PLEG): container finished" podID="23b4c1a1-b3ca-46a5-982f-0781472ccfa6" containerID="967ad750ff0377e4aa1d61cff611774e468201221e110f8d8fad5e5d28e4706a" exitCode=0 Nov 03 19:57:52.676954 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:52.676800 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" event={"ID":"23b4c1a1-b3ca-46a5-982f-0781472ccfa6","Type":"ContainerDied","Data":"967ad750ff0377e4aa1d61cff611774e468201221e110f8d8fad5e5d28e4706a"} Nov 03 19:57:52.677378 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:52.677351 5059 scope.go:122] "RemoveContainer" containerID="967ad750ff0377e4aa1d61cff611774e468201221e110f8d8fad5e5d28e4706a" Nov 03 19:57:52.681932 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:52.681900 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:57:52.682003 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:52.681989 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:57:52.682051 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:52.682035 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:57:53.695972 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:53.695941 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" event={"ID":"23b4c1a1-b3ca-46a5-982f-0781472ccfa6","Type":"ContainerStarted","Data":"4d81cf7be3bd9e6866c51d25719e9c5d695accded2632ca78991834db2f0fb22"} Nov 03 19:57:53.868687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:53.860389 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c55f0fac-4308-423f-bc06-994606b40c14" path="/var/lib/kubelet/pods/c55f0fac-4308-423f-bc06-994606b40c14/volumes" Nov 03 19:57:54.239854 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:54.238060 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:57:54.239854 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:54.238226 5059 patch_prober.go:29] interesting pod/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 03 19:57:54.239854 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:54.238273 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" podUID="23b4c1a1-b3ca-46a5-982f-0781472ccfa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 03 19:57:55.107284 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.107256 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:57:55.137110 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.137066 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-workload-socket\") pod \"0e53359d-3f88-4a9c-b50a-050984844949\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " Nov 03 19:57:55.137279 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.137129 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e53359d-3f88-4a9c-b50a-050984844949-kube-api-access-kd9vp\" (UniqueName: \"kubernetes.io/projected/0e53359d-3f88-4a9c-b50a-050984844949-kube-api-access-kd9vp\") pod \"0e53359d-3f88-4a9c-b50a-050984844949\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " Nov 03 19:57:55.137279 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.137255 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-istio-envoy\") pod \"0e53359d-3f88-4a9c-b50a-050984844949\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " Nov 03 19:57:55.137395 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.137311 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-credential-socket\") pod \"0e53359d-3f88-4a9c-b50a-050984844949\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " Nov 03 19:57:55.137395 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.137356 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e53359d-3f88-4a9c-b50a-050984844949-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e53359d-3f88-4a9c-b50a-050984844949-istio-podinfo\") pod \"0e53359d-3f88-4a9c-b50a-050984844949\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " Nov 03 19:57:55.137517 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.137438 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e53359d-3f88-4a9c-b50a-050984844949-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e53359d-3f88-4a9c-b50a-050984844949-istiod-ca-cert\") pod \"0e53359d-3f88-4a9c-b50a-050984844949\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " Nov 03 19:57:55.137517 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.137472 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-istio-data\") pod \"0e53359d-3f88-4a9c-b50a-050984844949\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " Nov 03 19:57:55.137517 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.137512 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-workload-certs\") pod \"0e53359d-3f88-4a9c-b50a-050984844949\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " Nov 03 19:57:55.137706 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.137680 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e53359d-3f88-4a9c-b50a-050984844949-istio-token\" (UniqueName: \"kubernetes.io/projected/0e53359d-3f88-4a9c-b50a-050984844949-istio-token\") pod \"0e53359d-3f88-4a9c-b50a-050984844949\" (UID: \"0e53359d-3f88-4a9c-b50a-050984844949\") " Nov 03 19:57:55.138048 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.138022 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-credential-socket" pod "0e53359d-3f88-4a9c-b50a-050984844949" (UID: "0e53359d-3f88-4a9c-b50a-050984844949"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:55.138402 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.138379 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:55.138779 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.138758 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-workload-socket" pod "0e53359d-3f88-4a9c-b50a-050984844949" (UID: "0e53359d-3f88-4a9c-b50a-050984844949"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:55.139808 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.139353 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-istio-data" pod "0e53359d-3f88-4a9c-b50a-050984844949" (UID: "0e53359d-3f88-4a9c-b50a-050984844949"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:55.139808 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.139539 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-workload-certs" pod "0e53359d-3f88-4a9c-b50a-050984844949" (UID: "0e53359d-3f88-4a9c-b50a-050984844949"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:55.140636 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.140615 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e53359d-3f88-4a9c-b50a-050984844949-istiod-ca-cert" pod "0e53359d-3f88-4a9c-b50a-050984844949" (UID: "0e53359d-3f88-4a9c-b50a-050984844949"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:57:55.152104 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.151809 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-istio-envoy" pod "0e53359d-3f88-4a9c-b50a-050984844949" (UID: "0e53359d-3f88-4a9c-b50a-050984844949"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:55.152104 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.151881 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e53359d-3f88-4a9c-b50a-050984844949-istio-podinfo" pod "0e53359d-3f88-4a9c-b50a-050984844949" (UID: "0e53359d-3f88-4a9c-b50a-050984844949"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:57:55.152104 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.151979 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e53359d-3f88-4a9c-b50a-050984844949-istio-token" pod "0e53359d-3f88-4a9c-b50a-050984844949" (UID: "0e53359d-3f88-4a9c-b50a-050984844949"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:57:55.152520 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.152501 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e53359d-3f88-4a9c-b50a-050984844949-kube-api-access-kd9vp" pod "0e53359d-3f88-4a9c-b50a-050984844949" (UID: "0e53359d-3f88-4a9c-b50a-050984844949"). InnerVolumeSpecName "kube-api-access-kd9vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:57:55.232406 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.232373 5059 patch_prober.go:29] interesting pod/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 03 19:57:55.232558 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.232437 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" podUID="23b4c1a1-b3ca-46a5-982f-0781472ccfa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 03 19:57:55.239840 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.239808 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:55.239959 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.239843 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e53359d-3f88-4a9c-b50a-050984844949-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:55.239959 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.239858 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e53359d-3f88-4a9c-b50a-050984844949-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:55.239959 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.239873 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:55.239959 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.239887 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:55.239959 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.239899 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e53359d-3f88-4a9c-b50a-050984844949-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:55.239959 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.239912 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e53359d-3f88-4a9c-b50a-050984844949-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:55.239959 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.239925 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd9vp\" (UniqueName: \"kubernetes.io/projected/0e53359d-3f88-4a9c-b50a-050984844949-kube-api-access-kd9vp\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:55.596820 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.596796 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:57:55.652426 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.652389 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-podinfo\") pod \"3a73f1d8-301f-40db-91e8-b4963410e1a0\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " Nov 03 19:57:55.652786 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.652761 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-credential-socket\") pod \"3a73f1d8-301f-40db-91e8-b4963410e1a0\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " Nov 03 19:57:55.652952 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.652935 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-token\" (UniqueName: \"kubernetes.io/projected/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-token\") pod \"3a73f1d8-301f-40db-91e8-b4963410e1a0\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " Nov 03 19:57:55.653100 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.653084 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-workload-certs\") pod \"3a73f1d8-301f-40db-91e8-b4963410e1a0\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " Nov 03 19:57:55.653231 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.653215 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-envoy\") pod \"3a73f1d8-301f-40db-91e8-b4963410e1a0\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " Nov 03 19:57:55.653384 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.653369 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-data\") pod \"3a73f1d8-301f-40db-91e8-b4963410e1a0\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " Nov 03 19:57:55.653520 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.653504 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a73f1d8-301f-40db-91e8-b4963410e1a0-kube-api-access-gjjmx\" (UniqueName: \"kubernetes.io/projected/3a73f1d8-301f-40db-91e8-b4963410e1a0-kube-api-access-gjjmx\") pod \"3a73f1d8-301f-40db-91e8-b4963410e1a0\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " Nov 03 19:57:55.653689 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.653652 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a73f1d8-301f-40db-91e8-b4963410e1a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a73f1d8-301f-40db-91e8-b4963410e1a0-istiod-ca-cert\") pod \"3a73f1d8-301f-40db-91e8-b4963410e1a0\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " Nov 03 19:57:55.653832 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.653817 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-workload-socket\") pod \"3a73f1d8-301f-40db-91e8-b4963410e1a0\" (UID: \"3a73f1d8-301f-40db-91e8-b4963410e1a0\") " Nov 03 19:57:55.654895 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.654873 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-workload-socket" pod "3a73f1d8-301f-40db-91e8-b4963410e1a0" (UID: "3a73f1d8-301f-40db-91e8-b4963410e1a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:55.657159 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.656284 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-data" pod "3a73f1d8-301f-40db-91e8-b4963410e1a0" (UID: "3a73f1d8-301f-40db-91e8-b4963410e1a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:55.657159 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.656641 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a73f1d8-301f-40db-91e8-b4963410e1a0-istiod-ca-cert" pod "3a73f1d8-301f-40db-91e8-b4963410e1a0" (UID: "3a73f1d8-301f-40db-91e8-b4963410e1a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:57:55.657159 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.656912 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-workload-certs" pod "3a73f1d8-301f-40db-91e8-b4963410e1a0" (UID: "3a73f1d8-301f-40db-91e8-b4963410e1a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:55.657159 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.657135 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-credential-socket" pod "3a73f1d8-301f-40db-91e8-b4963410e1a0" (UID: "3a73f1d8-301f-40db-91e8-b4963410e1a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:55.689069 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.689040 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-envoy" pod "3a73f1d8-301f-40db-91e8-b4963410e1a0" (UID: "3a73f1d8-301f-40db-91e8-b4963410e1a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:55.689238 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.689067 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-podinfo" pod "3a73f1d8-301f-40db-91e8-b4963410e1a0" (UID: "3a73f1d8-301f-40db-91e8-b4963410e1a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:57:55.689238 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.689090 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a73f1d8-301f-40db-91e8-b4963410e1a0-kube-api-access-gjjmx" pod "3a73f1d8-301f-40db-91e8-b4963410e1a0" (UID: "3a73f1d8-301f-40db-91e8-b4963410e1a0"). InnerVolumeSpecName "kube-api-access-gjjmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:57:55.690567 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.690531 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-token" pod "3a73f1d8-301f-40db-91e8-b4963410e1a0" (UID: "3a73f1d8-301f-40db-91e8-b4963410e1a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:57:55.715041 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.715016 5059 generic.go:356] "Generic (PLEG): container finished" podID="3a73f1d8-301f-40db-91e8-b4963410e1a0" containerID="c44d5229a7ea4579d97216bf3a68c0945f698e93fa10deb42a260172c495bada" exitCode=0 Nov 03 19:57:55.715149 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.715135 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" event={"ID":"3a73f1d8-301f-40db-91e8-b4963410e1a0","Type":"ContainerDied","Data":"c44d5229a7ea4579d97216bf3a68c0945f698e93fa10deb42a260172c495bada"} Nov 03 19:57:55.715194 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.715176 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" event={"ID":"3a73f1d8-301f-40db-91e8-b4963410e1a0","Type":"ContainerDied","Data":"d6c40070166be85674df1b5ba4d1477a146efb5f13c6855289fc790faa6b519b"} Nov 03 19:57:55.715231 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.715197 5059 scope.go:122] "RemoveContainer" containerID="c44d5229a7ea4579d97216bf3a68c0945f698e93fa10deb42a260172c495bada" Nov 03 19:57:55.715314 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.715305 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl" Nov 03 19:57:55.725292 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.725264 5059 generic.go:356] "Generic (PLEG): container finished" podID="0e53359d-3f88-4a9c-b50a-050984844949" containerID="61d4bffc78086245d3366e6ac7434f3e64b57bca64e4d3b1bfa2cc08dc31fded" exitCode=0 Nov 03 19:57:55.725390 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.725377 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" event={"ID":"0e53359d-3f88-4a9c-b50a-050984844949","Type":"ContainerDied","Data":"61d4bffc78086245d3366e6ac7434f3e64b57bca64e4d3b1bfa2cc08dc31fded"} Nov 03 19:57:55.725434 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.725413 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" event={"ID":"0e53359d-3f88-4a9c-b50a-050984844949","Type":"ContainerDied","Data":"7830b1eea2e42b5af9da0c974f0b976a19dbf7c8be5f25f7d8e50807e7ff5470"} Nov 03 19:57:55.725472 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.725463 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c" Nov 03 19:57:55.740829 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.740806 5059 scope.go:122] "RemoveContainer" containerID="c44d5229a7ea4579d97216bf3a68c0945f698e93fa10deb42a260172c495bada" Nov 03 19:57:55.741531 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:57:55.741503 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44d5229a7ea4579d97216bf3a68c0945f698e93fa10deb42a260172c495bada\": container with ID starting with c44d5229a7ea4579d97216bf3a68c0945f698e93fa10deb42a260172c495bada not found: ID does not exist" containerID="c44d5229a7ea4579d97216bf3a68c0945f698e93fa10deb42a260172c495bada" Nov 03 19:57:55.741610 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.741537 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44d5229a7ea4579d97216bf3a68c0945f698e93fa10deb42a260172c495bada"} err="failed to get container status \"c44d5229a7ea4579d97216bf3a68c0945f698e93fa10deb42a260172c495bada\": rpc error: code = NotFound desc = could not find container \"c44d5229a7ea4579d97216bf3a68c0945f698e93fa10deb42a260172c495bada\": container with ID starting with c44d5229a7ea4579d97216bf3a68c0945f698e93fa10deb42a260172c495bada not found: ID does not exist" Nov 03 19:57:55.741610 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.741553 5059 scope.go:122] "RemoveContainer" containerID="61d4bffc78086245d3366e6ac7434f3e64b57bca64e4d3b1bfa2cc08dc31fded" Nov 03 19:57:55.791727 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.785829 5059 scope.go:122] "RemoveContainer" containerID="61d4bffc78086245d3366e6ac7434f3e64b57bca64e4d3b1bfa2cc08dc31fded" Nov 03 19:57:55.799648 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.799212 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:55.799648 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.799243 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:55.799648 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.799259 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:55.799648 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.799273 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:55.799648 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.799286 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:55.799648 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.799299 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:55.799648 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.799312 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjjmx\" (UniqueName: \"kubernetes.io/projected/3a73f1d8-301f-40db-91e8-b4963410e1a0-kube-api-access-gjjmx\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:55.799648 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.799343 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a73f1d8-301f-40db-91e8-b4963410e1a0-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:55.799648 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.799358 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a73f1d8-301f-40db-91e8-b4963410e1a0-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:55.800262 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:57:55.800236 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61d4bffc78086245d3366e6ac7434f3e64b57bca64e4d3b1bfa2cc08dc31fded\": container with ID starting with 61d4bffc78086245d3366e6ac7434f3e64b57bca64e4d3b1bfa2cc08dc31fded not found: ID does not exist" containerID="61d4bffc78086245d3366e6ac7434f3e64b57bca64e4d3b1bfa2cc08dc31fded" Nov 03 19:57:55.800311 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.800275 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61d4bffc78086245d3366e6ac7434f3e64b57bca64e4d3b1bfa2cc08dc31fded"} err="failed to get container status \"61d4bffc78086245d3366e6ac7434f3e64b57bca64e4d3b1bfa2cc08dc31fded\": rpc error: code = NotFound desc = could not find container \"61d4bffc78086245d3366e6ac7434f3e64b57bca64e4d3b1bfa2cc08dc31fded\": container with ID starting with 61d4bffc78086245d3366e6ac7434f3e64b57bca64e4d3b1bfa2cc08dc31fded not found: ID does not exist" Nov 03 19:57:55.800732 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.800713 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl"] Nov 03 19:57:55.805578 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.805557 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xhqjl"] Nov 03 19:57:55.815291 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.815271 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c"] Nov 03 19:57:55.820466 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.820444 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-7mt5c"] Nov 03 19:57:55.886815 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.886738 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e53359d-3f88-4a9c-b50a-050984844949" path="/var/lib/kubelet/pods/0e53359d-3f88-4a9c-b50a-050984844949/volumes" Nov 03 19:57:55.887283 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:55.887268 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a73f1d8-301f-40db-91e8-b4963410e1a0" path="/var/lib/kubelet/pods/3a73f1d8-301f-40db-91e8-b4963410e1a0/volumes" Nov 03 19:57:56.233081 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:56.232999 5059 patch_prober.go:29] interesting pod/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 03 19:57:56.233081 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:56.233062 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" podUID="23b4c1a1-b3ca-46a5-982f-0781472ccfa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 03 19:57:57.247142 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:57.241465 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:57:57.253567 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:57.252687 5059 patch_prober.go:29] interesting pod/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 03 19:57:57.253567 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:57.252749 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" podUID="23b4c1a1-b3ca-46a5-982f-0781472ccfa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 03 19:57:57.305400 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:57.305370 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:57:58.002102 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:58.002067 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:57:58.254759 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:58.253239 5059 patch_prober.go:29] interesting pod/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 03 19:57:58.254759 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:58.253317 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" podUID="23b4c1a1-b3ca-46a5-982f-0781472ccfa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 03 19:57:59.429714 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:59.429657 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:57:59.431891 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:59.431867 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:57:59.433762 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:57:59.433735 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:58:03.026585 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:03.026547 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7820f1dc-g629w"] Nov 03 19:58:03.030738 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:03.030709 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7820f1dc-g629w"] Nov 03 19:58:03.860644 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:03.860611 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61622474-878f-4bf9-bb70-e8e4ce4e2bf0" path="/var/lib/kubelet/pods/61622474-878f-4bf9-bb70-e8e4ce4e2bf0/volumes" Nov 03 19:58:08.048723 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:08.048686 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e4f739-np4pk"] Nov 03 19:58:08.053674 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:08.053631 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e4f739-np4pk"] Nov 03 19:58:09.859497 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:09.859460 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29f9fdd0-72c1-46b1-888a-b413f9756773" path="/var/lib/kubelet/pods/29f9fdd0-72c1-46b1-888a-b413f9756773/volumes" Nov 03 19:58:10.606534 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:58:10.606501 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:58:15.702131 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:15.702092 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc"] Nov 03 19:58:15.702636 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:15.702309 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc" podUID="e1e5ddf9-f49d-4c38-bd61-955f3106fd9c" containerName="payload-processing" containerID="cri-o://6ce214f2b3f09dd2bdfc2c7aba5baf90166ae4a6a209973ee958956fc00c02d7" gracePeriod=30 Nov 03 19:58:15.712904 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:15.712744 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb"] Nov 03 19:58:15.713914 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:15.713518 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb" podUID="c1f480c4-4dd7-4622-b480-a5059adc4abb" containerName="payload-pre-processing" containerID="cri-o://99b8ec4f73a050506aa44d6d768433fdbccab298d6fa3e50832e4e1c88c7fc3d" gracePeriod=30 Nov 03 19:58:16.191949 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:16.191896 5059 generic.go:356] "Generic (PLEG): container finished" podID="e1e5ddf9-f49d-4c38-bd61-955f3106fd9c" containerID="6ce214f2b3f09dd2bdfc2c7aba5baf90166ae4a6a209973ee958956fc00c02d7" exitCode=0 Nov 03 19:58:16.192351 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:16.192327 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc" event={"ID":"e1e5ddf9-f49d-4c38-bd61-955f3106fd9c","Type":"ContainerDied","Data":"6ce214f2b3f09dd2bdfc2c7aba5baf90166ae4a6a209973ee958956fc00c02d7"} Nov 03 19:58:16.195017 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:16.194992 5059 generic.go:356] "Generic (PLEG): container finished" podID="c1f480c4-4dd7-4622-b480-a5059adc4abb" containerID="99b8ec4f73a050506aa44d6d768433fdbccab298d6fa3e50832e4e1c88c7fc3d" exitCode=0 Nov 03 19:58:16.195266 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:16.195251 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb" event={"ID":"c1f480c4-4dd7-4622-b480-a5059adc4abb","Type":"ContainerDied","Data":"99b8ec4f73a050506aa44d6d768433fdbccab298d6fa3e50832e4e1c88c7fc3d"} Nov 03 19:58:16.260371 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:16.260336 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc" Nov 03 19:58:16.321431 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:16.321398 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1e5ddf9-f49d-4c38-bd61-955f3106fd9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e5ddf9-f49d-4c38-bd61-955f3106fd9c-plugins-config-volume\") pod \"e1e5ddf9-f49d-4c38-bd61-955f3106fd9c\" (UID: \"e1e5ddf9-f49d-4c38-bd61-955f3106fd9c\") " Nov 03 19:58:16.321552 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:16.321480 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1e5ddf9-f49d-4c38-bd61-955f3106fd9c-kube-api-access-4qp65\" (UniqueName: \"kubernetes.io/projected/e1e5ddf9-f49d-4c38-bd61-955f3106fd9c-kube-api-access-4qp65\") pod \"e1e5ddf9-f49d-4c38-bd61-955f3106fd9c\" (UID: \"e1e5ddf9-f49d-4c38-bd61-955f3106fd9c\") " Nov 03 19:58:16.322036 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:16.322001 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e5ddf9-f49d-4c38-bd61-955f3106fd9c-plugins-config-volume" pod "e1e5ddf9-f49d-4c38-bd61-955f3106fd9c" (UID: "e1e5ddf9-f49d-4c38-bd61-955f3106fd9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:16.322383 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:16.322351 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e5ddf9-f49d-4c38-bd61-955f3106fd9c-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:16.324644 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:16.324619 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e5ddf9-f49d-4c38-bd61-955f3106fd9c-kube-api-access-4qp65" pod "e1e5ddf9-f49d-4c38-bd61-955f3106fd9c" (UID: "e1e5ddf9-f49d-4c38-bd61-955f3106fd9c"). InnerVolumeSpecName "kube-api-access-4qp65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:16.337283 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:16.337261 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb" Nov 03 19:58:16.426012 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:16.424272 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f480c4-4dd7-4622-b480-a5059adc4abb-kube-api-access-k8mlm\" (UniqueName: \"kubernetes.io/projected/c1f480c4-4dd7-4622-b480-a5059adc4abb-kube-api-access-k8mlm\") pod \"c1f480c4-4dd7-4622-b480-a5059adc4abb\" (UID: \"c1f480c4-4dd7-4622-b480-a5059adc4abb\") " Nov 03 19:58:16.426012 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:16.424330 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1f480c4-4dd7-4622-b480-a5059adc4abb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f480c4-4dd7-4622-b480-a5059adc4abb-plugins-config-volume\") pod \"c1f480c4-4dd7-4622-b480-a5059adc4abb\" (UID: \"c1f480c4-4dd7-4622-b480-a5059adc4abb\") " Nov 03 19:58:16.426012 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:16.425231 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qp65\" (UniqueName: \"kubernetes.io/projected/e1e5ddf9-f49d-4c38-bd61-955f3106fd9c-kube-api-access-4qp65\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:16.426724 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:16.426650 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1f480c4-4dd7-4622-b480-a5059adc4abb-plugins-config-volume" pod "c1f480c4-4dd7-4622-b480-a5059adc4abb" (UID: "c1f480c4-4dd7-4622-b480-a5059adc4abb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:16.429453 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:16.429420 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f480c4-4dd7-4622-b480-a5059adc4abb-kube-api-access-k8mlm" pod "c1f480c4-4dd7-4622-b480-a5059adc4abb" (UID: "c1f480c4-4dd7-4622-b480-a5059adc4abb"). InnerVolumeSpecName "kube-api-access-k8mlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:16.526695 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:16.526628 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8mlm\" (UniqueName: \"kubernetes.io/projected/c1f480c4-4dd7-4622-b480-a5059adc4abb-kube-api-access-k8mlm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:16.526695 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:16.526656 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1f480c4-4dd7-4622-b480-a5059adc4abb-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:17.093826 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.093788 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz"] Nov 03 19:58:17.094714 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.094693 5059 cpu_manager.go:414] "containerMap: removing container" podUID="0e53359d-3f88-4a9c-b50a-050984844949" containerName="istio-proxy" Nov 03 19:58:17.094714 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.094714 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e53359d-3f88-4a9c-b50a-050984844949" containerName="istio-proxy" Nov 03 19:58:17.094910 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.094765 5059 cpu_manager.go:414] "containerMap: removing container" podUID="3a73f1d8-301f-40db-91e8-b4963410e1a0" containerName="istio-proxy" Nov 03 19:58:17.094910 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.094773 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a73f1d8-301f-40db-91e8-b4963410e1a0" containerName="istio-proxy" Nov 03 19:58:17.094910 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.094795 5059 cpu_manager.go:414] "containerMap: removing container" podUID="e1e5ddf9-f49d-4c38-bd61-955f3106fd9c" containerName="payload-processing" Nov 03 19:58:17.094910 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.094803 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1e5ddf9-f49d-4c38-bd61-955f3106fd9c" containerName="payload-processing" Nov 03 19:58:17.094910 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.094820 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c1f480c4-4dd7-4622-b480-a5059adc4abb" containerName="payload-pre-processing" Nov 03 19:58:17.094910 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.094827 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f480c4-4dd7-4622-b480-a5059adc4abb" containerName="payload-pre-processing" Nov 03 19:58:17.095210 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.094978 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e53359d-3f88-4a9c-b50a-050984844949" containerName="istio-proxy" Nov 03 19:58:17.095210 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.095001 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a73f1d8-301f-40db-91e8-b4963410e1a0" containerName="istio-proxy" Nov 03 19:58:17.095210 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.095014 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1e5ddf9-f49d-4c38-bd61-955f3106fd9c" containerName="payload-processing" Nov 03 19:58:17.095210 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.095025 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1f480c4-4dd7-4622-b480-a5059adc4abb" containerName="payload-pre-processing" Nov 03 19:58:17.098987 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.098966 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz" Nov 03 19:58:17.104106 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.104079 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq"] Nov 03 19:58:17.108353 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.108332 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq" Nov 03 19:58:17.111143 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.111120 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3b6e2b\"" Nov 03 19:58:17.115323 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.115304 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3b6e2b-dockercfg-dbmmn\"" Nov 03 19:58:17.116890 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.116866 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz"] Nov 03 19:58:17.131825 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.131800 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq"] Nov 03 19:58:17.135453 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.135430 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32e5da96-23df-4b40-a08e-7b2820065df5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq\" (UID: \"32e5da96-23df-4b40-a08e-7b2820065df5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq" Nov 03 19:58:17.135570 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.135461 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5wc5\" (UniqueName: \"kubernetes.io/projected/32e5da96-23df-4b40-a08e-7b2820065df5-kube-api-access-f5wc5\") pod \"payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq\" (UID: \"32e5da96-23df-4b40-a08e-7b2820065df5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq" Nov 03 19:58:17.135570 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.135485 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg75s\" (UniqueName: \"kubernetes.io/projected/651eb1ff-2364-47df-96b5-5ae746240b93-kube-api-access-tg75s\") pod \"payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz\" (UID: \"651eb1ff-2364-47df-96b5-5ae746240b93\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz" Nov 03 19:58:17.135734 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.135598 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651eb1ff-2364-47df-96b5-5ae746240b93-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz\" (UID: \"651eb1ff-2364-47df-96b5-5ae746240b93\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz" Nov 03 19:58:17.255739 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.253913 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32e5da96-23df-4b40-a08e-7b2820065df5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq\" (UID: \"32e5da96-23df-4b40-a08e-7b2820065df5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq" Nov 03 19:58:17.255739 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.253973 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5wc5\" (UniqueName: \"kubernetes.io/projected/32e5da96-23df-4b40-a08e-7b2820065df5-kube-api-access-f5wc5\") pod \"payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq\" (UID: \"32e5da96-23df-4b40-a08e-7b2820065df5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq" Nov 03 19:58:17.255739 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.254005 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg75s\" (UniqueName: \"kubernetes.io/projected/651eb1ff-2364-47df-96b5-5ae746240b93-kube-api-access-tg75s\") pod \"payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz\" (UID: \"651eb1ff-2364-47df-96b5-5ae746240b93\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz" Nov 03 19:58:17.255739 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.254120 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651eb1ff-2364-47df-96b5-5ae746240b93-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz\" (UID: \"651eb1ff-2364-47df-96b5-5ae746240b93\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz" Nov 03 19:58:17.255739 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.255342 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651eb1ff-2364-47df-96b5-5ae746240b93-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz\" (UID: \"651eb1ff-2364-47df-96b5-5ae746240b93\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz" Nov 03 19:58:17.256201 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.256174 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32e5da96-23df-4b40-a08e-7b2820065df5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq\" (UID: \"32e5da96-23df-4b40-a08e-7b2820065df5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq" Nov 03 19:58:17.310684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.289705 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg75s\" (UniqueName: \"kubernetes.io/projected/651eb1ff-2364-47df-96b5-5ae746240b93-kube-api-access-tg75s\") pod \"payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz\" (UID: \"651eb1ff-2364-47df-96b5-5ae746240b93\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz" Nov 03 19:58:17.310684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.289832 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5wc5\" (UniqueName: \"kubernetes.io/projected/32e5da96-23df-4b40-a08e-7b2820065df5-kube-api-access-f5wc5\") pod \"payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq\" (UID: \"32e5da96-23df-4b40-a08e-7b2820065df5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq" Nov 03 19:58:17.310684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.298277 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc" event={"ID":"e1e5ddf9-f49d-4c38-bd61-955f3106fd9c","Type":"ContainerDied","Data":"72f1e131db82a9d04fe1a935327831d092eea2ad230aa7f48750717e378a9d49"} Nov 03 19:58:17.310684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.298343 5059 scope.go:122] "RemoveContainer" containerID="6ce214f2b3f09dd2bdfc2c7aba5baf90166ae4a6a209973ee958956fc00c02d7" Nov 03 19:58:17.310684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.298515 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc" Nov 03 19:58:17.311448 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.311320 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb" event={"ID":"c1f480c4-4dd7-4622-b480-a5059adc4abb","Type":"ContainerDied","Data":"7312c4affde3be8ac2b56324e9a61ef88f95f37f3490da366b1a9907781eebba"} Nov 03 19:58:17.311541 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.311477 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb" Nov 03 19:58:17.354991 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.352853 5059 scope.go:122] "RemoveContainer" containerID="99b8ec4f73a050506aa44d6d768433fdbccab298d6fa3e50832e4e1c88c7fc3d" Nov 03 19:58:17.380681 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.364047 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc"] Nov 03 19:58:17.380681 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.372631 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-7fbd7d57f4-7lljc"] Nov 03 19:58:17.405549 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.405516 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb"] Nov 03 19:58:17.411736 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.409815 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-67fb7cfd79-v9sjb"] Nov 03 19:58:17.421715 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.417848 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz" Nov 03 19:58:17.428179 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.426293 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq" Nov 03 19:58:17.796114 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.791598 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq"] Nov 03 19:58:17.830007 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.829976 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz"] Nov 03 19:58:17.833936 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:58:17.833906 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651eb1ff_2364_47df_96b5_5ae746240b93.slice/crio-113f30cd75fd14c486cd636f48d5f86808f2f402be1223a01413382023c30234 WatchSource:0}: Error finding container 113f30cd75fd14c486cd636f48d5f86808f2f402be1223a01413382023c30234: Status 404 returned error can't find the container with id 113f30cd75fd14c486cd636f48d5f86808f2f402be1223a01413382023c30234 Nov 03 19:58:17.880722 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.880687 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1f480c4-4dd7-4622-b480-a5059adc4abb" path="/var/lib/kubelet/pods/c1f480c4-4dd7-4622-b480-a5059adc4abb/volumes" Nov 03 19:58:17.881651 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:17.881621 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1e5ddf9-f49d-4c38-bd61-955f3106fd9c" path="/var/lib/kubelet/pods/e1e5ddf9-f49d-4c38-bd61-955f3106fd9c/volumes" Nov 03 19:58:18.324414 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:18.324377 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz" event={"ID":"651eb1ff-2364-47df-96b5-5ae746240b93","Type":"ContainerStarted","Data":"03bf2f26a83d43a9b790f05e408c4cff1d6803400918cfa45262e8ecbbb54561"} Nov 03 19:58:18.324870 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:18.324431 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz" event={"ID":"651eb1ff-2364-47df-96b5-5ae746240b93","Type":"ContainerStarted","Data":"113f30cd75fd14c486cd636f48d5f86808f2f402be1223a01413382023c30234"} Nov 03 19:58:18.324870 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:18.324453 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz" Nov 03 19:58:18.326027 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:18.326001 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq" event={"ID":"32e5da96-23df-4b40-a08e-7b2820065df5","Type":"ContainerStarted","Data":"f72fbd00f6d07d8fa957ff9cc5b40a950f89f07b6f10d37ae3f632ca24c1469f"} Nov 03 19:58:18.326139 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:18.326047 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq" Nov 03 19:58:18.326139 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:18.326065 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq" event={"ID":"32e5da96-23df-4b40-a08e-7b2820065df5","Type":"ContainerStarted","Data":"fb8ac644ac69b76b76f9f4bae926f091ac917d63c65ff313475c32e274eb51ed"} Nov 03 19:58:18.349370 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:18.349317 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz" podStartSLOduration=1.349299577 podStartE2EDuration="1.349299577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:18.34421844 +0000 UTC m=+1292.602015709" watchObservedRunningTime="2026-11-03 19:58:18.349299577 +0000 UTC m=+1292.607096841" Nov 03 19:58:18.366015 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:18.365964 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq" podStartSLOduration=1.365948052 podStartE2EDuration="1.365948052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:18.364541712 +0000 UTC m=+1292.622338973" watchObservedRunningTime="2026-11-03 19:58:18.365948052 +0000 UTC m=+1292.623745311" Nov 03 19:58:19.984594 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:19.984555 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h"] Nov 03 19:58:19.985126 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:19.984819 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" podUID="5dd98cc4-6a97-4b79-90c6-7118b01966c4" containerName="istio-proxy" containerID="cri-o://673937df819966ed0cd13bd7341daeeeaeb5458f8d2020750cb7dac184b4e320" gracePeriod=30 Nov 03 19:58:24.026537 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:24.026499 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-361e6db7-k66z9"] Nov 03 19:58:24.030246 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:24.030220 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-361e6db7-k66z9"] Nov 03 19:58:25.279607 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.279581 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:58:25.383843 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.383806 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-token\" (UniqueName: \"kubernetes.io/projected/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-token\") pod \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " Nov 03 19:58:25.384015 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.383872 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-podinfo\") pod \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " Nov 03 19:58:25.384015 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.383937 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-data\") pod \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " Nov 03 19:58:25.384015 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.383973 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-workload-certs\") pod \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " Nov 03 19:58:25.384140 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.384022 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istiod-ca-cert\") pod \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " Nov 03 19:58:25.384140 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.384064 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-envoy\") pod \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " Nov 03 19:58:25.384140 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.384117 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dd98cc4-6a97-4b79-90c6-7118b01966c4-kube-api-access-xwqqw\" (UniqueName: \"kubernetes.io/projected/5dd98cc4-6a97-4b79-90c6-7118b01966c4-kube-api-access-xwqqw\") pod \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " Nov 03 19:58:25.384273 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.384245 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-workload-certs" pod "5dd98cc4-6a97-4b79-90c6-7118b01966c4" (UID: "5dd98cc4-6a97-4b79-90c6-7118b01966c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:25.384339 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.384259 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-data" pod "5dd98cc4-6a97-4b79-90c6-7118b01966c4" (UID: "5dd98cc4-6a97-4b79-90c6-7118b01966c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:25.384339 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.384269 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-credential-socket\") pod \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " Nov 03 19:58:25.384445 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.384348 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-workload-socket\") pod \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\" (UID: \"5dd98cc4-6a97-4b79-90c6-7118b01966c4\") " Nov 03 19:58:25.384503 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.384465 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-credential-socket" pod "5dd98cc4-6a97-4b79-90c6-7118b01966c4" (UID: "5dd98cc4-6a97-4b79-90c6-7118b01966c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:25.384734 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.384707 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-workload-socket" pod "5dd98cc4-6a97-4b79-90c6-7118b01966c4" (UID: "5dd98cc4-6a97-4b79-90c6-7118b01966c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:25.384855 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.384768 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istiod-ca-cert" pod "5dd98cc4-6a97-4b79-90c6-7118b01966c4" (UID: "5dd98cc4-6a97-4b79-90c6-7118b01966c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:25.385233 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.385211 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:25.385319 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.385240 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:25.385319 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.385254 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:25.385319 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.385269 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:25.385319 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.385282 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:25.386301 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.386275 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-token" pod "5dd98cc4-6a97-4b79-90c6-7118b01966c4" (UID: "5dd98cc4-6a97-4b79-90c6-7118b01966c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:25.386531 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.386513 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-envoy" pod "5dd98cc4-6a97-4b79-90c6-7118b01966c4" (UID: "5dd98cc4-6a97-4b79-90c6-7118b01966c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:25.386593 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.386530 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-podinfo" pod "5dd98cc4-6a97-4b79-90c6-7118b01966c4" (UID: "5dd98cc4-6a97-4b79-90c6-7118b01966c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:58:25.386731 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.386709 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd98cc4-6a97-4b79-90c6-7118b01966c4-kube-api-access-xwqqw" pod "5dd98cc4-6a97-4b79-90c6-7118b01966c4" (UID: "5dd98cc4-6a97-4b79-90c6-7118b01966c4"). InnerVolumeSpecName "kube-api-access-xwqqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:25.407250 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.407218 5059 generic.go:356] "Generic (PLEG): container finished" podID="5dd98cc4-6a97-4b79-90c6-7118b01966c4" containerID="673937df819966ed0cd13bd7341daeeeaeb5458f8d2020750cb7dac184b4e320" exitCode=0 Nov 03 19:58:25.407343 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.407266 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" Nov 03 19:58:25.407343 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.407268 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" event={"ID":"5dd98cc4-6a97-4b79-90c6-7118b01966c4","Type":"ContainerDied","Data":"673937df819966ed0cd13bd7341daeeeaeb5458f8d2020750cb7dac184b4e320"} Nov 03 19:58:25.407343 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.407327 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h" event={"ID":"5dd98cc4-6a97-4b79-90c6-7118b01966c4","Type":"ContainerDied","Data":"749ce945dc8c15080f39655443b04f11afa38a578f35358d39d58a9dd62892ca"} Nov 03 19:58:25.407446 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.407364 5059 scope.go:122] "RemoveContainer" containerID="673937df819966ed0cd13bd7341daeeeaeb5458f8d2020750cb7dac184b4e320" Nov 03 19:58:25.420364 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.420342 5059 scope.go:122] "RemoveContainer" containerID="715614d1ed23234dba3aadf5d1940c312a3de8d2840a50fac06793443a431813" Nov 03 19:58:25.430857 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.430838 5059 scope.go:122] "RemoveContainer" containerID="673937df819966ed0cd13bd7341daeeeaeb5458f8d2020750cb7dac184b4e320" Nov 03 19:58:25.431152 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:58:25.431127 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"673937df819966ed0cd13bd7341daeeeaeb5458f8d2020750cb7dac184b4e320\": container with ID starting with 673937df819966ed0cd13bd7341daeeeaeb5458f8d2020750cb7dac184b4e320 not found: ID does not exist" containerID="673937df819966ed0cd13bd7341daeeeaeb5458f8d2020750cb7dac184b4e320" Nov 03 19:58:25.431225 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.431162 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"673937df819966ed0cd13bd7341daeeeaeb5458f8d2020750cb7dac184b4e320"} err="failed to get container status \"673937df819966ed0cd13bd7341daeeeaeb5458f8d2020750cb7dac184b4e320\": rpc error: code = NotFound desc = could not find container \"673937df819966ed0cd13bd7341daeeeaeb5458f8d2020750cb7dac184b4e320\": container with ID starting with 673937df819966ed0cd13bd7341daeeeaeb5458f8d2020750cb7dac184b4e320 not found: ID does not exist" Nov 03 19:58:25.431225 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.431179 5059 scope.go:122] "RemoveContainer" containerID="715614d1ed23234dba3aadf5d1940c312a3de8d2840a50fac06793443a431813" Nov 03 19:58:25.431467 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:58:25.431448 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"715614d1ed23234dba3aadf5d1940c312a3de8d2840a50fac06793443a431813\": container with ID starting with 715614d1ed23234dba3aadf5d1940c312a3de8d2840a50fac06793443a431813 not found: ID does not exist" containerID="715614d1ed23234dba3aadf5d1940c312a3de8d2840a50fac06793443a431813" Nov 03 19:58:25.431517 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.431474 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"715614d1ed23234dba3aadf5d1940c312a3de8d2840a50fac06793443a431813"} err="failed to get container status \"715614d1ed23234dba3aadf5d1940c312a3de8d2840a50fac06793443a431813\": rpc error: code = NotFound desc = could not find container \"715614d1ed23234dba3aadf5d1940c312a3de8d2840a50fac06793443a431813\": container with ID starting with 715614d1ed23234dba3aadf5d1940c312a3de8d2840a50fac06793443a431813 not found: ID does not exist" Nov 03 19:58:25.434792 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.434768 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h"] Nov 03 19:58:25.439472 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.439453 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6cd7f4-openshift-default-84d8d46b99-8cw7h"] Nov 03 19:58:25.486845 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.486822 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:25.486845 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.486844 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:25.486969 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.486854 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd98cc4-6a97-4b79-90c6-7118b01966c4-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:25.486969 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.486863 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwqqw\" (UniqueName: \"kubernetes.io/projected/5dd98cc4-6a97-4b79-90c6-7118b01966c4-kube-api-access-xwqqw\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:25.864292 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.864200 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dd98cc4-6a97-4b79-90c6-7118b01966c4" path="/var/lib/kubelet/pods/5dd98cc4-6a97-4b79-90c6-7118b01966c4/volumes" Nov 03 19:58:25.865064 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:25.865034 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a" path="/var/lib/kubelet/pods/c53ca540-ed8f-49a9-b9a3-c2ffe2e2013a/volumes" Nov 03 19:58:29.337455 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:29.337415 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz" Nov 03 19:58:29.337989 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:29.337776 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq" Nov 03 19:58:31.032137 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:31.032098 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac00d17d-466c6"] Nov 03 19:58:31.037286 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:31.037256 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac00d17d-466c6"] Nov 03 19:58:31.867964 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:31.867928 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b1001f7-8d10-41a9-b180-e42edcdb8faf" path="/var/lib/kubelet/pods/6b1001f7-8d10-41a9-b180-e42edcdb8faf/volumes" Nov 03 19:58:33.631018 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.630983 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s"] Nov 03 19:58:33.631921 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.631900 5059 cpu_manager.go:414] "containerMap: removing container" podUID="5dd98cc4-6a97-4b79-90c6-7118b01966c4" containerName="istio-proxy" Nov 03 19:58:33.631921 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.631922 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dd98cc4-6a97-4b79-90c6-7118b01966c4" containerName="istio-proxy" Nov 03 19:58:33.632108 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.631954 5059 cpu_manager.go:414] "containerMap: removing container" podUID="5dd98cc4-6a97-4b79-90c6-7118b01966c4" containerName="istio-proxy" Nov 03 19:58:33.632108 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.631962 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dd98cc4-6a97-4b79-90c6-7118b01966c4" containerName="istio-proxy" Nov 03 19:58:33.632220 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.632190 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dd98cc4-6a97-4b79-90c6-7118b01966c4" containerName="istio-proxy" Nov 03 19:58:33.632220 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.632208 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dd98cc4-6a97-4b79-90c6-7118b01966c4" containerName="istio-proxy" Nov 03 19:58:33.636081 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.636056 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.658074 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.658050 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-f493f7-openshift-default-dockercfg-8kwp4\"" Nov 03 19:58:33.662831 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.662804 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s"] Nov 03 19:58:33.756896 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.756866 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-credential-socket\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.757089 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.756919 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-workload-socket\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.757089 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.757068 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-envoy\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.757234 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.757147 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-podinfo\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.757234 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.757202 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-workload-certs\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.757364 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.757349 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad642be-b0b8-4a68-9392-ec28c87264a3-istiod-ca-cert\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.757418 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.757386 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-token\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.757463 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.757452 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-data\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.757510 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.757480 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfn4t\" (UniqueName: \"kubernetes.io/projected/7ad642be-b0b8-4a68-9392-ec28c87264a3-kube-api-access-lfn4t\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.877298 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.875170 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad642be-b0b8-4a68-9392-ec28c87264a3-istiod-ca-cert\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.877298 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.875231 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-token\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.877298 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.875269 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-data\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.877298 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.875306 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfn4t\" (UniqueName: \"kubernetes.io/projected/7ad642be-b0b8-4a68-9392-ec28c87264a3-kube-api-access-lfn4t\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.877298 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.875415 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-credential-socket\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.877298 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.875463 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-workload-socket\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.877298 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.875544 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-envoy\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.877298 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.875615 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-podinfo\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.877298 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.875679 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-workload-certs\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.877298 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.876132 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-workload-certs\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.877298 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.876730 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad642be-b0b8-4a68-9392-ec28c87264a3-istiod-ca-cert\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.877298 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.877014 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-credential-socket\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.877298 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.877026 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-workload-socket\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.903987 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.877582 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-data\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.920842 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.918014 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-envoy\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.985694 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.971960 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfn4t\" (UniqueName: \"kubernetes.io/projected/7ad642be-b0b8-4a68-9392-ec28c87264a3-kube-api-access-lfn4t\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.985694 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.973396 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-podinfo\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:33.985694 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:33.982590 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-token\") pod \"e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:34.274009 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:34.273932 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:34.567404 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:34.567375 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s"] Nov 03 19:58:34.568739 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:58:34.568708 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad642be_b0b8_4a68_9392_ec28c87264a3.slice/crio-96d996aaa01afe9809b705054a293f987e93df8eca20bda1eed713607b99fc7b WatchSource:0}: Error finding container 96d996aaa01afe9809b705054a293f987e93df8eca20bda1eed713607b99fc7b: Status 404 returned error can't find the container with id 96d996aaa01afe9809b705054a293f987e93df8eca20bda1eed713607b99fc7b Nov 03 19:58:34.571562 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:34.571529 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:58:34.571635 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:34.571609 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:58:34.571698 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:34.571637 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:58:35.440932 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.440898 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz"] Nov 03 19:58:35.445012 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.444988 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.454878 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.454133 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-277dc9-openshift-default-dockercfg-r8kb9\"" Nov 03 19:58:35.464290 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.464267 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz"] Nov 03 19:58:35.503284 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.503254 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" event={"ID":"7ad642be-b0b8-4a68-9392-ec28c87264a3","Type":"ContainerStarted","Data":"70e22e24b3a0416be1693328143270603f0b640b05256d39bbdc39f123cbedc9"} Nov 03 19:58:35.503454 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.503317 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" event={"ID":"7ad642be-b0b8-4a68-9392-ec28c87264a3","Type":"ContainerStarted","Data":"96d996aaa01afe9809b705054a293f987e93df8eca20bda1eed713607b99fc7b"} Nov 03 19:58:35.518309 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.518281 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-envoy\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.518419 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.518323 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-data\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.518479 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.518436 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-workload-certs\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.518577 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.518564 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-workload-socket\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.518622 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.518591 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-token\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.518740 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.518724 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d94pw\" (UniqueName: \"kubernetes.io/projected/87c2d4eb-392d-4a66-8622-9ff5eee0b761-kube-api-access-d94pw\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.518804 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.518790 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istiod-ca-cert\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.518847 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.518821 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-credential-socket\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.518889 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.518851 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-podinfo\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.810757 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.810221 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d94pw\" (UniqueName: \"kubernetes.io/projected/87c2d4eb-392d-4a66-8622-9ff5eee0b761-kube-api-access-d94pw\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.811357 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.811072 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istiod-ca-cert\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.812082 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.812048 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-credential-socket\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.812258 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.811537 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-credential-socket\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.812751 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.812722 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-podinfo\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.812931 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.812907 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-envoy\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.820062 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.820036 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-data\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.820201 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.813888 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-data\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.867693 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.820881 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-workload-certs\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.877962 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.876934 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-envoy\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.897135 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.878849 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-workload-certs\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.897135 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.881274 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-podinfo\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.922952 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.902950 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d94pw\" (UniqueName: \"kubernetes.io/projected/87c2d4eb-392d-4a66-8622-9ff5eee0b761-kube-api-access-d94pw\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.922952 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.904054 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-workload-socket\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.922952 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.904476 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-workload-socket\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.922952 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.904544 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-token\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.922952 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.905228 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istiod-ca-cert\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:35.936412 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:35.936385 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-token\") pod \"e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:36.032033 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:36.031968 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" podStartSLOduration=3.031949933 podStartE2EDuration="3.031949933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:35.575000561 +0000 UTC m=+1309.832797822" watchObservedRunningTime="2026-11-03 19:58:36.031949933 +0000 UTC m=+1310.289747193" Nov 03 19:58:36.033913 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:36.033882 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f4349e5-7kcw6"] Nov 03 19:58:36.037779 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:36.037753 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f4349e5-7kcw6"] Nov 03 19:58:36.071231 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:36.071186 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:36.274144 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:36.274110 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:36.280071 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:36.280043 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:36.399267 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:36.399241 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz"] Nov 03 19:58:36.403621 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:58:36.403592 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c2d4eb_392d_4a66_8622_9ff5eee0b761.slice/crio-a420d5704886fa69470ca01c235a6cd33201750d1931351d4575322408a0fc3d WatchSource:0}: Error finding container a420d5704886fa69470ca01c235a6cd33201750d1931351d4575322408a0fc3d: Status 404 returned error can't find the container with id a420d5704886fa69470ca01c235a6cd33201750d1931351d4575322408a0fc3d Nov 03 19:58:36.409139 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:36.409090 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:58:36.409237 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:36.409191 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:58:36.409287 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:36.409233 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:58:36.520353 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:36.519264 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" event={"ID":"87c2d4eb-392d-4a66-8622-9ff5eee0b761","Type":"ContainerStarted","Data":"ff2052ce9b253a2c2fbda40231cab9fe34232bc1021d11e8648c5fee496ace71"} Nov 03 19:58:36.520353 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:36.519369 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" event={"ID":"87c2d4eb-392d-4a66-8622-9ff5eee0b761","Type":"ContainerStarted","Data":"a420d5704886fa69470ca01c235a6cd33201750d1931351d4575322408a0fc3d"} Nov 03 19:58:36.520353 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:36.520252 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:36.521797 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:36.521773 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:58:36.547138 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:36.547089 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" podStartSLOduration=1.5470750949999998 podStartE2EDuration="1.547075095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:36.544746362 +0000 UTC m=+1310.802543651" watchObservedRunningTime="2026-11-03 19:58:36.547075095 +0000 UTC m=+1310.804872351" Nov 03 19:58:37.071887 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:37.071858 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:37.077655 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:37.077627 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:37.527827 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:37.527741 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:37.528433 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:37.528415 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:58:37.859372 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:37.859331 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10d3059c-9b84-4630-9e19-e4eef5d458f1" path="/var/lib/kubelet/pods/10d3059c-9b84-4630-9e19-e4eef5d458f1/volumes" Nov 03 19:58:38.195893 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:38.195805 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk"] Nov 03 19:58:38.196086 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:38.196055 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" podUID="23b4c1a1-b3ca-46a5-982f-0781472ccfa6" containerName="istio-proxy" containerID="cri-o://4d81cf7be3bd9e6866c51d25719e9c5d695accded2632ca78991834db2f0fb22" gracePeriod=30 Nov 03 19:58:38.831649 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:38.831591 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl"] Nov 03 19:58:38.832243 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:38.831973 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl" podUID="978df79a-3b55-4b2e-8736-8d95fff03873" containerName="payload-pre-processing" containerID="cri-o://5305028fa3d14c77186070210eed9eeeb4b54feab85b25eae4c54149649b4af7" gracePeriod=30 Nov 03 19:58:39.294683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:39.289365 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl" Nov 03 19:58:39.415530 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:39.409403 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/978df79a-3b55-4b2e-8736-8d95fff03873-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978df79a-3b55-4b2e-8736-8d95fff03873-plugins-config-volume\") pod \"978df79a-3b55-4b2e-8736-8d95fff03873\" (UID: \"978df79a-3b55-4b2e-8736-8d95fff03873\") " Nov 03 19:58:39.415530 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:39.409548 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/978df79a-3b55-4b2e-8736-8d95fff03873-kube-api-access-d2fkv\" (UniqueName: \"kubernetes.io/projected/978df79a-3b55-4b2e-8736-8d95fff03873-kube-api-access-d2fkv\") pod \"978df79a-3b55-4b2e-8736-8d95fff03873\" (UID: \"978df79a-3b55-4b2e-8736-8d95fff03873\") " Nov 03 19:58:39.415530 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:39.410453 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978df79a-3b55-4b2e-8736-8d95fff03873-plugins-config-volume" pod "978df79a-3b55-4b2e-8736-8d95fff03873" (UID: "978df79a-3b55-4b2e-8736-8d95fff03873"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:39.415530 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:39.411333 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978df79a-3b55-4b2e-8736-8d95fff03873-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:39.421096 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:39.421066 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978df79a-3b55-4b2e-8736-8d95fff03873-kube-api-access-d2fkv" pod "978df79a-3b55-4b2e-8736-8d95fff03873" (UID: "978df79a-3b55-4b2e-8736-8d95fff03873"). InnerVolumeSpecName "kube-api-access-d2fkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:39.512602 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:39.512526 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2fkv\" (UniqueName: \"kubernetes.io/projected/978df79a-3b55-4b2e-8736-8d95fff03873-kube-api-access-d2fkv\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:39.549678 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:39.549629 5059 generic.go:356] "Generic (PLEG): container finished" podID="978df79a-3b55-4b2e-8736-8d95fff03873" containerID="5305028fa3d14c77186070210eed9eeeb4b54feab85b25eae4c54149649b4af7" exitCode=0 Nov 03 19:58:39.549823 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:39.549694 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl" Nov 03 19:58:39.549823 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:39.549724 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl" event={"ID":"978df79a-3b55-4b2e-8736-8d95fff03873","Type":"ContainerDied","Data":"5305028fa3d14c77186070210eed9eeeb4b54feab85b25eae4c54149649b4af7"} Nov 03 19:58:39.549823 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:39.549782 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl" event={"ID":"978df79a-3b55-4b2e-8736-8d95fff03873","Type":"ContainerDied","Data":"55371203a8a704bd153980042d105006bccc27d607e22a079a06a7875f94e612"} Nov 03 19:58:39.549823 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:39.549803 5059 scope.go:122] "RemoveContainer" containerID="5305028fa3d14c77186070210eed9eeeb4b54feab85b25eae4c54149649b4af7" Nov 03 19:58:39.562743 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:39.562716 5059 scope.go:122] "RemoveContainer" containerID="5305028fa3d14c77186070210eed9eeeb4b54feab85b25eae4c54149649b4af7" Nov 03 19:58:39.563059 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:58:39.563037 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5305028fa3d14c77186070210eed9eeeb4b54feab85b25eae4c54149649b4af7\": container with ID starting with 5305028fa3d14c77186070210eed9eeeb4b54feab85b25eae4c54149649b4af7 not found: ID does not exist" containerID="5305028fa3d14c77186070210eed9eeeb4b54feab85b25eae4c54149649b4af7" Nov 03 19:58:39.563133 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:39.563067 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5305028fa3d14c77186070210eed9eeeb4b54feab85b25eae4c54149649b4af7"} err="failed to get container status \"5305028fa3d14c77186070210eed9eeeb4b54feab85b25eae4c54149649b4af7\": rpc error: code = NotFound desc = could not find container \"5305028fa3d14c77186070210eed9eeeb4b54feab85b25eae4c54149649b4af7\": container with ID starting with 5305028fa3d14c77186070210eed9eeeb4b54feab85b25eae4c54149649b4af7 not found: ID does not exist" Nov 03 19:58:39.587431 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:39.587351 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl"] Nov 03 19:58:39.617138 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:39.617106 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-6b776b55dc-ss9pl"] Nov 03 19:58:39.862463 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:39.862428 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="978df79a-3b55-4b2e-8736-8d95fff03873" path="/var/lib/kubelet/pods/978df79a-3b55-4b2e-8736-8d95fff03873/volumes" Nov 03 19:58:40.070683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:40.066726 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e23cb3e8-47thd"] Nov 03 19:58:40.070683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:40.070561 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e23cb3e8-47thd"] Nov 03 19:58:40.610614 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:58:40.610576 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:58:41.858348 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:41.858313 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="914d427c-bf1c-43a4-836c-168626ee5397" path="/var/lib/kubelet/pods/914d427c-bf1c-43a4-836c-168626ee5397/volumes" Nov 03 19:58:43.509886 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.509858 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:58:43.523150 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.523125 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-kube-api-access-fdss4\" (UniqueName: \"kubernetes.io/projected/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-kube-api-access-fdss4\") pod \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " Nov 03 19:58:43.523264 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.523216 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-workload-certs\") pod \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " Nov 03 19:58:43.523334 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.523323 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-credential-socket\") pod \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " Nov 03 19:58:43.523382 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.523371 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istiod-ca-cert\") pod \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " Nov 03 19:58:43.523418 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.523395 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-workload-socket\") pod \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " Nov 03 19:58:43.523459 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.523421 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-envoy\") pod \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " Nov 03 19:58:43.523505 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.523485 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-data\") pod \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " Nov 03 19:58:43.523558 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.523496 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-workload-certs" pod "23b4c1a1-b3ca-46a5-982f-0781472ccfa6" (UID: "23b4c1a1-b3ca-46a5-982f-0781472ccfa6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:43.523558 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.523544 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-token\" (UniqueName: \"kubernetes.io/projected/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-token\") pod \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " Nov 03 19:58:43.523701 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.523615 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-credential-socket" pod "23b4c1a1-b3ca-46a5-982f-0781472ccfa6" (UID: "23b4c1a1-b3ca-46a5-982f-0781472ccfa6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:43.523701 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.523629 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-podinfo\") pod \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\" (UID: \"23b4c1a1-b3ca-46a5-982f-0781472ccfa6\") " Nov 03 19:58:43.523825 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.523698 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-workload-socket" pod "23b4c1a1-b3ca-46a5-982f-0781472ccfa6" (UID: "23b4c1a1-b3ca-46a5-982f-0781472ccfa6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:43.523825 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.523733 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istiod-ca-cert" pod "23b4c1a1-b3ca-46a5-982f-0781472ccfa6" (UID: "23b4c1a1-b3ca-46a5-982f-0781472ccfa6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:43.523929 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.523862 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-data" pod "23b4c1a1-b3ca-46a5-982f-0781472ccfa6" (UID: "23b4c1a1-b3ca-46a5-982f-0781472ccfa6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:43.524376 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.524277 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:43.524376 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.524306 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:43.524376 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.524324 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:43.524376 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.524340 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:43.524376 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.524354 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:43.525643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.525620 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-envoy" pod "23b4c1a1-b3ca-46a5-982f-0781472ccfa6" (UID: "23b4c1a1-b3ca-46a5-982f-0781472ccfa6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:43.525749 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.525650 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-kube-api-access-fdss4" pod "23b4c1a1-b3ca-46a5-982f-0781472ccfa6" (UID: "23b4c1a1-b3ca-46a5-982f-0781472ccfa6"). InnerVolumeSpecName "kube-api-access-fdss4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:43.525795 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.525767 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-podinfo" pod "23b4c1a1-b3ca-46a5-982f-0781472ccfa6" (UID: "23b4c1a1-b3ca-46a5-982f-0781472ccfa6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:58:43.525857 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.525839 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-token" pod "23b4c1a1-b3ca-46a5-982f-0781472ccfa6" (UID: "23b4c1a1-b3ca-46a5-982f-0781472ccfa6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:43.585542 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.585492 5059 generic.go:356] "Generic (PLEG): container finished" podID="23b4c1a1-b3ca-46a5-982f-0781472ccfa6" containerID="4d81cf7be3bd9e6866c51d25719e9c5d695accded2632ca78991834db2f0fb22" exitCode=0 Nov 03 19:58:43.585542 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.585540 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" Nov 03 19:58:43.585791 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.585559 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" event={"ID":"23b4c1a1-b3ca-46a5-982f-0781472ccfa6","Type":"ContainerDied","Data":"4d81cf7be3bd9e6866c51d25719e9c5d695accded2632ca78991834db2f0fb22"} Nov 03 19:58:43.585791 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.585619 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk" event={"ID":"23b4c1a1-b3ca-46a5-982f-0781472ccfa6","Type":"ContainerDied","Data":"3ae280fbf39328e52d7a008eafea01787ce9d2f33fe169aa6f52af1aa4697cc6"} Nov 03 19:58:43.585791 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.585650 5059 scope.go:122] "RemoveContainer" containerID="4d81cf7be3bd9e6866c51d25719e9c5d695accded2632ca78991834db2f0fb22" Nov 03 19:58:43.601643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.601621 5059 scope.go:122] "RemoveContainer" containerID="967ad750ff0377e4aa1d61cff611774e468201221e110f8d8fad5e5d28e4706a" Nov 03 19:58:43.613805 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.613781 5059 scope.go:122] "RemoveContainer" containerID="4d81cf7be3bd9e6866c51d25719e9c5d695accded2632ca78991834db2f0fb22" Nov 03 19:58:43.614092 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:58:43.614074 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d81cf7be3bd9e6866c51d25719e9c5d695accded2632ca78991834db2f0fb22\": container with ID starting with 4d81cf7be3bd9e6866c51d25719e9c5d695accded2632ca78991834db2f0fb22 not found: ID does not exist" containerID="4d81cf7be3bd9e6866c51d25719e9c5d695accded2632ca78991834db2f0fb22" Nov 03 19:58:43.614189 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.614099 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d81cf7be3bd9e6866c51d25719e9c5d695accded2632ca78991834db2f0fb22"} err="failed to get container status \"4d81cf7be3bd9e6866c51d25719e9c5d695accded2632ca78991834db2f0fb22\": rpc error: code = NotFound desc = could not find container \"4d81cf7be3bd9e6866c51d25719e9c5d695accded2632ca78991834db2f0fb22\": container with ID starting with 4d81cf7be3bd9e6866c51d25719e9c5d695accded2632ca78991834db2f0fb22 not found: ID does not exist" Nov 03 19:58:43.614189 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.614118 5059 scope.go:122] "RemoveContainer" containerID="967ad750ff0377e4aa1d61cff611774e468201221e110f8d8fad5e5d28e4706a" Nov 03 19:58:43.614423 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:58:43.614396 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"967ad750ff0377e4aa1d61cff611774e468201221e110f8d8fad5e5d28e4706a\": container with ID starting with 967ad750ff0377e4aa1d61cff611774e468201221e110f8d8fad5e5d28e4706a not found: ID does not exist" containerID="967ad750ff0377e4aa1d61cff611774e468201221e110f8d8fad5e5d28e4706a" Nov 03 19:58:43.614476 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.614430 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967ad750ff0377e4aa1d61cff611774e468201221e110f8d8fad5e5d28e4706a"} err="failed to get container status \"967ad750ff0377e4aa1d61cff611774e468201221e110f8d8fad5e5d28e4706a\": rpc error: code = NotFound desc = could not find container \"967ad750ff0377e4aa1d61cff611774e468201221e110f8d8fad5e5d28e4706a\": container with ID starting with 967ad750ff0377e4aa1d61cff611774e468201221e110f8d8fad5e5d28e4706a not found: ID does not exist" Nov 03 19:58:43.616973 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.616952 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk"] Nov 03 19:58:43.621324 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.621304 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6a7dc9-openshift-default-6dc86bd9c7-756xk"] Nov 03 19:58:43.625905 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.625883 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:43.625905 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.625906 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:43.626021 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.625916 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdss4\" (UniqueName: \"kubernetes.io/projected/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-kube-api-access-fdss4\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:43.626021 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.625928 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23b4c1a1-b3ca-46a5-982f-0781472ccfa6-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:43.860575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:43.860534 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23b4c1a1-b3ca-46a5-982f-0781472ccfa6" path="/var/lib/kubelet/pods/23b4c1a1-b3ca-46a5-982f-0781472ccfa6/volumes" Nov 03 19:58:44.732519 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:44.732482 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9"] Nov 03 19:58:44.732949 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:44.732734 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" podUID="c149bde6-19ca-4ce5-9685-c1cb12fe1740" containerName="istio-proxy" containerID="cri-o://2e7e20cce29e971c016746efa172864a6f8aa4433e310137cd128ff94cfaf22a" gracePeriod=30 Nov 03 19:58:46.991762 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:46.989147 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8"] Nov 03 19:58:47.007453 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:46.994246 5059 cpu_manager.go:414] "containerMap: removing container" podUID="23b4c1a1-b3ca-46a5-982f-0781472ccfa6" containerName="istio-proxy" Nov 03 19:58:47.007453 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:46.994271 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23b4c1a1-b3ca-46a5-982f-0781472ccfa6" containerName="istio-proxy" Nov 03 19:58:47.007453 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:46.994285 5059 cpu_manager.go:414] "containerMap: removing container" podUID="978df79a-3b55-4b2e-8736-8d95fff03873" containerName="payload-pre-processing" Nov 03 19:58:47.007453 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:46.994293 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="978df79a-3b55-4b2e-8736-8d95fff03873" containerName="payload-pre-processing" Nov 03 19:58:47.007453 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:46.994682 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="23b4c1a1-b3ca-46a5-982f-0781472ccfa6" containerName="istio-proxy" Nov 03 19:58:47.007453 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:46.994857 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="978df79a-3b55-4b2e-8736-8d95fff03873" containerName="payload-pre-processing" Nov 03 19:58:47.035726 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.009722 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8"] Nov 03 19:58:47.035726 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.009834 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.117685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.116804 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-token\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.117685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.116880 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-workload-socket\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.117685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.116938 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-workload-certs\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.117685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.117042 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/856d3ff7-8514-46b9-8884-f99b674dd6e6-istiod-ca-cert\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.117685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.117084 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ngjp\" (UniqueName: \"kubernetes.io/projected/856d3ff7-8514-46b9-8884-f99b674dd6e6-kube-api-access-8ngjp\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.117685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.117139 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-envoy\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.117685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.117183 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-podinfo\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.117685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.117554 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-data\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.117685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.117602 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-credential-socket\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.167894 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.165864 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-3faa07-openshift-default-dockercfg-7lkkf\"" Nov 03 19:58:47.188764 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.186911 5059 scope.go:122] "RemoveContainer" containerID="91c16f32baf87d5cbf46c52ad29a472d3bcba41d8f42bd22e8fd5e8960113c9a" Nov 03 19:58:47.212889 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.212762 5059 scope.go:122] "RemoveContainer" containerID="34d4fc10c1c858abdc76cfe27baafae85d7e2920066716b6747e0a3ea01e8c5c" Nov 03 19:58:47.220140 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.220102 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-envoy\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.220260 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.220145 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-podinfo\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.220260 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.220226 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-data\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.220384 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.220271 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-credential-socket\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.220438 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.220408 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-token\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.220495 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.220456 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-workload-socket\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.220551 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.220504 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-workload-certs\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.220681 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.220597 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/856d3ff7-8514-46b9-8884-f99b674dd6e6-istiod-ca-cert\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.220681 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.220629 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-data\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.221074 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.221053 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-credential-socket\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.221494 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.221474 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/856d3ff7-8514-46b9-8884-f99b674dd6e6-istiod-ca-cert\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.221683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.221652 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-workload-certs\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.221759 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.221716 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ngjp\" (UniqueName: \"kubernetes.io/projected/856d3ff7-8514-46b9-8884-f99b674dd6e6-kube-api-access-8ngjp\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.222130 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.222108 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-workload-socket\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.223628 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.223567 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-envoy\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.229700 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.225401 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-podinfo\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.233462 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.233444 5059 scope.go:122] "RemoveContainer" containerID="13779577fa40a32b08441e2d1eb1c8db93189edcd27b739284273db8de4cd358" Nov 03 19:58:47.260440 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.260420 5059 scope.go:122] "RemoveContainer" containerID="557d36a75a0b862d875b6ef517e2887667b563f9858880dacb18ba7d5eb26388" Nov 03 19:58:47.269773 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.269752 5059 scope.go:122] "RemoveContainer" containerID="0d325d1d6cb0e466855f3633d6c42e87c402787f80961edd9a8b1e39361399e8" Nov 03 19:58:47.288697 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.273116 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-token\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.288697 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.273181 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ngjp\" (UniqueName: \"kubernetes.io/projected/856d3ff7-8514-46b9-8884-f99b674dd6e6-kube-api-access-8ngjp\") pod \"e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.336568 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.336533 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:47.345509 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.345464 5059 scope.go:122] "RemoveContainer" containerID="7dff1f9587b4e2eaf860585edcee5f798f89c74d723d658603fc095d3d74f9ac" Nov 03 19:58:47.360879 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.360855 5059 scope.go:122] "RemoveContainer" containerID="b584bffd1d14ca3ade450f93059b444a13328671b77f2598cb2af2a34a34d9e3" Nov 03 19:58:47.378327 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.378295 5059 scope.go:122] "RemoveContainer" containerID="f82cdb32968c387a7fe51d66808be881c555ea2cb20f58f591b2aa51f7e27a51" Nov 03 19:58:47.390395 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.390375 5059 scope.go:122] "RemoveContainer" containerID="2da6adaacd3f5eebd686a6a724ad7e81b471cb64db18684a94146f78687ff606" Nov 03 19:58:47.397450 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.397424 5059 patch_prober.go:29] interesting pod/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:58:47.397575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.397492 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" podUID="c149bde6-19ca-4ce5-9685-c1cb12fe1740" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:58:47.403753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.403720 5059 scope.go:122] "RemoveContainer" containerID="99fd9fcecfbd449cf96d2c83c139b6ca85b554e622be3e8403a5c1b6000c9911" Nov 03 19:58:47.421713 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.421332 5059 scope.go:122] "RemoveContainer" containerID="3319d5ea3dcafc5635eeae15e137d718aa7e14cc6f1ebbb044c276b51e2330d6" Nov 03 19:58:47.651202 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.651173 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8"] Nov 03 19:58:47.653770 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:58:47.653737 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856d3ff7_8514_46b9_8884_f99b674dd6e6.slice/crio-39c3b03e0ec688c9c6957a7cb3a5346d6dbbb14d394b58b2710b232760035905 WatchSource:0}: Error finding container 39c3b03e0ec688c9c6957a7cb3a5346d6dbbb14d394b58b2710b232760035905: Status 404 returned error can't find the container with id 39c3b03e0ec688c9c6957a7cb3a5346d6dbbb14d394b58b2710b232760035905 Nov 03 19:58:47.656947 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.656913 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:58:47.657048 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.656983 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:58:47.657048 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:47.657015 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:58:48.665306 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:48.665271 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" event={"ID":"856d3ff7-8514-46b9-8884-f99b674dd6e6","Type":"ContainerStarted","Data":"4dfd6dd522fa140050dad5e5f74edeaeb56816f8fbb43c92919142c90af8a88b"} Nov 03 19:58:48.665770 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:48.665329 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" event={"ID":"856d3ff7-8514-46b9-8884-f99b674dd6e6","Type":"ContainerStarted","Data":"39c3b03e0ec688c9c6957a7cb3a5346d6dbbb14d394b58b2710b232760035905"} Nov 03 19:58:48.734894 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:48.734845 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" podStartSLOduration=2.734831889 podStartE2EDuration="2.734831889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:48.732182613 +0000 UTC m=+1322.989979870" watchObservedRunningTime="2026-11-03 19:58:48.734831889 +0000 UTC m=+1322.992629146" Nov 03 19:58:49.337415 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:49.337383 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:49.342564 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:49.342537 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:49.695967 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:49.695871 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:49.696906 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:49.696870 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:58:50.023120 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.023098 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:58:50.082596 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.082566 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-token\" (UniqueName: \"kubernetes.io/projected/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-token\") pod \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " Nov 03 19:58:50.082758 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.082612 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-podinfo\") pod \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " Nov 03 19:58:50.082758 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.082644 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istiod-ca-cert\") pod \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " Nov 03 19:58:50.082875 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.082796 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-workload-socket\") pod \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " Nov 03 19:58:50.082970 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.082944 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-data\") pod \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " Nov 03 19:58:50.083051 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.083020 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-workload-certs\") pod \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " Nov 03 19:58:50.083113 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.083071 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-workload-socket" pod "c149bde6-19ca-4ce5-9685-c1cb12fe1740" (UID: "c149bde6-19ca-4ce5-9685-c1cb12fe1740"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:50.083182 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.083166 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-envoy\") pod \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " Nov 03 19:58:50.083243 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.083196 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c149bde6-19ca-4ce5-9685-c1cb12fe1740-kube-api-access-2gp6r\" (UniqueName: \"kubernetes.io/projected/c149bde6-19ca-4ce5-9685-c1cb12fe1740-kube-api-access-2gp6r\") pod \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " Nov 03 19:58:50.083243 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.083199 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istiod-ca-cert" pod "c149bde6-19ca-4ce5-9685-c1cb12fe1740" (UID: "c149bde6-19ca-4ce5-9685-c1cb12fe1740"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:50.083490 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.083277 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-workload-certs" pod "c149bde6-19ca-4ce5-9685-c1cb12fe1740" (UID: "c149bde6-19ca-4ce5-9685-c1cb12fe1740"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:50.083490 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.083292 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-credential-socket\") pod \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\" (UID: \"c149bde6-19ca-4ce5-9685-c1cb12fe1740\") " Nov 03 19:58:50.083490 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.083455 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-data" pod "c149bde6-19ca-4ce5-9685-c1cb12fe1740" (UID: "c149bde6-19ca-4ce5-9685-c1cb12fe1740"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:50.084457 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.083855 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-credential-socket" pod "c149bde6-19ca-4ce5-9685-c1cb12fe1740" (UID: "c149bde6-19ca-4ce5-9685-c1cb12fe1740"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:50.084457 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.084052 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:50.084457 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.084075 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:50.084457 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.084092 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:50.084457 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.084106 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:50.084457 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.084120 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:50.085220 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.085192 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-podinfo" pod "c149bde6-19ca-4ce5-9685-c1cb12fe1740" (UID: "c149bde6-19ca-4ce5-9685-c1cb12fe1740"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:58:50.085643 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.085626 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-envoy" pod "c149bde6-19ca-4ce5-9685-c1cb12fe1740" (UID: "c149bde6-19ca-4ce5-9685-c1cb12fe1740"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:50.085877 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.085853 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-token" pod "c149bde6-19ca-4ce5-9685-c1cb12fe1740" (UID: "c149bde6-19ca-4ce5-9685-c1cb12fe1740"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:50.085965 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.085917 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c149bde6-19ca-4ce5-9685-c1cb12fe1740-kube-api-access-2gp6r" pod "c149bde6-19ca-4ce5-9685-c1cb12fe1740" (UID: "c149bde6-19ca-4ce5-9685-c1cb12fe1740"). InnerVolumeSpecName "kube-api-access-2gp6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:50.185397 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.185366 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:50.185397 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.185393 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:50.185397 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.185403 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gp6r\" (UniqueName: \"kubernetes.io/projected/c149bde6-19ca-4ce5-9685-c1cb12fe1740-kube-api-access-2gp6r\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:50.185597 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.185412 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c149bde6-19ca-4ce5-9685-c1cb12fe1740-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:50.707323 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.707280 5059 generic.go:356] "Generic (PLEG): container finished" podID="c149bde6-19ca-4ce5-9685-c1cb12fe1740" containerID="2e7e20cce29e971c016746efa172864a6f8aa4433e310137cd128ff94cfaf22a" exitCode=0 Nov 03 19:58:50.708461 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.708434 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" Nov 03 19:58:50.709895 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.709864 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" event={"ID":"c149bde6-19ca-4ce5-9685-c1cb12fe1740","Type":"ContainerDied","Data":"2e7e20cce29e971c016746efa172864a6f8aa4433e310137cd128ff94cfaf22a"} Nov 03 19:58:50.710025 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.709936 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9" event={"ID":"c149bde6-19ca-4ce5-9685-c1cb12fe1740","Type":"ContainerDied","Data":"f5e10f9d5a1709642f9c13d32c9862f5b43c5672d7b3393bacdb58ea416f13e0"} Nov 03 19:58:50.710025 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.709970 5059 scope.go:122] "RemoveContainer" containerID="2e7e20cce29e971c016746efa172864a6f8aa4433e310137cd128ff94cfaf22a" Nov 03 19:58:50.725472 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.725453 5059 scope.go:122] "RemoveContainer" containerID="2f8abc715842645f0a29b43c181cb7842a915e01b85268981ed036ea730585a7" Nov 03 19:58:50.737041 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.737018 5059 scope.go:122] "RemoveContainer" containerID="2e7e20cce29e971c016746efa172864a6f8aa4433e310137cd128ff94cfaf22a" Nov 03 19:58:50.737402 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:58:50.737380 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7e20cce29e971c016746efa172864a6f8aa4433e310137cd128ff94cfaf22a\": container with ID starting with 2e7e20cce29e971c016746efa172864a6f8aa4433e310137cd128ff94cfaf22a not found: ID does not exist" containerID="2e7e20cce29e971c016746efa172864a6f8aa4433e310137cd128ff94cfaf22a" Nov 03 19:58:50.737485 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.737411 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7e20cce29e971c016746efa172864a6f8aa4433e310137cd128ff94cfaf22a"} err="failed to get container status \"2e7e20cce29e971c016746efa172864a6f8aa4433e310137cd128ff94cfaf22a\": rpc error: code = NotFound desc = could not find container \"2e7e20cce29e971c016746efa172864a6f8aa4433e310137cd128ff94cfaf22a\": container with ID starting with 2e7e20cce29e971c016746efa172864a6f8aa4433e310137cd128ff94cfaf22a not found: ID does not exist" Nov 03 19:58:50.737485 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.737430 5059 scope.go:122] "RemoveContainer" containerID="2f8abc715842645f0a29b43c181cb7842a915e01b85268981ed036ea730585a7" Nov 03 19:58:50.737764 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:58:50.737736 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f8abc715842645f0a29b43c181cb7842a915e01b85268981ed036ea730585a7\": container with ID starting with 2f8abc715842645f0a29b43c181cb7842a915e01b85268981ed036ea730585a7 not found: ID does not exist" containerID="2f8abc715842645f0a29b43c181cb7842a915e01b85268981ed036ea730585a7" Nov 03 19:58:50.737838 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.737771 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f8abc715842645f0a29b43c181cb7842a915e01b85268981ed036ea730585a7"} err="failed to get container status \"2f8abc715842645f0a29b43c181cb7842a915e01b85268981ed036ea730585a7\": rpc error: code = NotFound desc = could not find container \"2f8abc715842645f0a29b43c181cb7842a915e01b85268981ed036ea730585a7\": container with ID starting with 2f8abc715842645f0a29b43c181cb7842a915e01b85268981ed036ea730585a7 not found: ID does not exist" Nov 03 19:58:50.765913 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.765877 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9"] Nov 03 19:58:50.779204 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:50.779174 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-9b3619-openshift-default-56bb9797b9-sbjs9"] Nov 03 19:58:51.858987 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:58:51.858955 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c149bde6-19ca-4ce5-9685-c1cb12fe1740" path="/var/lib/kubelet/pods/c149bde6-19ca-4ce5-9685-c1cb12fe1740/volumes" Nov 03 19:59:06.558568 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.558530 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz"] Nov 03 19:59:06.559803 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.559776 5059 cpu_manager.go:414] "containerMap: removing container" podUID="23b4c1a1-b3ca-46a5-982f-0781472ccfa6" containerName="istio-proxy" Nov 03 19:59:06.559803 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.559801 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23b4c1a1-b3ca-46a5-982f-0781472ccfa6" containerName="istio-proxy" Nov 03 19:59:06.559975 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.559822 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c149bde6-19ca-4ce5-9685-c1cb12fe1740" containerName="istio-proxy" Nov 03 19:59:06.559975 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.559831 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c149bde6-19ca-4ce5-9685-c1cb12fe1740" containerName="istio-proxy" Nov 03 19:59:06.559975 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.559875 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c149bde6-19ca-4ce5-9685-c1cb12fe1740" containerName="istio-proxy" Nov 03 19:59:06.559975 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.559883 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c149bde6-19ca-4ce5-9685-c1cb12fe1740" containerName="istio-proxy" Nov 03 19:59:06.560123 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.560085 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c149bde6-19ca-4ce5-9685-c1cb12fe1740" containerName="istio-proxy" Nov 03 19:59:06.560123 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.560104 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="23b4c1a1-b3ca-46a5-982f-0781472ccfa6" containerName="istio-proxy" Nov 03 19:59:06.568734 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.564084 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.593740 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.593709 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz"] Nov 03 19:59:06.637772 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.635232 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c6092e-openshift-default-dockercfg-85h4c\"" Nov 03 19:59:06.735686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.725789 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-istio-data\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.735686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.725888 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-workload-socket\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.735686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.725937 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8531b4f3-5179-4948-86ce-52e13576ea11-istiod-ca-cert\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.735686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.725971 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-istio-envoy\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.735686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.726005 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-credential-socket\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.735686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.726137 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-workload-certs\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.735686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.726270 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4mfn\" (UniqueName: \"kubernetes.io/projected/8531b4f3-5179-4948-86ce-52e13576ea11-kube-api-access-f4mfn\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.735686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.726320 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8531b4f3-5179-4948-86ce-52e13576ea11-istio-token\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.735686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.726419 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8531b4f3-5179-4948-86ce-52e13576ea11-istio-podinfo\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.834837 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.832991 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8531b4f3-5179-4948-86ce-52e13576ea11-istio-token\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.834837 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.833090 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8531b4f3-5179-4948-86ce-52e13576ea11-istio-podinfo\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.834837 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.833132 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-istio-data\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.834837 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.833201 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-workload-socket\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.834837 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.833238 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8531b4f3-5179-4948-86ce-52e13576ea11-istiod-ca-cert\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.834837 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.833269 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-istio-envoy\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.834837 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.833295 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-credential-socket\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.834837 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.833402 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-workload-certs\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.834837 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.833510 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4mfn\" (UniqueName: \"kubernetes.io/projected/8531b4f3-5179-4948-86ce-52e13576ea11-kube-api-access-f4mfn\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.871785 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.852728 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-istio-data\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.871785 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.853013 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-credential-socket\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.871785 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.858071 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-workload-certs\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.871785 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.858330 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-workload-socket\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.898720 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.882430 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8531b4f3-5179-4948-86ce-52e13576ea11-istiod-ca-cert\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.957345 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.957313 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8531b4f3-5179-4948-86ce-52e13576ea11-istio-token\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.957521 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.957500 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8531b4f3-5179-4948-86ce-52e13576ea11-istio-podinfo\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.957583 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.957511 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4mfn\" (UniqueName: \"kubernetes.io/projected/8531b4f3-5179-4948-86ce-52e13576ea11-kube-api-access-f4mfn\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:06.958735 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:06.958709 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-istio-envoy\") pod \"e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:07.195475 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:07.195386 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:07.520572 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:59:07.520544 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8531b4f3_5179_4948_86ce_52e13576ea11.slice/crio-9950b053e8368a1e4a2fc23f9001b946a63062f035c882f354fab5963f763b71 WatchSource:0}: Error finding container 9950b053e8368a1e4a2fc23f9001b946a63062f035c882f354fab5963f763b71: Status 404 returned error can't find the container with id 9950b053e8368a1e4a2fc23f9001b946a63062f035c882f354fab5963f763b71 Nov 03 19:59:07.520915 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:07.520890 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz"] Nov 03 19:59:07.523210 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:07.523175 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:59:07.523282 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:07.523242 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:59:07.523282 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:07.523269 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:59:08.011739 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:08.003845 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" event={"ID":"8531b4f3-5179-4948-86ce-52e13576ea11","Type":"ContainerStarted","Data":"75e02563ce0318f1dbc83f69e4cc5d059fe289723fa324d9d3379736d1b57fdf"} Nov 03 19:59:08.011739 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:08.003926 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" event={"ID":"8531b4f3-5179-4948-86ce-52e13576ea11","Type":"ContainerStarted","Data":"9950b053e8368a1e4a2fc23f9001b946a63062f035c882f354fab5963f763b71"} Nov 03 19:59:08.079688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:08.073482 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" podStartSLOduration=2.073461074 podStartE2EDuration="2.073461074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:08.049111089 +0000 UTC m=+1342.306908362" watchObservedRunningTime="2026-11-03 19:59:08.073461074 +0000 UTC m=+1342.331258332" Nov 03 19:59:08.195807 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:08.195773 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:08.208004 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:08.207959 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:09.011165 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:09.011128 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:09.011992 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:09.011970 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 19:59:10.608738 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:59:10.608700 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:59:12.792879 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:12.792848 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6"] Nov 03 19:59:12.793719 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:12.793702 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c149bde6-19ca-4ce5-9685-c1cb12fe1740" containerName="istio-proxy" Nov 03 19:59:12.798518 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:12.798499 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 19:59:12.808232 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:12.808190 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6"] Nov 03 19:59:12.808648 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:12.808624 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-1b1d04\"/\"kube-root-ca.crt\"" Nov 03 19:59:12.808841 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:12.808722 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-1b1d04\"/\"praxis-route-1b1d04-kserve-self-signed-certs\"" Nov 03 19:59:12.808841 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:12.808816 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-1b1d04\"/\"openshift-service-ca.crt\"" Nov 03 19:59:12.809897 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:12.809877 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-1b1d04\"/\"default-dockercfg-k94ph\"" Nov 03 19:59:12.887789 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:12.887756 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-tmp-dir\") pod \"praxis-route-1b1d04-kserve-656b956d69-dcrz6\" (UID: \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\") " pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 19:59:12.887947 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:12.887814 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-model-cache\") pod \"praxis-route-1b1d04-kserve-656b956d69-dcrz6\" (UID: \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\") " pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 19:59:12.887990 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:12.887947 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq29m\" (UniqueName: \"kubernetes.io/projected/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-kube-api-access-tq29m\") pod \"praxis-route-1b1d04-kserve-656b956d69-dcrz6\" (UID: \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\") " pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 19:59:12.888048 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:12.888034 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-tls-certs\") pod \"praxis-route-1b1d04-kserve-656b956d69-dcrz6\" (UID: \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\") " pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 19:59:12.888116 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:12.888104 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-dshm\") pod \"praxis-route-1b1d04-kserve-656b956d69-dcrz6\" (UID: \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\") " pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 19:59:12.888170 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:12.888158 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-home\") pod \"praxis-route-1b1d04-kserve-656b956d69-dcrz6\" (UID: \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\") " pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 19:59:13.011683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:12.994518 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-tls-certs\") pod \"praxis-route-1b1d04-kserve-656b956d69-dcrz6\" (UID: \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\") " pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 19:59:13.011683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:12.994626 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-dshm\") pod \"praxis-route-1b1d04-kserve-656b956d69-dcrz6\" (UID: \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\") " pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 19:59:13.011683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:12.994735 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-home\") pod \"praxis-route-1b1d04-kserve-656b956d69-dcrz6\" (UID: \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\") " pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 19:59:13.011683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:12.994823 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-tmp-dir\") pod \"praxis-route-1b1d04-kserve-656b956d69-dcrz6\" (UID: \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\") " pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 19:59:13.011683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:12.994896 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-model-cache\") pod \"praxis-route-1b1d04-kserve-656b956d69-dcrz6\" (UID: \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\") " pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 19:59:13.011683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:12.995027 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq29m\" (UniqueName: \"kubernetes.io/projected/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-kube-api-access-tq29m\") pod \"praxis-route-1b1d04-kserve-656b956d69-dcrz6\" (UID: \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\") " pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 19:59:13.011683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:13.001093 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-home\") pod \"praxis-route-1b1d04-kserve-656b956d69-dcrz6\" (UID: \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\") " pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 19:59:13.026436 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:13.016015 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-model-cache\") pod \"praxis-route-1b1d04-kserve-656b956d69-dcrz6\" (UID: \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\") " pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 19:59:13.026436 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:13.016279 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-tmp-dir\") pod \"praxis-route-1b1d04-kserve-656b956d69-dcrz6\" (UID: \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\") " pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 19:59:13.041342 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:13.027646 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-tls-certs\") pod \"praxis-route-1b1d04-kserve-656b956d69-dcrz6\" (UID: \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\") " pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 19:59:13.041342 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:13.031997 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-dshm\") pod \"praxis-route-1b1d04-kserve-656b956d69-dcrz6\" (UID: \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\") " pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 19:59:13.065277 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:13.044745 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq29m\" (UniqueName: \"kubernetes.io/projected/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-kube-api-access-tq29m\") pod \"praxis-route-1b1d04-kserve-656b956d69-dcrz6\" (UID: \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\") " pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 19:59:13.124218 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:13.124068 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 19:59:13.427106 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:13.427075 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6"] Nov 03 19:59:13.427805 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:59:13.427780 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb995a77c_4a8e_44c5_82cc_c8ad99b3e048.slice/crio-9f0c24d0e478577aec8c18edc725005c6f6879719412a751416d187db3ebed0a WatchSource:0}: Error finding container 9f0c24d0e478577aec8c18edc725005c6f6879719412a751416d187db3ebed0a: Status 404 returned error can't find the container with id 9f0c24d0e478577aec8c18edc725005c6f6879719412a751416d187db3ebed0a Nov 03 19:59:14.143507 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:14.143469 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" event={"ID":"b995a77c-4a8e-44c5-82cc-c8ad99b3e048","Type":"ContainerStarted","Data":"9c4115725cabe7e524d49294d673b3fcca584650d9410e615185eeec6f338449"} Nov 03 19:59:14.143916 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:14.143521 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" event={"ID":"b995a77c-4a8e-44c5-82cc-c8ad99b3e048","Type":"ContainerStarted","Data":"9f0c24d0e478577aec8c18edc725005c6f6879719412a751416d187db3ebed0a"} Nov 03 19:59:14.164084 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:14.164035 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" podStartSLOduration=2.164022105 podStartE2EDuration="2.164022105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:14.162119588 +0000 UTC m=+1348.419916842" watchObservedRunningTime="2026-11-03 19:59:14.164022105 +0000 UTC m=+1348.421819361" Nov 03 19:59:15.125168 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:15.125131 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 19:59:22.893070 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:22.893034 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w"] Nov 03 19:59:22.899407 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:22.899385 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w" Nov 03 19:59:22.906252 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:22.906227 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-277dc9\"" Nov 03 19:59:22.906443 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:22.906244 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-277dc9-dockercfg-xmr5b\"" Nov 03 19:59:22.906905 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:22.906882 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm"] Nov 03 19:59:22.927883 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:22.914060 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm" Nov 03 19:59:22.932422 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:22.932395 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w"] Nov 03 19:59:22.950714 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:22.950654 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm"] Nov 03 19:59:23.009897 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:23.009861 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wrv4\" (UniqueName: \"kubernetes.io/projected/04a05159-3d1d-4317-9890-504243a56f46-kube-api-access-2wrv4\") pod \"payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w\" (UID: \"04a05159-3d1d-4317-9890-504243a56f46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w" Nov 03 19:59:23.010229 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:23.010196 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04a05159-3d1d-4317-9890-504243a56f46-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w\" (UID: \"04a05159-3d1d-4317-9890-504243a56f46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w" Nov 03 19:59:23.010345 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:23.010264 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trbjp\" (UniqueName: \"kubernetes.io/projected/b5075f9d-997f-4fb3-be38-92f1f15f814b-kube-api-access-trbjp\") pod \"payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm\" (UID: \"b5075f9d-997f-4fb3-be38-92f1f15f814b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm" Nov 03 19:59:23.010345 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:23.010326 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5075f9d-997f-4fb3-be38-92f1f15f814b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm\" (UID: \"b5075f9d-997f-4fb3-be38-92f1f15f814b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm" Nov 03 19:59:23.135983 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:23.116935 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04a05159-3d1d-4317-9890-504243a56f46-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w\" (UID: \"04a05159-3d1d-4317-9890-504243a56f46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w" Nov 03 19:59:23.135983 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:23.117014 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trbjp\" (UniqueName: \"kubernetes.io/projected/b5075f9d-997f-4fb3-be38-92f1f15f814b-kube-api-access-trbjp\") pod \"payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm\" (UID: \"b5075f9d-997f-4fb3-be38-92f1f15f814b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm" Nov 03 19:59:23.135983 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:23.117040 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5075f9d-997f-4fb3-be38-92f1f15f814b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm\" (UID: \"b5075f9d-997f-4fb3-be38-92f1f15f814b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm" Nov 03 19:59:23.135983 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:23.117180 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wrv4\" (UniqueName: \"kubernetes.io/projected/04a05159-3d1d-4317-9890-504243a56f46-kube-api-access-2wrv4\") pod \"payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w\" (UID: \"04a05159-3d1d-4317-9890-504243a56f46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w" Nov 03 19:59:23.155406 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:23.155330 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04a05159-3d1d-4317-9890-504243a56f46-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w\" (UID: \"04a05159-3d1d-4317-9890-504243a56f46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w" Nov 03 19:59:23.171982 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:23.156678 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5075f9d-997f-4fb3-be38-92f1f15f814b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm\" (UID: \"b5075f9d-997f-4fb3-be38-92f1f15f814b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm" Nov 03 19:59:23.248766 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:23.203501 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 19:59:23.248766 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:23.204435 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wrv4\" (UniqueName: \"kubernetes.io/projected/04a05159-3d1d-4317-9890-504243a56f46-kube-api-access-2wrv4\") pod \"payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w\" (UID: \"04a05159-3d1d-4317-9890-504243a56f46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w" Nov 03 19:59:23.249521 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:23.249487 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w" Nov 03 19:59:23.250648 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:23.250616 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trbjp\" (UniqueName: \"kubernetes.io/projected/b5075f9d-997f-4fb3-be38-92f1f15f814b-kube-api-access-trbjp\") pod \"payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm\" (UID: \"b5075f9d-997f-4fb3-be38-92f1f15f814b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm" Nov 03 19:59:23.368792 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:23.368765 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 19:59:23.388568 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:23.388400 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 19:59:23.544908 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:23.544855 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm" Nov 03 19:59:23.961803 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:23.961776 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w"] Nov 03 19:59:24.086605 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:24.086576 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm"] Nov 03 19:59:24.086801 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:59:24.086779 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5075f9d_997f_4fb3_be38_92f1f15f814b.slice/crio-bab8dda750fe0e930671310ed339124b0ac8c2294638f63cfca1c3c8d578baf5 WatchSource:0}: Error finding container bab8dda750fe0e930671310ed339124b0ac8c2294638f63cfca1c3c8d578baf5: Status 404 returned error can't find the container with id bab8dda750fe0e930671310ed339124b0ac8c2294638f63cfca1c3c8d578baf5 Nov 03 19:59:24.347509 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:24.347431 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w" event={"ID":"04a05159-3d1d-4317-9890-504243a56f46","Type":"ContainerStarted","Data":"76be8b86166f50303d522afcc47ba967d9f3589ed5869b9eb67426987ffe1462"} Nov 03 19:59:24.347509 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:24.347480 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w" event={"ID":"04a05159-3d1d-4317-9890-504243a56f46","Type":"ContainerStarted","Data":"4ce0359ab22e236023ac9f3fdda2c178358949c5f703f60717a8012edba03555"} Nov 03 19:59:24.347811 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:24.347516 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w" Nov 03 19:59:24.349275 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:24.349247 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm" event={"ID":"b5075f9d-997f-4fb3-be38-92f1f15f814b","Type":"ContainerStarted","Data":"bf8ccc04da4bae3640d2c90ed8eea9cb1d7047793c870a0d2e441989d4390524"} Nov 03 19:59:24.349422 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:24.349296 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm" event={"ID":"b5075f9d-997f-4fb3-be38-92f1f15f814b","Type":"ContainerStarted","Data":"bab8dda750fe0e930671310ed339124b0ac8c2294638f63cfca1c3c8d578baf5"} Nov 03 19:59:24.349422 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:24.349379 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm" Nov 03 19:59:24.371337 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:24.371277 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w" podStartSLOduration=2.371258763 podStartE2EDuration="2.371258763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:24.366103043 +0000 UTC m=+1358.623900313" watchObservedRunningTime="2026-11-03 19:59:24.371258763 +0000 UTC m=+1358.629056021" Nov 03 19:59:24.388475 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:24.388425 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm" podStartSLOduration=2.3884126869999998 podStartE2EDuration="2.388412687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:24.385091764 +0000 UTC m=+1358.642889022" watchObservedRunningTime="2026-11-03 19:59:24.388412687 +0000 UTC m=+1358.646209943" Nov 03 19:59:30.557212 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:30.556368 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr"] Nov 03 19:59:30.557212 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:30.556582 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" podUID="9d117f5b-5d47-488d-9d42-0ee87596813b" containerName="istio-proxy" containerID="cri-o://c547d1b656a3bb125078038eede3d512f20261edfee3ed4dd5d1bdb89c29881e" gracePeriod=30 Nov 03 19:59:30.567980 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:30.567900 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8_1f33bc6f-5525-469a-a399-b8d63af713e5/payload-processing/0.log" Nov 03 19:59:30.616103 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:30.616064 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv"] Nov 03 19:59:30.616575 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:30.616540 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" podUID="ced1fbaf-4771-4fab-8df0-a661dffb0d1f" containerName="istio-proxy" containerID="cri-o://ea3db214c78be67a09c85ab42f65ff26dbb8345e6f3312da74da713db8595587" gracePeriod=30 Nov 03 19:59:31.279010 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:31.278975 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7_1463c361-4736-4158-83f1-e31353a338f4/payload-processing/0.log" Nov 03 19:59:31.452096 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:31.452067 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7_1463c361-4736-4158-83f1-e31353a338f4/payload-processing/0.log" Nov 03 19:59:32.036212 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:32.036124 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2876b2ef-jfzsx"] Nov 03 19:59:32.039948 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:32.039922 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2876b2ef-jfzsx"] Nov 03 19:59:33.880269 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:33.880225 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e71b43c-8c02-43ea-bb4a-9051fd3d26b6" path="/var/lib/kubelet/pods/1e71b43c-8c02-43ea-bb4a-9051fd3d26b6/volumes" Nov 03 19:59:35.371847 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:35.371799 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm" Nov 03 19:59:35.372332 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:35.372249 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w" Nov 03 19:59:35.871540 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:35.871501 5059 patch_prober.go:29] interesting pod/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:59:35.871810 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:35.871557 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" podUID="856d3ff7-8514-46b9-8884-f99b674dd6e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:59:36.648679 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.644712 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:59:36.758933 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.758899 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-token\" (UniqueName: \"kubernetes.io/projected/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-token\") pod \"9d117f5b-5d47-488d-9d42-0ee87596813b\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " Nov 03 19:59:36.759161 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.758944 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-podinfo\") pod \"9d117f5b-5d47-488d-9d42-0ee87596813b\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " Nov 03 19:59:36.759161 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.758988 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-data\") pod \"9d117f5b-5d47-488d-9d42-0ee87596813b\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " Nov 03 19:59:36.759161 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.759034 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d117f5b-5d47-488d-9d42-0ee87596813b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d117f5b-5d47-488d-9d42-0ee87596813b-istiod-ca-cert\") pod \"9d117f5b-5d47-488d-9d42-0ee87596813b\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " Nov 03 19:59:36.759161 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.759083 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-workload-socket\") pod \"9d117f5b-5d47-488d-9d42-0ee87596813b\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " Nov 03 19:59:36.759161 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.759110 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-credential-socket\") pod \"9d117f5b-5d47-488d-9d42-0ee87596813b\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " Nov 03 19:59:36.759548 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.759251 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-envoy\") pod \"9d117f5b-5d47-488d-9d42-0ee87596813b\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " Nov 03 19:59:36.759548 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.759290 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-workload-certs\") pod \"9d117f5b-5d47-488d-9d42-0ee87596813b\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " Nov 03 19:59:36.759548 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.759316 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d117f5b-5d47-488d-9d42-0ee87596813b-kube-api-access-fc4bk\" (UniqueName: \"kubernetes.io/projected/9d117f5b-5d47-488d-9d42-0ee87596813b-kube-api-access-fc4bk\") pod \"9d117f5b-5d47-488d-9d42-0ee87596813b\" (UID: \"9d117f5b-5d47-488d-9d42-0ee87596813b\") " Nov 03 19:59:36.760306 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.760275 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-workload-socket" pod "9d117f5b-5d47-488d-9d42-0ee87596813b" (UID: "9d117f5b-5d47-488d-9d42-0ee87596813b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:59:36.760921 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.760610 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-data" pod "9d117f5b-5d47-488d-9d42-0ee87596813b" (UID: "9d117f5b-5d47-488d-9d42-0ee87596813b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:59:36.760921 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.760614 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d117f5b-5d47-488d-9d42-0ee87596813b-istiod-ca-cert" pod "9d117f5b-5d47-488d-9d42-0ee87596813b" (UID: "9d117f5b-5d47-488d-9d42-0ee87596813b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:59:36.760921 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.760777 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-credential-socket" pod "9d117f5b-5d47-488d-9d42-0ee87596813b" (UID: "9d117f5b-5d47-488d-9d42-0ee87596813b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:59:36.760921 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.760890 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-workload-certs" pod "9d117f5b-5d47-488d-9d42-0ee87596813b" (UID: "9d117f5b-5d47-488d-9d42-0ee87596813b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:59:36.762787 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.762153 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d117f5b-5d47-488d-9d42-0ee87596813b-kube-api-access-fc4bk" pod "9d117f5b-5d47-488d-9d42-0ee87596813b" (UID: "9d117f5b-5d47-488d-9d42-0ee87596813b"). InnerVolumeSpecName "kube-api-access-fc4bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:59:36.763191 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.763168 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-envoy" pod "9d117f5b-5d47-488d-9d42-0ee87596813b" (UID: "9d117f5b-5d47-488d-9d42-0ee87596813b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:59:36.764116 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.764088 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-podinfo" pod "9d117f5b-5d47-488d-9d42-0ee87596813b" (UID: "9d117f5b-5d47-488d-9d42-0ee87596813b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:59:36.767899 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.767875 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-token" pod "9d117f5b-5d47-488d-9d42-0ee87596813b" (UID: "9d117f5b-5d47-488d-9d42-0ee87596813b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:59:36.862723 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.862533 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:36.862723 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.862574 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:36.862723 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.862591 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:36.862723 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.862605 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d117f5b-5d47-488d-9d42-0ee87596813b-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:36.862723 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.862619 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:36.862723 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.862633 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:36.862723 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.862650 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:36.862723 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.862681 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d117f5b-5d47-488d-9d42-0ee87596813b-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:36.862723 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.862695 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc4bk\" (UniqueName: \"kubernetes.io/projected/9d117f5b-5d47-488d-9d42-0ee87596813b-kube-api-access-fc4bk\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:36.950389 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.950353 5059 generic.go:356] "Generic (PLEG): container finished" podID="ced1fbaf-4771-4fab-8df0-a661dffb0d1f" containerID="ea3db214c78be67a09c85ab42f65ff26dbb8345e6f3312da74da713db8595587" exitCode=0 Nov 03 19:59:36.950523 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.950428 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" event={"ID":"ced1fbaf-4771-4fab-8df0-a661dffb0d1f","Type":"ContainerDied","Data":"ea3db214c78be67a09c85ab42f65ff26dbb8345e6f3312da74da713db8595587"} Nov 03 19:59:36.950523 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.950499 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" event={"ID":"ced1fbaf-4771-4fab-8df0-a661dffb0d1f","Type":"ContainerDied","Data":"f6b35f6edc6789fb490759be5752bf83f6655e7e34518f17957ddaab18a9c802"} Nov 03 19:59:36.950637 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.950525 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6b35f6edc6789fb490759be5752bf83f6655e7e34518f17957ddaab18a9c802" Nov 03 19:59:36.952278 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.952253 5059 generic.go:356] "Generic (PLEG): container finished" podID="9d117f5b-5d47-488d-9d42-0ee87596813b" containerID="c547d1b656a3bb125078038eede3d512f20261edfee3ed4dd5d1bdb89c29881e" exitCode=0 Nov 03 19:59:36.952419 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.952297 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" Nov 03 19:59:36.952419 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.952318 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" event={"ID":"9d117f5b-5d47-488d-9d42-0ee87596813b","Type":"ContainerDied","Data":"c547d1b656a3bb125078038eede3d512f20261edfee3ed4dd5d1bdb89c29881e"} Nov 03 19:59:36.952419 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.952363 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr" event={"ID":"9d117f5b-5d47-488d-9d42-0ee87596813b","Type":"ContainerDied","Data":"aa3f8b5aa191e8dc15d26636a56b15c767e55d2621b24a40a08e8b2056e39567"} Nov 03 19:59:36.952419 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.952392 5059 scope.go:122] "RemoveContainer" containerID="c547d1b656a3bb125078038eede3d512f20261edfee3ed4dd5d1bdb89c29881e" Nov 03 19:59:36.976348 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.976185 5059 scope.go:122] "RemoveContainer" containerID="c547d1b656a3bb125078038eede3d512f20261edfee3ed4dd5d1bdb89c29881e" Nov 03 19:59:36.976523 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:59:36.976494 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c547d1b656a3bb125078038eede3d512f20261edfee3ed4dd5d1bdb89c29881e\": container with ID starting with c547d1b656a3bb125078038eede3d512f20261edfee3ed4dd5d1bdb89c29881e not found: ID does not exist" containerID="c547d1b656a3bb125078038eede3d512f20261edfee3ed4dd5d1bdb89c29881e" Nov 03 19:59:36.976580 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:36.976535 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c547d1b656a3bb125078038eede3d512f20261edfee3ed4dd5d1bdb89c29881e"} err="failed to get container status \"c547d1b656a3bb125078038eede3d512f20261edfee3ed4dd5d1bdb89c29881e\": rpc error: code = NotFound desc = could not find container \"c547d1b656a3bb125078038eede3d512f20261edfee3ed4dd5d1bdb89c29881e\": container with ID starting with c547d1b656a3bb125078038eede3d512f20261edfee3ed4dd5d1bdb89c29881e not found: ID does not exist" Nov 03 19:59:37.406392 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.405567 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:59:37.431803 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.431767 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr"] Nov 03 19:59:37.438036 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.437990 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-599xr"] Nov 03 19:59:37.442708 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.442639 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m"] Nov 03 19:59:37.478003 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.472002 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-workload-certs\") pod \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " Nov 03 19:59:37.478003 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.472140 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istiod-ca-cert\") pod \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " Nov 03 19:59:37.478003 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.472290 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-token\" (UniqueName: \"kubernetes.io/projected/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-token\") pod \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " Nov 03 19:59:37.478003 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.472353 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-data\") pod \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " Nov 03 19:59:37.478003 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.472384 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-envoy\") pod \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " Nov 03 19:59:37.478003 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.472447 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-podinfo\") pod \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " Nov 03 19:59:37.478003 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.472480 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-credential-socket\") pod \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " Nov 03 19:59:37.478003 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.472528 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-kube-api-access-dswk2\" (UniqueName: \"kubernetes.io/projected/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-kube-api-access-dswk2\") pod \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " Nov 03 19:59:37.478003 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.472557 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-workload-socket\") pod \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\" (UID: \"ced1fbaf-4771-4fab-8df0-a661dffb0d1f\") " Nov 03 19:59:37.482830 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.480983 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-workload-socket" pod "ced1fbaf-4771-4fab-8df0-a661dffb0d1f" (UID: "ced1fbaf-4771-4fab-8df0-a661dffb0d1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:59:37.482830 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.481439 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-workload-certs" pod "ced1fbaf-4771-4fab-8df0-a661dffb0d1f" (UID: "ced1fbaf-4771-4fab-8df0-a661dffb0d1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:59:37.482830 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.481790 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istiod-ca-cert" pod "ced1fbaf-4771-4fab-8df0-a661dffb0d1f" (UID: "ced1fbaf-4771-4fab-8df0-a661dffb0d1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:59:37.482830 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.482379 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-data" pod "ced1fbaf-4771-4fab-8df0-a661dffb0d1f" (UID: "ced1fbaf-4771-4fab-8df0-a661dffb0d1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:59:37.482830 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.482575 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-credential-socket" pod "ced1fbaf-4771-4fab-8df0-a661dffb0d1f" (UID: "ced1fbaf-4771-4fab-8df0-a661dffb0d1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:59:37.493929 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.489240 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-token" pod "ced1fbaf-4771-4fab-8df0-a661dffb0d1f" (UID: "ced1fbaf-4771-4fab-8df0-a661dffb0d1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:59:37.509793 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.502120 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-podinfo" pod "ced1fbaf-4771-4fab-8df0-a661dffb0d1f" (UID: "ced1fbaf-4771-4fab-8df0-a661dffb0d1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:59:37.509793 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.503010 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-kube-api-access-dswk2" pod "ced1fbaf-4771-4fab-8df0-a661dffb0d1f" (UID: "ced1fbaf-4771-4fab-8df0-a661dffb0d1f"). InnerVolumeSpecName "kube-api-access-dswk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:59:37.509793 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.506806 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-envoy" pod "ced1fbaf-4771-4fab-8df0-a661dffb0d1f" (UID: "ced1fbaf-4771-4fab-8df0-a661dffb0d1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:59:37.563276 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.562796 5059 patch_prober.go:29] interesting pod/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:59:37.563276 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.562863 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" podUID="7ad642be-b0b8-4a68-9392-ec28c87264a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:59:37.582455 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.582281 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:37.582455 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.582319 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:37.582455 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.582333 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:37.582455 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.582347 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:37.582455 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.582361 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:37.582455 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.582375 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:37.582455 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.582390 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:37.582455 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.582405 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dswk2\" (UniqueName: \"kubernetes.io/projected/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-kube-api-access-dswk2\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:37.582455 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.582420 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced1fbaf-4771-4fab-8df0-a661dffb0d1f-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:37.864344 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.864299 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d117f5b-5d47-488d-9d42-0ee87596813b" path="/var/lib/kubelet/pods/9d117f5b-5d47-488d-9d42-0ee87596813b/volumes" Nov 03 19:59:37.964715 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.964647 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv" Nov 03 19:59:37.986343 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.986306 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv"] Nov 03 19:59:37.990239 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:37.990216 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-z9xmv"] Nov 03 19:59:38.576419 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:38.574784 5059 patch_prober.go:29] interesting pod/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:59:38.576419 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:38.574855 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" podUID="87c2d4eb-392d-4a66-8622-9ff5eee0b761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:59:39.663654 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:39.663626 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm_b5075f9d-997f-4fb3-be38-92f1f15f814b/payload-processing/0.log" Nov 03 19:59:39.859124 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:39.859090 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ced1fbaf-4771-4fab-8df0-a661dffb0d1f" path="/var/lib/kubelet/pods/ced1fbaf-4771-4fab-8df0-a661dffb0d1f/volumes" Nov 03 19:59:40.850964 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:59:40.610867 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 19:59:40.994622 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:40.990504 5059 generic.go:356] "Generic (PLEG): container finished" podID="856d3ff7-8514-46b9-8884-f99b674dd6e6" containerID="4dfd6dd522fa140050dad5e5f74edeaeb56816f8fbb43c92919142c90af8a88b" exitCode=0 Nov 03 19:59:40.994622 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:40.990533 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" event={"ID":"856d3ff7-8514-46b9-8884-f99b674dd6e6","Type":"ContainerDied","Data":"4dfd6dd522fa140050dad5e5f74edeaeb56816f8fbb43c92919142c90af8a88b"} Nov 03 19:59:40.994622 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:40.990974 5059 scope.go:122] "RemoveContainer" containerID="4dfd6dd522fa140050dad5e5f74edeaeb56816f8fbb43c92919142c90af8a88b" Nov 03 19:59:40.994622 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:40.992842 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:59:40.994622 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:40.992928 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:59:40.994622 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:40.992971 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:59:42.007173 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:42.007135 5059 generic.go:356] "Generic (PLEG): container finished" podID="87c2d4eb-392d-4a66-8622-9ff5eee0b761" containerID="ff2052ce9b253a2c2fbda40231cab9fe34232bc1021d11e8648c5fee496ace71" exitCode=0 Nov 03 19:59:42.007630 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:42.007212 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" event={"ID":"87c2d4eb-392d-4a66-8622-9ff5eee0b761","Type":"ContainerDied","Data":"ff2052ce9b253a2c2fbda40231cab9fe34232bc1021d11e8648c5fee496ace71"} Nov 03 19:59:42.007794 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:42.007775 5059 scope.go:122] "RemoveContainer" containerID="ff2052ce9b253a2c2fbda40231cab9fe34232bc1021d11e8648c5fee496ace71" Nov 03 19:59:42.009600 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:42.009570 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:59:42.009735 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:42.009586 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" event={"ID":"856d3ff7-8514-46b9-8884-f99b674dd6e6","Type":"ContainerStarted","Data":"fb95eba5d616de91bd7fc8a639b7f9be441869b05ec13d63169baa5585f37f2b"} Nov 03 19:59:42.009735 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:42.009717 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:59:42.009860 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:42.009760 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:59:42.013841 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:42.013814 5059 generic.go:356] "Generic (PLEG): container finished" podID="7ad642be-b0b8-4a68-9392-ec28c87264a3" containerID="70e22e24b3a0416be1693328143270603f0b640b05256d39bbdc39f123cbedc9" exitCode=0 Nov 03 19:59:42.013957 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:42.013851 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" event={"ID":"7ad642be-b0b8-4a68-9392-ec28c87264a3","Type":"ContainerDied","Data":"70e22e24b3a0416be1693328143270603f0b640b05256d39bbdc39f123cbedc9"} Nov 03 19:59:42.014566 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:42.014242 5059 scope.go:122] "RemoveContainer" containerID="70e22e24b3a0416be1693328143270603f0b640b05256d39bbdc39f123cbedc9" Nov 03 19:59:42.021808 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:42.021774 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:59:42.021896 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:42.021864 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:59:42.021981 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:42.021908 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:59:42.337727 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:42.337590 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:59:42.337727 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:42.337730 5059 patch_prober.go:29] interesting pod/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:59:42.337999 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:42.337775 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" podUID="856d3ff7-8514-46b9-8884-f99b674dd6e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:59:43.044215 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:43.044169 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" event={"ID":"7ad642be-b0b8-4a68-9392-ec28c87264a3","Type":"ContainerStarted","Data":"f57104699dec2690120dbe503676167d6c6ecc3dab6a2e4344002cc2e3cd3d5a"} Nov 03 19:59:43.047769 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:43.047744 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" event={"ID":"87c2d4eb-392d-4a66-8622-9ff5eee0b761","Type":"ContainerStarted","Data":"25f1d35ccdcc578f635b59cc635791b85534cdee7e6f5f467fb2fee218e979cc"} Nov 03 19:59:43.103683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:43.092763 5059 patch_prober.go:29] interesting pod/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:59:43.103683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:43.092826 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" podUID="87c2d4eb-392d-4a66-8622-9ff5eee0b761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:59:43.103683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:43.093558 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:59:43.285688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:43.278068 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:59:43.285688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:43.278207 5059 patch_prober.go:29] interesting pod/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 19:59:43.285688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:43.278252 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" podUID="7ad642be-b0b8-4a68-9392-ec28c87264a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 19:59:43.347695 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:43.347615 5059 patch_prober.go:29] interesting pod/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:59:43.347915 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:43.347696 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" podUID="856d3ff7-8514-46b9-8884-f99b674dd6e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:59:44.086689 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:44.081049 5059 patch_prober.go:29] interesting pod/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:59:44.086689 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:44.081206 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" podUID="87c2d4eb-392d-4a66-8622-9ff5eee0b761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:59:44.286687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:44.285931 5059 patch_prober.go:29] interesting pod/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 19:59:44.286687 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:44.286001 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" podUID="7ad642be-b0b8-4a68-9392-ec28c87264a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 19:59:44.337253 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:44.337153 5059 patch_prober.go:29] interesting pod/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:59:44.337253 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:44.337219 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" podUID="856d3ff7-8514-46b9-8884-f99b674dd6e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:59:45.085691 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:45.085080 5059 patch_prober.go:29] interesting pod/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:59:45.085691 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:45.085148 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" podUID="87c2d4eb-392d-4a66-8622-9ff5eee0b761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:59:45.275103 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:45.275063 5059 patch_prober.go:29] interesting pod/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 19:59:45.275723 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:45.275687 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" podUID="7ad642be-b0b8-4a68-9392-ec28c87264a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 19:59:45.337784 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:45.337697 5059 patch_prober.go:29] interesting pod/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:59:45.337784 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:45.337749 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" podUID="856d3ff7-8514-46b9-8884-f99b674dd6e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:59:46.084684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:46.084010 5059 patch_prober.go:29] interesting pod/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:59:46.084684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:46.084076 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" podUID="87c2d4eb-392d-4a66-8622-9ff5eee0b761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:59:46.282686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:46.282134 5059 patch_prober.go:29] interesting pod/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 19:59:46.282686 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:46.282220 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" podUID="7ad642be-b0b8-4a68-9392-ec28c87264a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 19:59:46.339722 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:46.338087 5059 patch_prober.go:29] interesting pod/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:59:46.339722 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:46.338148 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" podUID="856d3ff7-8514-46b9-8884-f99b674dd6e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:59:47.100155 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.100115 5059 patch_prober.go:29] interesting pod/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:59:47.100578 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.100543 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" podUID="87c2d4eb-392d-4a66-8622-9ff5eee0b761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:59:47.336684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.333063 5059 patch_prober.go:29] interesting pod/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 19:59:47.336684 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.333138 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" podUID="7ad642be-b0b8-4a68-9392-ec28c87264a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 19:59:47.485054 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.484966 5059 patch_prober.go:29] interesting pod/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:59:47.485342 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.485306 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" podUID="856d3ff7-8514-46b9-8884-f99b674dd6e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:59:47.486542 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.486514 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2"] Nov 03 19:59:47.489316 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.489293 5059 cpu_manager.go:414] "containerMap: removing container" podUID="ced1fbaf-4771-4fab-8df0-a661dffb0d1f" containerName="istio-proxy" Nov 03 19:59:47.498753 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.498727 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced1fbaf-4771-4fab-8df0-a661dffb0d1f" containerName="istio-proxy" Nov 03 19:59:47.498929 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.498918 5059 cpu_manager.go:414] "containerMap: removing container" podUID="9d117f5b-5d47-488d-9d42-0ee87596813b" containerName="istio-proxy" Nov 03 19:59:47.499021 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.499011 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d117f5b-5d47-488d-9d42-0ee87596813b" containerName="istio-proxy" Nov 03 19:59:47.499352 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.499340 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced1fbaf-4771-4fab-8df0-a661dffb0d1f" containerName="istio-proxy" Nov 03 19:59:47.499444 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.499435 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d117f5b-5d47-488d-9d42-0ee87596813b" containerName="istio-proxy" Nov 03 19:59:47.618938 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.618891 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2" Nov 03 19:59:47.669433 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.669399 5059 scope.go:122] "RemoveContainer" containerID="ea3db214c78be67a09c85ab42f65ff26dbb8345e6f3312da74da713db8595587" Nov 03 19:59:47.676478 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.676174 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj2kk\" (UniqueName: \"kubernetes.io/projected/539fad2d-5ad7-4007-8b7f-94c325a943db-kube-api-access-sj2kk\") pod \"payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2\" (UID: \"539fad2d-5ad7-4007-8b7f-94c325a943db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2" Nov 03 19:59:47.676478 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.676230 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/539fad2d-5ad7-4007-8b7f-94c325a943db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2\" (UID: \"539fad2d-5ad7-4007-8b7f-94c325a943db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2" Nov 03 19:59:47.683682 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.680081 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2"] Nov 03 19:59:47.704710 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.703944 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c6092e-dockercfg-xmq8r\"" Nov 03 19:59:47.785440 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.785318 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj2kk\" (UniqueName: \"kubernetes.io/projected/539fad2d-5ad7-4007-8b7f-94c325a943db-kube-api-access-sj2kk\") pod \"payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2\" (UID: \"539fad2d-5ad7-4007-8b7f-94c325a943db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2" Nov 03 19:59:47.785715 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.785693 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/539fad2d-5ad7-4007-8b7f-94c325a943db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2\" (UID: \"539fad2d-5ad7-4007-8b7f-94c325a943db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2" Nov 03 19:59:47.816725 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.815397 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c6092e\"" Nov 03 19:59:47.840458 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.829840 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj2kk\" (UniqueName: \"kubernetes.io/projected/539fad2d-5ad7-4007-8b7f-94c325a943db-kube-api-access-sj2kk\") pod \"payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2\" (UID: \"539fad2d-5ad7-4007-8b7f-94c325a943db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2" Nov 03 19:59:47.887840 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.843327 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/539fad2d-5ad7-4007-8b7f-94c325a943db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2\" (UID: \"539fad2d-5ad7-4007-8b7f-94c325a943db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2" Nov 03 19:59:47.970077 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:47.970044 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2" Nov 03 19:59:48.029016 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:48.028621 5059 scope.go:122] "RemoveContainer" containerID="d8cac496f3ba3b85c9a475d4c226c3e86199e7e674f24280b56e8eefa76ea668" Nov 03 19:59:48.048808 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:48.048760 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" podUID="5f12977e-2949-4de3-8c7b-2e42dbc495e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.153:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:59:48.075698 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:48.075253 5059 patch_prober.go:29] interesting pod/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:59:48.075698 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:48.075314 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" podUID="87c2d4eb-392d-4a66-8622-9ff5eee0b761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:59:48.232867 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:48.232818 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:59:48.244864 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:48.241859 5059 scope.go:122] "RemoveContainer" containerID="6ff5e1dfb0ee483d176ff2e04b3217dba0c168d58625e1cb163e71d5b8eeadfd" Nov 03 19:59:48.281808 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:48.281772 5059 patch_prober.go:29] interesting pod/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 19:59:48.282096 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:48.282065 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" podUID="7ad642be-b0b8-4a68-9392-ec28c87264a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 19:59:48.339171 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:48.339136 5059 patch_prober.go:29] interesting pod/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 19:59:48.345792 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:48.345751 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" podUID="856d3ff7-8514-46b9-8884-f99b674dd6e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 19:59:48.353064 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:48.349953 5059 scope.go:122] "RemoveContainer" containerID="ce1f3ec2b07fb6c3130cce90f9ce0da535e7d8528d2cfa3f212189b7edff7789" Nov 03 19:59:48.442830 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:48.442794 5059 scope.go:122] "RemoveContainer" containerID="25d3556b8415448f4f453e3b13f7abdb1e9b1baea12dab448e0e72865ddbbef7" Nov 03 19:59:48.527452 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:48.527425 5059 scope.go:122] "RemoveContainer" containerID="46b72c3b34b23f6c120c42912125e6cd5c93dffc15b678cbee23afdd51a40c8a" Nov 03 19:59:48.549977 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:48.549850 5059 scope.go:122] "RemoveContainer" containerID="5c3861f203c47baef9469e3007567abfe7482aa19062187054c5a5411e088e7a" Nov 03 19:59:48.654679 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:48.645007 5059 scope.go:122] "RemoveContainer" containerID="2eb28b2903edbb43c568a1d84342cee7d2e5444d4a0d07d27b1512860ef218a4" Nov 03 19:59:48.951630 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:48.951597 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2"] Nov 03 19:59:49.164987 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:49.164947 5059 patch_prober.go:29] interesting pod/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 19:59:49.165415 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:49.165271 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" podUID="87c2d4eb-392d-4a66-8622-9ff5eee0b761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 19:59:49.339699 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:49.338812 5059 patch_prober.go:29] interesting pod/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 19:59:49.339699 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:49.338897 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" podUID="7ad642be-b0b8-4a68-9392-ec28c87264a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 19:59:49.379036 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:49.379002 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:59:49.402688 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:49.380917 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:59:49.442203 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:49.442080 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 19:59:50.105867 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:50.105833 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:59:50.106583 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:50.106556 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:59:50.108118 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:50.108060 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 19:59:50.398747 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:50.374034 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2" event={"ID":"539fad2d-5ad7-4007-8b7f-94c325a943db","Type":"ContainerStarted","Data":"5372c3fa517a24caedc3959e64453282ab3ba4fbe715df9d40382c0ee3686dae"} Nov 03 19:59:50.399377 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:50.399345 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2" event={"ID":"539fad2d-5ad7-4007-8b7f-94c325a943db","Type":"ContainerStarted","Data":"ed33dcd767378db33e642cb1ed6ec23e81ebd964e1fbe02a07793e13b53aa59d"} Nov 03 19:59:50.399523 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:50.399508 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2" Nov 03 19:59:50.429681 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:50.408317 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2" podStartSLOduration=3.408298146 podStartE2EDuration="3.408298146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:50.398077645 +0000 UTC m=+1384.655874904" watchObservedRunningTime="2026-11-03 19:59:50.408298146 +0000 UTC m=+1384.666095405" Nov 03 19:59:50.444028 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:50.443983 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:59:50.446128 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:50.446105 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:59:50.464685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:50.458930 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 19:59:52.488489 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.488449 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" podUID="5f12977e-2949-4de3-8c7b-2e42dbc495e7" containerName="main" containerID="cri-o://14d62fb25c7489609d927c24fdefb001dac0545d210b73a0e4401b5dbf2e8e08" gracePeriod=45 Nov 03 19:59:52.736478 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.736438 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq"] Nov 03 19:59:52.741081 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.741029 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.754216 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.754187 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq"] Nov 03 19:59:52.790692 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.754881 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-1b1d04-openshift-default-dockercfg-gl687\"" Nov 03 19:59:52.833819 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.833793 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:59:52.836683 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.836652 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-token\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.836774 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.836753 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-envoy\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.836843 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.836804 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-data\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.836978 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.836963 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-credential-socket\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.837019 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.837000 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2bwx\" (UniqueName: \"kubernetes.io/projected/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-kube-api-access-r2bwx\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.837093 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.837078 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istiod-ca-cert\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.837221 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.837207 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-workload-socket\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.837286 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.837265 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-podinfo\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.837343 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.837302 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-workload-certs\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.938100 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.938065 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-home\" (UniqueName: \"kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-home\") pod \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\" (UID: \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\") " Nov 03 19:59:52.938318 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.938304 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f12977e-2949-4de3-8c7b-2e42dbc495e7-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f12977e-2949-4de3-8c7b-2e42dbc495e7-tls-certs\") pod \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\" (UID: \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\") " Nov 03 19:59:52.938393 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.938364 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-home" pod "5f12977e-2949-4de3-8c7b-2e42dbc495e7" (UID: "5f12977e-2949-4de3-8c7b-2e42dbc495e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:59:52.938487 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.938477 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-dshm\") pod \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\" (UID: \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\") " Nov 03 19:59:52.938637 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.938628 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-tmp-dir\") pod \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\" (UID: \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\") " Nov 03 19:59:52.938880 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.938868 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f12977e-2949-4de3-8c7b-2e42dbc495e7-kube-api-access-7bd2c\" (UniqueName: \"kubernetes.io/projected/5f12977e-2949-4de3-8c7b-2e42dbc495e7-kube-api-access-7bd2c\") pod \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\" (UID: \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\") " Nov 03 19:59:52.939003 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.938989 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-model-cache\") pod \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\" (UID: \"5f12977e-2949-4de3-8c7b-2e42dbc495e7\") " Nov 03 19:59:52.939100 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.939079 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-tmp-dir" pod "5f12977e-2949-4de3-8c7b-2e42dbc495e7" (UID: "5f12977e-2949-4de3-8c7b-2e42dbc495e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:59:52.939219 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.939196 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-model-cache" pod "5f12977e-2949-4de3-8c7b-2e42dbc495e7" (UID: "5f12977e-2949-4de3-8c7b-2e42dbc495e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:59:52.939546 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.939521 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-workload-socket\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.939743 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.939721 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-podinfo\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.939897 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.939880 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-workload-certs\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.940138 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.940118 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-token\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.940364 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.940255 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-workload-certs\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.940364 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.940256 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-envoy\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.940364 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.940324 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-data\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.940915 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.939927 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-workload-socket\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.940915 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.940702 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-credential-socket\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.940915 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.940756 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2bwx\" (UniqueName: \"kubernetes.io/projected/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-kube-api-access-r2bwx\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.940915 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.940808 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istiod-ca-cert\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.940915 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.940848 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-data\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.942126 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.941319 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-credential-socket\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.942126 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.941335 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:52.942126 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.941366 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:52.942126 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.941384 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:52.942126 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.941836 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f12977e-2949-4de3-8c7b-2e42dbc495e7-tls-certs" pod "5f12977e-2949-4de3-8c7b-2e42dbc495e7" (UID: "5f12977e-2949-4de3-8c7b-2e42dbc495e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:59:52.942126 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.941980 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istiod-ca-cert\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.942126 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.942100 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-dshm" pod "5f12977e-2949-4de3-8c7b-2e42dbc495e7" (UID: "5f12977e-2949-4de3-8c7b-2e42dbc495e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:59:52.944253 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.944227 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-envoy\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.944712 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.944677 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-podinfo\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.944897 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.944873 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f12977e-2949-4de3-8c7b-2e42dbc495e7-kube-api-access-7bd2c" pod "5f12977e-2949-4de3-8c7b-2e42dbc495e7" (UID: "5f12977e-2949-4de3-8c7b-2e42dbc495e7"). InnerVolumeSpecName "kube-api-access-7bd2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:59:52.953403 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.953383 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2bwx\" (UniqueName: \"kubernetes.io/projected/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-kube-api-access-r2bwx\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:52.953544 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:52.953468 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-token\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:53.042614 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.042526 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bd2c\" (UniqueName: \"kubernetes.io/projected/5f12977e-2949-4de3-8c7b-2e42dbc495e7-kube-api-access-7bd2c\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:53.042614 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.042559 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f12977e-2949-4de3-8c7b-2e42dbc495e7-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:53.042614 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.042573 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f12977e-2949-4de3-8c7b-2e42dbc495e7-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:53.094538 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.094502 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:53.295145 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.295058 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz"] Nov 03 19:59:53.295371 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.295322 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz" podUID="651eb1ff-2364-47df-96b5-5ae746240b93" containerName="payload-processing" containerID="cri-o://03bf2f26a83d43a9b790f05e408c4cff1d6803400918cfa45262e8ecbbb54561" gracePeriod=30 Nov 03 19:59:53.299548 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.299518 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq"] Nov 03 19:59:53.300100 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.299815 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq" podUID="32e5da96-23df-4b40-a08e-7b2820065df5" containerName="payload-pre-processing" containerID="cri-o://f72fbd00f6d07d8fa957ff9cc5b40a950f89f07b6f10d37ae3f632ca24c1469f" gracePeriod=30 Nov 03 19:59:53.396142 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.396076 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq"] Nov 03 19:59:53.421740 ip-10-0-132-83 kubenswrapper[5059]: W1103 19:59:53.421710 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01f88a8_c8c2_44ba_ab1d_bd69d2b4084f.slice/crio-6305ec56316a207af186bbc8d932641ea1e2d368e68880100b7c59997f11b353 WatchSource:0}: Error finding container 6305ec56316a207af186bbc8d932641ea1e2d368e68880100b7c59997f11b353: Status 404 returned error can't find the container with id 6305ec56316a207af186bbc8d932641ea1e2d368e68880100b7c59997f11b353 Nov 03 19:59:53.425965 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.425507 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:59:53.425965 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.425600 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:59:53.425965 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.425644 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 19:59:53.447845 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.447381 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" event={"ID":"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f","Type":"ContainerStarted","Data":"6305ec56316a207af186bbc8d932641ea1e2d368e68880100b7c59997f11b353"} Nov 03 19:59:53.449872 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.449630 5059 generic.go:356] "Generic (PLEG): container finished" podID="651eb1ff-2364-47df-96b5-5ae746240b93" containerID="03bf2f26a83d43a9b790f05e408c4cff1d6803400918cfa45262e8ecbbb54561" exitCode=0 Nov 03 19:59:53.449872 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.449771 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz" event={"ID":"651eb1ff-2364-47df-96b5-5ae746240b93","Type":"ContainerDied","Data":"03bf2f26a83d43a9b790f05e408c4cff1d6803400918cfa45262e8ecbbb54561"} Nov 03 19:59:53.453121 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.451670 5059 generic.go:356] "Generic (PLEG): container finished" podID="32e5da96-23df-4b40-a08e-7b2820065df5" containerID="f72fbd00f6d07d8fa957ff9cc5b40a950f89f07b6f10d37ae3f632ca24c1469f" exitCode=0 Nov 03 19:59:53.453121 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.451806 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq" event={"ID":"32e5da96-23df-4b40-a08e-7b2820065df5","Type":"ContainerDied","Data":"f72fbd00f6d07d8fa957ff9cc5b40a950f89f07b6f10d37ae3f632ca24c1469f"} Nov 03 19:59:53.454480 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.454456 5059 generic.go:356] "Generic (PLEG): container finished" podID="5f12977e-2949-4de3-8c7b-2e42dbc495e7" containerID="14d62fb25c7489609d927c24fdefb001dac0545d210b73a0e4401b5dbf2e8e08" exitCode=0 Nov 03 19:59:53.454581 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.454521 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" Nov 03 19:59:53.454581 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.454542 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" event={"ID":"5f12977e-2949-4de3-8c7b-2e42dbc495e7","Type":"ContainerDied","Data":"14d62fb25c7489609d927c24fdefb001dac0545d210b73a0e4401b5dbf2e8e08"} Nov 03 19:59:53.454717 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.454587 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m" event={"ID":"5f12977e-2949-4de3-8c7b-2e42dbc495e7","Type":"ContainerDied","Data":"49068727121bc447a4e79ad61c8bfc083985965712406baec96f1999754ef2d2"} Nov 03 19:59:53.454717 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.454612 5059 scope.go:122] "RemoveContainer" containerID="14d62fb25c7489609d927c24fdefb001dac0545d210b73a0e4401b5dbf2e8e08" Nov 03 19:59:53.745271 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.731156 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m"] Nov 03 19:59:53.801239 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.801022 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-389e6d/ipp-route-389e6d-kserve-9fc78d8d5-cfv6m"] Nov 03 19:59:53.823698 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.823573 5059 scope.go:122] "RemoveContainer" containerID="14d62fb25c7489609d927c24fdefb001dac0545d210b73a0e4401b5dbf2e8e08" Nov 03 19:59:53.824005 ip-10-0-132-83 kubenswrapper[5059]: E1103 19:59:53.823942 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14d62fb25c7489609d927c24fdefb001dac0545d210b73a0e4401b5dbf2e8e08\": container with ID starting with 14d62fb25c7489609d927c24fdefb001dac0545d210b73a0e4401b5dbf2e8e08 not found: ID does not exist" containerID="14d62fb25c7489609d927c24fdefb001dac0545d210b73a0e4401b5dbf2e8e08" Nov 03 19:59:53.824134 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.824010 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d62fb25c7489609d927c24fdefb001dac0545d210b73a0e4401b5dbf2e8e08"} err="failed to get container status \"14d62fb25c7489609d927c24fdefb001dac0545d210b73a0e4401b5dbf2e8e08\": rpc error: code = NotFound desc = could not find container \"14d62fb25c7489609d927c24fdefb001dac0545d210b73a0e4401b5dbf2e8e08\": container with ID starting with 14d62fb25c7489609d927c24fdefb001dac0545d210b73a0e4401b5dbf2e8e08 not found: ID does not exist" Nov 03 19:59:53.904302 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.904249 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f12977e-2949-4de3-8c7b-2e42dbc495e7" path="/var/lib/kubelet/pods/5f12977e-2949-4de3-8c7b-2e42dbc495e7/volumes" Nov 03 19:59:53.913823 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.913798 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz" Nov 03 19:59:54.000685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.999412 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651eb1ff-2364-47df-96b5-5ae746240b93-kube-api-access-tg75s\" (UniqueName: \"kubernetes.io/projected/651eb1ff-2364-47df-96b5-5ae746240b93-kube-api-access-tg75s\") pod \"651eb1ff-2364-47df-96b5-5ae746240b93\" (UID: \"651eb1ff-2364-47df-96b5-5ae746240b93\") " Nov 03 19:59:54.000685 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:53.999502 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/651eb1ff-2364-47df-96b5-5ae746240b93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651eb1ff-2364-47df-96b5-5ae746240b93-plugins-config-volume\") pod \"651eb1ff-2364-47df-96b5-5ae746240b93\" (UID: \"651eb1ff-2364-47df-96b5-5ae746240b93\") " Nov 03 19:59:54.086020 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:54.085886 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/651eb1ff-2364-47df-96b5-5ae746240b93-plugins-config-volume" pod "651eb1ff-2364-47df-96b5-5ae746240b93" (UID: "651eb1ff-2364-47df-96b5-5ae746240b93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:59:54.099692 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:54.094962 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651eb1ff-2364-47df-96b5-5ae746240b93-kube-api-access-tg75s" pod "651eb1ff-2364-47df-96b5-5ae746240b93" (UID: "651eb1ff-2364-47df-96b5-5ae746240b93"). InnerVolumeSpecName "kube-api-access-tg75s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:59:54.106694 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:54.103561 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg75s\" (UniqueName: \"kubernetes.io/projected/651eb1ff-2364-47df-96b5-5ae746240b93-kube-api-access-tg75s\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:54.106694 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:54.103586 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651eb1ff-2364-47df-96b5-5ae746240b93-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:54.106694 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:54.103908 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq" Nov 03 19:59:54.204616 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:54.204529 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32e5da96-23df-4b40-a08e-7b2820065df5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32e5da96-23df-4b40-a08e-7b2820065df5-plugins-config-volume\") pod \"32e5da96-23df-4b40-a08e-7b2820065df5\" (UID: \"32e5da96-23df-4b40-a08e-7b2820065df5\") " Nov 03 19:59:54.204804 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:54.204736 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32e5da96-23df-4b40-a08e-7b2820065df5-kube-api-access-f5wc5\" (UniqueName: \"kubernetes.io/projected/32e5da96-23df-4b40-a08e-7b2820065df5-kube-api-access-f5wc5\") pod \"32e5da96-23df-4b40-a08e-7b2820065df5\" (UID: \"32e5da96-23df-4b40-a08e-7b2820065df5\") " Nov 03 19:59:54.204983 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:54.204951 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32e5da96-23df-4b40-a08e-7b2820065df5-plugins-config-volume" pod "32e5da96-23df-4b40-a08e-7b2820065df5" (UID: "32e5da96-23df-4b40-a08e-7b2820065df5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:59:54.205415 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:54.205392 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32e5da96-23df-4b40-a08e-7b2820065df5-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:54.206916 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:54.206893 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e5da96-23df-4b40-a08e-7b2820065df5-kube-api-access-f5wc5" pod "32e5da96-23df-4b40-a08e-7b2820065df5" (UID: "32e5da96-23df-4b40-a08e-7b2820065df5"). InnerVolumeSpecName "kube-api-access-f5wc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:59:54.307239 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:54.307201 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5wc5\" (UniqueName: \"kubernetes.io/projected/32e5da96-23df-4b40-a08e-7b2820065df5-kube-api-access-f5wc5\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:54.517191 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:54.517094 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" event={"ID":"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f","Type":"ContainerStarted","Data":"5ef15580b65fdc48c306f5da5fe11dc6b97f20f1719efaf5b37bb08947787e4b"} Nov 03 19:59:54.518745 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:54.518725 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz" Nov 03 19:59:54.518974 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:54.518952 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz" event={"ID":"651eb1ff-2364-47df-96b5-5ae746240b93","Type":"ContainerDied","Data":"113f30cd75fd14c486cd636f48d5f86808f2f402be1223a01413382023c30234"} Nov 03 19:59:54.519035 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:54.519004 5059 scope.go:122] "RemoveContainer" containerID="03bf2f26a83d43a9b790f05e408c4cff1d6803400918cfa45262e8ecbbb54561" Nov 03 19:59:54.520637 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:54.520613 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq" event={"ID":"32e5da96-23df-4b40-a08e-7b2820065df5","Type":"ContainerDied","Data":"fb8ac644ac69b76b76f9f4bae926f091ac917d63c65ff313475c32e274eb51ed"} Nov 03 19:59:54.520747 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:54.520729 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq" Nov 03 19:59:54.533199 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:54.533095 5059 scope.go:122] "RemoveContainer" containerID="f72fbd00f6d07d8fa957ff9cc5b40a950f89f07b6f10d37ae3f632ca24c1469f" Nov 03 19:59:54.543354 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:54.543312 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" podStartSLOduration=2.543300121 podStartE2EDuration="2.543300121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:54.54107532 +0000 UTC m=+1388.798872578" watchObservedRunningTime="2026-11-03 19:59:54.543300121 +0000 UTC m=+1388.801097377" Nov 03 19:59:54.594559 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:54.573165 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz"] Nov 03 19:59:54.615010 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:54.614977 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-5f74cbdd58-xl2wz"] Nov 03 19:59:54.620490 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:54.620459 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq"] Nov 03 19:59:54.624188 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:54.624162 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-df8756bc9-w59hq"] Nov 03 19:59:55.095031 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:55.095000 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 19:59:55.096728 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:55.096700 5059 patch_prober.go:29] interesting pod/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 19:59:55.096805 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:55.096763 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" podUID="b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 19:59:55.870747 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:55.867153 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32e5da96-23df-4b40-a08e-7b2820065df5" path="/var/lib/kubelet/pods/32e5da96-23df-4b40-a08e-7b2820065df5/volumes" Nov 03 19:59:55.870747 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:55.869612 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="651eb1ff-2364-47df-96b5-5ae746240b93" path="/var/lib/kubelet/pods/651eb1ff-2364-47df-96b5-5ae746240b93/volumes" Nov 03 19:59:56.095520 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:56.095486 5059 patch_prober.go:29] interesting pod/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 19:59:56.095884 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:56.095543 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" podUID="b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 19:59:57.094939 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:57.094903 5059 patch_prober.go:29] interesting pod/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 19:59:57.095123 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:57.094971 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" podUID="b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 19:59:58.095841 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:58.095794 5059 patch_prober.go:29] interesting pod/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 19:59:58.096342 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:58.095859 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" podUID="b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 19:59:59.095533 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:59.095452 5059 patch_prober.go:29] interesting pod/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 19:59:59.095533 ip-10-0-132-83 kubenswrapper[5059]: I1103 19:59:59.095521 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" podUID="b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 20:00:00.102685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.095473 5059 patch_prober.go:29] interesting pod/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:00:00.102685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.095579 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" podUID="b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 20:00:00.142829 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.142763 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2"] Nov 03 20:00:00.144122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.144091 5059 cpu_manager.go:414] "containerMap: removing container" podUID="32e5da96-23df-4b40-a08e-7b2820065df5" containerName="payload-pre-processing" Nov 03 20:00:00.144122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.144116 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e5da96-23df-4b40-a08e-7b2820065df5" containerName="payload-pre-processing" Nov 03 20:00:00.144327 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.144147 5059 cpu_manager.go:414] "containerMap: removing container" podUID="5f12977e-2949-4de3-8c7b-2e42dbc495e7" containerName="main" Nov 03 20:00:00.144327 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.144155 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f12977e-2949-4de3-8c7b-2e42dbc495e7" containerName="main" Nov 03 20:00:00.144327 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.144174 5059 cpu_manager.go:414] "containerMap: removing container" podUID="651eb1ff-2364-47df-96b5-5ae746240b93" containerName="payload-processing" Nov 03 20:00:00.144327 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.144182 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651eb1ff-2364-47df-96b5-5ae746240b93" containerName="payload-processing" Nov 03 20:00:00.144529 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.144365 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e5da96-23df-4b40-a08e-7b2820065df5" containerName="payload-pre-processing" Nov 03 20:00:00.144529 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.144380 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f12977e-2949-4de3-8c7b-2e42dbc495e7" containerName="main" Nov 03 20:00:00.144529 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.144392 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="651eb1ff-2364-47df-96b5-5ae746240b93" containerName="payload-processing" Nov 03 20:00:00.148598 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.148573 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2" Nov 03 20:00:00.154714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.152593 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6"] Nov 03 20:00:00.160466 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.160426 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:00:00.197003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.176578 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw"] Nov 03 20:00:00.197003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.176706 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6" Nov 03 20:00:00.197003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.190017 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x"] Nov 03 20:00:00.197003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.190797 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw" Nov 03 20:00:00.197003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.195527 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd"] Nov 03 20:00:00.197003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.196144 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x" Nov 03 20:00:00.203184 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.199883 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2"] Nov 03 20:00:00.203184 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.199923 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6"] Nov 03 20:00:00.203184 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.200006 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd" Nov 03 20:00:00.215079 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.214866 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw"] Nov 03 20:00:00.226511 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.224183 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd"] Nov 03 20:00:00.231751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.231721 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb"] Nov 03 20:00:00.233045 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.233013 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsqzz\" (UniqueName: \"kubernetes.io/projected/518d7432-1ff3-4ebf-8700-0a234e678892-kube-api-access-hsqzz\") pod \"maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2\" (UID: \"518d7432-1ff3-4ebf-8700-0a234e678892\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2" Nov 03 20:00:00.233453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.233429 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/518d7432-1ff3-4ebf-8700-0a234e678892-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2\" (UID: \"518d7432-1ff3-4ebf-8700-0a234e678892\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2" Nov 03 20:00:00.239954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.239929 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb" Nov 03 20:00:00.253720 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.245278 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl"] Nov 03 20:00:00.253720 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.249975 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl" Nov 03 20:00:00.263474 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.262762 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb"] Nov 03 20:00:00.273706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.270834 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl"] Nov 03 20:00:00.278050 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.278027 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5"] Nov 03 20:00:00.284713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.283972 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x"] Nov 03 20:00:00.284713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.284083 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5" Nov 03 20:00:00.287086 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.287065 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d"] Nov 03 20:00:00.290916 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.290763 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d" Nov 03 20:00:00.294054 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.294034 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5"] Nov 03 20:00:00.298014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.297992 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d"] Nov 03 20:00:00.335650 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.335608 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5zkh\" (UniqueName: \"kubernetes.io/projected/c3c45aa4-c24e-4674-a258-38226dad4b13-kube-api-access-g5zkh\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl\" (UID: \"c3c45aa4-c24e-4674-a258-38226dad4b13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl" Nov 03 20:00:00.335849 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.335756 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsqzz\" (UniqueName: \"kubernetes.io/projected/518d7432-1ff3-4ebf-8700-0a234e678892-kube-api-access-hsqzz\") pod \"maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2\" (UID: \"518d7432-1ff3-4ebf-8700-0a234e678892\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2" Nov 03 20:00:00.335919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.335874 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/799b3fa3-1f7a-46c8-868f-1a1624b1fd9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd\" (UID: \"799b3fa3-1f7a-46c8-868f-1a1624b1fd9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd" Nov 03 20:00:00.335994 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.335980 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99fe4ede-51a2-4f4a-a5ce-96faef90590c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb\" (UID: \"99fe4ede-51a2-4f4a-a5ce-96faef90590c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb" Nov 03 20:00:00.336128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.336003 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37f7af3e-9c65-42c8-b761-306fb918190c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x\" (UID: \"37f7af3e-9c65-42c8-b761-306fb918190c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x" Nov 03 20:00:00.336210 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.336183 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwn4m\" (UniqueName: \"kubernetes.io/projected/b8602537-2de3-4fc6-84f5-7306b45cc4fe-kube-api-access-wwn4m\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6\" (UID: \"b8602537-2de3-4fc6-84f5-7306b45cc4fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6" Nov 03 20:00:00.336270 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.336230 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzc29\" (UniqueName: \"kubernetes.io/projected/799b3fa3-1f7a-46c8-868f-1a1624b1fd9c-kube-api-access-bzc29\") pod \"maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd\" (UID: \"799b3fa3-1f7a-46c8-868f-1a1624b1fd9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd" Nov 03 20:00:00.336329 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.336283 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8602537-2de3-4fc6-84f5-7306b45cc4fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6\" (UID: \"b8602537-2de3-4fc6-84f5-7306b45cc4fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6" Nov 03 20:00:00.336407 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.336331 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz47r\" (UniqueName: \"kubernetes.io/projected/37f7af3e-9c65-42c8-b761-306fb918190c-kube-api-access-zz47r\") pod \"maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x\" (UID: \"37f7af3e-9c65-42c8-b761-306fb918190c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x" Nov 03 20:00:00.336530 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.336510 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df8b2fc-3ed3-4396-a215-1f66b95e0937-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw\" (UID: \"5df8b2fc-3ed3-4396-a215-1f66b95e0937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw" Nov 03 20:00:00.336611 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.336573 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/518d7432-1ff3-4ebf-8700-0a234e678892-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2\" (UID: \"518d7432-1ff3-4ebf-8700-0a234e678892\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2" Nov 03 20:00:00.336611 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.336603 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwdqw\" (UniqueName: \"kubernetes.io/projected/99fe4ede-51a2-4f4a-a5ce-96faef90590c-kube-api-access-jwdqw\") pod \"maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb\" (UID: \"99fe4ede-51a2-4f4a-a5ce-96faef90590c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb" Nov 03 20:00:00.336778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.336643 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcksp\" (UniqueName: \"kubernetes.io/projected/5df8b2fc-3ed3-4396-a215-1f66b95e0937-kube-api-access-vcksp\") pod \"maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw\" (UID: \"5df8b2fc-3ed3-4396-a215-1f66b95e0937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw" Nov 03 20:00:00.336778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.336718 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c45aa4-c24e-4674-a258-38226dad4b13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl\" (UID: \"c3c45aa4-c24e-4674-a258-38226dad4b13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl" Nov 03 20:00:00.337578 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.337551 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/518d7432-1ff3-4ebf-8700-0a234e678892-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2\" (UID: \"518d7432-1ff3-4ebf-8700-0a234e678892\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2" Nov 03 20:00:00.349399 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.349367 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsqzz\" (UniqueName: \"kubernetes.io/projected/518d7432-1ff3-4ebf-8700-0a234e678892-kube-api-access-hsqzz\") pod \"maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2\" (UID: \"518d7432-1ff3-4ebf-8700-0a234e678892\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2" Nov 03 20:00:00.438373 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.438292 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cclk\" (UniqueName: \"kubernetes.io/projected/2b8f9144-8073-42b2-b859-bdde4c32ecd6-kube-api-access-4cclk\") pod \"maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d\" (UID: \"2b8f9144-8073-42b2-b859-bdde4c32ecd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d" Nov 03 20:00:00.438514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.438426 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5zkh\" (UniqueName: \"kubernetes.io/projected/c3c45aa4-c24e-4674-a258-38226dad4b13-kube-api-access-g5zkh\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl\" (UID: \"c3c45aa4-c24e-4674-a258-38226dad4b13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl" Nov 03 20:00:00.438514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.438467 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75da83f6-395e-4ca2-8709-7956a4dd7eba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5\" (UID: \"75da83f6-395e-4ca2-8709-7956a4dd7eba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5" Nov 03 20:00:00.438514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.438505 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/799b3fa3-1f7a-46c8-868f-1a1624b1fd9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd\" (UID: \"799b3fa3-1f7a-46c8-868f-1a1624b1fd9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd" Nov 03 20:00:00.438703 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.438582 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b8f9144-8073-42b2-b859-bdde4c32ecd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d\" (UID: \"2b8f9144-8073-42b2-b859-bdde4c32ecd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d" Nov 03 20:00:00.438782 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.438764 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99fe4ede-51a2-4f4a-a5ce-96faef90590c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb\" (UID: \"99fe4ede-51a2-4f4a-a5ce-96faef90590c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb" Nov 03 20:00:00.438845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.438791 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37f7af3e-9c65-42c8-b761-306fb918190c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x\" (UID: \"37f7af3e-9c65-42c8-b761-306fb918190c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x" Nov 03 20:00:00.438845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.438810 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwn4m\" (UniqueName: \"kubernetes.io/projected/b8602537-2de3-4fc6-84f5-7306b45cc4fe-kube-api-access-wwn4m\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6\" (UID: \"b8602537-2de3-4fc6-84f5-7306b45cc4fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6" Nov 03 20:00:00.438845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.438841 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzc29\" (UniqueName: \"kubernetes.io/projected/799b3fa3-1f7a-46c8-868f-1a1624b1fd9c-kube-api-access-bzc29\") pod \"maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd\" (UID: \"799b3fa3-1f7a-46c8-868f-1a1624b1fd9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd" Nov 03 20:00:00.439008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.438887 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8602537-2de3-4fc6-84f5-7306b45cc4fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6\" (UID: \"b8602537-2de3-4fc6-84f5-7306b45cc4fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6" Nov 03 20:00:00.439008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.438931 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz47r\" (UniqueName: \"kubernetes.io/projected/37f7af3e-9c65-42c8-b761-306fb918190c-kube-api-access-zz47r\") pod \"maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x\" (UID: \"37f7af3e-9c65-42c8-b761-306fb918190c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x" Nov 03 20:00:00.439008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.438972 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kggm\" (UniqueName: \"kubernetes.io/projected/75da83f6-395e-4ca2-8709-7956a4dd7eba-kube-api-access-2kggm\") pod \"maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5\" (UID: \"75da83f6-395e-4ca2-8709-7956a4dd7eba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5" Nov 03 20:00:00.439171 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.439041 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df8b2fc-3ed3-4396-a215-1f66b95e0937-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw\" (UID: \"5df8b2fc-3ed3-4396-a215-1f66b95e0937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw" Nov 03 20:00:00.439171 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.439101 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwdqw\" (UniqueName: \"kubernetes.io/projected/99fe4ede-51a2-4f4a-a5ce-96faef90590c-kube-api-access-jwdqw\") pod \"maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb\" (UID: \"99fe4ede-51a2-4f4a-a5ce-96faef90590c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb" Nov 03 20:00:00.439171 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.439142 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcksp\" (UniqueName: \"kubernetes.io/projected/5df8b2fc-3ed3-4396-a215-1f66b95e0937-kube-api-access-vcksp\") pod \"maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw\" (UID: \"5df8b2fc-3ed3-4396-a215-1f66b95e0937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw" Nov 03 20:00:00.439332 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.439170 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c45aa4-c24e-4674-a258-38226dad4b13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl\" (UID: \"c3c45aa4-c24e-4674-a258-38226dad4b13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl" Nov 03 20:00:00.439599 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.439385 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/799b3fa3-1f7a-46c8-868f-1a1624b1fd9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd\" (UID: \"799b3fa3-1f7a-46c8-868f-1a1624b1fd9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd" Nov 03 20:00:00.439740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.439638 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99fe4ede-51a2-4f4a-a5ce-96faef90590c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb\" (UID: \"99fe4ede-51a2-4f4a-a5ce-96faef90590c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb" Nov 03 20:00:00.439740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.439712 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37f7af3e-9c65-42c8-b761-306fb918190c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x\" (UID: \"37f7af3e-9c65-42c8-b761-306fb918190c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x" Nov 03 20:00:00.439869 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.439747 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8602537-2de3-4fc6-84f5-7306b45cc4fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6\" (UID: \"b8602537-2de3-4fc6-84f5-7306b45cc4fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6" Nov 03 20:00:00.439934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.439919 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df8b2fc-3ed3-4396-a215-1f66b95e0937-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw\" (UID: \"5df8b2fc-3ed3-4396-a215-1f66b95e0937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw" Nov 03 20:00:00.439934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.439927 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c45aa4-c24e-4674-a258-38226dad4b13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl\" (UID: \"c3c45aa4-c24e-4674-a258-38226dad4b13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl" Nov 03 20:00:00.520741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.482127 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzc29\" (UniqueName: \"kubernetes.io/projected/799b3fa3-1f7a-46c8-868f-1a1624b1fd9c-kube-api-access-bzc29\") pod \"maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd\" (UID: \"799b3fa3-1f7a-46c8-868f-1a1624b1fd9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd" Nov 03 20:00:00.520741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.487299 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5zkh\" (UniqueName: \"kubernetes.io/projected/c3c45aa4-c24e-4674-a258-38226dad4b13-kube-api-access-g5zkh\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl\" (UID: \"c3c45aa4-c24e-4674-a258-38226dad4b13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl" Nov 03 20:00:00.520741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.489031 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2" Nov 03 20:00:00.520741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.491372 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwdqw\" (UniqueName: \"kubernetes.io/projected/99fe4ede-51a2-4f4a-a5ce-96faef90590c-kube-api-access-jwdqw\") pod \"maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb\" (UID: \"99fe4ede-51a2-4f4a-a5ce-96faef90590c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb" Nov 03 20:00:00.520741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.493755 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcksp\" (UniqueName: \"kubernetes.io/projected/5df8b2fc-3ed3-4396-a215-1f66b95e0937-kube-api-access-vcksp\") pod \"maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw\" (UID: \"5df8b2fc-3ed3-4396-a215-1f66b95e0937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw" Nov 03 20:00:00.520741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.494846 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwn4m\" (UniqueName: \"kubernetes.io/projected/b8602537-2de3-4fc6-84f5-7306b45cc4fe-kube-api-access-wwn4m\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6\" (UID: \"b8602537-2de3-4fc6-84f5-7306b45cc4fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6" Nov 03 20:00:00.520741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.506246 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz47r\" (UniqueName: \"kubernetes.io/projected/37f7af3e-9c65-42c8-b761-306fb918190c-kube-api-access-zz47r\") pod \"maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x\" (UID: \"37f7af3e-9c65-42c8-b761-306fb918190c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x" Nov 03 20:00:00.530003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.529971 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6" Nov 03 20:00:00.540803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.540773 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kggm\" (UniqueName: \"kubernetes.io/projected/75da83f6-395e-4ca2-8709-7956a4dd7eba-kube-api-access-2kggm\") pod \"maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5\" (UID: \"75da83f6-395e-4ca2-8709-7956a4dd7eba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5" Nov 03 20:00:00.540984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.540962 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cclk\" (UniqueName: \"kubernetes.io/projected/2b8f9144-8073-42b2-b859-bdde4c32ecd6-kube-api-access-4cclk\") pod \"maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d\" (UID: \"2b8f9144-8073-42b2-b859-bdde4c32ecd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d" Nov 03 20:00:00.542109 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.541072 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75da83f6-395e-4ca2-8709-7956a4dd7eba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5\" (UID: \"75da83f6-395e-4ca2-8709-7956a4dd7eba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5" Nov 03 20:00:00.542109 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.541162 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b8f9144-8073-42b2-b859-bdde4c32ecd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d\" (UID: \"2b8f9144-8073-42b2-b859-bdde4c32ecd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d" Nov 03 20:00:00.542109 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.542040 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b8f9144-8073-42b2-b859-bdde4c32ecd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d\" (UID: \"2b8f9144-8073-42b2-b859-bdde4c32ecd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d" Nov 03 20:00:00.542109 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.542066 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75da83f6-395e-4ca2-8709-7956a4dd7eba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5\" (UID: \"75da83f6-395e-4ca2-8709-7956a4dd7eba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5" Nov 03 20:00:00.598986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.551214 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw" Nov 03 20:00:00.598986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.562802 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x" Nov 03 20:00:00.598986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.564347 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kggm\" (UniqueName: \"kubernetes.io/projected/75da83f6-395e-4ca2-8709-7956a4dd7eba-kube-api-access-2kggm\") pod \"maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5\" (UID: \"75da83f6-395e-4ca2-8709-7956a4dd7eba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5" Nov 03 20:00:00.598986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.578863 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd" Nov 03 20:00:00.598986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.591977 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb" Nov 03 20:00:00.603091 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.603038 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cclk\" (UniqueName: \"kubernetes.io/projected/2b8f9144-8073-42b2-b859-bdde4c32ecd6-kube-api-access-4cclk\") pod \"maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d\" (UID: \"2b8f9144-8073-42b2-b859-bdde4c32ecd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d" Nov 03 20:00:00.605698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.605671 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl" Nov 03 20:00:00.618423 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.618381 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5" Nov 03 20:00:00.630504 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:00.630458 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d" Nov 03 20:00:01.065336 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:00:01.065301 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod518d7432_1ff3_4ebf_8700_0a234e678892.slice/crio-70b9863c4b051ff81944b8885ce681bcc092c3937e11da991fc35c59aa161dec WatchSource:0}: Error finding container 70b9863c4b051ff81944b8885ce681bcc092c3937e11da991fc35c59aa161dec: Status 404 returned error can't find the container with id 70b9863c4b051ff81944b8885ce681bcc092c3937e11da991fc35c59aa161dec Nov 03 20:00:01.065863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.065838 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2"] Nov 03 20:00:01.097736 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.097708 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6"] Nov 03 20:00:01.100789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.100767 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 20:00:01.101251 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.101232 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 20:00:01.101988 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.101970 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 20:00:01.173698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.172813 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5"] Nov 03 20:00:01.232518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.232488 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw"] Nov 03 20:00:01.236793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.236767 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb"] Nov 03 20:00:01.242980 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.242900 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl"] Nov 03 20:00:01.247740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.247717 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd"] Nov 03 20:00:01.251105 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.251083 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x"] Nov 03 20:00:01.340747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.340721 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d"] Nov 03 20:00:01.361283 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:00:01.361243 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df8b2fc_3ed3_4396_a215_1f66b95e0937.slice/crio-2dab556ed97636f91e4317ffdced56a72e8ef250889ad7a73e83c8994cf8e244 WatchSource:0}: Error finding container 2dab556ed97636f91e4317ffdced56a72e8ef250889ad7a73e83c8994cf8e244: Status 404 returned error can't find the container with id 2dab556ed97636f91e4317ffdced56a72e8ef250889ad7a73e83c8994cf8e244 Nov 03 20:00:01.362258 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:00:01.362234 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99fe4ede_51a2_4f4a_a5ce_96faef90590c.slice/crio-dfd3c40fcde38ffebc140d10ad17c255199bdc648090d6e82cbc3ac11a0c5abb WatchSource:0}: Error finding container dfd3c40fcde38ffebc140d10ad17c255199bdc648090d6e82cbc3ac11a0c5abb: Status 404 returned error can't find the container with id dfd3c40fcde38ffebc140d10ad17c255199bdc648090d6e82cbc3ac11a0c5abb Nov 03 20:00:01.363834 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:00:01.363811 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c45aa4_c24e_4674_a258_38226dad4b13.slice/crio-f98b857a275b2d503ba9c9f119299e2eeba30b75d72a9a7ceb713b48b47b4fb3 WatchSource:0}: Error finding container f98b857a275b2d503ba9c9f119299e2eeba30b75d72a9a7ceb713b48b47b4fb3: Status 404 returned error can't find the container with id f98b857a275b2d503ba9c9f119299e2eeba30b75d72a9a7ceb713b48b47b4fb3 Nov 03 20:00:01.427985 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.426885 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2" Nov 03 20:00:01.694498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.694466 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x" event={"ID":"37f7af3e-9c65-42c8-b761-306fb918190c","Type":"ContainerStarted","Data":"81eff7dba9d1970fa536c36097474a7ed33b61fde1fe5bd0084634e025d5bef4"} Nov 03 20:00:01.695959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.695932 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl" event={"ID":"c3c45aa4-c24e-4674-a258-38226dad4b13","Type":"ContainerStarted","Data":"f98b857a275b2d503ba9c9f119299e2eeba30b75d72a9a7ceb713b48b47b4fb3"} Nov 03 20:00:01.697255 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.697202 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d" event={"ID":"2b8f9144-8073-42b2-b859-bdde4c32ecd6","Type":"ContainerStarted","Data":"400e8939540a0fb5ecbcbaec7f6f02e2413ea24523004f0fb12e91588c1aa8b6"} Nov 03 20:00:01.698447 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.698423 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb" event={"ID":"99fe4ede-51a2-4f4a-a5ce-96faef90590c","Type":"ContainerStarted","Data":"dfd3c40fcde38ffebc140d10ad17c255199bdc648090d6e82cbc3ac11a0c5abb"} Nov 03 20:00:01.699609 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.699587 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw" event={"ID":"5df8b2fc-3ed3-4396-a215-1f66b95e0937","Type":"ContainerStarted","Data":"2dab556ed97636f91e4317ffdced56a72e8ef250889ad7a73e83c8994cf8e244"} Nov 03 20:00:01.700776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.700755 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd" event={"ID":"799b3fa3-1f7a-46c8-868f-1a1624b1fd9c","Type":"ContainerStarted","Data":"086e56594f78343cb98cda3379d29d04d228d1d7a7863e4478e129d3cdbb6daa"} Nov 03 20:00:01.702231 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.702208 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6" event={"ID":"b8602537-2de3-4fc6-84f5-7306b45cc4fe","Type":"ContainerStarted","Data":"313f06767babde83dc131261a73503200d8113db2287f718ba8190e3b327d894"} Nov 03 20:00:01.702333 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.702246 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6" event={"ID":"b8602537-2de3-4fc6-84f5-7306b45cc4fe","Type":"ContainerStarted","Data":"0825df6e29841a9b09b305b7b04ff124a7470ef14c658431e5ced47dc14d6334"} Nov 03 20:00:01.703479 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.703458 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5" event={"ID":"75da83f6-395e-4ca2-8709-7956a4dd7eba","Type":"ContainerStarted","Data":"e97f68217e5465f19bb2d0cb7b204c719361ab0e755aace6f1b195002ddfb145"} Nov 03 20:00:01.705103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.705083 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2" event={"ID":"518d7432-1ff3-4ebf-8700-0a234e678892","Type":"ContainerStarted","Data":"10ba7846f6d0c78de624af9a65b0a3ade4aaffb4f53c47ff230c3fefcd27c62c"} Nov 03 20:00:01.705199 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.705121 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2" event={"ID":"518d7432-1ff3-4ebf-8700-0a234e678892","Type":"ContainerStarted","Data":"70b9863c4b051ff81944b8885ce681bcc092c3937e11da991fc35c59aa161dec"} Nov 03 20:00:01.706608 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.706583 5059 generic.go:356] "Generic (PLEG): container finished" podID="8531b4f3-5179-4948-86ce-52e13576ea11" containerID="75e02563ce0318f1dbc83f69e4cc5d059fe289723fa324d9d3379736d1b57fdf" exitCode=0 Nov 03 20:00:01.706723 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.706654 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" event={"ID":"8531b4f3-5179-4948-86ce-52e13576ea11","Type":"ContainerDied","Data":"75e02563ce0318f1dbc83f69e4cc5d059fe289723fa324d9d3379736d1b57fdf"} Nov 03 20:00:01.707268 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.707253 5059 scope.go:122] "RemoveContainer" containerID="75e02563ce0318f1dbc83f69e4cc5d059fe289723fa324d9d3379736d1b57fdf" Nov 03 20:00:01.708883 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.708849 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:00:01.708959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.708937 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:00:01.709002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.708991 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:00:01.743629 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.743556 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6" podStartSLOduration=1.74354211 podStartE2EDuration="1.74354211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:01.72104019 +0000 UTC m=+1395.978837447" watchObservedRunningTime="2026-11-03 20:00:01.74354211 +0000 UTC m=+1396.001339367" Nov 03 20:00:01.771735 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:01.769067 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2" podStartSLOduration=1.769053041 podStartE2EDuration="1.769053041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:01.744884238 +0000 UTC m=+1396.002681496" watchObservedRunningTime="2026-11-03 20:00:01.769053041 +0000 UTC m=+1396.026850297" Nov 03 20:00:02.718840 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.718791 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd" event={"ID":"799b3fa3-1f7a-46c8-868f-1a1624b1fd9c","Type":"ContainerStarted","Data":"565c68d3bacbe4bb87c33a320396deaee022558594fda308b682877cc3029d67"} Nov 03 20:00:02.720636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.720606 5059 generic.go:356] "Generic (PLEG): container finished" podID="b8602537-2de3-4fc6-84f5-7306b45cc4fe" containerID="313f06767babde83dc131261a73503200d8113db2287f718ba8190e3b327d894" exitCode=60 Nov 03 20:00:02.720821 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.720736 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6" event={"ID":"b8602537-2de3-4fc6-84f5-7306b45cc4fe","Type":"ContainerDied","Data":"313f06767babde83dc131261a73503200d8113db2287f718ba8190e3b327d894"} Nov 03 20:00:02.721078 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.721058 5059 scope.go:122] "RemoveContainer" containerID="313f06767babde83dc131261a73503200d8113db2287f718ba8190e3b327d894" Nov 03 20:00:02.722459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.722434 5059 generic.go:356] "Generic (PLEG): container finished" podID="75da83f6-395e-4ca2-8709-7956a4dd7eba" containerID="092893fda72edead9b05d269ca741d73fae833ea1fdf98ec48d9e75bc85bcb89" exitCode=60 Nov 03 20:00:02.722561 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.722499 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5" event={"ID":"75da83f6-395e-4ca2-8709-7956a4dd7eba","Type":"ContainerDied","Data":"092893fda72edead9b05d269ca741d73fae833ea1fdf98ec48d9e75bc85bcb89"} Nov 03 20:00:02.722812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.722794 5059 scope.go:122] "RemoveContainer" containerID="092893fda72edead9b05d269ca741d73fae833ea1fdf98ec48d9e75bc85bcb89" Nov 03 20:00:02.725203 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.725179 5059 generic.go:356] "Generic (PLEG): container finished" podID="518d7432-1ff3-4ebf-8700-0a234e678892" containerID="10ba7846f6d0c78de624af9a65b0a3ade4aaffb4f53c47ff230c3fefcd27c62c" exitCode=60 Nov 03 20:00:02.725280 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.725260 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2" event={"ID":"518d7432-1ff3-4ebf-8700-0a234e678892","Type":"ContainerDied","Data":"10ba7846f6d0c78de624af9a65b0a3ade4aaffb4f53c47ff230c3fefcd27c62c"} Nov 03 20:00:02.725498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.725481 5059 scope.go:122] "RemoveContainer" containerID="10ba7846f6d0c78de624af9a65b0a3ade4aaffb4f53c47ff230c3fefcd27c62c" Nov 03 20:00:02.728654 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.728632 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" event={"ID":"8531b4f3-5179-4948-86ce-52e13576ea11","Type":"ContainerStarted","Data":"214a53ef694c00238d5e914252f452ff7a8677228ee8c69a81ba92d007afde19"} Nov 03 20:00:02.730361 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.730334 5059 generic.go:356] "Generic (PLEG): container finished" podID="37f7af3e-9c65-42c8-b761-306fb918190c" containerID="773a43a1d399d97775a58d7bbadcfc95b3f692c54301dd6543b08a4944f67ce0" exitCode=60 Nov 03 20:00:02.730463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.730414 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x" event={"ID":"37f7af3e-9c65-42c8-b761-306fb918190c","Type":"ContainerDied","Data":"773a43a1d399d97775a58d7bbadcfc95b3f692c54301dd6543b08a4944f67ce0"} Nov 03 20:00:02.730608 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.730592 5059 scope.go:122] "RemoveContainer" containerID="773a43a1d399d97775a58d7bbadcfc95b3f692c54301dd6543b08a4944f67ce0" Nov 03 20:00:02.731888 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.731869 5059 generic.go:356] "Generic (PLEG): container finished" podID="c3c45aa4-c24e-4674-a258-38226dad4b13" containerID="c3636fa11e95e1bcfee803e2a7ac00ff7a85416aded39c735a4194842fdb117d" exitCode=60 Nov 03 20:00:02.731978 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.731961 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl" event={"ID":"c3c45aa4-c24e-4674-a258-38226dad4b13","Type":"ContainerDied","Data":"c3636fa11e95e1bcfee803e2a7ac00ff7a85416aded39c735a4194842fdb117d"} Nov 03 20:00:02.732159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.732143 5059 scope.go:122] "RemoveContainer" containerID="c3636fa11e95e1bcfee803e2a7ac00ff7a85416aded39c735a4194842fdb117d" Nov 03 20:00:02.735112 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.735089 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d" event={"ID":"2b8f9144-8073-42b2-b859-bdde4c32ecd6","Type":"ContainerStarted","Data":"75c912f9ac6e94144af4a232fcba7af26217d7d8e64756841799fa2d3dc1af91"} Nov 03 20:00:02.743526 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.742218 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd" podStartSLOduration=2.74218875 podStartE2EDuration="2.74218875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:02.737300528 +0000 UTC m=+1396.995097792" watchObservedRunningTime="2026-11-03 20:00:02.74218875 +0000 UTC m=+1396.999986009" Nov 03 20:00:02.743526 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.742781 5059 generic.go:356] "Generic (PLEG): container finished" podID="99fe4ede-51a2-4f4a-a5ce-96faef90590c" containerID="ee89649eb5c99d8c60000c91f0de332dfcaccd5864dc759afe77ff0c9f735c62" exitCode=60 Nov 03 20:00:02.743526 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.742937 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb" event={"ID":"99fe4ede-51a2-4f4a-a5ce-96faef90590c","Type":"ContainerDied","Data":"ee89649eb5c99d8c60000c91f0de332dfcaccd5864dc759afe77ff0c9f735c62"} Nov 03 20:00:02.743526 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.743349 5059 scope.go:122] "RemoveContainer" containerID="ee89649eb5c99d8c60000c91f0de332dfcaccd5864dc759afe77ff0c9f735c62" Nov 03 20:00:02.774258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.773797 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw" event={"ID":"5df8b2fc-3ed3-4396-a215-1f66b95e0937","Type":"ContainerStarted","Data":"0dda1f081e22d78273f5f0603530f23ba716acfd9252ce53e4fd32f69bb8de66"} Nov 03 20:00:02.778685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.778226 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d" podStartSLOduration=2.778209155 podStartE2EDuration="2.778209155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:02.756268834 +0000 UTC m=+1397.014066092" watchObservedRunningTime="2026-11-03 20:00:02.778209155 +0000 UTC m=+1397.036006413" Nov 03 20:00:02.926802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:02.926750 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw" podStartSLOduration=2.926736187 podStartE2EDuration="2.926736187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:02.924042884 +0000 UTC m=+1397.181840141" watchObservedRunningTime="2026-11-03 20:00:02.926736187 +0000 UTC m=+1397.184533444" Nov 03 20:00:03.214257 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:03.214076 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 20:00:03.246440 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:03.245399 5059 patch_prober.go:29] interesting pod/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 20:00:03.246440 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:03.245482 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" podUID="8531b4f3-5179-4948-86ce-52e13576ea11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 20:00:03.797632 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:03.797548 5059 generic.go:356] "Generic (PLEG): container finished" podID="799b3fa3-1f7a-46c8-868f-1a1624b1fd9c" containerID="565c68d3bacbe4bb87c33a320396deaee022558594fda308b682877cc3029d67" exitCode=60 Nov 03 20:00:03.798091 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:03.797962 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd" event={"ID":"799b3fa3-1f7a-46c8-868f-1a1624b1fd9c","Type":"ContainerDied","Data":"565c68d3bacbe4bb87c33a320396deaee022558594fda308b682877cc3029d67"} Nov 03 20:00:03.798091 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:03.798018 5059 scope.go:122] "RemoveContainer" containerID="565c68d3bacbe4bb87c33a320396deaee022558594fda308b682877cc3029d67" Nov 03 20:00:03.801051 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:03.801028 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6" event={"ID":"b8602537-2de3-4fc6-84f5-7306b45cc4fe","Type":"ContainerStarted","Data":"5746e718d3c459f9d4121a5d6686cc2013fae7c0f8ac73b30acd15942a58ad97"} Nov 03 20:00:03.803170 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:03.803143 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5" event={"ID":"75da83f6-395e-4ca2-8709-7956a4dd7eba","Type":"ContainerStarted","Data":"9c0550441ef8d0ab0b92e3de1d087a1323d5062f85c8d8ab9ff6a119ac571734"} Nov 03 20:00:03.805548 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:03.805522 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2" event={"ID":"518d7432-1ff3-4ebf-8700-0a234e678892","Type":"ContainerStarted","Data":"180b1dddad368b4811d3379474f62cecb9201a1a708e105c9ba4d7966702c4b5"} Nov 03 20:00:03.807478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:03.807459 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x" event={"ID":"37f7af3e-9c65-42c8-b761-306fb918190c","Type":"ContainerStarted","Data":"e10a073f920433a0608c7289fc84d3e11e924ea8ab02864f13bd49fb50ae56e8"} Nov 03 20:00:03.809481 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:03.809464 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl" event={"ID":"c3c45aa4-c24e-4674-a258-38226dad4b13","Type":"ContainerStarted","Data":"ecebbd87cd15bd645ee6d97deb6ce8dc69db97441c11cc06d06d767e917262cf"} Nov 03 20:00:03.812222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:03.812010 5059 generic.go:356] "Generic (PLEG): container finished" podID="2b8f9144-8073-42b2-b859-bdde4c32ecd6" containerID="75c912f9ac6e94144af4a232fcba7af26217d7d8e64756841799fa2d3dc1af91" exitCode=60 Nov 03 20:00:03.812222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:03.812101 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d" event={"ID":"2b8f9144-8073-42b2-b859-bdde4c32ecd6","Type":"ContainerDied","Data":"75c912f9ac6e94144af4a232fcba7af26217d7d8e64756841799fa2d3dc1af91"} Nov 03 20:00:03.812361 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:03.812341 5059 scope.go:122] "RemoveContainer" containerID="75c912f9ac6e94144af4a232fcba7af26217d7d8e64756841799fa2d3dc1af91" Nov 03 20:00:03.814590 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:03.814463 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb" event={"ID":"99fe4ede-51a2-4f4a-a5ce-96faef90590c","Type":"ContainerStarted","Data":"15f43a219d75b82f14f4d66cb36d5dfdf5c7ca080f1798d22665281fe9381ace"} Nov 03 20:00:03.816720 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:03.816046 5059 generic.go:356] "Generic (PLEG): container finished" podID="5df8b2fc-3ed3-4396-a215-1f66b95e0937" containerID="0dda1f081e22d78273f5f0603530f23ba716acfd9252ce53e4fd32f69bb8de66" exitCode=60 Nov 03 20:00:03.816720 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:03.816432 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw" event={"ID":"5df8b2fc-3ed3-4396-a215-1f66b95e0937","Type":"ContainerDied","Data":"0dda1f081e22d78273f5f0603530f23ba716acfd9252ce53e4fd32f69bb8de66"} Nov 03 20:00:03.816720 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:03.816697 5059 scope.go:122] "RemoveContainer" containerID="0dda1f081e22d78273f5f0603530f23ba716acfd9252ce53e4fd32f69bb8de66" Nov 03 20:00:03.909396 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:03.909348 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x" podStartSLOduration=3.909334159 podStartE2EDuration="3.909334159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:03.904493513 +0000 UTC m=+1398.162290771" watchObservedRunningTime="2026-11-03 20:00:03.909334159 +0000 UTC m=+1398.167131436" Nov 03 20:00:03.942509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:03.942454 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5" podStartSLOduration=3.942437254 podStartE2EDuration="3.942437254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:03.939994465 +0000 UTC m=+1398.197791723" watchObservedRunningTime="2026-11-03 20:00:03.942437254 +0000 UTC m=+1398.200234512" Nov 03 20:00:03.992723 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:03.992656 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb" podStartSLOduration=3.992640549 podStartE2EDuration="3.992640549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:03.979625781 +0000 UTC m=+1398.237423040" watchObservedRunningTime="2026-11-03 20:00:03.992640549 +0000 UTC m=+1398.250437807" Nov 03 20:00:04.001582 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.001531 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl" podStartSLOduration=4.001517859 podStartE2EDuration="4.001517859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:03.997471191 +0000 UTC m=+1398.255268449" watchObservedRunningTime="2026-11-03 20:00:04.001517859 +0000 UTC m=+1398.259315116" Nov 03 20:00:04.196371 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.196340 5059 patch_prober.go:29] interesting pod/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 20:00:04.196509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.196398 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" podUID="8531b4f3-5179-4948-86ce-52e13576ea11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 20:00:04.831534 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.831503 5059 generic.go:356] "Generic (PLEG): container finished" podID="37f7af3e-9c65-42c8-b761-306fb918190c" containerID="e10a073f920433a0608c7289fc84d3e11e924ea8ab02864f13bd49fb50ae56e8" exitCode=60 Nov 03 20:00:04.832042 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.831589 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x" event={"ID":"37f7af3e-9c65-42c8-b761-306fb918190c","Type":"ContainerDied","Data":"e10a073f920433a0608c7289fc84d3e11e924ea8ab02864f13bd49fb50ae56e8"} Nov 03 20:00:04.832042 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.831650 5059 scope.go:122] "RemoveContainer" containerID="773a43a1d399d97775a58d7bbadcfc95b3f692c54301dd6543b08a4944f67ce0" Nov 03 20:00:04.832162 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.832063 5059 scope.go:122] "RemoveContainer" containerID="e10a073f920433a0608c7289fc84d3e11e924ea8ab02864f13bd49fb50ae56e8" Nov 03 20:00:04.832503 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:04.832478 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x_odh-ai-gateway-infra(37f7af3e-9c65-42c8-b761-306fb918190c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x" podUID="37f7af3e-9c65-42c8-b761-306fb918190c" Nov 03 20:00:04.833654 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.833633 5059 generic.go:356] "Generic (PLEG): container finished" podID="c3c45aa4-c24e-4674-a258-38226dad4b13" containerID="ecebbd87cd15bd645ee6d97deb6ce8dc69db97441c11cc06d06d767e917262cf" exitCode=60 Nov 03 20:00:04.833760 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.833702 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl" event={"ID":"c3c45aa4-c24e-4674-a258-38226dad4b13","Type":"ContainerDied","Data":"ecebbd87cd15bd645ee6d97deb6ce8dc69db97441c11cc06d06d767e917262cf"} Nov 03 20:00:04.834026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.834009 5059 scope.go:122] "RemoveContainer" containerID="ecebbd87cd15bd645ee6d97deb6ce8dc69db97441c11cc06d06d767e917262cf" Nov 03 20:00:04.834373 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:04.834352 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl_odh-ai-gateway-infra(c3c45aa4-c24e-4674-a258-38226dad4b13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl" podUID="c3c45aa4-c24e-4674-a258-38226dad4b13" Nov 03 20:00:04.835879 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.835852 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d" event={"ID":"2b8f9144-8073-42b2-b859-bdde4c32ecd6","Type":"ContainerStarted","Data":"39c7a7dde8374e819bdeb402eb88e2d6e9ad97d5edae7b4d6f6b4541af886b89"} Nov 03 20:00:04.837893 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.837870 5059 generic.go:356] "Generic (PLEG): container finished" podID="99fe4ede-51a2-4f4a-a5ce-96faef90590c" containerID="15f43a219d75b82f14f4d66cb36d5dfdf5c7ca080f1798d22665281fe9381ace" exitCode=60 Nov 03 20:00:04.837971 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.837952 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb" event={"ID":"99fe4ede-51a2-4f4a-a5ce-96faef90590c","Type":"ContainerDied","Data":"15f43a219d75b82f14f4d66cb36d5dfdf5c7ca080f1798d22665281fe9381ace"} Nov 03 20:00:04.838159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.838143 5059 scope.go:122] "RemoveContainer" containerID="15f43a219d75b82f14f4d66cb36d5dfdf5c7ca080f1798d22665281fe9381ace" Nov 03 20:00:04.838439 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:04.838418 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb_odh-ai-gateway-infra(99fe4ede-51a2-4f4a-a5ce-96faef90590c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb" podUID="99fe4ede-51a2-4f4a-a5ce-96faef90590c" Nov 03 20:00:04.840425 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.840400 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw" event={"ID":"5df8b2fc-3ed3-4396-a215-1f66b95e0937","Type":"ContainerStarted","Data":"739e62af15791d5595eb2c3a6605a48f1162b32044199f4c20056b171073ac9a"} Nov 03 20:00:04.842349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.842321 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd" event={"ID":"799b3fa3-1f7a-46c8-868f-1a1624b1fd9c","Type":"ContainerStarted","Data":"bed2ab51f3356bf607250345f75c4b094209e76d33f1c9fd191089f31736627d"} Nov 03 20:00:04.844272 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.844249 5059 generic.go:356] "Generic (PLEG): container finished" podID="b8602537-2de3-4fc6-84f5-7306b45cc4fe" containerID="5746e718d3c459f9d4121a5d6686cc2013fae7c0f8ac73b30acd15942a58ad97" exitCode=60 Nov 03 20:00:04.844365 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.844329 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6" event={"ID":"b8602537-2de3-4fc6-84f5-7306b45cc4fe","Type":"ContainerDied","Data":"5746e718d3c459f9d4121a5d6686cc2013fae7c0f8ac73b30acd15942a58ad97"} Nov 03 20:00:04.844501 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.844487 5059 scope.go:122] "RemoveContainer" containerID="5746e718d3c459f9d4121a5d6686cc2013fae7c0f8ac73b30acd15942a58ad97" Nov 03 20:00:04.844742 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:04.844719 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6_odh-ai-gateway-infra(b8602537-2de3-4fc6-84f5-7306b45cc4fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6" podUID="b8602537-2de3-4fc6-84f5-7306b45cc4fe" Nov 03 20:00:04.846297 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.846275 5059 generic.go:356] "Generic (PLEG): container finished" podID="75da83f6-395e-4ca2-8709-7956a4dd7eba" containerID="9c0550441ef8d0ab0b92e3de1d087a1323d5062f85c8d8ab9ff6a119ac571734" exitCode=60 Nov 03 20:00:04.846395 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.846350 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5" event={"ID":"75da83f6-395e-4ca2-8709-7956a4dd7eba","Type":"ContainerDied","Data":"9c0550441ef8d0ab0b92e3de1d087a1323d5062f85c8d8ab9ff6a119ac571734"} Nov 03 20:00:04.846554 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.846537 5059 scope.go:122] "RemoveContainer" containerID="9c0550441ef8d0ab0b92e3de1d087a1323d5062f85c8d8ab9ff6a119ac571734" Nov 03 20:00:04.846950 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:04.846925 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5_odh-ai-gateway-infra(75da83f6-395e-4ca2-8709-7956a4dd7eba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5" podUID="75da83f6-395e-4ca2-8709-7956a4dd7eba" Nov 03 20:00:04.847527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.847505 5059 scope.go:122] "RemoveContainer" containerID="c3636fa11e95e1bcfee803e2a7ac00ff7a85416aded39c735a4194842fdb117d" Nov 03 20:00:04.848621 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.848598 5059 generic.go:356] "Generic (PLEG): container finished" podID="518d7432-1ff3-4ebf-8700-0a234e678892" containerID="180b1dddad368b4811d3379474f62cecb9201a1a708e105c9ba4d7966702c4b5" exitCode=60 Nov 03 20:00:04.848742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.848723 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2" event={"ID":"518d7432-1ff3-4ebf-8700-0a234e678892","Type":"ContainerDied","Data":"180b1dddad368b4811d3379474f62cecb9201a1a708e105c9ba4d7966702c4b5"} Nov 03 20:00:04.849014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.848997 5059 scope.go:122] "RemoveContainer" containerID="180b1dddad368b4811d3379474f62cecb9201a1a708e105c9ba4d7966702c4b5" Nov 03 20:00:04.849939 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:04.849674 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2_odh-ai-gateway-infra(518d7432-1ff3-4ebf-8700-0a234e678892)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2" podUID="518d7432-1ff3-4ebf-8700-0a234e678892" Nov 03 20:00:04.897779 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.897749 5059 scope.go:122] "RemoveContainer" containerID="ee89649eb5c99d8c60000c91f0de332dfcaccd5864dc759afe77ff0c9f735c62" Nov 03 20:00:04.919135 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.919111 5059 scope.go:122] "RemoveContainer" containerID="313f06767babde83dc131261a73503200d8113db2287f718ba8190e3b327d894" Nov 03 20:00:04.936864 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.936843 5059 scope.go:122] "RemoveContainer" containerID="092893fda72edead9b05d269ca741d73fae833ea1fdf98ec48d9e75bc85bcb89" Nov 03 20:00:04.955139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:04.955112 5059 scope.go:122] "RemoveContainer" containerID="10ba7846f6d0c78de624af9a65b0a3ade4aaffb4f53c47ff230c3fefcd27c62c" Nov 03 20:00:05.197157 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:05.197117 5059 patch_prober.go:29] interesting pod/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 20:00:05.197338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:05.197184 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" podUID="8531b4f3-5179-4948-86ce-52e13576ea11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 20:00:05.889211 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:05.889175 5059 scope.go:122] "RemoveContainer" containerID="e10a073f920433a0608c7289fc84d3e11e924ea8ab02864f13bd49fb50ae56e8" Nov 03 20:00:05.889611 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:05.889471 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x_odh-ai-gateway-infra(37f7af3e-9c65-42c8-b761-306fb918190c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x" podUID="37f7af3e-9c65-42c8-b761-306fb918190c" Nov 03 20:00:05.890494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:05.890475 5059 scope.go:122] "RemoveContainer" containerID="ecebbd87cd15bd645ee6d97deb6ce8dc69db97441c11cc06d06d767e917262cf" Nov 03 20:00:05.890732 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:05.890708 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl_odh-ai-gateway-infra(c3c45aa4-c24e-4674-a258-38226dad4b13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl" podUID="c3c45aa4-c24e-4674-a258-38226dad4b13" Nov 03 20:00:05.892255 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:05.892233 5059 generic.go:356] "Generic (PLEG): container finished" podID="2b8f9144-8073-42b2-b859-bdde4c32ecd6" containerID="39c7a7dde8374e819bdeb402eb88e2d6e9ad97d5edae7b4d6f6b4541af886b89" exitCode=60 Nov 03 20:00:05.892343 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:05.892312 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d" event={"ID":"2b8f9144-8073-42b2-b859-bdde4c32ecd6","Type":"ContainerDied","Data":"39c7a7dde8374e819bdeb402eb88e2d6e9ad97d5edae7b4d6f6b4541af886b89"} Nov 03 20:00:05.892415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:05.892370 5059 scope.go:122] "RemoveContainer" containerID="75c912f9ac6e94144af4a232fcba7af26217d7d8e64756841799fa2d3dc1af91" Nov 03 20:00:05.892575 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:05.892556 5059 scope.go:122] "RemoveContainer" containerID="39c7a7dde8374e819bdeb402eb88e2d6e9ad97d5edae7b4d6f6b4541af886b89" Nov 03 20:00:05.892855 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:05.892827 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d_odh-ai-gateway-infra(2b8f9144-8073-42b2-b859-bdde4c32ecd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d" podUID="2b8f9144-8073-42b2-b859-bdde4c32ecd6" Nov 03 20:00:05.894182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:05.894157 5059 scope.go:122] "RemoveContainer" containerID="15f43a219d75b82f14f4d66cb36d5dfdf5c7ca080f1798d22665281fe9381ace" Nov 03 20:00:05.894383 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:05.894363 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb_odh-ai-gateway-infra(99fe4ede-51a2-4f4a-a5ce-96faef90590c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb" podUID="99fe4ede-51a2-4f4a-a5ce-96faef90590c" Nov 03 20:00:05.895862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:05.895840 5059 generic.go:356] "Generic (PLEG): container finished" podID="5df8b2fc-3ed3-4396-a215-1f66b95e0937" containerID="739e62af15791d5595eb2c3a6605a48f1162b32044199f4c20056b171073ac9a" exitCode=60 Nov 03 20:00:05.895954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:05.895915 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw" event={"ID":"5df8b2fc-3ed3-4396-a215-1f66b95e0937","Type":"ContainerDied","Data":"739e62af15791d5595eb2c3a6605a48f1162b32044199f4c20056b171073ac9a"} Nov 03 20:00:05.896110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:05.896096 5059 scope.go:122] "RemoveContainer" containerID="739e62af15791d5595eb2c3a6605a48f1162b32044199f4c20056b171073ac9a" Nov 03 20:00:05.896402 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:05.896380 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw_odh-ai-gateway-infra(5df8b2fc-3ed3-4396-a215-1f66b95e0937)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw" podUID="5df8b2fc-3ed3-4396-a215-1f66b95e0937" Nov 03 20:00:05.897845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:05.897822 5059 generic.go:356] "Generic (PLEG): container finished" podID="799b3fa3-1f7a-46c8-868f-1a1624b1fd9c" containerID="bed2ab51f3356bf607250345f75c4b094209e76d33f1c9fd191089f31736627d" exitCode=60 Nov 03 20:00:05.897926 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:05.897896 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd" event={"ID":"799b3fa3-1f7a-46c8-868f-1a1624b1fd9c","Type":"ContainerDied","Data":"bed2ab51f3356bf607250345f75c4b094209e76d33f1c9fd191089f31736627d"} Nov 03 20:00:05.898062 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:05.898047 5059 scope.go:122] "RemoveContainer" containerID="bed2ab51f3356bf607250345f75c4b094209e76d33f1c9fd191089f31736627d" Nov 03 20:00:05.898276 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:05.898256 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd_odh-ai-gateway-infra(799b3fa3-1f7a-46c8-868f-1a1624b1fd9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd" podUID="799b3fa3-1f7a-46c8-868f-1a1624b1fd9c" Nov 03 20:00:05.899644 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:05.899626 5059 scope.go:122] "RemoveContainer" containerID="5746e718d3c459f9d4121a5d6686cc2013fae7c0f8ac73b30acd15942a58ad97" Nov 03 20:00:05.899920 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:05.899897 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6_odh-ai-gateway-infra(b8602537-2de3-4fc6-84f5-7306b45cc4fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6" podUID="b8602537-2de3-4fc6-84f5-7306b45cc4fe" Nov 03 20:00:05.901473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:05.901456 5059 scope.go:122] "RemoveContainer" containerID="9c0550441ef8d0ab0b92e3de1d087a1323d5062f85c8d8ab9ff6a119ac571734" Nov 03 20:00:05.901708 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:05.901688 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5_odh-ai-gateway-infra(75da83f6-395e-4ca2-8709-7956a4dd7eba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5" podUID="75da83f6-395e-4ca2-8709-7956a4dd7eba" Nov 03 20:00:05.903040 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:05.903025 5059 scope.go:122] "RemoveContainer" containerID="180b1dddad368b4811d3379474f62cecb9201a1a708e105c9ba4d7966702c4b5" Nov 03 20:00:05.903217 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:05.903200 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2_odh-ai-gateway-infra(518d7432-1ff3-4ebf-8700-0a234e678892)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2" podUID="518d7432-1ff3-4ebf-8700-0a234e678892" Nov 03 20:00:05.907305 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:05.907283 5059 scope.go:122] "RemoveContainer" containerID="0dda1f081e22d78273f5f0603530f23ba716acfd9252ce53e4fd32f69bb8de66" Nov 03 20:00:05.923326 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:05.923295 5059 scope.go:122] "RemoveContainer" containerID="565c68d3bacbe4bb87c33a320396deaee022558594fda308b682877cc3029d67" Nov 03 20:00:06.203283 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:06.200789 5059 patch_prober.go:29] interesting pod/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 20:00:06.203283 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:06.200849 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" podUID="8531b4f3-5179-4948-86ce-52e13576ea11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 20:00:06.923412 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:06.922401 5059 scope.go:122] "RemoveContainer" containerID="bed2ab51f3356bf607250345f75c4b094209e76d33f1c9fd191089f31736627d" Nov 03 20:00:06.923412 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:06.922885 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd_odh-ai-gateway-infra(799b3fa3-1f7a-46c8-868f-1a1624b1fd9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd" podUID="799b3fa3-1f7a-46c8-868f-1a1624b1fd9c" Nov 03 20:00:06.929833 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:06.929806 5059 scope.go:122] "RemoveContainer" containerID="39c7a7dde8374e819bdeb402eb88e2d6e9ad97d5edae7b4d6f6b4541af886b89" Nov 03 20:00:06.930365 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:06.930335 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d_odh-ai-gateway-infra(2b8f9144-8073-42b2-b859-bdde4c32ecd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d" podUID="2b8f9144-8073-42b2-b859-bdde4c32ecd6" Nov 03 20:00:06.934908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:06.934887 5059 scope.go:122] "RemoveContainer" containerID="739e62af15791d5595eb2c3a6605a48f1162b32044199f4c20056b171073ac9a" Nov 03 20:00:06.935472 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:06.935439 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw_odh-ai-gateway-infra(5df8b2fc-3ed3-4396-a215-1f66b95e0937)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw" podUID="5df8b2fc-3ed3-4396-a215-1f66b95e0937" Nov 03 20:00:07.211364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:07.211270 5059 patch_prober.go:29] interesting pod/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 20:00:07.211548 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:07.211371 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" podUID="8531b4f3-5179-4948-86ce-52e13576ea11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 20:00:08.196055 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:08.196015 5059 patch_prober.go:29] interesting pod/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 20:00:08.196477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:08.196107 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" podUID="8531b4f3-5179-4948-86ce-52e13576ea11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 20:00:09.015710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:09.011951 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 20:00:09.200988 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:09.200956 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 20:00:09.976306 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:09.976281 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 20:00:10.606757 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:10.606723 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:00:13.033434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:13.033399 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80a16c0d-mnn86"] Nov 03 20:00:13.038978 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:13.038954 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80a16c0d-mnn86"] Nov 03 20:00:13.749072 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:13.749034 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa494873-tvtzx"] Nov 03 20:00:13.755241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:13.755219 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa494873-tvtzx" Nov 03 20:00:13.764827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:13.764802 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:00:13.768247 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:13.768208 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa494873-tvtzx"] Nov 03 20:00:13.864459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:13.864362 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14" path="/var/lib/kubelet/pods/ac84d4c1-fb3c-47f6-9ee0-33b89f83ae14/volumes" Nov 03 20:00:13.865645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:13.865612 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4793e8b-0775-4d3f-ac3c-60197ed96852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa494873-tvtzx\" (UID: \"b4793e8b-0775-4d3f-ac3c-60197ed96852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa494873-tvtzx" Nov 03 20:00:13.866017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:13.865991 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4793e8b-0775-4d3f-ac3c-60197ed96852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa494873-tvtzx\" (UID: \"b4793e8b-0775-4d3f-ac3c-60197ed96852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa494873-tvtzx" Nov 03 20:00:13.866141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:13.866080 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hsb6\" (UniqueName: \"kubernetes.io/projected/b4793e8b-0775-4d3f-ac3c-60197ed96852-kube-api-access-4hsb6\") pod \"maas-api-revoke-sub-aa494873-tvtzx\" (UID: \"b4793e8b-0775-4d3f-ac3c-60197ed96852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa494873-tvtzx" Nov 03 20:00:13.967484 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:13.967452 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4793e8b-0775-4d3f-ac3c-60197ed96852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa494873-tvtzx\" (UID: \"b4793e8b-0775-4d3f-ac3c-60197ed96852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa494873-tvtzx" Nov 03 20:00:13.969211 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:13.967956 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4793e8b-0775-4d3f-ac3c-60197ed96852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa494873-tvtzx\" (UID: \"b4793e8b-0775-4d3f-ac3c-60197ed96852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa494873-tvtzx" Nov 03 20:00:13.969211 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:13.968093 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hsb6\" (UniqueName: \"kubernetes.io/projected/b4793e8b-0775-4d3f-ac3c-60197ed96852-kube-api-access-4hsb6\") pod \"maas-api-revoke-sub-aa494873-tvtzx\" (UID: \"b4793e8b-0775-4d3f-ac3c-60197ed96852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa494873-tvtzx" Nov 03 20:00:13.969211 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:13.968889 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4793e8b-0775-4d3f-ac3c-60197ed96852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa494873-tvtzx\" (UID: \"b4793e8b-0775-4d3f-ac3c-60197ed96852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa494873-tvtzx" Nov 03 20:00:13.970066 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:13.970045 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm"] Nov 03 20:00:13.970277 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:13.970249 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm" podUID="b5075f9d-997f-4fb3-be38-92f1f15f814b" containerName="payload-processing" containerID="cri-o://bf8ccc04da4bae3640d2c90ed8eea9cb1d7047793c870a0d2e441989d4390524" gracePeriod=30 Nov 03 20:00:13.976479 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:13.976460 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4793e8b-0775-4d3f-ac3c-60197ed96852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa494873-tvtzx\" (UID: \"b4793e8b-0775-4d3f-ac3c-60197ed96852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa494873-tvtzx" Nov 03 20:00:14.001161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:14.000656 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w"] Nov 03 20:00:14.001161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:14.000859 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w" podUID="04a05159-3d1d-4317-9890-504243a56f46" containerName="payload-pre-processing" containerID="cri-o://76be8b86166f50303d522afcc47ba967d9f3589ed5869b9eb67426987ffe1462" gracePeriod=30 Nov 03 20:00:14.019148 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:14.019122 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hsb6\" (UniqueName: \"kubernetes.io/projected/b4793e8b-0775-4d3f-ac3c-60197ed96852-kube-api-access-4hsb6\") pod \"maas-api-revoke-sub-aa494873-tvtzx\" (UID: \"b4793e8b-0775-4d3f-ac3c-60197ed96852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa494873-tvtzx" Nov 03 20:00:14.076963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:14.076927 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa494873-tvtzx" Nov 03 20:00:14.595741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:14.595631 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa494873-tvtzx"] Nov 03 20:00:14.598891 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:00:14.597880 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4793e8b_0775_4d3f_ac3c_60197ed96852.slice/crio-26a3674361d4dbe93b6c2fb96e9b5a40ba690ffd5a69c2226366ffba396fd357 WatchSource:0}: Error finding container 26a3674361d4dbe93b6c2fb96e9b5a40ba690ffd5a69c2226366ffba396fd357: Status 404 returned error can't find the container with id 26a3674361d4dbe93b6c2fb96e9b5a40ba690ffd5a69c2226366ffba396fd357 Nov 03 20:00:14.729185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:14.728842 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm" Nov 03 20:00:14.768701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:14.768424 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5075f9d-997f-4fb3-be38-92f1f15f814b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5075f9d-997f-4fb3-be38-92f1f15f814b-plugins-config-volume\") pod \"b5075f9d-997f-4fb3-be38-92f1f15f814b\" (UID: \"b5075f9d-997f-4fb3-be38-92f1f15f814b\") " Nov 03 20:00:14.768701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:14.768603 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5075f9d-997f-4fb3-be38-92f1f15f814b-kube-api-access-trbjp\" (UniqueName: \"kubernetes.io/projected/b5075f9d-997f-4fb3-be38-92f1f15f814b-kube-api-access-trbjp\") pod \"b5075f9d-997f-4fb3-be38-92f1f15f814b\" (UID: \"b5075f9d-997f-4fb3-be38-92f1f15f814b\") " Nov 03 20:00:14.769817 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:14.769724 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w" Nov 03 20:00:14.793684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:14.779871 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5075f9d-997f-4fb3-be38-92f1f15f814b-plugins-config-volume" pod "b5075f9d-997f-4fb3-be38-92f1f15f814b" (UID: "b5075f9d-997f-4fb3-be38-92f1f15f814b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:14.793684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:14.792691 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5075f9d-997f-4fb3-be38-92f1f15f814b-kube-api-access-trbjp" pod "b5075f9d-997f-4fb3-be38-92f1f15f814b" (UID: "b5075f9d-997f-4fb3-be38-92f1f15f814b"). InnerVolumeSpecName "kube-api-access-trbjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:14.797680 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:14.795176 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5075f9d-997f-4fb3-be38-92f1f15f814b-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:14.797680 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:14.795220 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trbjp\" (UniqueName: \"kubernetes.io/projected/b5075f9d-997f-4fb3-be38-92f1f15f814b-kube-api-access-trbjp\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:14.853683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:14.850212 5059 scope.go:122] "RemoveContainer" containerID="180b1dddad368b4811d3379474f62cecb9201a1a708e105c9ba4d7966702c4b5" Nov 03 20:00:14.946683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:14.944876 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a05159-3d1d-4317-9890-504243a56f46-kube-api-access-2wrv4\" (UniqueName: \"kubernetes.io/projected/04a05159-3d1d-4317-9890-504243a56f46-kube-api-access-2wrv4\") pod \"04a05159-3d1d-4317-9890-504243a56f46\" (UID: \"04a05159-3d1d-4317-9890-504243a56f46\") " Nov 03 20:00:14.946683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:14.944970 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04a05159-3d1d-4317-9890-504243a56f46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04a05159-3d1d-4317-9890-504243a56f46-plugins-config-volume\") pod \"04a05159-3d1d-4317-9890-504243a56f46\" (UID: \"04a05159-3d1d-4317-9890-504243a56f46\") " Nov 03 20:00:14.947078 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:14.947040 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a05159-3d1d-4317-9890-504243a56f46-plugins-config-volume" pod "04a05159-3d1d-4317-9890-504243a56f46" (UID: "04a05159-3d1d-4317-9890-504243a56f46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:14.949211 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:14.949165 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a05159-3d1d-4317-9890-504243a56f46-kube-api-access-2wrv4" pod "04a05159-3d1d-4317-9890-504243a56f46" (UID: "04a05159-3d1d-4317-9890-504243a56f46"). InnerVolumeSpecName "kube-api-access-2wrv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:14.955316 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:14.955254 5059 scope.go:122] "RemoveContainer" containerID="15f43a219d75b82f14f4d66cb36d5dfdf5c7ca080f1798d22665281fe9381ace" Nov 03 20:00:15.035361 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.035333 5059 generic.go:356] "Generic (PLEG): container finished" podID="04a05159-3d1d-4317-9890-504243a56f46" containerID="76be8b86166f50303d522afcc47ba967d9f3589ed5869b9eb67426987ffe1462" exitCode=0 Nov 03 20:00:15.035503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.035377 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w" Nov 03 20:00:15.035503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.035415 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w" event={"ID":"04a05159-3d1d-4317-9890-504243a56f46","Type":"ContainerDied","Data":"76be8b86166f50303d522afcc47ba967d9f3589ed5869b9eb67426987ffe1462"} Nov 03 20:00:15.035503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.035486 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w" event={"ID":"04a05159-3d1d-4317-9890-504243a56f46","Type":"ContainerDied","Data":"4ce0359ab22e236023ac9f3fdda2c178358949c5f703f60717a8012edba03555"} Nov 03 20:00:15.035611 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.035515 5059 scope.go:122] "RemoveContainer" containerID="76be8b86166f50303d522afcc47ba967d9f3589ed5869b9eb67426987ffe1462" Nov 03 20:00:15.037053 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.037031 5059 generic.go:356] "Generic (PLEG): container finished" podID="b5075f9d-997f-4fb3-be38-92f1f15f814b" containerID="bf8ccc04da4bae3640d2c90ed8eea9cb1d7047793c870a0d2e441989d4390524" exitCode=0 Nov 03 20:00:15.037173 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.037087 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm" Nov 03 20:00:15.037173 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.037110 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm" event={"ID":"b5075f9d-997f-4fb3-be38-92f1f15f814b","Type":"ContainerDied","Data":"bf8ccc04da4bae3640d2c90ed8eea9cb1d7047793c870a0d2e441989d4390524"} Nov 03 20:00:15.037173 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.037157 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm" event={"ID":"b5075f9d-997f-4fb3-be38-92f1f15f814b","Type":"ContainerDied","Data":"bab8dda750fe0e930671310ed339124b0ac8c2294638f63cfca1c3c8d578baf5"} Nov 03 20:00:15.048129 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.048103 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wrv4\" (UniqueName: \"kubernetes.io/projected/04a05159-3d1d-4317-9890-504243a56f46-kube-api-access-2wrv4\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:15.048350 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.048334 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04a05159-3d1d-4317-9890-504243a56f46-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:15.048503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.048472 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa494873-tvtzx" event={"ID":"b4793e8b-0775-4d3f-ac3c-60197ed96852","Type":"ContainerStarted","Data":"26a3674361d4dbe93b6c2fb96e9b5a40ba690ffd5a69c2226366ffba396fd357"} Nov 03 20:00:15.058202 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.058185 5059 scope.go:122] "RemoveContainer" containerID="76be8b86166f50303d522afcc47ba967d9f3589ed5869b9eb67426987ffe1462" Nov 03 20:00:15.058544 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:15.058511 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76be8b86166f50303d522afcc47ba967d9f3589ed5869b9eb67426987ffe1462\": container with ID starting with 76be8b86166f50303d522afcc47ba967d9f3589ed5869b9eb67426987ffe1462 not found: ID does not exist" containerID="76be8b86166f50303d522afcc47ba967d9f3589ed5869b9eb67426987ffe1462" Nov 03 20:00:15.058708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.058552 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76be8b86166f50303d522afcc47ba967d9f3589ed5869b9eb67426987ffe1462"} err="failed to get container status \"76be8b86166f50303d522afcc47ba967d9f3589ed5869b9eb67426987ffe1462\": rpc error: code = NotFound desc = could not find container \"76be8b86166f50303d522afcc47ba967d9f3589ed5869b9eb67426987ffe1462\": container with ID starting with 76be8b86166f50303d522afcc47ba967d9f3589ed5869b9eb67426987ffe1462 not found: ID does not exist" Nov 03 20:00:15.058708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.058574 5059 scope.go:122] "RemoveContainer" containerID="bf8ccc04da4bae3640d2c90ed8eea9cb1d7047793c870a0d2e441989d4390524" Nov 03 20:00:15.080860 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.075169 5059 scope.go:122] "RemoveContainer" containerID="bf8ccc04da4bae3640d2c90ed8eea9cb1d7047793c870a0d2e441989d4390524" Nov 03 20:00:15.080860 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:15.075644 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf8ccc04da4bae3640d2c90ed8eea9cb1d7047793c870a0d2e441989d4390524\": container with ID starting with bf8ccc04da4bae3640d2c90ed8eea9cb1d7047793c870a0d2e441989d4390524 not found: ID does not exist" containerID="bf8ccc04da4bae3640d2c90ed8eea9cb1d7047793c870a0d2e441989d4390524" Nov 03 20:00:15.080860 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.075729 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf8ccc04da4bae3640d2c90ed8eea9cb1d7047793c870a0d2e441989d4390524"} err="failed to get container status \"bf8ccc04da4bae3640d2c90ed8eea9cb1d7047793c870a0d2e441989d4390524\": rpc error: code = NotFound desc = could not find container \"bf8ccc04da4bae3640d2c90ed8eea9cb1d7047793c870a0d2e441989d4390524\": container with ID starting with bf8ccc04da4bae3640d2c90ed8eea9cb1d7047793c870a0d2e441989d4390524 not found: ID does not exist" Nov 03 20:00:15.080860 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.079715 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm"] Nov 03 20:00:15.086268 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.086241 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-277dc9-b8848778d-2kslm"] Nov 03 20:00:15.091100 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.091080 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w"] Nov 03 20:00:15.095733 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.095711 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-65b9775fc5-7xb9w"] Nov 03 20:00:15.857205 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.857170 5059 scope.go:122] "RemoveContainer" containerID="39c7a7dde8374e819bdeb402eb88e2d6e9ad97d5edae7b4d6f6b4541af886b89" Nov 03 20:00:15.857524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.857503 5059 scope.go:122] "RemoveContainer" containerID="9c0550441ef8d0ab0b92e3de1d087a1323d5062f85c8d8ab9ff6a119ac571734" Nov 03 20:00:15.864000 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.863978 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04a05159-3d1d-4317-9890-504243a56f46" path="/var/lib/kubelet/pods/04a05159-3d1d-4317-9890-504243a56f46/volumes" Nov 03 20:00:15.864447 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:15.864425 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5075f9d-997f-4fb3-be38-92f1f15f814b" path="/var/lib/kubelet/pods/b5075f9d-997f-4fb3-be38-92f1f15f814b/volumes" Nov 03 20:00:16.062957 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:16.062926 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb" event={"ID":"99fe4ede-51a2-4f4a-a5ce-96faef90590c","Type":"ContainerStarted","Data":"6f19b97eae4554b4728deedd9bb7e47c2047f8a569c695357b44e2e27e466c17"} Nov 03 20:00:16.064727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:16.064698 5059 generic.go:356] "Generic (PLEG): container finished" podID="b4793e8b-0775-4d3f-ac3c-60197ed96852" containerID="8841df5b8138c83d4d8639f4f198bfeca78a20e9f540ae8ecce06a600afa7b23" exitCode=0 Nov 03 20:00:16.064852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:16.064777 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa494873-tvtzx" event={"ID":"b4793e8b-0775-4d3f-ac3c-60197ed96852","Type":"ContainerDied","Data":"8841df5b8138c83d4d8639f4f198bfeca78a20e9f540ae8ecce06a600afa7b23"} Nov 03 20:00:16.067226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:16.067200 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2" event={"ID":"518d7432-1ff3-4ebf-8700-0a234e678892","Type":"ContainerStarted","Data":"7faf803c7b5c9b1c20407d6d38481805dac4b5aa3914ff737b0fd11a849c9321"} Nov 03 20:00:17.112015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.111982 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb"] Nov 03 20:00:17.116965 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.116927 5059 generic.go:356] "Generic (PLEG): container finished" podID="2b8f9144-8073-42b2-b859-bdde4c32ecd6" containerID="365b325f54fea6681cf0f5af3e33716940eb1067d44d3afbe99507d1a94c8571" exitCode=60 Nov 03 20:00:17.117117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.116974 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d" event={"ID":"2b8f9144-8073-42b2-b859-bdde4c32ecd6","Type":"ContainerDied","Data":"365b325f54fea6681cf0f5af3e33716940eb1067d44d3afbe99507d1a94c8571"} Nov 03 20:00:17.117117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.117036 5059 scope.go:122] "RemoveContainer" containerID="39c7a7dde8374e819bdeb402eb88e2d6e9ad97d5edae7b4d6f6b4541af886b89" Nov 03 20:00:17.117549 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.117532 5059 scope.go:122] "RemoveContainer" containerID="365b325f54fea6681cf0f5af3e33716940eb1067d44d3afbe99507d1a94c8571" Nov 03 20:00:17.118022 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:17.117997 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d_odh-ai-gateway-infra(2b8f9144-8073-42b2-b859-bdde4c32ecd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d" podUID="2b8f9144-8073-42b2-b859-bdde4c32ecd6" Nov 03 20:00:17.119093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.119022 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2"] Nov 03 20:00:17.119729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.119704 5059 generic.go:356] "Generic (PLEG): container finished" podID="99fe4ede-51a2-4f4a-a5ce-96faef90590c" containerID="6f19b97eae4554b4728deedd9bb7e47c2047f8a569c695357b44e2e27e466c17" exitCode=60 Nov 03 20:00:17.120404 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.120377 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb" event={"ID":"99fe4ede-51a2-4f4a-a5ce-96faef90590c","Type":"ContainerDied","Data":"6f19b97eae4554b4728deedd9bb7e47c2047f8a569c695357b44e2e27e466c17"} Nov 03 20:00:17.124135 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.124104 5059 generic.go:356] "Generic (PLEG): container finished" podID="75da83f6-395e-4ca2-8709-7956a4dd7eba" containerID="477f760770089e2c203aa75fad1b651baabfa5893c83c2178a93b0520b5bd026" exitCode=60 Nov 03 20:00:17.124135 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.124124 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5" event={"ID":"75da83f6-395e-4ca2-8709-7956a4dd7eba","Type":"ContainerDied","Data":"477f760770089e2c203aa75fad1b651baabfa5893c83c2178a93b0520b5bd026"} Nov 03 20:00:17.124547 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.124527 5059 scope.go:122] "RemoveContainer" containerID="477f760770089e2c203aa75fad1b651baabfa5893c83c2178a93b0520b5bd026" Nov 03 20:00:17.124916 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:17.124887 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5_odh-ai-gateway-infra(75da83f6-395e-4ca2-8709-7956a4dd7eba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5" podUID="75da83f6-395e-4ca2-8709-7956a4dd7eba" Nov 03 20:00:17.127460 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.126860 5059 generic.go:356] "Generic (PLEG): container finished" podID="518d7432-1ff3-4ebf-8700-0a234e678892" containerID="7faf803c7b5c9b1c20407d6d38481805dac4b5aa3914ff737b0fd11a849c9321" exitCode=60 Nov 03 20:00:17.127460 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.127252 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2" event={"ID":"518d7432-1ff3-4ebf-8700-0a234e678892","Type":"ContainerDied","Data":"7faf803c7b5c9b1c20407d6d38481805dac4b5aa3914ff737b0fd11a849c9321"} Nov 03 20:00:17.143025 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.142753 5059 scope.go:122] "RemoveContainer" containerID="15f43a219d75b82f14f4d66cb36d5dfdf5c7ca080f1798d22665281fe9381ace" Nov 03 20:00:17.192065 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.192034 5059 scope.go:122] "RemoveContainer" containerID="9c0550441ef8d0ab0b92e3de1d087a1323d5062f85c8d8ab9ff6a119ac571734" Nov 03 20:00:17.216097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.216075 5059 scope.go:122] "RemoveContainer" containerID="180b1dddad368b4811d3379474f62cecb9201a1a708e105c9ba4d7966702c4b5" Nov 03 20:00:17.394477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.394154 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb" Nov 03 20:00:17.449516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.449486 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa494873-tvtzx" Nov 03 20:00:17.455443 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.455407 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2" Nov 03 20:00:17.560494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.559269 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4793e8b-0775-4d3f-ac3c-60197ed96852-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4793e8b-0775-4d3f-ac3c-60197ed96852-maas-api-openshift-ca\") pod \"b4793e8b-0775-4d3f-ac3c-60197ed96852\" (UID: \"b4793e8b-0775-4d3f-ac3c-60197ed96852\") " Nov 03 20:00:17.560494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.559369 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99fe4ede-51a2-4f4a-a5ce-96faef90590c-kube-api-access-jwdqw\" (UniqueName: \"kubernetes.io/projected/99fe4ede-51a2-4f4a-a5ce-96faef90590c-kube-api-access-jwdqw\") pod \"99fe4ede-51a2-4f4a-a5ce-96faef90590c\" (UID: \"99fe4ede-51a2-4f4a-a5ce-96faef90590c\") " Nov 03 20:00:17.560494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.559437 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4793e8b-0775-4d3f-ac3c-60197ed96852-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4793e8b-0775-4d3f-ac3c-60197ed96852-maas-api-xks-ca\") pod \"b4793e8b-0775-4d3f-ac3c-60197ed96852\" (UID: \"b4793e8b-0775-4d3f-ac3c-60197ed96852\") " Nov 03 20:00:17.560494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.559524 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/518d7432-1ff3-4ebf-8700-0a234e678892-kube-api-access-hsqzz\" (UniqueName: \"kubernetes.io/projected/518d7432-1ff3-4ebf-8700-0a234e678892-kube-api-access-hsqzz\") pod \"518d7432-1ff3-4ebf-8700-0a234e678892\" (UID: \"518d7432-1ff3-4ebf-8700-0a234e678892\") " Nov 03 20:00:17.560494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.559609 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4793e8b-0775-4d3f-ac3c-60197ed96852-kube-api-access-4hsb6\" (UniqueName: \"kubernetes.io/projected/b4793e8b-0775-4d3f-ac3c-60197ed96852-kube-api-access-4hsb6\") pod \"b4793e8b-0775-4d3f-ac3c-60197ed96852\" (UID: \"b4793e8b-0775-4d3f-ac3c-60197ed96852\") " Nov 03 20:00:17.560494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.559767 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/518d7432-1ff3-4ebf-8700-0a234e678892-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/518d7432-1ff3-4ebf-8700-0a234e678892-maas-api-service-ca\") pod \"518d7432-1ff3-4ebf-8700-0a234e678892\" (UID: \"518d7432-1ff3-4ebf-8700-0a234e678892\") " Nov 03 20:00:17.560494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.559810 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99fe4ede-51a2-4f4a-a5ce-96faef90590c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99fe4ede-51a2-4f4a-a5ce-96faef90590c-maas-api-service-ca\") pod \"99fe4ede-51a2-4f4a-a5ce-96faef90590c\" (UID: \"99fe4ede-51a2-4f4a-a5ce-96faef90590c\") " Nov 03 20:00:17.562803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.562151 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4793e8b-0775-4d3f-ac3c-60197ed96852-maas-api-openshift-ca" pod "b4793e8b-0775-4d3f-ac3c-60197ed96852" (UID: "b4793e8b-0775-4d3f-ac3c-60197ed96852"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:17.569386 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.567565 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99fe4ede-51a2-4f4a-a5ce-96faef90590c-kube-api-access-jwdqw" pod "99fe4ede-51a2-4f4a-a5ce-96faef90590c" (UID: "99fe4ede-51a2-4f4a-a5ce-96faef90590c"). InnerVolumeSpecName "kube-api-access-jwdqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:17.569386 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.568087 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/518d7432-1ff3-4ebf-8700-0a234e678892-maas-api-service-ca" pod "518d7432-1ff3-4ebf-8700-0a234e678892" (UID: "518d7432-1ff3-4ebf-8700-0a234e678892"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:17.569638 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.569413 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/518d7432-1ff3-4ebf-8700-0a234e678892-kube-api-access-hsqzz" pod "518d7432-1ff3-4ebf-8700-0a234e678892" (UID: "518d7432-1ff3-4ebf-8700-0a234e678892"). InnerVolumeSpecName "kube-api-access-hsqzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:17.569904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.569866 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99fe4ede-51a2-4f4a-a5ce-96faef90590c-maas-api-service-ca" pod "99fe4ede-51a2-4f4a-a5ce-96faef90590c" (UID: "99fe4ede-51a2-4f4a-a5ce-96faef90590c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:17.570157 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.570130 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4793e8b-0775-4d3f-ac3c-60197ed96852-maas-api-xks-ca" pod "b4793e8b-0775-4d3f-ac3c-60197ed96852" (UID: "b4793e8b-0775-4d3f-ac3c-60197ed96852"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:00:17.570839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.570809 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4793e8b-0775-4d3f-ac3c-60197ed96852-kube-api-access-4hsb6" pod "b4793e8b-0775-4d3f-ac3c-60197ed96852" (UID: "b4793e8b-0775-4d3f-ac3c-60197ed96852"). InnerVolumeSpecName "kube-api-access-4hsb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:17.661595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.661563 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwdqw\" (UniqueName: \"kubernetes.io/projected/99fe4ede-51a2-4f4a-a5ce-96faef90590c-kube-api-access-jwdqw\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:17.661595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.661594 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4793e8b-0775-4d3f-ac3c-60197ed96852-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:17.661828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.661608 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsqzz\" (UniqueName: \"kubernetes.io/projected/518d7432-1ff3-4ebf-8700-0a234e678892-kube-api-access-hsqzz\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:17.661828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.661621 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hsb6\" (UniqueName: \"kubernetes.io/projected/b4793e8b-0775-4d3f-ac3c-60197ed96852-kube-api-access-4hsb6\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:17.661828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.661634 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/518d7432-1ff3-4ebf-8700-0a234e678892-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:17.661828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.661646 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99fe4ede-51a2-4f4a-a5ce-96faef90590c-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:17.661828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.661701 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4793e8b-0775-4d3f-ac3c-60197ed96852-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:17.850146 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.850118 5059 scope.go:122] "RemoveContainer" containerID="bed2ab51f3356bf607250345f75c4b094209e76d33f1c9fd191089f31736627d" Nov 03 20:00:17.850306 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.850271 5059 scope.go:122] "RemoveContainer" containerID="e10a073f920433a0608c7289fc84d3e11e924ea8ab02864f13bd49fb50ae56e8" Nov 03 20:00:17.850384 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:17.850366 5059 scope.go:122] "RemoveContainer" containerID="5746e718d3c459f9d4121a5d6686cc2013fae7c0f8ac73b30acd15942a58ad97" Nov 03 20:00:18.137984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.137919 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2" Nov 03 20:00:18.137984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.137966 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2" event={"ID":"518d7432-1ff3-4ebf-8700-0a234e678892","Type":"ContainerDied","Data":"70b9863c4b051ff81944b8885ce681bcc092c3937e11da991fc35c59aa161dec"} Nov 03 20:00:18.138455 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.138017 5059 scope.go:122] "RemoveContainer" containerID="7faf803c7b5c9b1c20407d6d38481805dac4b5aa3914ff737b0fd11a849c9321" Nov 03 20:00:18.141438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.141410 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb" Nov 03 20:00:18.141557 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.141456 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb" event={"ID":"99fe4ede-51a2-4f4a-a5ce-96faef90590c","Type":"ContainerDied","Data":"dfd3c40fcde38ffebc140d10ad17c255199bdc648090d6e82cbc3ac11a0c5abb"} Nov 03 20:00:18.142915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.142893 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa494873-tvtzx" event={"ID":"b4793e8b-0775-4d3f-ac3c-60197ed96852","Type":"ContainerDied","Data":"26a3674361d4dbe93b6c2fb96e9b5a40ba690ffd5a69c2226366ffba396fd357"} Nov 03 20:00:18.143003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.142917 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa494873-tvtzx" Nov 03 20:00:18.143003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.142927 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26a3674361d4dbe93b6c2fb96e9b5a40ba690ffd5a69c2226366ffba396fd357" Nov 03 20:00:18.150486 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.150465 5059 scope.go:122] "RemoveContainer" containerID="6f19b97eae4554b4728deedd9bb7e47c2047f8a569c695357b44e2e27e466c17" Nov 03 20:00:18.153792 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.153771 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d"] Nov 03 20:00:18.173023 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.173002 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2"] Nov 03 20:00:18.184619 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.184584 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895600-97vc2"] Nov 03 20:00:18.198024 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.198002 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb"] Nov 03 20:00:18.209575 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.209538 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6092e-29895600-5b9mb"] Nov 03 20:00:18.270274 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.270246 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5"] Nov 03 20:00:18.465308 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.465279 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d" Nov 03 20:00:18.494432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.489293 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b8f9144-8073-42b2-b859-bdde4c32ecd6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b8f9144-8073-42b2-b859-bdde4c32ecd6-maas-api-service-ca\") pod \"2b8f9144-8073-42b2-b859-bdde4c32ecd6\" (UID: \"2b8f9144-8073-42b2-b859-bdde4c32ecd6\") " Nov 03 20:00:18.494432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.489417 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b8f9144-8073-42b2-b859-bdde4c32ecd6-kube-api-access-4cclk\" (UniqueName: \"kubernetes.io/projected/2b8f9144-8073-42b2-b859-bdde4c32ecd6-kube-api-access-4cclk\") pod \"2b8f9144-8073-42b2-b859-bdde4c32ecd6\" (UID: \"2b8f9144-8073-42b2-b859-bdde4c32ecd6\") " Nov 03 20:00:18.494432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.490252 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b8f9144-8073-42b2-b859-bdde4c32ecd6-maas-api-service-ca" pod "2b8f9144-8073-42b2-b859-bdde4c32ecd6" (UID: "2b8f9144-8073-42b2-b859-bdde4c32ecd6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:18.495911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.495552 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b8f9144-8073-42b2-b859-bdde4c32ecd6-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:18.504779 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.504745 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b8f9144-8073-42b2-b859-bdde4c32ecd6-kube-api-access-4cclk" pod "2b8f9144-8073-42b2-b859-bdde4c32ecd6" (UID: "2b8f9144-8073-42b2-b859-bdde4c32ecd6"). InnerVolumeSpecName "kube-api-access-4cclk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:18.588803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.588778 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5" Nov 03 20:00:18.596521 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.596499 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cclk\" (UniqueName: \"kubernetes.io/projected/2b8f9144-8073-42b2-b859-bdde4c32ecd6-kube-api-access-4cclk\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:18.697915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.697843 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75da83f6-395e-4ca2-8709-7956a4dd7eba-kube-api-access-2kggm\" (UniqueName: \"kubernetes.io/projected/75da83f6-395e-4ca2-8709-7956a4dd7eba-kube-api-access-2kggm\") pod \"75da83f6-395e-4ca2-8709-7956a4dd7eba\" (UID: \"75da83f6-395e-4ca2-8709-7956a4dd7eba\") " Nov 03 20:00:18.698081 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.698025 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75da83f6-395e-4ca2-8709-7956a4dd7eba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75da83f6-395e-4ca2-8709-7956a4dd7eba-maas-api-service-ca\") pod \"75da83f6-395e-4ca2-8709-7956a4dd7eba\" (UID: \"75da83f6-395e-4ca2-8709-7956a4dd7eba\") " Nov 03 20:00:18.698376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.698350 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75da83f6-395e-4ca2-8709-7956a4dd7eba-maas-api-service-ca" pod "75da83f6-395e-4ca2-8709-7956a4dd7eba" (UID: "75da83f6-395e-4ca2-8709-7956a4dd7eba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:18.698675 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.698646 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75da83f6-395e-4ca2-8709-7956a4dd7eba-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:18.699875 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.699844 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75da83f6-395e-4ca2-8709-7956a4dd7eba-kube-api-access-2kggm" pod "75da83f6-395e-4ca2-8709-7956a4dd7eba" (UID: "75da83f6-395e-4ca2-8709-7956a4dd7eba"). InnerVolumeSpecName "kube-api-access-2kggm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:18.800345 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.800317 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kggm\" (UniqueName: \"kubernetes.io/projected/75da83f6-395e-4ca2-8709-7956a4dd7eba-kube-api-access-2kggm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:18.849832 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.849804 5059 scope.go:122] "RemoveContainer" containerID="ecebbd87cd15bd645ee6d97deb6ce8dc69db97441c11cc06d06d767e917262cf" Nov 03 20:00:18.850138 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:18.850117 5059 scope.go:122] "RemoveContainer" containerID="739e62af15791d5595eb2c3a6605a48f1162b32044199f4c20056b171073ac9a" Nov 03 20:00:19.154844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.154810 5059 generic.go:356] "Generic (PLEG): container finished" podID="799b3fa3-1f7a-46c8-868f-1a1624b1fd9c" containerID="ef89bcd1766f710e5368189a44fc461e4bf7c5226204fc7186c942429d12463c" exitCode=60 Nov 03 20:00:19.155324 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.154875 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd" event={"ID":"799b3fa3-1f7a-46c8-868f-1a1624b1fd9c","Type":"ContainerDied","Data":"ef89bcd1766f710e5368189a44fc461e4bf7c5226204fc7186c942429d12463c"} Nov 03 20:00:19.155324 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.154949 5059 scope.go:122] "RemoveContainer" containerID="bed2ab51f3356bf607250345f75c4b094209e76d33f1c9fd191089f31736627d" Nov 03 20:00:19.155510 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.155491 5059 scope.go:122] "RemoveContainer" containerID="ef89bcd1766f710e5368189a44fc461e4bf7c5226204fc7186c942429d12463c" Nov 03 20:00:19.155982 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:19.155954 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd_odh-ai-gateway-infra(799b3fa3-1f7a-46c8-868f-1a1624b1fd9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd" podUID="799b3fa3-1f7a-46c8-868f-1a1624b1fd9c" Nov 03 20:00:19.160594 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.160572 5059 generic.go:356] "Generic (PLEG): container finished" podID="b8602537-2de3-4fc6-84f5-7306b45cc4fe" containerID="c033d3d00c9a0729a058e2370f60d1720c2aa914fd94072e5fcb9db89bc436ad" exitCode=60 Nov 03 20:00:19.160772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.160703 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6" event={"ID":"b8602537-2de3-4fc6-84f5-7306b45cc4fe","Type":"ContainerDied","Data":"c033d3d00c9a0729a058e2370f60d1720c2aa914fd94072e5fcb9db89bc436ad"} Nov 03 20:00:19.161420 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.161398 5059 scope.go:122] "RemoveContainer" containerID="c033d3d00c9a0729a058e2370f60d1720c2aa914fd94072e5fcb9db89bc436ad" Nov 03 20:00:19.161985 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:19.161955 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6_odh-ai-gateway-infra(b8602537-2de3-4fc6-84f5-7306b45cc4fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6" podUID="b8602537-2de3-4fc6-84f5-7306b45cc4fe" Nov 03 20:00:19.163784 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.163766 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5" Nov 03 20:00:19.164209 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.164184 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5" event={"ID":"75da83f6-395e-4ca2-8709-7956a4dd7eba","Type":"ContainerDied","Data":"e97f68217e5465f19bb2d0cb7b204c719361ab0e755aace6f1b195002ddfb145"} Nov 03 20:00:19.167992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.167832 5059 generic.go:356] "Generic (PLEG): container finished" podID="37f7af3e-9c65-42c8-b761-306fb918190c" containerID="f167454d9ab74ba1d9b443522559ac2ef412e3acf907f58eae5df4905c368ae5" exitCode=60 Nov 03 20:00:19.167992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.167956 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x" event={"ID":"37f7af3e-9c65-42c8-b761-306fb918190c","Type":"ContainerDied","Data":"f167454d9ab74ba1d9b443522559ac2ef412e3acf907f58eae5df4905c368ae5"} Nov 03 20:00:19.204007 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.185152 5059 scope.go:122] "RemoveContainer" containerID="f167454d9ab74ba1d9b443522559ac2ef412e3acf907f58eae5df4905c368ae5" Nov 03 20:00:19.204007 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:19.185584 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x_odh-ai-gateway-infra(37f7af3e-9c65-42c8-b761-306fb918190c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x" podUID="37f7af3e-9c65-42c8-b761-306fb918190c" Nov 03 20:00:19.205102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.205039 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d" event={"ID":"2b8f9144-8073-42b2-b859-bdde4c32ecd6","Type":"ContainerDied","Data":"400e8939540a0fb5ecbcbaec7f6f02e2413ea24523004f0fb12e91588c1aa8b6"} Nov 03 20:00:19.205265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.205137 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d" Nov 03 20:00:19.235829 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.233140 5059 scope.go:122] "RemoveContainer" containerID="5746e718d3c459f9d4121a5d6686cc2013fae7c0f8ac73b30acd15942a58ad97" Nov 03 20:00:19.254395 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.254371 5059 scope.go:122] "RemoveContainer" containerID="477f760770089e2c203aa75fad1b651baabfa5893c83c2178a93b0520b5bd026" Nov 03 20:00:19.264947 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.264919 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d"] Nov 03 20:00:19.267452 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.267415 5059 scope.go:122] "RemoveContainer" containerID="e10a073f920433a0608c7289fc84d3e11e924ea8ab02864f13bd49fb50ae56e8" Nov 03 20:00:19.270086 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.270065 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1d04-29895600-vbs4d"] Nov 03 20:00:19.281925 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.281898 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5"] Nov 03 20:00:19.282135 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.282113 5059 scope.go:122] "RemoveContainer" containerID="365b325f54fea6681cf0f5af3e33716940eb1067d44d3afbe99507d1a94c8571" Nov 03 20:00:19.291016 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.290994 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f493f7-29895600-7jtv5"] Nov 03 20:00:19.870888 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.870835 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b8f9144-8073-42b2-b859-bdde4c32ecd6" path="/var/lib/kubelet/pods/2b8f9144-8073-42b2-b859-bdde4c32ecd6/volumes" Nov 03 20:00:19.871589 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.871561 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="518d7432-1ff3-4ebf-8700-0a234e678892" path="/var/lib/kubelet/pods/518d7432-1ff3-4ebf-8700-0a234e678892/volumes" Nov 03 20:00:19.872276 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.872249 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75da83f6-395e-4ca2-8709-7956a4dd7eba" path="/var/lib/kubelet/pods/75da83f6-395e-4ca2-8709-7956a4dd7eba/volumes" Nov 03 20:00:19.873082 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:19.873054 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99fe4ede-51a2-4f4a-a5ce-96faef90590c" path="/var/lib/kubelet/pods/99fe4ede-51a2-4f4a-a5ce-96faef90590c/volumes" Nov 03 20:00:20.223178 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.223100 5059 generic.go:356] "Generic (PLEG): container finished" podID="c3c45aa4-c24e-4674-a258-38226dad4b13" containerID="6d1475e3da1735d0bfd834e231b3df736e3230d8b89f140989d161f88eb31f06" exitCode=60 Nov 03 20:00:20.223585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.223170 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl" event={"ID":"c3c45aa4-c24e-4674-a258-38226dad4b13","Type":"ContainerDied","Data":"6d1475e3da1735d0bfd834e231b3df736e3230d8b89f140989d161f88eb31f06"} Nov 03 20:00:20.223585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.223223 5059 scope.go:122] "RemoveContainer" containerID="ecebbd87cd15bd645ee6d97deb6ce8dc69db97441c11cc06d06d767e917262cf" Nov 03 20:00:20.223789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.223653 5059 scope.go:122] "RemoveContainer" containerID="6d1475e3da1735d0bfd834e231b3df736e3230d8b89f140989d161f88eb31f06" Nov 03 20:00:20.224059 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:20.224034 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl_odh-ai-gateway-infra(c3c45aa4-c24e-4674-a258-38226dad4b13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl" podUID="c3c45aa4-c24e-4674-a258-38226dad4b13" Nov 03 20:00:20.226165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.226144 5059 generic.go:356] "Generic (PLEG): container finished" podID="5df8b2fc-3ed3-4396-a215-1f66b95e0937" containerID="a416d4fe4cfa8dda811521e676da72659501e3be3421a8b0379fc91e86b7313d" exitCode=60 Nov 03 20:00:20.226280 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.226218 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw" event={"ID":"5df8b2fc-3ed3-4396-a215-1f66b95e0937","Type":"ContainerDied","Data":"a416d4fe4cfa8dda811521e676da72659501e3be3421a8b0379fc91e86b7313d"} Nov 03 20:00:20.226571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.226553 5059 scope.go:122] "RemoveContainer" containerID="a416d4fe4cfa8dda811521e676da72659501e3be3421a8b0379fc91e86b7313d" Nov 03 20:00:20.226895 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:20.226875 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw_odh-ai-gateway-infra(5df8b2fc-3ed3-4396-a215-1f66b95e0937)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw" podUID="5df8b2fc-3ed3-4396-a215-1f66b95e0937" Nov 03 20:00:20.236441 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.236425 5059 scope.go:122] "RemoveContainer" containerID="739e62af15791d5595eb2c3a6605a48f1162b32044199f4c20056b171073ac9a" Nov 03 20:00:20.249139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.249114 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd"] Nov 03 20:00:20.266008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.265984 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6"] Nov 03 20:00:20.282858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.282832 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x"] Nov 03 20:00:20.452037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.452011 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd" Nov 03 20:00:20.552462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.552432 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/799b3fa3-1f7a-46c8-868f-1a1624b1fd9c-kube-api-access-bzc29\" (UniqueName: \"kubernetes.io/projected/799b3fa3-1f7a-46c8-868f-1a1624b1fd9c-kube-api-access-bzc29\") pod \"799b3fa3-1f7a-46c8-868f-1a1624b1fd9c\" (UID: \"799b3fa3-1f7a-46c8-868f-1a1624b1fd9c\") " Nov 03 20:00:20.552576 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.552563 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/799b3fa3-1f7a-46c8-868f-1a1624b1fd9c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/799b3fa3-1f7a-46c8-868f-1a1624b1fd9c-maas-api-service-ca\") pod \"799b3fa3-1f7a-46c8-868f-1a1624b1fd9c\" (UID: \"799b3fa3-1f7a-46c8-868f-1a1624b1fd9c\") " Nov 03 20:00:20.552948 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.552924 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/799b3fa3-1f7a-46c8-868f-1a1624b1fd9c-maas-api-service-ca" pod "799b3fa3-1f7a-46c8-868f-1a1624b1fd9c" (UID: "799b3fa3-1f7a-46c8-868f-1a1624b1fd9c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:20.553204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.553185 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/799b3fa3-1f7a-46c8-868f-1a1624b1fd9c-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:20.554638 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.554615 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/799b3fa3-1f7a-46c8-868f-1a1624b1fd9c-kube-api-access-bzc29" pod "799b3fa3-1f7a-46c8-868f-1a1624b1fd9c" (UID: "799b3fa3-1f7a-46c8-868f-1a1624b1fd9c"). InnerVolumeSpecName "kube-api-access-bzc29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:20.572373 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.572354 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6" Nov 03 20:00:20.575802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.575785 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x" Nov 03 20:00:20.654494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.654460 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8602537-2de3-4fc6-84f5-7306b45cc4fe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8602537-2de3-4fc6-84f5-7306b45cc4fe-maas-api-service-ca\") pod \"b8602537-2de3-4fc6-84f5-7306b45cc4fe\" (UID: \"b8602537-2de3-4fc6-84f5-7306b45cc4fe\") " Nov 03 20:00:20.654671 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.654508 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37f7af3e-9c65-42c8-b761-306fb918190c-kube-api-access-zz47r\" (UniqueName: \"kubernetes.io/projected/37f7af3e-9c65-42c8-b761-306fb918190c-kube-api-access-zz47r\") pod \"37f7af3e-9c65-42c8-b761-306fb918190c\" (UID: \"37f7af3e-9c65-42c8-b761-306fb918190c\") " Nov 03 20:00:20.654671 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.654526 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37f7af3e-9c65-42c8-b761-306fb918190c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37f7af3e-9c65-42c8-b761-306fb918190c-maas-api-service-ca\") pod \"37f7af3e-9c65-42c8-b761-306fb918190c\" (UID: \"37f7af3e-9c65-42c8-b761-306fb918190c\") " Nov 03 20:00:20.654671 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.654559 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8602537-2de3-4fc6-84f5-7306b45cc4fe-kube-api-access-wwn4m\" (UniqueName: \"kubernetes.io/projected/b8602537-2de3-4fc6-84f5-7306b45cc4fe-kube-api-access-wwn4m\") pod \"b8602537-2de3-4fc6-84f5-7306b45cc4fe\" (UID: \"b8602537-2de3-4fc6-84f5-7306b45cc4fe\") " Nov 03 20:00:20.655278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.655249 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzc29\" (UniqueName: \"kubernetes.io/projected/799b3fa3-1f7a-46c8-868f-1a1624b1fd9c-kube-api-access-bzc29\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:20.655941 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.655912 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8602537-2de3-4fc6-84f5-7306b45cc4fe-maas-api-service-ca" pod "b8602537-2de3-4fc6-84f5-7306b45cc4fe" (UID: "b8602537-2de3-4fc6-84f5-7306b45cc4fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:20.656144 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.656111 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f7af3e-9c65-42c8-b761-306fb918190c-maas-api-service-ca" pod "37f7af3e-9c65-42c8-b761-306fb918190c" (UID: "37f7af3e-9c65-42c8-b761-306fb918190c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:20.659162 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.659139 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8602537-2de3-4fc6-84f5-7306b45cc4fe-kube-api-access-wwn4m" pod "b8602537-2de3-4fc6-84f5-7306b45cc4fe" (UID: "b8602537-2de3-4fc6-84f5-7306b45cc4fe"). InnerVolumeSpecName "kube-api-access-wwn4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:20.659258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.659169 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f7af3e-9c65-42c8-b761-306fb918190c-kube-api-access-zz47r" pod "37f7af3e-9c65-42c8-b761-306fb918190c" (UID: "37f7af3e-9c65-42c8-b761-306fb918190c"). InnerVolumeSpecName "kube-api-access-zz47r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:20.756868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.756829 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8602537-2de3-4fc6-84f5-7306b45cc4fe-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:20.756868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.756861 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz47r\" (UniqueName: \"kubernetes.io/projected/37f7af3e-9c65-42c8-b761-306fb918190c-kube-api-access-zz47r\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:20.757025 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.756876 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37f7af3e-9c65-42c8-b761-306fb918190c-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:20.757025 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:20.756890 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwn4m\" (UniqueName: \"kubernetes.io/projected/b8602537-2de3-4fc6-84f5-7306b45cc4fe-kube-api-access-wwn4m\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:21.243214 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.243173 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x" event={"ID":"37f7af3e-9c65-42c8-b761-306fb918190c","Type":"ContainerDied","Data":"81eff7dba9d1970fa536c36097474a7ed33b61fde1fe5bd0084634e025d5bef4"} Nov 03 20:00:21.243821 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.243799 5059 scope.go:122] "RemoveContainer" containerID="f167454d9ab74ba1d9b443522559ac2ef412e3acf907f58eae5df4905c368ae5" Nov 03 20:00:21.244070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.244053 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x" Nov 03 20:00:21.268485 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.268453 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd" event={"ID":"799b3fa3-1f7a-46c8-868f-1a1624b1fd9c","Type":"ContainerDied","Data":"086e56594f78343cb98cda3379d29d04d228d1d7a7863e4478e129d3cdbb6daa"} Nov 03 20:00:21.268814 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.268795 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd" Nov 03 20:00:21.341716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.321823 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6" event={"ID":"b8602537-2de3-4fc6-84f5-7306b45cc4fe","Type":"ContainerDied","Data":"0825df6e29841a9b09b305b7b04ff124a7470ef14c658431e5ced47dc14d6334"} Nov 03 20:00:21.341716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.321945 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6" Nov 03 20:00:21.372745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.372701 5059 scope.go:122] "RemoveContainer" containerID="ef89bcd1766f710e5368189a44fc461e4bf7c5226204fc7186c942429d12463c" Nov 03 20:00:21.395929 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.389254 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl"] Nov 03 20:00:21.411424 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.411388 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw"] Nov 03 20:00:21.432957 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.432927 5059 scope.go:122] "RemoveContainer" containerID="c033d3d00c9a0729a058e2370f60d1720c2aa914fd94072e5fcb9db89bc436ad" Nov 03 20:00:21.438863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.438837 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6"] Nov 03 20:00:21.445241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.445212 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895600-x44d6"] Nov 03 20:00:21.450843 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.450785 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd"] Nov 03 20:00:21.468710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.467340 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895600-c5smd"] Nov 03 20:00:21.491852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.491819 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x"] Nov 03 20:00:21.496318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.496245 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-389e6d-29895600-2c24x"] Nov 03 20:00:21.670966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.670926 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl" Nov 03 20:00:21.696081 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.695273 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3c45aa4-c24e-4674-a258-38226dad4b13-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c45aa4-c24e-4674-a258-38226dad4b13-maas-api-service-ca\") pod \"c3c45aa4-c24e-4674-a258-38226dad4b13\" (UID: \"c3c45aa4-c24e-4674-a258-38226dad4b13\") " Nov 03 20:00:21.696081 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.695372 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3c45aa4-c24e-4674-a258-38226dad4b13-kube-api-access-g5zkh\" (UniqueName: \"kubernetes.io/projected/c3c45aa4-c24e-4674-a258-38226dad4b13-kube-api-access-g5zkh\") pod \"c3c45aa4-c24e-4674-a258-38226dad4b13\" (UID: \"c3c45aa4-c24e-4674-a258-38226dad4b13\") " Nov 03 20:00:21.696535 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.696501 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c45aa4-c24e-4674-a258-38226dad4b13-maas-api-service-ca" pod "c3c45aa4-c24e-4674-a258-38226dad4b13" (UID: "c3c45aa4-c24e-4674-a258-38226dad4b13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:21.698588 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.698553 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c45aa4-c24e-4674-a258-38226dad4b13-kube-api-access-g5zkh" pod "c3c45aa4-c24e-4674-a258-38226dad4b13" (UID: "c3c45aa4-c24e-4674-a258-38226dad4b13"). InnerVolumeSpecName "kube-api-access-g5zkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:21.711543 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.711521 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw" Nov 03 20:00:21.833005 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.822989 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c45aa4-c24e-4674-a258-38226dad4b13-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:21.833005 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.823030 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5zkh\" (UniqueName: \"kubernetes.io/projected/c3c45aa4-c24e-4674-a258-38226dad4b13-kube-api-access-g5zkh\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:21.865698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.865646 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37f7af3e-9c65-42c8-b761-306fb918190c" path="/var/lib/kubelet/pods/37f7af3e-9c65-42c8-b761-306fb918190c/volumes" Nov 03 20:00:21.866156 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.866139 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="799b3fa3-1f7a-46c8-868f-1a1624b1fd9c" path="/var/lib/kubelet/pods/799b3fa3-1f7a-46c8-868f-1a1624b1fd9c/volumes" Nov 03 20:00:21.866556 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.866541 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8602537-2de3-4fc6-84f5-7306b45cc4fe" path="/var/lib/kubelet/pods/b8602537-2de3-4fc6-84f5-7306b45cc4fe/volumes" Nov 03 20:00:21.924156 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.924127 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5df8b2fc-3ed3-4396-a215-1f66b95e0937-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df8b2fc-3ed3-4396-a215-1f66b95e0937-maas-api-service-ca\") pod \"5df8b2fc-3ed3-4396-a215-1f66b95e0937\" (UID: \"5df8b2fc-3ed3-4396-a215-1f66b95e0937\") " Nov 03 20:00:21.924334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.924288 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5df8b2fc-3ed3-4396-a215-1f66b95e0937-kube-api-access-vcksp\" (UniqueName: \"kubernetes.io/projected/5df8b2fc-3ed3-4396-a215-1f66b95e0937-kube-api-access-vcksp\") pod \"5df8b2fc-3ed3-4396-a215-1f66b95e0937\" (UID: \"5df8b2fc-3ed3-4396-a215-1f66b95e0937\") " Nov 03 20:00:21.924490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.924468 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5df8b2fc-3ed3-4396-a215-1f66b95e0937-maas-api-service-ca" pod "5df8b2fc-3ed3-4396-a215-1f66b95e0937" (UID: "5df8b2fc-3ed3-4396-a215-1f66b95e0937"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:21.924908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.924889 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df8b2fc-3ed3-4396-a215-1f66b95e0937-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:21.926480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:21.926460 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df8b2fc-3ed3-4396-a215-1f66b95e0937-kube-api-access-vcksp" pod "5df8b2fc-3ed3-4396-a215-1f66b95e0937" (UID: "5df8b2fc-3ed3-4396-a215-1f66b95e0937"). InnerVolumeSpecName "kube-api-access-vcksp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:22.026853 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:22.026817 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcksp\" (UniqueName: \"kubernetes.io/projected/5df8b2fc-3ed3-4396-a215-1f66b95e0937-kube-api-access-vcksp\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:22.340742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:22.340700 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl" event={"ID":"c3c45aa4-c24e-4674-a258-38226dad4b13","Type":"ContainerDied","Data":"f98b857a275b2d503ba9c9f119299e2eeba30b75d72a9a7ceb713b48b47b4fb3"} Nov 03 20:00:22.340742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:22.340738 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl" Nov 03 20:00:22.341275 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:22.340782 5059 scope.go:122] "RemoveContainer" containerID="6d1475e3da1735d0bfd834e231b3df736e3230d8b89f140989d161f88eb31f06" Nov 03 20:00:22.345086 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:22.345018 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw" event={"ID":"5df8b2fc-3ed3-4396-a215-1f66b95e0937","Type":"ContainerDied","Data":"2dab556ed97636f91e4317ffdced56a72e8ef250889ad7a73e83c8994cf8e244"} Nov 03 20:00:22.345233 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:22.345089 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw" Nov 03 20:00:22.364319 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:22.364298 5059 scope.go:122] "RemoveContainer" containerID="a416d4fe4cfa8dda811521e676da72659501e3be3421a8b0379fc91e86b7313d" Nov 03 20:00:22.379627 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:22.379601 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl"] Nov 03 20:00:22.383363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:22.383341 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895600-wmrtl"] Nov 03 20:00:22.394459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:22.394428 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw"] Nov 03 20:00:22.404696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:22.404629 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895600-mstvw"] Nov 03 20:00:22.559253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:22.559215 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz"] Nov 03 20:00:22.559729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:22.559695 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" podUID="87c2d4eb-392d-4a66-8622-9ff5eee0b761" containerName="istio-proxy" containerID="cri-o://25f1d35ccdcc578f635b59cc635791b85534cdee7e6f5f467fb2fee218e979cc" gracePeriod=30 Nov 03 20:00:23.858778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:23.858734 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5df8b2fc-3ed3-4396-a215-1f66b95e0937" path="/var/lib/kubelet/pods/5df8b2fc-3ed3-4396-a215-1f66b95e0937/volumes" Nov 03 20:00:23.859266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:23.859249 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3c45aa4-c24e-4674-a258-38226dad4b13" path="/var/lib/kubelet/pods/c3c45aa4-c24e-4674-a258-38226dad4b13/volumes" Nov 03 20:00:27.903255 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:27.903229 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 20:00:27.941820 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:27.941783 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-podinfo\") pod \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " Nov 03 20:00:27.941978 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:27.941852 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-workload-certs\") pod \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " Nov 03 20:00:27.941978 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:27.941925 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-token\" (UniqueName: \"kubernetes.io/projected/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-token\") pod \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " Nov 03 20:00:27.942103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:27.942021 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-data\") pod \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " Nov 03 20:00:27.942103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:27.942079 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istiod-ca-cert\") pod \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " Nov 03 20:00:27.942249 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:27.942229 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-envoy\") pod \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " Nov 03 20:00:27.942363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:27.942347 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-credential-socket\") pod \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " Nov 03 20:00:27.942433 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:27.942388 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87c2d4eb-392d-4a66-8622-9ff5eee0b761-kube-api-access-d94pw\" (UniqueName: \"kubernetes.io/projected/87c2d4eb-392d-4a66-8622-9ff5eee0b761-kube-api-access-d94pw\") pod \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " Nov 03 20:00:27.942511 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:27.942497 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-workload-socket\") pod \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\" (UID: \"87c2d4eb-392d-4a66-8622-9ff5eee0b761\") " Nov 03 20:00:27.944263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:27.944207 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-credential-socket" pod "87c2d4eb-392d-4a66-8622-9ff5eee0b761" (UID: "87c2d4eb-392d-4a66-8622-9ff5eee0b761"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:00:27.944419 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:27.944395 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-workload-certs" pod "87c2d4eb-392d-4a66-8622-9ff5eee0b761" (UID: "87c2d4eb-392d-4a66-8622-9ff5eee0b761"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:00:27.944888 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:27.944859 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-workload-socket" pod "87c2d4eb-392d-4a66-8622-9ff5eee0b761" (UID: "87c2d4eb-392d-4a66-8622-9ff5eee0b761"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:00:27.945441 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:27.945410 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-data" pod "87c2d4eb-392d-4a66-8622-9ff5eee0b761" (UID: "87c2d4eb-392d-4a66-8622-9ff5eee0b761"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:00:27.945546 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:27.945516 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istiod-ca-cert" pod "87c2d4eb-392d-4a66-8622-9ff5eee0b761" (UID: "87c2d4eb-392d-4a66-8622-9ff5eee0b761"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:27.947243 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:27.947212 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-token" pod "87c2d4eb-392d-4a66-8622-9ff5eee0b761" (UID: "87c2d4eb-392d-4a66-8622-9ff5eee0b761"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:27.947489 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:27.947459 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-podinfo" pod "87c2d4eb-392d-4a66-8622-9ff5eee0b761" (UID: "87c2d4eb-392d-4a66-8622-9ff5eee0b761"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:00:27.959134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:27.959103 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c2d4eb-392d-4a66-8622-9ff5eee0b761-kube-api-access-d94pw" pod "87c2d4eb-392d-4a66-8622-9ff5eee0b761" (UID: "87c2d4eb-392d-4a66-8622-9ff5eee0b761"). InnerVolumeSpecName "kube-api-access-d94pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:27.960806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:27.960759 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-envoy" pod "87c2d4eb-392d-4a66-8622-9ff5eee0b761" (UID: "87c2d4eb-392d-4a66-8622-9ff5eee0b761"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:00:28.045358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:28.045203 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:28.045358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:28.045241 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:28.045358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:28.045256 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:28.045358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:28.045269 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d94pw\" (UniqueName: \"kubernetes.io/projected/87c2d4eb-392d-4a66-8622-9ff5eee0b761-kube-api-access-d94pw\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:28.045358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:28.045282 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:28.045358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:28.045295 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:28.045358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:28.045308 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:28.045358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:28.045321 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:28.045358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:28.045333 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c2d4eb-392d-4a66-8622-9ff5eee0b761-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:28.072307 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:28.070814 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8"] Nov 03 20:00:28.072307 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:28.071061 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" podUID="856d3ff7-8514-46b9-8884-f99b674dd6e6" containerName="istio-proxy" containerID="cri-o://fb95eba5d616de91bd7fc8a639b7f9be441869b05ec13d63169baa5585f37f2b" gracePeriod=30 Nov 03 20:00:28.408005 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:28.407965 5059 generic.go:356] "Generic (PLEG): container finished" podID="87c2d4eb-392d-4a66-8622-9ff5eee0b761" containerID="25f1d35ccdcc578f635b59cc635791b85534cdee7e6f5f467fb2fee218e979cc" exitCode=0 Nov 03 20:00:28.408183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:28.408013 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" event={"ID":"87c2d4eb-392d-4a66-8622-9ff5eee0b761","Type":"ContainerDied","Data":"25f1d35ccdcc578f635b59cc635791b85534cdee7e6f5f467fb2fee218e979cc"} Nov 03 20:00:28.408183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:28.408019 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" Nov 03 20:00:28.408183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:28.408060 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz" event={"ID":"87c2d4eb-392d-4a66-8622-9ff5eee0b761","Type":"ContainerDied","Data":"a420d5704886fa69470ca01c235a6cd33201750d1931351d4575322408a0fc3d"} Nov 03 20:00:28.408183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:28.408083 5059 scope.go:122] "RemoveContainer" containerID="25f1d35ccdcc578f635b59cc635791b85534cdee7e6f5f467fb2fee218e979cc" Nov 03 20:00:28.421637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:28.421612 5059 scope.go:122] "RemoveContainer" containerID="ff2052ce9b253a2c2fbda40231cab9fe34232bc1021d11e8648c5fee496ace71" Nov 03 20:00:28.433198 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:28.433153 5059 scope.go:122] "RemoveContainer" containerID="25f1d35ccdcc578f635b59cc635791b85534cdee7e6f5f467fb2fee218e979cc" Nov 03 20:00:28.433567 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:28.433533 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25f1d35ccdcc578f635b59cc635791b85534cdee7e6f5f467fb2fee218e979cc\": container with ID starting with 25f1d35ccdcc578f635b59cc635791b85534cdee7e6f5f467fb2fee218e979cc not found: ID does not exist" containerID="25f1d35ccdcc578f635b59cc635791b85534cdee7e6f5f467fb2fee218e979cc" Nov 03 20:00:28.433675 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:28.433576 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f1d35ccdcc578f635b59cc635791b85534cdee7e6f5f467fb2fee218e979cc"} err="failed to get container status \"25f1d35ccdcc578f635b59cc635791b85534cdee7e6f5f467fb2fee218e979cc\": rpc error: code = NotFound desc = could not find container \"25f1d35ccdcc578f635b59cc635791b85534cdee7e6f5f467fb2fee218e979cc\": container with ID starting with 25f1d35ccdcc578f635b59cc635791b85534cdee7e6f5f467fb2fee218e979cc not found: ID does not exist" Nov 03 20:00:28.433675 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:28.433597 5059 scope.go:122] "RemoveContainer" containerID="ff2052ce9b253a2c2fbda40231cab9fe34232bc1021d11e8648c5fee496ace71" Nov 03 20:00:28.433934 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:28.433903 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff2052ce9b253a2c2fbda40231cab9fe34232bc1021d11e8648c5fee496ace71\": container with ID starting with ff2052ce9b253a2c2fbda40231cab9fe34232bc1021d11e8648c5fee496ace71 not found: ID does not exist" containerID="ff2052ce9b253a2c2fbda40231cab9fe34232bc1021d11e8648c5fee496ace71" Nov 03 20:00:28.433984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:28.433933 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff2052ce9b253a2c2fbda40231cab9fe34232bc1021d11e8648c5fee496ace71"} err="failed to get container status \"ff2052ce9b253a2c2fbda40231cab9fe34232bc1021d11e8648c5fee496ace71\": rpc error: code = NotFound desc = could not find container \"ff2052ce9b253a2c2fbda40231cab9fe34232bc1021d11e8648c5fee496ace71\": container with ID starting with ff2052ce9b253a2c2fbda40231cab9fe34232bc1021d11e8648c5fee496ace71 not found: ID does not exist" Nov 03 20:00:28.436837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:28.436813 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz"] Nov 03 20:00:28.444762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:28.444740 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-277dc9-openshift-default-fb6787465-ljpkz"] Nov 03 20:00:29.024144 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:29.024109 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a81ac7b8-flpxz"] Nov 03 20:00:29.027945 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:29.027915 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a81ac7b8-flpxz"] Nov 03 20:00:29.859401 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:29.859366 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87c2d4eb-392d-4a66-8622-9ff5eee0b761" path="/var/lib/kubelet/pods/87c2d4eb-392d-4a66-8622-9ff5eee0b761/volumes" Nov 03 20:00:29.859942 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:29.859924 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9502112f-4584-47aa-853a-13c2e33a8e99" path="/var/lib/kubelet/pods/9502112f-4584-47aa-853a-13c2e33a8e99/volumes" Nov 03 20:00:30.152242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:30.152165 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8"] Nov 03 20:00:30.152681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:30.152393 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8" podUID="1f33bc6f-5525-469a-a399-b8d63af713e5" containerName="payload-processing" containerID="cri-o://61f32bf28db3a3751a39d2b47c1cc533d07a803d70f1b0f678e78acebefbc84f" gracePeriod=30 Nov 03 20:00:30.163549 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:30.163518 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt"] Nov 03 20:00:30.164067 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:30.164035 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt" podUID="eb0391f6-c81d-40ca-9542-3d8976d0dbb3" containerName="payload-pre-processing" containerID="cri-o://19e4b68c2ec255f8b7aff70ee2855886ce031facb0be51bbad492b5b4b3080f2" gracePeriod=30 Nov 03 20:00:30.594866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:30.594592 5059 generic.go:356] "Generic (PLEG): container finished" podID="1f33bc6f-5525-469a-a399-b8d63af713e5" containerID="61f32bf28db3a3751a39d2b47c1cc533d07a803d70f1b0f678e78acebefbc84f" exitCode=0 Nov 03 20:00:30.594866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:30.594793 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8" event={"ID":"1f33bc6f-5525-469a-a399-b8d63af713e5","Type":"ContainerDied","Data":"61f32bf28db3a3751a39d2b47c1cc533d07a803d70f1b0f678e78acebefbc84f"} Nov 03 20:00:30.596908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:30.596883 5059 generic.go:356] "Generic (PLEG): container finished" podID="eb0391f6-c81d-40ca-9542-3d8976d0dbb3" containerID="19e4b68c2ec255f8b7aff70ee2855886ce031facb0be51bbad492b5b4b3080f2" exitCode=0 Nov 03 20:00:30.597051 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:30.597029 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt" event={"ID":"eb0391f6-c81d-40ca-9542-3d8976d0dbb3","Type":"ContainerDied","Data":"19e4b68c2ec255f8b7aff70ee2855886ce031facb0be51bbad492b5b4b3080f2"} Nov 03 20:00:30.695272 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:30.685915 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8" Nov 03 20:00:30.755119 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:30.755079 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt" Nov 03 20:00:30.815876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:30.815820 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f33bc6f-5525-469a-a399-b8d63af713e5-kube-api-access-7mwcn\" (UniqueName: \"kubernetes.io/projected/1f33bc6f-5525-469a-a399-b8d63af713e5-kube-api-access-7mwcn\") pod \"1f33bc6f-5525-469a-a399-b8d63af713e5\" (UID: \"1f33bc6f-5525-469a-a399-b8d63af713e5\") " Nov 03 20:00:30.817595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:30.815910 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb0391f6-c81d-40ca-9542-3d8976d0dbb3-kube-api-access-hdgr4\" (UniqueName: \"kubernetes.io/projected/eb0391f6-c81d-40ca-9542-3d8976d0dbb3-kube-api-access-hdgr4\") pod \"eb0391f6-c81d-40ca-9542-3d8976d0dbb3\" (UID: \"eb0391f6-c81d-40ca-9542-3d8976d0dbb3\") " Nov 03 20:00:30.817595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:30.816137 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f33bc6f-5525-469a-a399-b8d63af713e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f33bc6f-5525-469a-a399-b8d63af713e5-plugins-config-volume\") pod \"1f33bc6f-5525-469a-a399-b8d63af713e5\" (UID: \"1f33bc6f-5525-469a-a399-b8d63af713e5\") " Nov 03 20:00:30.817595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:30.816210 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb0391f6-c81d-40ca-9542-3d8976d0dbb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb0391f6-c81d-40ca-9542-3d8976d0dbb3-plugins-config-volume\") pod \"eb0391f6-c81d-40ca-9542-3d8976d0dbb3\" (UID: \"eb0391f6-c81d-40ca-9542-3d8976d0dbb3\") " Nov 03 20:00:30.817595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:30.817295 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb0391f6-c81d-40ca-9542-3d8976d0dbb3-plugins-config-volume" pod "eb0391f6-c81d-40ca-9542-3d8976d0dbb3" (UID: "eb0391f6-c81d-40ca-9542-3d8976d0dbb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:30.820328 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:30.820304 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f33bc6f-5525-469a-a399-b8d63af713e5-plugins-config-volume" pod "1f33bc6f-5525-469a-a399-b8d63af713e5" (UID: "1f33bc6f-5525-469a-a399-b8d63af713e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:30.820526 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:30.820503 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb0391f6-c81d-40ca-9542-3d8976d0dbb3-kube-api-access-hdgr4" pod "eb0391f6-c81d-40ca-9542-3d8976d0dbb3" (UID: "eb0391f6-c81d-40ca-9542-3d8976d0dbb3"). InnerVolumeSpecName "kube-api-access-hdgr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:30.820600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:30.820538 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f33bc6f-5525-469a-a399-b8d63af713e5-kube-api-access-7mwcn" pod "1f33bc6f-5525-469a-a399-b8d63af713e5" (UID: "1f33bc6f-5525-469a-a399-b8d63af713e5"). InnerVolumeSpecName "kube-api-access-7mwcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:30.918409 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:30.918324 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdgr4\" (UniqueName: \"kubernetes.io/projected/eb0391f6-c81d-40ca-9542-3d8976d0dbb3-kube-api-access-hdgr4\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:30.918409 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:30.918358 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f33bc6f-5525-469a-a399-b8d63af713e5-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:30.918409 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:30.918372 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb0391f6-c81d-40ca-9542-3d8976d0dbb3-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:30.918409 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:30.918385 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mwcn\" (UniqueName: \"kubernetes.io/projected/1f33bc6f-5525-469a-a399-b8d63af713e5-kube-api-access-7mwcn\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:31.616715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:31.607344 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8" Nov 03 20:00:31.616715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:31.613602 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8" event={"ID":"1f33bc6f-5525-469a-a399-b8d63af713e5","Type":"ContainerDied","Data":"f83146740c56fb27a67cc7636301086c6858ffe3c98aa49cc44bf43167b67918"} Nov 03 20:00:31.616715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:31.613711 5059 scope.go:122] "RemoveContainer" containerID="61f32bf28db3a3751a39d2b47c1cc533d07a803d70f1b0f678e78acebefbc84f" Nov 03 20:00:31.616715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:31.615775 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt" event={"ID":"eb0391f6-c81d-40ca-9542-3d8976d0dbb3","Type":"ContainerDied","Data":"cb28695f8016f727bb062544b854b31b4a8b61b19e1051545dcd32c1d6235b63"} Nov 03 20:00:31.616715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:31.615839 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt" Nov 03 20:00:31.636762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:31.636737 5059 scope.go:122] "RemoveContainer" containerID="19e4b68c2ec255f8b7aff70ee2855886ce031facb0be51bbad492b5b4b3080f2" Nov 03 20:00:31.654930 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:31.654903 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8"] Nov 03 20:00:31.664784 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:31.662719 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-389e6d-57dbc5b744-jj7g8"] Nov 03 20:00:31.666881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:31.666856 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt"] Nov 03 20:00:31.669940 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:31.669916 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-86f8b7cc75-rstzt"] Nov 03 20:00:31.859434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:31.859395 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f33bc6f-5525-469a-a399-b8d63af713e5" path="/var/lib/kubelet/pods/1f33bc6f-5525-469a-a399-b8d63af713e5/volumes" Nov 03 20:00:31.859847 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:31.859830 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb0391f6-c81d-40ca-9542-3d8976d0dbb3" path="/var/lib/kubelet/pods/eb0391f6-c81d-40ca-9542-3d8976d0dbb3/volumes" Nov 03 20:00:32.254323 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.254290 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11bdb454-xk9wd"] Nov 03 20:00:32.254983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.254966 5059 cpu_manager.go:414] "containerMap: removing container" podUID="b8602537-2de3-4fc6-84f5-7306b45cc4fe" containerName="cleanup" Nov 03 20:00:32.254983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.254981 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8602537-2de3-4fc6-84f5-7306b45cc4fe" containerName="cleanup" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.254993 5059 cpu_manager.go:414] "containerMap: removing container" podUID="5df8b2fc-3ed3-4396-a215-1f66b95e0937" containerName="cleanup" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.254999 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df8b2fc-3ed3-4396-a215-1f66b95e0937" containerName="cleanup" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255009 5059 cpu_manager.go:414] "containerMap: removing container" podUID="b4793e8b-0775-4d3f-ac3c-60197ed96852" containerName="revoke-keys" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255014 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4793e8b-0775-4d3f-ac3c-60197ed96852" containerName="revoke-keys" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255026 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c3c45aa4-c24e-4674-a258-38226dad4b13" containerName="cleanup" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255031 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c45aa4-c24e-4674-a258-38226dad4b13" containerName="cleanup" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255041 5059 cpu_manager.go:414] "containerMap: removing container" podUID="518d7432-1ff3-4ebf-8700-0a234e678892" containerName="cleanup" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255045 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="518d7432-1ff3-4ebf-8700-0a234e678892" containerName="cleanup" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255056 5059 cpu_manager.go:414] "containerMap: removing container" podUID="87c2d4eb-392d-4a66-8622-9ff5eee0b761" containerName="istio-proxy" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255060 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c2d4eb-392d-4a66-8622-9ff5eee0b761" containerName="istio-proxy" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255067 5059 cpu_manager.go:414] "containerMap: removing container" podUID="5df8b2fc-3ed3-4396-a215-1f66b95e0937" containerName="cleanup" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255071 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df8b2fc-3ed3-4396-a215-1f66b95e0937" containerName="cleanup" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255079 5059 cpu_manager.go:414] "containerMap: removing container" podUID="75da83f6-395e-4ca2-8709-7956a4dd7eba" containerName="cleanup" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255084 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75da83f6-395e-4ca2-8709-7956a4dd7eba" containerName="cleanup" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255091 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c3c45aa4-c24e-4674-a258-38226dad4b13" containerName="cleanup" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255095 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c45aa4-c24e-4674-a258-38226dad4b13" containerName="cleanup" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255101 5059 cpu_manager.go:414] "containerMap: removing container" podUID="799b3fa3-1f7a-46c8-868f-1a1624b1fd9c" containerName="cleanup" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255105 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="799b3fa3-1f7a-46c8-868f-1a1624b1fd9c" containerName="cleanup" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255111 5059 cpu_manager.go:414] "containerMap: removing container" podUID="99fe4ede-51a2-4f4a-a5ce-96faef90590c" containerName="cleanup" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255115 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99fe4ede-51a2-4f4a-a5ce-96faef90590c" containerName="cleanup" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255122 5059 cpu_manager.go:414] "containerMap: removing container" podUID="518d7432-1ff3-4ebf-8700-0a234e678892" containerName="cleanup" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255126 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="518d7432-1ff3-4ebf-8700-0a234e678892" containerName="cleanup" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255133 5059 cpu_manager.go:414] "containerMap: removing container" podUID="37f7af3e-9c65-42c8-b761-306fb918190c" containerName="cleanup" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255138 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f7af3e-9c65-42c8-b761-306fb918190c" containerName="cleanup" Nov 03 20:00:32.255130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255147 5059 cpu_manager.go:414] "containerMap: removing container" podUID="75da83f6-395e-4ca2-8709-7956a4dd7eba" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255152 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75da83f6-395e-4ca2-8709-7956a4dd7eba" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255160 5059 cpu_manager.go:414] "containerMap: removing container" podUID="799b3fa3-1f7a-46c8-868f-1a1624b1fd9c" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255164 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="799b3fa3-1f7a-46c8-868f-1a1624b1fd9c" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255171 5059 cpu_manager.go:414] "containerMap: removing container" podUID="99fe4ede-51a2-4f4a-a5ce-96faef90590c" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255175 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99fe4ede-51a2-4f4a-a5ce-96faef90590c" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255183 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1f33bc6f-5525-469a-a399-b8d63af713e5" containerName="payload-processing" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255188 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f33bc6f-5525-469a-a399-b8d63af713e5" containerName="payload-processing" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255198 5059 cpu_manager.go:414] "containerMap: removing container" podUID="b8602537-2de3-4fc6-84f5-7306b45cc4fe" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255202 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8602537-2de3-4fc6-84f5-7306b45cc4fe" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255208 5059 cpu_manager.go:414] "containerMap: removing container" podUID="518d7432-1ff3-4ebf-8700-0a234e678892" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255212 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="518d7432-1ff3-4ebf-8700-0a234e678892" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255228 5059 cpu_manager.go:414] "containerMap: removing container" podUID="99fe4ede-51a2-4f4a-a5ce-96faef90590c" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255233 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99fe4ede-51a2-4f4a-a5ce-96faef90590c" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255240 5059 cpu_manager.go:414] "containerMap: removing container" podUID="2b8f9144-8073-42b2-b859-bdde4c32ecd6" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255244 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b8f9144-8073-42b2-b859-bdde4c32ecd6" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255252 5059 cpu_manager.go:414] "containerMap: removing container" podUID="87c2d4eb-392d-4a66-8622-9ff5eee0b761" containerName="istio-proxy" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255256 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c2d4eb-392d-4a66-8622-9ff5eee0b761" containerName="istio-proxy" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255264 5059 cpu_manager.go:414] "containerMap: removing container" podUID="75da83f6-395e-4ca2-8709-7956a4dd7eba" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255268 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75da83f6-395e-4ca2-8709-7956a4dd7eba" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255276 5059 cpu_manager.go:414] "containerMap: removing container" podUID="37f7af3e-9c65-42c8-b761-306fb918190c" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255280 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f7af3e-9c65-42c8-b761-306fb918190c" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255287 5059 cpu_manager.go:414] "containerMap: removing container" podUID="799b3fa3-1f7a-46c8-868f-1a1624b1fd9c" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255291 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="799b3fa3-1f7a-46c8-868f-1a1624b1fd9c" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255297 5059 cpu_manager.go:414] "containerMap: removing container" podUID="04a05159-3d1d-4317-9890-504243a56f46" containerName="payload-pre-processing" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255302 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a05159-3d1d-4317-9890-504243a56f46" containerName="payload-pre-processing" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255311 5059 cpu_manager.go:414] "containerMap: removing container" podUID="eb0391f6-c81d-40ca-9542-3d8976d0dbb3" containerName="payload-pre-processing" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255315 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb0391f6-c81d-40ca-9542-3d8976d0dbb3" containerName="payload-pre-processing" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255323 5059 cpu_manager.go:414] "containerMap: removing container" podUID="b5075f9d-997f-4fb3-be38-92f1f15f814b" containerName="payload-processing" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255327 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5075f9d-997f-4fb3-be38-92f1f15f814b" containerName="payload-processing" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255337 5059 cpu_manager.go:414] "containerMap: removing container" podUID="2b8f9144-8073-42b2-b859-bdde4c32ecd6" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255342 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b8f9144-8073-42b2-b859-bdde4c32ecd6" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255430 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c45aa4-c24e-4674-a258-38226dad4b13" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255438 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="518d7432-1ff3-4ebf-8700-0a234e678892" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255445 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="87c2d4eb-392d-4a66-8622-9ff5eee0b761" containerName="istio-proxy" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255450 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df8b2fc-3ed3-4396-a215-1f66b95e0937" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255456 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="75da83f6-395e-4ca2-8709-7956a4dd7eba" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255462 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c45aa4-c24e-4674-a258-38226dad4b13" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255466 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="799b3fa3-1f7a-46c8-868f-1a1624b1fd9c" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255471 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="99fe4ede-51a2-4f4a-a5ce-96faef90590c" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255476 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="518d7432-1ff3-4ebf-8700-0a234e678892" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255482 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8602537-2de3-4fc6-84f5-7306b45cc4fe" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255487 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="37f7af3e-9c65-42c8-b761-306fb918190c" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255492 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="75da83f6-395e-4ca2-8709-7956a4dd7eba" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255498 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="99fe4ede-51a2-4f4a-a5ce-96faef90590c" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255504 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f33bc6f-5525-469a-a399-b8d63af713e5" containerName="payload-processing" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255511 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df8b2fc-3ed3-4396-a215-1f66b95e0937" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255517 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8602537-2de3-4fc6-84f5-7306b45cc4fe" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255521 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="518d7432-1ff3-4ebf-8700-0a234e678892" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255527 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="99fe4ede-51a2-4f4a-a5ce-96faef90590c" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255532 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b8f9144-8073-42b2-b859-bdde4c32ecd6" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255538 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="37f7af3e-9c65-42c8-b761-306fb918190c" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255543 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="37f7af3e-9c65-42c8-b761-306fb918190c" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255549 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="799b3fa3-1f7a-46c8-868f-1a1624b1fd9c" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255554 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a05159-3d1d-4317-9890-504243a56f46" containerName="payload-pre-processing" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255559 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb0391f6-c81d-40ca-9542-3d8976d0dbb3" containerName="payload-pre-processing" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255566 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5075f9d-997f-4fb3-be38-92f1f15f814b" containerName="payload-processing" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255573 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b8f9144-8073-42b2-b859-bdde4c32ecd6" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255579 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b8f9144-8073-42b2-b859-bdde4c32ecd6" containerName="cleanup" Nov 03 20:00:32.255907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.255585 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4793e8b-0775-4d3f-ac3c-60197ed96852" containerName="revoke-keys" Nov 03 20:00:32.260416 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.260400 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bdb454-xk9wd" Nov 03 20:00:32.281565 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.271430 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11bdb454-xk9wd"] Nov 03 20:00:32.281565 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.272112 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:00:32.281565 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.272761 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:00:32.432802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.432759 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qlqt\" (UniqueName: \"kubernetes.io/projected/fea8feed-e6f9-44ca-ab14-d355aea06ed8-kube-api-access-9qlqt\") pod \"maas-api-revoke-sub-11bdb454-xk9wd\" (UID: \"fea8feed-e6f9-44ca-ab14-d355aea06ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bdb454-xk9wd" Nov 03 20:00:32.459612 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.433204 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea8feed-e6f9-44ca-ab14-d355aea06ed8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11bdb454-xk9wd\" (UID: \"fea8feed-e6f9-44ca-ab14-d355aea06ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bdb454-xk9wd" Nov 03 20:00:32.477689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.460118 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea8feed-e6f9-44ca-ab14-d355aea06ed8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11bdb454-xk9wd\" (UID: \"fea8feed-e6f9-44ca-ab14-d355aea06ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bdb454-xk9wd" Nov 03 20:00:32.579448 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.579411 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea8feed-e6f9-44ca-ab14-d355aea06ed8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11bdb454-xk9wd\" (UID: \"fea8feed-e6f9-44ca-ab14-d355aea06ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bdb454-xk9wd" Nov 03 20:00:32.579628 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.579537 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qlqt\" (UniqueName: \"kubernetes.io/projected/fea8feed-e6f9-44ca-ab14-d355aea06ed8-kube-api-access-9qlqt\") pod \"maas-api-revoke-sub-11bdb454-xk9wd\" (UID: \"fea8feed-e6f9-44ca-ab14-d355aea06ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bdb454-xk9wd" Nov 03 20:00:32.579718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.579657 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea8feed-e6f9-44ca-ab14-d355aea06ed8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11bdb454-xk9wd\" (UID: \"fea8feed-e6f9-44ca-ab14-d355aea06ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bdb454-xk9wd" Nov 03 20:00:32.580140 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.580112 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea8feed-e6f9-44ca-ab14-d355aea06ed8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11bdb454-xk9wd\" (UID: \"fea8feed-e6f9-44ca-ab14-d355aea06ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bdb454-xk9wd" Nov 03 20:00:32.582071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.582040 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea8feed-e6f9-44ca-ab14-d355aea06ed8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11bdb454-xk9wd\" (UID: \"fea8feed-e6f9-44ca-ab14-d355aea06ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bdb454-xk9wd" Nov 03 20:00:32.597559 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.597533 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qlqt\" (UniqueName: \"kubernetes.io/projected/fea8feed-e6f9-44ca-ab14-d355aea06ed8-kube-api-access-9qlqt\") pod \"maas-api-revoke-sub-11bdb454-xk9wd\" (UID: \"fea8feed-e6f9-44ca-ab14-d355aea06ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bdb454-xk9wd" Nov 03 20:00:32.630120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.630094 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bdb454-xk9wd" Nov 03 20:00:32.923517 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:00:32.921922 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea8feed_e6f9_44ca_ab14_d355aea06ed8.slice/crio-8fb359725a813ffa100c9e031cba2b3bf648c59a21d77ab401e90e29bd476741 WatchSource:0}: Error finding container 8fb359725a813ffa100c9e031cba2b3bf648c59a21d77ab401e90e29bd476741: Status 404 returned error can't find the container with id 8fb359725a813ffa100c9e031cba2b3bf648c59a21d77ab401e90e29bd476741 Nov 03 20:00:32.923517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:32.922882 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11bdb454-xk9wd"] Nov 03 20:00:33.332551 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.332524 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 20:00:33.495481 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.495399 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-podinfo\") pod \"856d3ff7-8514-46b9-8884-f99b674dd6e6\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " Nov 03 20:00:33.495481 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.495441 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-workload-socket\") pod \"856d3ff7-8514-46b9-8884-f99b674dd6e6\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " Nov 03 20:00:33.495481 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.495474 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-data\") pod \"856d3ff7-8514-46b9-8884-f99b674dd6e6\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " Nov 03 20:00:33.495815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.495504 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-token\" (UniqueName: \"kubernetes.io/projected/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-token\") pod \"856d3ff7-8514-46b9-8884-f99b674dd6e6\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " Nov 03 20:00:33.495815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.495530 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-envoy\") pod \"856d3ff7-8514-46b9-8884-f99b674dd6e6\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " Nov 03 20:00:33.495815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.495640 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/856d3ff7-8514-46b9-8884-f99b674dd6e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/856d3ff7-8514-46b9-8884-f99b674dd6e6-istiod-ca-cert\") pod \"856d3ff7-8514-46b9-8884-f99b674dd6e6\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " Nov 03 20:00:33.495815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.495739 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/856d3ff7-8514-46b9-8884-f99b674dd6e6-kube-api-access-8ngjp\" (UniqueName: \"kubernetes.io/projected/856d3ff7-8514-46b9-8884-f99b674dd6e6-kube-api-access-8ngjp\") pod \"856d3ff7-8514-46b9-8884-f99b674dd6e6\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " Nov 03 20:00:33.495815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.495786 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-credential-socket\") pod \"856d3ff7-8514-46b9-8884-f99b674dd6e6\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " Nov 03 20:00:33.496063 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.495846 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-workload-socket" pod "856d3ff7-8514-46b9-8884-f99b674dd6e6" (UID: "856d3ff7-8514-46b9-8884-f99b674dd6e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:00:33.496063 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.495893 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-workload-certs\") pod \"856d3ff7-8514-46b9-8884-f99b674dd6e6\" (UID: \"856d3ff7-8514-46b9-8884-f99b674dd6e6\") " Nov 03 20:00:33.496063 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.496027 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-data" pod "856d3ff7-8514-46b9-8884-f99b674dd6e6" (UID: "856d3ff7-8514-46b9-8884-f99b674dd6e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:00:33.496280 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.496144 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856d3ff7-8514-46b9-8884-f99b674dd6e6-istiod-ca-cert" pod "856d3ff7-8514-46b9-8884-f99b674dd6e6" (UID: "856d3ff7-8514-46b9-8884-f99b674dd6e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:33.496339 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.496291 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-credential-socket" pod "856d3ff7-8514-46b9-8884-f99b674dd6e6" (UID: "856d3ff7-8514-46b9-8884-f99b674dd6e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:00:33.496339 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.496328 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-workload-certs" pod "856d3ff7-8514-46b9-8884-f99b674dd6e6" (UID: "856d3ff7-8514-46b9-8884-f99b674dd6e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:00:33.496630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.496609 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:33.496741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.496635 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:33.496741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.496649 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:33.496741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.496696 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/856d3ff7-8514-46b9-8884-f99b674dd6e6-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:33.496741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.496712 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:33.497876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.497855 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-envoy" pod "856d3ff7-8514-46b9-8884-f99b674dd6e6" (UID: "856d3ff7-8514-46b9-8884-f99b674dd6e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:00:33.498197 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.498178 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-podinfo" pod "856d3ff7-8514-46b9-8884-f99b674dd6e6" (UID: "856d3ff7-8514-46b9-8884-f99b674dd6e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:00:33.498253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.498205 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-token" pod "856d3ff7-8514-46b9-8884-f99b674dd6e6" (UID: "856d3ff7-8514-46b9-8884-f99b674dd6e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:33.498253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.498242 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856d3ff7-8514-46b9-8884-f99b674dd6e6-kube-api-access-8ngjp" pod "856d3ff7-8514-46b9-8884-f99b674dd6e6" (UID: "856d3ff7-8514-46b9-8884-f99b674dd6e6"). InnerVolumeSpecName "kube-api-access-8ngjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:33.598722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.598688 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:33.598722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.598721 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:33.598878 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.598738 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/856d3ff7-8514-46b9-8884-f99b674dd6e6-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:33.598878 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.598753 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ngjp\" (UniqueName: \"kubernetes.io/projected/856d3ff7-8514-46b9-8884-f99b674dd6e6-kube-api-access-8ngjp\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:33.637170 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.637132 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bdb454-xk9wd" event={"ID":"fea8feed-e6f9-44ca-ab14-d355aea06ed8","Type":"ContainerStarted","Data":"cc82e1e6accb53514993ed4694c26d66956bbd1838ba6e3d04decfe587bf5fc1"} Nov 03 20:00:33.637588 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.637194 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bdb454-xk9wd" event={"ID":"fea8feed-e6f9-44ca-ab14-d355aea06ed8","Type":"ContainerStarted","Data":"8fb359725a813ffa100c9e031cba2b3bf648c59a21d77ab401e90e29bd476741"} Nov 03 20:00:33.639433 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.639400 5059 generic.go:356] "Generic (PLEG): container finished" podID="856d3ff7-8514-46b9-8884-f99b674dd6e6" containerID="fb95eba5d616de91bd7fc8a639b7f9be441869b05ec13d63169baa5585f37f2b" exitCode=0 Nov 03 20:00:33.639541 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.639529 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" event={"ID":"856d3ff7-8514-46b9-8884-f99b674dd6e6","Type":"ContainerDied","Data":"fb95eba5d616de91bd7fc8a639b7f9be441869b05ec13d63169baa5585f37f2b"} Nov 03 20:00:33.639596 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.639576 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" event={"ID":"856d3ff7-8514-46b9-8884-f99b674dd6e6","Type":"ContainerDied","Data":"39c3b03e0ec688c9c6957a7cb3a5346d6dbbb14d394b58b2710b232760035905"} Nov 03 20:00:33.639638 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.639608 5059 scope.go:122] "RemoveContainer" containerID="fb95eba5d616de91bd7fc8a639b7f9be441869b05ec13d63169baa5585f37f2b" Nov 03 20:00:33.639830 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.639814 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8" Nov 03 20:00:33.716686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.712926 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bdb454-xk9wd" podStartSLOduration=1.712906833 podStartE2EDuration="1.712906833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:33.699280409 +0000 UTC m=+1427.957077667" watchObservedRunningTime="2026-11-03 20:00:33.712906833 +0000 UTC m=+1427.970704092" Nov 03 20:00:33.872719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.869444 5059 scope.go:122] "RemoveContainer" containerID="4dfd6dd522fa140050dad5e5f74edeaeb56816f8fbb43c92919142c90af8a88b" Nov 03 20:00:33.931086 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.931040 5059 scope.go:122] "RemoveContainer" containerID="fb95eba5d616de91bd7fc8a639b7f9be441869b05ec13d63169baa5585f37f2b" Nov 03 20:00:33.931494 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:33.931460 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb95eba5d616de91bd7fc8a639b7f9be441869b05ec13d63169baa5585f37f2b\": container with ID starting with fb95eba5d616de91bd7fc8a639b7f9be441869b05ec13d63169baa5585f37f2b not found: ID does not exist" containerID="fb95eba5d616de91bd7fc8a639b7f9be441869b05ec13d63169baa5585f37f2b" Nov 03 20:00:33.931623 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.931508 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb95eba5d616de91bd7fc8a639b7f9be441869b05ec13d63169baa5585f37f2b"} err="failed to get container status \"fb95eba5d616de91bd7fc8a639b7f9be441869b05ec13d63169baa5585f37f2b\": rpc error: code = NotFound desc = could not find container \"fb95eba5d616de91bd7fc8a639b7f9be441869b05ec13d63169baa5585f37f2b\": container with ID starting with fb95eba5d616de91bd7fc8a639b7f9be441869b05ec13d63169baa5585f37f2b not found: ID does not exist" Nov 03 20:00:33.931623 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.931535 5059 scope.go:122] "RemoveContainer" containerID="4dfd6dd522fa140050dad5e5f74edeaeb56816f8fbb43c92919142c90af8a88b" Nov 03 20:00:33.931973 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:33.931932 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dfd6dd522fa140050dad5e5f74edeaeb56816f8fbb43c92919142c90af8a88b\": container with ID starting with 4dfd6dd522fa140050dad5e5f74edeaeb56816f8fbb43c92919142c90af8a88b not found: ID does not exist" containerID="4dfd6dd522fa140050dad5e5f74edeaeb56816f8fbb43c92919142c90af8a88b" Nov 03 20:00:33.932114 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.931968 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dfd6dd522fa140050dad5e5f74edeaeb56816f8fbb43c92919142c90af8a88b"} err="failed to get container status \"4dfd6dd522fa140050dad5e5f74edeaeb56816f8fbb43c92919142c90af8a88b\": rpc error: code = NotFound desc = could not find container \"4dfd6dd522fa140050dad5e5f74edeaeb56816f8fbb43c92919142c90af8a88b\": container with ID starting with 4dfd6dd522fa140050dad5e5f74edeaeb56816f8fbb43c92919142c90af8a88b not found: ID does not exist" Nov 03 20:00:33.953807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.953773 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8"] Nov 03 20:00:33.957784 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:33.957756 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-3faa07-openshift-default-6f98bf9cd5-x9bd8"] Nov 03 20:00:34.690684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:34.684865 5059 generic.go:356] "Generic (PLEG): container finished" podID="fea8feed-e6f9-44ca-ab14-d355aea06ed8" containerID="cc82e1e6accb53514993ed4694c26d66956bbd1838ba6e3d04decfe587bf5fc1" exitCode=0 Nov 03 20:00:34.690684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:34.688765 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bdb454-xk9wd" event={"ID":"fea8feed-e6f9-44ca-ab14-d355aea06ed8","Type":"ContainerDied","Data":"cc82e1e6accb53514993ed4694c26d66956bbd1838ba6e3d04decfe587bf5fc1"} Nov 03 20:00:35.912209 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:35.912168 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="856d3ff7-8514-46b9-8884-f99b674dd6e6" path="/var/lib/kubelet/pods/856d3ff7-8514-46b9-8884-f99b674dd6e6/volumes" Nov 03 20:00:35.919367 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:35.919338 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bdb454-xk9wd" Nov 03 20:00:36.088971 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:36.088941 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea8feed-e6f9-44ca-ab14-d355aea06ed8-kube-api-access-9qlqt\" (UniqueName: \"kubernetes.io/projected/fea8feed-e6f9-44ca-ab14-d355aea06ed8-kube-api-access-9qlqt\") pod \"fea8feed-e6f9-44ca-ab14-d355aea06ed8\" (UID: \"fea8feed-e6f9-44ca-ab14-d355aea06ed8\") " Nov 03 20:00:36.089138 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:36.089105 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fea8feed-e6f9-44ca-ab14-d355aea06ed8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea8feed-e6f9-44ca-ab14-d355aea06ed8-maas-api-xks-ca\") pod \"fea8feed-e6f9-44ca-ab14-d355aea06ed8\" (UID: \"fea8feed-e6f9-44ca-ab14-d355aea06ed8\") " Nov 03 20:00:36.089180 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:36.089153 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fea8feed-e6f9-44ca-ab14-d355aea06ed8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea8feed-e6f9-44ca-ab14-d355aea06ed8-maas-api-openshift-ca\") pod \"fea8feed-e6f9-44ca-ab14-d355aea06ed8\" (UID: \"fea8feed-e6f9-44ca-ab14-d355aea06ed8\") " Nov 03 20:00:36.089526 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:36.089501 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fea8feed-e6f9-44ca-ab14-d355aea06ed8-maas-api-openshift-ca" pod "fea8feed-e6f9-44ca-ab14-d355aea06ed8" (UID: "fea8feed-e6f9-44ca-ab14-d355aea06ed8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:36.089621 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:36.089607 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea8feed-e6f9-44ca-ab14-d355aea06ed8-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:36.091130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:36.091106 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea8feed-e6f9-44ca-ab14-d355aea06ed8-kube-api-access-9qlqt" pod "fea8feed-e6f9-44ca-ab14-d355aea06ed8" (UID: "fea8feed-e6f9-44ca-ab14-d355aea06ed8"). InnerVolumeSpecName "kube-api-access-9qlqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:36.091213 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:36.091115 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fea8feed-e6f9-44ca-ab14-d355aea06ed8-maas-api-xks-ca" pod "fea8feed-e6f9-44ca-ab14-d355aea06ed8" (UID: "fea8feed-e6f9-44ca-ab14-d355aea06ed8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:00:36.191280 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:36.191248 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea8feed-e6f9-44ca-ab14-d355aea06ed8-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:36.191280 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:36.191278 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qlqt\" (UniqueName: \"kubernetes.io/projected/fea8feed-e6f9-44ca-ab14-d355aea06ed8-kube-api-access-9qlqt\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:36.728225 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:36.728194 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bdb454-xk9wd" Nov 03 20:00:36.728419 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:36.728234 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11bdb454-xk9wd" event={"ID":"fea8feed-e6f9-44ca-ab14-d355aea06ed8","Type":"ContainerDied","Data":"8fb359725a813ffa100c9e031cba2b3bf648c59a21d77ab401e90e29bd476741"} Nov 03 20:00:36.728419 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:36.728283 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fb359725a813ffa100c9e031cba2b3bf648c59a21d77ab401e90e29bd476741" Nov 03 20:00:37.290752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:37.290716 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln"] Nov 03 20:00:37.291167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:37.290977 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" podUID="67d6043a-3156-4ca3-992b-fe613a604883" containerName="istio-proxy" containerID="cri-o://7636f8a662a01ab07cddbeab56ed1cec283a0e38f9f59180d20792fd3241868f" gracePeriod=30 Nov 03 20:00:40.607413 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:40.607371 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:00:42.587477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.587450 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 20:00:42.600799 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.600771 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-workload-certs\") pod \"67d6043a-3156-4ca3-992b-fe613a604883\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " Nov 03 20:00:42.600916 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.600854 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d6043a-3156-4ca3-992b-fe613a604883-istio-token\" (UniqueName: \"kubernetes.io/projected/67d6043a-3156-4ca3-992b-fe613a604883-istio-token\") pod \"67d6043a-3156-4ca3-992b-fe613a604883\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " Nov 03 20:00:42.600916 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.600888 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-istio-envoy\") pod \"67d6043a-3156-4ca3-992b-fe613a604883\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " Nov 03 20:00:42.601009 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.600946 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67d6043a-3156-4ca3-992b-fe613a604883-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d6043a-3156-4ca3-992b-fe613a604883-istio-podinfo\") pod \"67d6043a-3156-4ca3-992b-fe613a604883\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " Nov 03 20:00:42.601009 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.600972 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d6043a-3156-4ca3-992b-fe613a604883-kube-api-access-gmmz7\" (UniqueName: \"kubernetes.io/projected/67d6043a-3156-4ca3-992b-fe613a604883-kube-api-access-gmmz7\") pod \"67d6043a-3156-4ca3-992b-fe613a604883\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " Nov 03 20:00:42.601110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.601020 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-credential-socket\") pod \"67d6043a-3156-4ca3-992b-fe613a604883\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " Nov 03 20:00:42.601110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.601046 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67d6043a-3156-4ca3-992b-fe613a604883-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d6043a-3156-4ca3-992b-fe613a604883-istiod-ca-cert\") pod \"67d6043a-3156-4ca3-992b-fe613a604883\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " Nov 03 20:00:42.601110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.601067 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-istio-data\") pod \"67d6043a-3156-4ca3-992b-fe613a604883\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " Nov 03 20:00:42.601270 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.601155 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-workload-socket\") pod \"67d6043a-3156-4ca3-992b-fe613a604883\" (UID: \"67d6043a-3156-4ca3-992b-fe613a604883\") " Nov 03 20:00:42.601795 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.601038 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-workload-certs" pod "67d6043a-3156-4ca3-992b-fe613a604883" (UID: "67d6043a-3156-4ca3-992b-fe613a604883"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:00:42.601795 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.601542 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d6043a-3156-4ca3-992b-fe613a604883-istiod-ca-cert" pod "67d6043a-3156-4ca3-992b-fe613a604883" (UID: "67d6043a-3156-4ca3-992b-fe613a604883"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:42.601795 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.601697 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-istio-data" pod "67d6043a-3156-4ca3-992b-fe613a604883" (UID: "67d6043a-3156-4ca3-992b-fe613a604883"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:00:42.602040 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.601829 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-credential-socket" pod "67d6043a-3156-4ca3-992b-fe613a604883" (UID: "67d6043a-3156-4ca3-992b-fe613a604883"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:00:42.602040 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.601838 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-workload-socket" pod "67d6043a-3156-4ca3-992b-fe613a604883" (UID: "67d6043a-3156-4ca3-992b-fe613a604883"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:00:42.602040 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.601950 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:42.602040 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.601969 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:42.602040 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.601978 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d6043a-3156-4ca3-992b-fe613a604883-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:42.602040 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.601987 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:42.602040 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.601995 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:42.603166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.603145 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-istio-envoy" pod "67d6043a-3156-4ca3-992b-fe613a604883" (UID: "67d6043a-3156-4ca3-992b-fe613a604883"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:00:42.603349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.603325 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d6043a-3156-4ca3-992b-fe613a604883-kube-api-access-gmmz7" pod "67d6043a-3156-4ca3-992b-fe613a604883" (UID: "67d6043a-3156-4ca3-992b-fe613a604883"). InnerVolumeSpecName "kube-api-access-gmmz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:42.603580 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.603553 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67d6043a-3156-4ca3-992b-fe613a604883-istio-podinfo" pod "67d6043a-3156-4ca3-992b-fe613a604883" (UID: "67d6043a-3156-4ca3-992b-fe613a604883"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:00:42.603709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.603579 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d6043a-3156-4ca3-992b-fe613a604883-istio-token" pod "67d6043a-3156-4ca3-992b-fe613a604883" (UID: "67d6043a-3156-4ca3-992b-fe613a604883"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:42.703857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.703828 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d6043a-3156-4ca3-992b-fe613a604883-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:42.703857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.703853 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d6043a-3156-4ca3-992b-fe613a604883-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:42.703857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.703863 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d6043a-3156-4ca3-992b-fe613a604883-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:42.704093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.703874 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmmz7\" (UniqueName: \"kubernetes.io/projected/67d6043a-3156-4ca3-992b-fe613a604883-kube-api-access-gmmz7\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:42.784496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.784462 5059 generic.go:356] "Generic (PLEG): container finished" podID="67d6043a-3156-4ca3-992b-fe613a604883" containerID="7636f8a662a01ab07cddbeab56ed1cec283a0e38f9f59180d20792fd3241868f" exitCode=0 Nov 03 20:00:42.784705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.784528 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" Nov 03 20:00:42.784705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.784553 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" event={"ID":"67d6043a-3156-4ca3-992b-fe613a604883","Type":"ContainerDied","Data":"7636f8a662a01ab07cddbeab56ed1cec283a0e38f9f59180d20792fd3241868f"} Nov 03 20:00:42.784705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.784624 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln" event={"ID":"67d6043a-3156-4ca3-992b-fe613a604883","Type":"ContainerDied","Data":"dd83de92c0866e0e87bb639ccdaf4f956756576c85e827fb56c50871bb63d4f5"} Nov 03 20:00:42.784705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.784674 5059 scope.go:122] "RemoveContainer" containerID="7636f8a662a01ab07cddbeab56ed1cec283a0e38f9f59180d20792fd3241868f" Nov 03 20:00:42.797047 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.797033 5059 scope.go:122] "RemoveContainer" containerID="1c0a4f71993101224f72fbd33bc2d0305b4c5c1b3891b51eb07284545f0536e7" Nov 03 20:00:42.806835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.806814 5059 scope.go:122] "RemoveContainer" containerID="7636f8a662a01ab07cddbeab56ed1cec283a0e38f9f59180d20792fd3241868f" Nov 03 20:00:42.807226 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:42.807198 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7636f8a662a01ab07cddbeab56ed1cec283a0e38f9f59180d20792fd3241868f\": container with ID starting with 7636f8a662a01ab07cddbeab56ed1cec283a0e38f9f59180d20792fd3241868f not found: ID does not exist" containerID="7636f8a662a01ab07cddbeab56ed1cec283a0e38f9f59180d20792fd3241868f" Nov 03 20:00:42.807341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.807233 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7636f8a662a01ab07cddbeab56ed1cec283a0e38f9f59180d20792fd3241868f"} err="failed to get container status \"7636f8a662a01ab07cddbeab56ed1cec283a0e38f9f59180d20792fd3241868f\": rpc error: code = NotFound desc = could not find container \"7636f8a662a01ab07cddbeab56ed1cec283a0e38f9f59180d20792fd3241868f\": container with ID starting with 7636f8a662a01ab07cddbeab56ed1cec283a0e38f9f59180d20792fd3241868f not found: ID does not exist" Nov 03 20:00:42.807341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.807255 5059 scope.go:122] "RemoveContainer" containerID="1c0a4f71993101224f72fbd33bc2d0305b4c5c1b3891b51eb07284545f0536e7" Nov 03 20:00:42.807598 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:42.807568 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c0a4f71993101224f72fbd33bc2d0305b4c5c1b3891b51eb07284545f0536e7\": container with ID starting with 1c0a4f71993101224f72fbd33bc2d0305b4c5c1b3891b51eb07284545f0536e7 not found: ID does not exist" containerID="1c0a4f71993101224f72fbd33bc2d0305b4c5c1b3891b51eb07284545f0536e7" Nov 03 20:00:42.807745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.807607 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c0a4f71993101224f72fbd33bc2d0305b4c5c1b3891b51eb07284545f0536e7"} err="failed to get container status \"1c0a4f71993101224f72fbd33bc2d0305b4c5c1b3891b51eb07284545f0536e7\": rpc error: code = NotFound desc = could not find container \"1c0a4f71993101224f72fbd33bc2d0305b4c5c1b3891b51eb07284545f0536e7\": container with ID starting with 1c0a4f71993101224f72fbd33bc2d0305b4c5c1b3891b51eb07284545f0536e7 not found: ID does not exist" Nov 03 20:00:42.811451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.811430 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln"] Nov 03 20:00:42.815111 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:42.815064 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-btcln"] Nov 03 20:00:43.859210 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.859178 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67d6043a-3156-4ca3-992b-fe613a604883" path="/var/lib/kubelet/pods/67d6043a-3156-4ca3-992b-fe613a604883/volumes" Nov 03 20:00:43.889043 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.889008 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4"] Nov 03 20:00:43.891100 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.891070 5059 cpu_manager.go:414] "containerMap: removing container" podUID="856d3ff7-8514-46b9-8884-f99b674dd6e6" containerName="istio-proxy" Nov 03 20:00:43.891277 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.891263 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="856d3ff7-8514-46b9-8884-f99b674dd6e6" containerName="istio-proxy" Nov 03 20:00:43.891371 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.891360 5059 cpu_manager.go:414] "containerMap: removing container" podUID="2b8f9144-8073-42b2-b859-bdde4c32ecd6" containerName="cleanup" Nov 03 20:00:43.891442 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.891431 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b8f9144-8073-42b2-b859-bdde4c32ecd6" containerName="cleanup" Nov 03 20:00:43.891527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.891517 5059 cpu_manager.go:414] "containerMap: removing container" podUID="fea8feed-e6f9-44ca-ab14-d355aea06ed8" containerName="revoke-keys" Nov 03 20:00:43.891676 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.891648 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fea8feed-e6f9-44ca-ab14-d355aea06ed8" containerName="revoke-keys" Nov 03 20:00:43.891782 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.891771 5059 cpu_manager.go:414] "containerMap: removing container" podUID="67d6043a-3156-4ca3-992b-fe613a604883" containerName="istio-proxy" Nov 03 20:00:43.891850 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.891841 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d6043a-3156-4ca3-992b-fe613a604883" containerName="istio-proxy" Nov 03 20:00:43.891942 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.891933 5059 cpu_manager.go:414] "containerMap: removing container" podUID="67d6043a-3156-4ca3-992b-fe613a604883" containerName="istio-proxy" Nov 03 20:00:43.892019 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.892010 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d6043a-3156-4ca3-992b-fe613a604883" containerName="istio-proxy" Nov 03 20:00:43.892139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.892130 5059 cpu_manager.go:414] "containerMap: removing container" podUID="5df8b2fc-3ed3-4396-a215-1f66b95e0937" containerName="cleanup" Nov 03 20:00:43.892215 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.892206 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df8b2fc-3ed3-4396-a215-1f66b95e0937" containerName="cleanup" Nov 03 20:00:43.892291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.892283 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c3c45aa4-c24e-4674-a258-38226dad4b13" containerName="cleanup" Nov 03 20:00:43.892353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.892344 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c45aa4-c24e-4674-a258-38226dad4b13" containerName="cleanup" Nov 03 20:00:43.892472 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.892462 5059 cpu_manager.go:414] "containerMap: removing container" podUID="b8602537-2de3-4fc6-84f5-7306b45cc4fe" containerName="cleanup" Nov 03 20:00:43.892538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.892529 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8602537-2de3-4fc6-84f5-7306b45cc4fe" containerName="cleanup" Nov 03 20:00:43.892634 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.892625 5059 cpu_manager.go:414] "containerMap: removing container" podUID="37f7af3e-9c65-42c8-b761-306fb918190c" containerName="cleanup" Nov 03 20:00:43.892720 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.892710 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f7af3e-9c65-42c8-b761-306fb918190c" containerName="cleanup" Nov 03 20:00:43.893098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.893081 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="75da83f6-395e-4ca2-8709-7956a4dd7eba" containerName="cleanup" Nov 03 20:00:43.893205 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.893195 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df8b2fc-3ed3-4396-a215-1f66b95e0937" containerName="cleanup" Nov 03 20:00:43.893287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.893276 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="856d3ff7-8514-46b9-8884-f99b674dd6e6" containerName="istio-proxy" Nov 03 20:00:43.893362 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.893353 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="799b3fa3-1f7a-46c8-868f-1a1624b1fd9c" containerName="cleanup" Nov 03 20:00:43.893476 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.893464 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="fea8feed-e6f9-44ca-ab14-d355aea06ed8" containerName="revoke-keys" Nov 03 20:00:43.893551 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.893543 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d6043a-3156-4ca3-992b-fe613a604883" containerName="istio-proxy" Nov 03 20:00:43.893637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.893628 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d6043a-3156-4ca3-992b-fe613a604883" containerName="istio-proxy" Nov 03 20:00:43.893738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.893728 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c45aa4-c24e-4674-a258-38226dad4b13" containerName="cleanup" Nov 03 20:00:43.893811 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.893803 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="87c2d4eb-392d-4a66-8622-9ff5eee0b761" containerName="istio-proxy" Nov 03 20:00:43.893895 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.893887 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8602537-2de3-4fc6-84f5-7306b45cc4fe" containerName="cleanup" Nov 03 20:00:43.903560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.903340 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4" Nov 03 20:00:43.907135 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.907111 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4"] Nov 03 20:00:43.920038 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.920011 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk4vs\" (UniqueName: \"kubernetes.io/projected/76e4398b-91ee-4b1f-b498-4b54d7322193-kube-api-access-sk4vs\") pod \"payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4\" (UID: \"76e4398b-91ee-4b1f-b498-4b54d7322193\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4" Nov 03 20:00:43.920124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.920086 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76e4398b-91ee-4b1f-b498-4b54d7322193-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4\" (UID: \"76e4398b-91ee-4b1f-b498-4b54d7322193\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4" Nov 03 20:00:43.926971 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.926933 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8b3020-dockercfg-t8fsh\"" Nov 03 20:00:43.931518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:43.931498 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8b3020\"" Nov 03 20:00:44.054686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.039460 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk4vs\" (UniqueName: \"kubernetes.io/projected/76e4398b-91ee-4b1f-b498-4b54d7322193-kube-api-access-sk4vs\") pod \"payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4\" (UID: \"76e4398b-91ee-4b1f-b498-4b54d7322193\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4" Nov 03 20:00:44.055031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.054999 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76e4398b-91ee-4b1f-b498-4b54d7322193-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4\" (UID: \"76e4398b-91ee-4b1f-b498-4b54d7322193\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4" Nov 03 20:00:44.069500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.066327 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv"] Nov 03 20:00:44.069500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.067364 5059 cpu_manager.go:414] "containerMap: removing container" podUID="856d3ff7-8514-46b9-8884-f99b674dd6e6" containerName="istio-proxy" Nov 03 20:00:44.069500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.067388 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="856d3ff7-8514-46b9-8884-f99b674dd6e6" containerName="istio-proxy" Nov 03 20:00:44.069500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.067587 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="856d3ff7-8514-46b9-8884-f99b674dd6e6" containerName="istio-proxy" Nov 03 20:00:44.081987 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.071834 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv" Nov 03 20:00:44.085833 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.083326 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76e4398b-91ee-4b1f-b498-4b54d7322193-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4\" (UID: \"76e4398b-91ee-4b1f-b498-4b54d7322193\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4" Nov 03 20:00:44.144120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.144046 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv"] Nov 03 20:00:44.157319 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.157276 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zr4h\" (UniqueName: \"kubernetes.io/projected/fbe92210-726b-458d-9766-452f49c237d7-kube-api-access-9zr4h\") pod \"payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv\" (UID: \"fbe92210-726b-458d-9766-452f49c237d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv" Nov 03 20:00:44.157686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.157639 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe92210-726b-458d-9766-452f49c237d7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv\" (UID: \"fbe92210-726b-458d-9766-452f49c237d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv" Nov 03 20:00:44.164034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.164007 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk4vs\" (UniqueName: \"kubernetes.io/projected/76e4398b-91ee-4b1f-b498-4b54d7322193-kube-api-access-sk4vs\") pod \"payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4\" (UID: \"76e4398b-91ee-4b1f-b498-4b54d7322193\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4" Nov 03 20:00:44.226725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.222860 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4" Nov 03 20:00:44.265962 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.265913 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zr4h\" (UniqueName: \"kubernetes.io/projected/fbe92210-726b-458d-9766-452f49c237d7-kube-api-access-9zr4h\") pod \"payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv\" (UID: \"fbe92210-726b-458d-9766-452f49c237d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv" Nov 03 20:00:44.266122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.266042 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe92210-726b-458d-9766-452f49c237d7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv\" (UID: \"fbe92210-726b-458d-9766-452f49c237d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv" Nov 03 20:00:44.267018 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.266992 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe92210-726b-458d-9766-452f49c237d7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv\" (UID: \"fbe92210-726b-458d-9766-452f49c237d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv" Nov 03 20:00:44.279392 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.279365 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zr4h\" (UniqueName: \"kubernetes.io/projected/fbe92210-726b-458d-9766-452f49c237d7-kube-api-access-9zr4h\") pod \"payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv\" (UID: \"fbe92210-726b-458d-9766-452f49c237d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv" Nov 03 20:00:44.293530 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.293498 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6"] Nov 03 20:00:44.503514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.503444 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv" Nov 03 20:00:44.524511 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.524487 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4"] Nov 03 20:00:44.525843 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:00:44.525813 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e4398b_91ee_4b1f_b498_4b54d7322193.slice/crio-68e8cbaf9052dfbc9df05dabe07bc13299af3bed2569860d93a8ed1df441f586 WatchSource:0}: Error finding container 68e8cbaf9052dfbc9df05dabe07bc13299af3bed2569860d93a8ed1df441f586: Status 404 returned error can't find the container with id 68e8cbaf9052dfbc9df05dabe07bc13299af3bed2569860d93a8ed1df441f586 Nov 03 20:00:44.799709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.799676 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv"] Nov 03 20:00:44.799847 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:00:44.799823 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe92210_726b_458d_9766_452f49c237d7.slice/crio-ff553bd533cdaea8146bc61ab282c16199de2b94c80d808a66ff7471abd0eb31 WatchSource:0}: Error finding container ff553bd533cdaea8146bc61ab282c16199de2b94c80d808a66ff7471abd0eb31: Status 404 returned error can't find the container with id ff553bd533cdaea8146bc61ab282c16199de2b94c80d808a66ff7471abd0eb31 Nov 03 20:00:44.806315 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.806285 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4" event={"ID":"76e4398b-91ee-4b1f-b498-4b54d7322193","Type":"ContainerStarted","Data":"8cffdefedb54423d797c3eae4993cb83799f831bb5e05c62ff386dd41339cf5f"} Nov 03 20:00:44.806422 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.806346 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4" event={"ID":"76e4398b-91ee-4b1f-b498-4b54d7322193","Type":"ContainerStarted","Data":"68e8cbaf9052dfbc9df05dabe07bc13299af3bed2569860d93a8ed1df441f586"} Nov 03 20:00:44.806422 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.806371 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4" Nov 03 20:00:44.828619 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:44.828581 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4" podStartSLOduration=1.828568857 podStartE2EDuration="1.828568857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:44.824807961 +0000 UTC m=+1439.082605218" watchObservedRunningTime="2026-11-03 20:00:44.828568857 +0000 UTC m=+1439.086366113" Nov 03 20:00:45.816040 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:45.816001 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv" event={"ID":"fbe92210-726b-458d-9766-452f49c237d7","Type":"ContainerStarted","Data":"a49ed5cb02983f39d30db29b7ff938dd01ab1f494991a6df6e616c84808342bd"} Nov 03 20:00:45.816438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:45.816058 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv" event={"ID":"fbe92210-726b-458d-9766-452f49c237d7","Type":"ContainerStarted","Data":"ff553bd533cdaea8146bc61ab282c16199de2b94c80d808a66ff7471abd0eb31"} Nov 03 20:00:45.816438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:45.816087 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv" Nov 03 20:00:45.838103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:45.838062 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv" podStartSLOduration=2.838049106 podStartE2EDuration="2.838049106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:45.837060641 +0000 UTC m=+1440.094857899" watchObservedRunningTime="2026-11-03 20:00:45.838049106 +0000 UTC m=+1440.095846362" Nov 03 20:00:48.764371 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:48.764338 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7"] Nov 03 20:00:48.764780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:48.764528 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7" podUID="1463c361-4736-4158-83f1-e31353a338f4" containerName="payload-processing" containerID="cri-o://da13d2704a055e1c385f688c426a9e0c1775173a7eed85d675fed505bb8d5572" gracePeriod=30 Nov 03 20:00:49.092716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:49.091495 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7" Nov 03 20:00:49.144688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:49.144355 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1463c361-4736-4158-83f1-e31353a338f4-kube-api-access-6hcfw\" (UniqueName: \"kubernetes.io/projected/1463c361-4736-4158-83f1-e31353a338f4-kube-api-access-6hcfw\") pod \"1463c361-4736-4158-83f1-e31353a338f4\" (UID: \"1463c361-4736-4158-83f1-e31353a338f4\") " Nov 03 20:00:49.144688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:49.144436 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1463c361-4736-4158-83f1-e31353a338f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1463c361-4736-4158-83f1-e31353a338f4-plugins-config-volume\") pod \"1463c361-4736-4158-83f1-e31353a338f4\" (UID: \"1463c361-4736-4158-83f1-e31353a338f4\") " Nov 03 20:00:49.148692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:49.146906 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1463c361-4736-4158-83f1-e31353a338f4-plugins-config-volume" pod "1463c361-4736-4158-83f1-e31353a338f4" (UID: "1463c361-4736-4158-83f1-e31353a338f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:49.153684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:49.148983 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1463c361-4736-4158-83f1-e31353a338f4-kube-api-access-6hcfw" pod "1463c361-4736-4158-83f1-e31353a338f4" (UID: "1463c361-4736-4158-83f1-e31353a338f4"). InnerVolumeSpecName "kube-api-access-6hcfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:49.246292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:49.246253 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hcfw\" (UniqueName: \"kubernetes.io/projected/1463c361-4736-4158-83f1-e31353a338f4-kube-api-access-6hcfw\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:49.246292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:49.246285 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1463c361-4736-4158-83f1-e31353a338f4-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:49.826425 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:49.826390 5059 scope.go:122] "RemoveContainer" containerID="6e88b3c669ba1ae0f42319195d2e11824d33d6baf9b5492a19b8a5d6d474ebb8" Nov 03 20:00:49.867933 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:49.867899 5059 generic.go:356] "Generic (PLEG): container finished" podID="1463c361-4736-4158-83f1-e31353a338f4" containerID="da13d2704a055e1c385f688c426a9e0c1775173a7eed85d675fed505bb8d5572" exitCode=0 Nov 03 20:00:49.868099 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:49.867942 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7" event={"ID":"1463c361-4736-4158-83f1-e31353a338f4","Type":"ContainerDied","Data":"da13d2704a055e1c385f688c426a9e0c1775173a7eed85d675fed505bb8d5572"} Nov 03 20:00:49.868211 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:49.868007 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7" Nov 03 20:00:49.868211 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:49.868136 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7" event={"ID":"1463c361-4736-4158-83f1-e31353a338f4","Type":"ContainerDied","Data":"33756232f4ae1ff9b6a01b9a4309abcb6c5b247aa0716b6d9a84ca990c4ff60e"} Nov 03 20:00:49.868211 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:49.868199 5059 scope.go:122] "RemoveContainer" containerID="da13d2704a055e1c385f688c426a9e0c1775173a7eed85d675fed505bb8d5572" Nov 03 20:00:49.881885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:49.881807 5059 scope.go:122] "RemoveContainer" containerID="da13d2704a055e1c385f688c426a9e0c1775173a7eed85d675fed505bb8d5572" Nov 03 20:00:49.882161 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:00:49.882077 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da13d2704a055e1c385f688c426a9e0c1775173a7eed85d675fed505bb8d5572\": container with ID starting with da13d2704a055e1c385f688c426a9e0c1775173a7eed85d675fed505bb8d5572 not found: ID does not exist" containerID="da13d2704a055e1c385f688c426a9e0c1775173a7eed85d675fed505bb8d5572" Nov 03 20:00:49.882161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:49.882110 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da13d2704a055e1c385f688c426a9e0c1775173a7eed85d675fed505bb8d5572"} err="failed to get container status \"da13d2704a055e1c385f688c426a9e0c1775173a7eed85d675fed505bb8d5572\": rpc error: code = NotFound desc = could not find container \"da13d2704a055e1c385f688c426a9e0c1775173a7eed85d675fed505bb8d5572\": container with ID starting with da13d2704a055e1c385f688c426a9e0c1775173a7eed85d675fed505bb8d5572 not found: ID does not exist" Nov 03 20:00:49.898802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:49.898781 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7"] Nov 03 20:00:49.903749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:49.903728 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-7d9b8cc5c8-kkqq7"] Nov 03 20:00:51.323919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:51.323886 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35852943-4n7m4"] Nov 03 20:00:51.324568 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:51.324542 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1463c361-4736-4158-83f1-e31353a338f4" containerName="payload-processing" Nov 03 20:00:51.324568 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:51.324559 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1463c361-4736-4158-83f1-e31353a338f4" containerName="payload-processing" Nov 03 20:00:51.324715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:51.324704 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1463c361-4736-4158-83f1-e31353a338f4" containerName="payload-processing" Nov 03 20:00:51.330233 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:51.330214 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35852943-4n7m4" Nov 03 20:00:51.334872 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:51.334162 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:00:51.335541 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:51.335410 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:00:51.338510 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:51.338485 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35852943-4n7m4"] Nov 03 20:00:51.491537 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:51.491020 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe8145b-bfa0-4b87-b570-6fdca730e088-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35852943-4n7m4\" (UID: \"afe8145b-bfa0-4b87-b570-6fdca730e088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35852943-4n7m4" Nov 03 20:00:51.491537 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:51.491163 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9mrh\" (UniqueName: \"kubernetes.io/projected/afe8145b-bfa0-4b87-b570-6fdca730e088-kube-api-access-p9mrh\") pod \"maas-api-revoke-sub-35852943-4n7m4\" (UID: \"afe8145b-bfa0-4b87-b570-6fdca730e088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35852943-4n7m4" Nov 03 20:00:51.491537 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:51.491323 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe8145b-bfa0-4b87-b570-6fdca730e088-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35852943-4n7m4\" (UID: \"afe8145b-bfa0-4b87-b570-6fdca730e088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35852943-4n7m4" Nov 03 20:00:51.593082 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:51.593003 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9mrh\" (UniqueName: \"kubernetes.io/projected/afe8145b-bfa0-4b87-b570-6fdca730e088-kube-api-access-p9mrh\") pod \"maas-api-revoke-sub-35852943-4n7m4\" (UID: \"afe8145b-bfa0-4b87-b570-6fdca730e088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35852943-4n7m4" Nov 03 20:00:51.593222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:51.593116 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe8145b-bfa0-4b87-b570-6fdca730e088-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35852943-4n7m4\" (UID: \"afe8145b-bfa0-4b87-b570-6fdca730e088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35852943-4n7m4" Nov 03 20:00:51.593274 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:51.593234 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe8145b-bfa0-4b87-b570-6fdca730e088-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35852943-4n7m4\" (UID: \"afe8145b-bfa0-4b87-b570-6fdca730e088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35852943-4n7m4" Nov 03 20:00:51.593828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:51.593806 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe8145b-bfa0-4b87-b570-6fdca730e088-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35852943-4n7m4\" (UID: \"afe8145b-bfa0-4b87-b570-6fdca730e088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35852943-4n7m4" Nov 03 20:00:51.595629 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:51.595599 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe8145b-bfa0-4b87-b570-6fdca730e088-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35852943-4n7m4\" (UID: \"afe8145b-bfa0-4b87-b570-6fdca730e088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35852943-4n7m4" Nov 03 20:00:51.605231 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:51.605211 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9mrh\" (UniqueName: \"kubernetes.io/projected/afe8145b-bfa0-4b87-b570-6fdca730e088-kube-api-access-p9mrh\") pod \"maas-api-revoke-sub-35852943-4n7m4\" (UID: \"afe8145b-bfa0-4b87-b570-6fdca730e088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35852943-4n7m4" Nov 03 20:00:51.646876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:51.646849 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35852943-4n7m4" Nov 03 20:00:51.858616 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:51.858534 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1463c361-4736-4158-83f1-e31353a338f4" path="/var/lib/kubelet/pods/1463c361-4736-4158-83f1-e31353a338f4/volumes" Nov 03 20:00:51.931418 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:51.931392 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35852943-4n7m4"] Nov 03 20:00:51.932234 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:00:51.932210 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe8145b_bfa0_4b87_b570_6fdca730e088.slice/crio-c2593dedec00872d04c04f6d9097e9945618aa11527406b46ba4f07d109a63ce WatchSource:0}: Error finding container c2593dedec00872d04c04f6d9097e9945618aa11527406b46ba4f07d109a63ce: Status 404 returned error can't find the container with id c2593dedec00872d04c04f6d9097e9945618aa11527406b46ba4f07d109a63ce Nov 03 20:00:52.894976 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:52.894943 5059 generic.go:356] "Generic (PLEG): container finished" podID="afe8145b-bfa0-4b87-b570-6fdca730e088" containerID="fe8c5965bdb63f76399a237cf281aad4805fa4963b78b83fdd05b1b2865c8f92" exitCode=0 Nov 03 20:00:52.895524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:52.894979 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35852943-4n7m4" event={"ID":"afe8145b-bfa0-4b87-b570-6fdca730e088","Type":"ContainerDied","Data":"fe8c5965bdb63f76399a237cf281aad4805fa4963b78b83fdd05b1b2865c8f92"} Nov 03 20:00:52.895649 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:52.895572 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35852943-4n7m4" event={"ID":"afe8145b-bfa0-4b87-b570-6fdca730e088","Type":"ContainerStarted","Data":"c2593dedec00872d04c04f6d9097e9945618aa11527406b46ba4f07d109a63ce"} Nov 03 20:00:54.059988 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:54.059963 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35852943-4n7m4" Nov 03 20:00:54.231730 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:54.231617 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afe8145b-bfa0-4b87-b570-6fdca730e088-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe8145b-bfa0-4b87-b570-6fdca730e088-maas-api-openshift-ca\") pod \"afe8145b-bfa0-4b87-b570-6fdca730e088\" (UID: \"afe8145b-bfa0-4b87-b570-6fdca730e088\") " Nov 03 20:00:54.231907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:54.231854 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afe8145b-bfa0-4b87-b570-6fdca730e088-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe8145b-bfa0-4b87-b570-6fdca730e088-maas-api-xks-ca\") pod \"afe8145b-bfa0-4b87-b570-6fdca730e088\" (UID: \"afe8145b-bfa0-4b87-b570-6fdca730e088\") " Nov 03 20:00:54.231976 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:54.231908 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe8145b-bfa0-4b87-b570-6fdca730e088-kube-api-access-p9mrh\" (UniqueName: \"kubernetes.io/projected/afe8145b-bfa0-4b87-b570-6fdca730e088-kube-api-access-p9mrh\") pod \"afe8145b-bfa0-4b87-b570-6fdca730e088\" (UID: \"afe8145b-bfa0-4b87-b570-6fdca730e088\") " Nov 03 20:00:54.232105 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:54.232078 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe8145b-bfa0-4b87-b570-6fdca730e088-maas-api-openshift-ca" pod "afe8145b-bfa0-4b87-b570-6fdca730e088" (UID: "afe8145b-bfa0-4b87-b570-6fdca730e088"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:54.232570 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:54.232547 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe8145b-bfa0-4b87-b570-6fdca730e088-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:54.234227 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:54.234199 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe8145b-bfa0-4b87-b570-6fdca730e088-kube-api-access-p9mrh" pod "afe8145b-bfa0-4b87-b570-6fdca730e088" (UID: "afe8145b-bfa0-4b87-b570-6fdca730e088"). InnerVolumeSpecName "kube-api-access-p9mrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:54.235806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:54.235778 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe8145b-bfa0-4b87-b570-6fdca730e088-maas-api-xks-ca" pod "afe8145b-bfa0-4b87-b570-6fdca730e088" (UID: "afe8145b-bfa0-4b87-b570-6fdca730e088"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:00:54.334326 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:54.334295 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe8145b-bfa0-4b87-b570-6fdca730e088-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:54.334326 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:54.334324 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9mrh\" (UniqueName: \"kubernetes.io/projected/afe8145b-bfa0-4b87-b570-6fdca730e088-kube-api-access-p9mrh\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:54.911825 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:54.911780 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35852943-4n7m4" event={"ID":"afe8145b-bfa0-4b87-b570-6fdca730e088","Type":"ContainerDied","Data":"c2593dedec00872d04c04f6d9097e9945618aa11527406b46ba4f07d109a63ce"} Nov 03 20:00:54.912013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:54.911834 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2593dedec00872d04c04f6d9097e9945618aa11527406b46ba4f07d109a63ce" Nov 03 20:00:54.912013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:54.911888 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35852943-4n7m4" Nov 03 20:00:54.945572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:54.944650 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-398e0990-tbplm"] Nov 03 20:00:54.945761 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:54.945649 5059 cpu_manager.go:414] "containerMap: removing container" podUID="afe8145b-bfa0-4b87-b570-6fdca730e088" containerName="revoke-keys" Nov 03 20:00:54.945761 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:54.945681 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe8145b-bfa0-4b87-b570-6fdca730e088" containerName="revoke-keys" Nov 03 20:00:54.945895 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:54.945880 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe8145b-bfa0-4b87-b570-6fdca730e088" containerName="revoke-keys" Nov 03 20:00:54.950031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:54.949986 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398e0990-tbplm" Nov 03 20:00:54.952605 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:54.952581 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-398e0990-tbplm"] Nov 03 20:00:54.953762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:54.953739 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:00:54.953931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:54.953911 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:00:55.149936 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:55.149879 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4c71f5-9a02-4740-b8a8-25a1c4178b23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-398e0990-tbplm\" (UID: \"0e4c71f5-9a02-4740-b8a8-25a1c4178b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398e0990-tbplm" Nov 03 20:00:55.150301 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:55.150083 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4c71f5-9a02-4740-b8a8-25a1c4178b23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-398e0990-tbplm\" (UID: \"0e4c71f5-9a02-4740-b8a8-25a1c4178b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398e0990-tbplm" Nov 03 20:00:55.150301 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:55.150117 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnx72\" (UniqueName: \"kubernetes.io/projected/0e4c71f5-9a02-4740-b8a8-25a1c4178b23-kube-api-access-rnx72\") pod \"maas-api-revoke-sub-398e0990-tbplm\" (UID: \"0e4c71f5-9a02-4740-b8a8-25a1c4178b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398e0990-tbplm" Nov 03 20:00:55.251307 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:55.251233 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4c71f5-9a02-4740-b8a8-25a1c4178b23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-398e0990-tbplm\" (UID: \"0e4c71f5-9a02-4740-b8a8-25a1c4178b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398e0990-tbplm" Nov 03 20:00:55.251438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:55.251405 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4c71f5-9a02-4740-b8a8-25a1c4178b23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-398e0990-tbplm\" (UID: \"0e4c71f5-9a02-4740-b8a8-25a1c4178b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398e0990-tbplm" Nov 03 20:00:55.251494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:55.251442 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnx72\" (UniqueName: \"kubernetes.io/projected/0e4c71f5-9a02-4740-b8a8-25a1c4178b23-kube-api-access-rnx72\") pod \"maas-api-revoke-sub-398e0990-tbplm\" (UID: \"0e4c71f5-9a02-4740-b8a8-25a1c4178b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398e0990-tbplm" Nov 03 20:00:55.251877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:55.251856 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4c71f5-9a02-4740-b8a8-25a1c4178b23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-398e0990-tbplm\" (UID: \"0e4c71f5-9a02-4740-b8a8-25a1c4178b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398e0990-tbplm" Nov 03 20:00:55.254008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:55.253987 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4c71f5-9a02-4740-b8a8-25a1c4178b23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-398e0990-tbplm\" (UID: \"0e4c71f5-9a02-4740-b8a8-25a1c4178b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398e0990-tbplm" Nov 03 20:00:55.263518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:55.263495 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnx72\" (UniqueName: \"kubernetes.io/projected/0e4c71f5-9a02-4740-b8a8-25a1c4178b23-kube-api-access-rnx72\") pod \"maas-api-revoke-sub-398e0990-tbplm\" (UID: \"0e4c71f5-9a02-4740-b8a8-25a1c4178b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-398e0990-tbplm" Nov 03 20:00:55.273334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:55.273314 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398e0990-tbplm" Nov 03 20:00:55.565348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:55.565319 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-398e0990-tbplm"] Nov 03 20:00:55.566762 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:00:55.566737 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4c71f5_9a02_4740_b8a8_25a1c4178b23.slice/crio-c3c9b23d56f0703e424f32f6a09428d3aab15485feec9263880bfb496a188e38 WatchSource:0}: Error finding container c3c9b23d56f0703e424f32f6a09428d3aab15485feec9263880bfb496a188e38: Status 404 returned error can't find the container with id c3c9b23d56f0703e424f32f6a09428d3aab15485feec9263880bfb496a188e38 Nov 03 20:00:55.817781 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:55.817697 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4" Nov 03 20:00:55.919856 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:55.919823 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398e0990-tbplm" event={"ID":"0e4c71f5-9a02-4740-b8a8-25a1c4178b23","Type":"ContainerStarted","Data":"c3c9b23d56f0703e424f32f6a09428d3aab15485feec9263880bfb496a188e38"} Nov 03 20:00:56.826115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:56.826087 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv" Nov 03 20:00:56.928299 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:56.928267 5059 generic.go:356] "Generic (PLEG): container finished" podID="0e4c71f5-9a02-4740-b8a8-25a1c4178b23" containerID="c2a6adb3375dc514cd970092ba1eb5db443fef7cb70fb4a6c349f817bfc707d8" exitCode=0 Nov 03 20:00:56.928450 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:56.928320 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398e0990-tbplm" event={"ID":"0e4c71f5-9a02-4740-b8a8-25a1c4178b23","Type":"ContainerDied","Data":"c2a6adb3375dc514cd970092ba1eb5db443fef7cb70fb4a6c349f817bfc707d8"} Nov 03 20:00:57.468203 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:57.468167 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh"] Nov 03 20:00:57.468620 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:57.468583 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" podUID="c7ae3583-b9fa-4fe3-9621-fe95f254bce7" containerName="istio-proxy" containerID="cri-o://7c09c7574e1818265f6c829187102752ced557d8a23f1942190802c9afb945e6" gracePeriod=30 Nov 03 20:00:58.090082 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:58.090053 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398e0990-tbplm" Nov 03 20:00:58.128437 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:58.128405 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e4c71f5-9a02-4740-b8a8-25a1c4178b23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4c71f5-9a02-4740-b8a8-25a1c4178b23-maas-api-xks-ca\") pod \"0e4c71f5-9a02-4740-b8a8-25a1c4178b23\" (UID: \"0e4c71f5-9a02-4740-b8a8-25a1c4178b23\") " Nov 03 20:00:58.128583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:58.128487 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e4c71f5-9a02-4740-b8a8-25a1c4178b23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4c71f5-9a02-4740-b8a8-25a1c4178b23-maas-api-openshift-ca\") pod \"0e4c71f5-9a02-4740-b8a8-25a1c4178b23\" (UID: \"0e4c71f5-9a02-4740-b8a8-25a1c4178b23\") " Nov 03 20:00:58.128583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:58.128513 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e4c71f5-9a02-4740-b8a8-25a1c4178b23-kube-api-access-rnx72\" (UniqueName: \"kubernetes.io/projected/0e4c71f5-9a02-4740-b8a8-25a1c4178b23-kube-api-access-rnx72\") pod \"0e4c71f5-9a02-4740-b8a8-25a1c4178b23\" (UID: \"0e4c71f5-9a02-4740-b8a8-25a1c4178b23\") " Nov 03 20:00:58.128949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:58.128916 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e4c71f5-9a02-4740-b8a8-25a1c4178b23-maas-api-openshift-ca" pod "0e4c71f5-9a02-4740-b8a8-25a1c4178b23" (UID: "0e4c71f5-9a02-4740-b8a8-25a1c4178b23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:58.129277 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:58.129253 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e4c71f5-9a02-4740-b8a8-25a1c4178b23-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:58.130730 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:58.130651 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4c71f5-9a02-4740-b8a8-25a1c4178b23-kube-api-access-rnx72" pod "0e4c71f5-9a02-4740-b8a8-25a1c4178b23" (UID: "0e4c71f5-9a02-4740-b8a8-25a1c4178b23"). InnerVolumeSpecName "kube-api-access-rnx72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:58.131219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:58.131188 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e4c71f5-9a02-4740-b8a8-25a1c4178b23-maas-api-xks-ca" pod "0e4c71f5-9a02-4740-b8a8-25a1c4178b23" (UID: "0e4c71f5-9a02-4740-b8a8-25a1c4178b23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:00:58.231017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:58.230933 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnx72\" (UniqueName: \"kubernetes.io/projected/0e4c71f5-9a02-4740-b8a8-25a1c4178b23-kube-api-access-rnx72\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:58.231017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:58.230967 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e4c71f5-9a02-4740-b8a8-25a1c4178b23-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:58.944905 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:58.944878 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398e0990-tbplm" Nov 03 20:00:58.945061 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:58.944879 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-398e0990-tbplm" event={"ID":"0e4c71f5-9a02-4740-b8a8-25a1c4178b23","Type":"ContainerDied","Data":"c3c9b23d56f0703e424f32f6a09428d3aab15485feec9263880bfb496a188e38"} Nov 03 20:00:58.945061 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:58.945011 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3c9b23d56f0703e424f32f6a09428d3aab15485feec9263880bfb496a188e38" Nov 03 20:00:59.325481 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.325443 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" podUID="b995a77c-4a8e-44c5-82cc-c8ad99b3e048" containerName="main" containerID="cri-o://9c4115725cabe7e524d49294d673b3fcca584650d9410e615185eeec6f338449" gracePeriod=45 Nov 03 20:00:59.370588 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.370552 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" podUID="b995a77c-4a8e-44c5-82cc-c8ad99b3e048" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": dial tcp 10.132.0.160:8000: connect: connection refused" Nov 03 20:00:59.622440 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.622411 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 20:00:59.651102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.651074 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-tmp-dir\") pod \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\" (UID: \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\") " Nov 03 20:00:59.651258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.651110 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-kube-api-access-tq29m\" (UniqueName: \"kubernetes.io/projected/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-kube-api-access-tq29m\") pod \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\" (UID: \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\") " Nov 03 20:00:59.651343 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.651272 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-dshm\") pod \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\" (UID: \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\") " Nov 03 20:00:59.651397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.651355 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-home\" (UniqueName: \"kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-home\") pod \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\" (UID: \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\") " Nov 03 20:00:59.651397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.651377 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-tmp-dir" pod "b995a77c-4a8e-44c5-82cc-c8ad99b3e048" (UID: "b995a77c-4a8e-44c5-82cc-c8ad99b3e048"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:00:59.651502 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.651455 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-model-cache\") pod \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\" (UID: \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\") " Nov 03 20:00:59.651557 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.651511 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-tls-certs\" (UniqueName: \"kubernetes.io/secret/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-tls-certs\") pod \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\" (UID: \"b995a77c-4a8e-44c5-82cc-c8ad99b3e048\") " Nov 03 20:00:59.651948 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.651836 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-home" pod "b995a77c-4a8e-44c5-82cc-c8ad99b3e048" (UID: "b995a77c-4a8e-44c5-82cc-c8ad99b3e048"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:00:59.652238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.652215 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:59.652361 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.652241 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:59.652361 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.652316 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-model-cache" pod "b995a77c-4a8e-44c5-82cc-c8ad99b3e048" (UID: "b995a77c-4a8e-44c5-82cc-c8ad99b3e048"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:00:59.653864 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.653841 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-dshm" pod "b995a77c-4a8e-44c5-82cc-c8ad99b3e048" (UID: "b995a77c-4a8e-44c5-82cc-c8ad99b3e048"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:00:59.654393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.654369 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-tls-certs" pod "b995a77c-4a8e-44c5-82cc-c8ad99b3e048" (UID: "b995a77c-4a8e-44c5-82cc-c8ad99b3e048"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:00:59.654649 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.654623 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-kube-api-access-tq29m" pod "b995a77c-4a8e-44c5-82cc-c8ad99b3e048" (UID: "b995a77c-4a8e-44c5-82cc-c8ad99b3e048"). InnerVolumeSpecName "kube-api-access-tq29m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:59.753478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.753448 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:59.753478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.753476 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:59.753656 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.753488 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq29m\" (UniqueName: \"kubernetes.io/projected/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-kube-api-access-tq29m\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:59.753656 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.753497 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b995a77c-4a8e-44c5-82cc-c8ad99b3e048-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:59.976369 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.971333 5059 generic.go:356] "Generic (PLEG): container finished" podID="b995a77c-4a8e-44c5-82cc-c8ad99b3e048" containerID="9c4115725cabe7e524d49294d673b3fcca584650d9410e615185eeec6f338449" exitCode=0 Nov 03 20:00:59.976369 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.971520 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" event={"ID":"b995a77c-4a8e-44c5-82cc-c8ad99b3e048","Type":"ContainerDied","Data":"9c4115725cabe7e524d49294d673b3fcca584650d9410e615185eeec6f338449"} Nov 03 20:00:59.976369 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.971567 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" event={"ID":"b995a77c-4a8e-44c5-82cc-c8ad99b3e048","Type":"ContainerDied","Data":"9f0c24d0e478577aec8c18edc725005c6f6879719412a751416d187db3ebed0a"} Nov 03 20:00:59.976369 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.971589 5059 scope.go:122] "RemoveContainer" containerID="9c4115725cabe7e524d49294d673b3fcca584650d9410e615185eeec6f338449" Nov 03 20:00:59.976369 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:00:59.971798 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6" Nov 03 20:01:00.042543 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:00.042482 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6"] Nov 03 20:01:00.043975 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:00.042927 5059 scope.go:122] "RemoveContainer" containerID="9c4115725cabe7e524d49294d673b3fcca584650d9410e615185eeec6f338449" Nov 03 20:01:00.043975 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:01:00.043362 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c4115725cabe7e524d49294d673b3fcca584650d9410e615185eeec6f338449\": container with ID starting with 9c4115725cabe7e524d49294d673b3fcca584650d9410e615185eeec6f338449 not found: ID does not exist" containerID="9c4115725cabe7e524d49294d673b3fcca584650d9410e615185eeec6f338449" Nov 03 20:01:00.043975 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:00.043406 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c4115725cabe7e524d49294d673b3fcca584650d9410e615185eeec6f338449"} err="failed to get container status \"9c4115725cabe7e524d49294d673b3fcca584650d9410e615185eeec6f338449\": rpc error: code = NotFound desc = could not find container \"9c4115725cabe7e524d49294d673b3fcca584650d9410e615185eeec6f338449\": container with ID starting with 9c4115725cabe7e524d49294d673b3fcca584650d9410e615185eeec6f338449 not found: ID does not exist" Nov 03 20:01:00.046910 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:00.046880 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-1b1d04/praxis-route-1b1d04-kserve-656b956d69-dcrz6"] Nov 03 20:01:00.754699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:00.754192 5059 patch_prober.go:29] interesting pod/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:01:00.754699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:00.754248 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" podUID="c7ae3583-b9fa-4fe3-9621-fe95f254bce7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:01:01.867686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:01.867379 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:01.941582 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:01.940018 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b995a77c-4a8e-44c5-82cc-c8ad99b3e048" path="/var/lib/kubelet/pods/b995a77c-4a8e-44c5-82cc-c8ad99b3e048/volumes" Nov 03 20:01:01.993426 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:01.993383 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22ec26b7-9rqqj"] Nov 03 20:01:01.994719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:01.994648 5059 cpu_manager.go:414] "containerMap: removing container" podUID="0e4c71f5-9a02-4740-b8a8-25a1c4178b23" containerName="revoke-keys" Nov 03 20:01:01.994862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:01.994847 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e4c71f5-9a02-4740-b8a8-25a1c4178b23" containerName="revoke-keys" Nov 03 20:01:01.994989 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:01.994977 5059 cpu_manager.go:414] "containerMap: removing container" podUID="b995a77c-4a8e-44c5-82cc-c8ad99b3e048" containerName="main" Nov 03 20:01:01.995089 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:01.995076 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b995a77c-4a8e-44c5-82cc-c8ad99b3e048" containerName="main" Nov 03 20:01:01.995373 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:01.995352 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e4c71f5-9a02-4740-b8a8-25a1c4178b23" containerName="revoke-keys" Nov 03 20:01:01.995502 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:01.995489 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b995a77c-4a8e-44c5-82cc-c8ad99b3e048" containerName="main" Nov 03 20:01:02.011848 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.011825 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ec26b7-9rqqj" Nov 03 20:01:02.017466 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.011724 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22ec26b7-9rqqj"] Nov 03 20:01:02.027684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.020973 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:01:02.027684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.021712 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:01:02.122780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.122638 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69cdf024-85ab-4f82-8441-87e8e04f2a7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22ec26b7-9rqqj\" (UID: \"69cdf024-85ab-4f82-8441-87e8e04f2a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ec26b7-9rqqj" Nov 03 20:01:02.129773 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.123126 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69cdf024-85ab-4f82-8441-87e8e04f2a7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22ec26b7-9rqqj\" (UID: \"69cdf024-85ab-4f82-8441-87e8e04f2a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ec26b7-9rqqj" Nov 03 20:01:02.129773 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.123243 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9gz7\" (UniqueName: \"kubernetes.io/projected/69cdf024-85ab-4f82-8441-87e8e04f2a7c-kube-api-access-s9gz7\") pod \"maas-api-revoke-sub-22ec26b7-9rqqj\" (UID: \"69cdf024-85ab-4f82-8441-87e8e04f2a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ec26b7-9rqqj" Nov 03 20:01:02.224939 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.224908 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69cdf024-85ab-4f82-8441-87e8e04f2a7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22ec26b7-9rqqj\" (UID: \"69cdf024-85ab-4f82-8441-87e8e04f2a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ec26b7-9rqqj" Nov 03 20:01:02.225256 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.225232 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9gz7\" (UniqueName: \"kubernetes.io/projected/69cdf024-85ab-4f82-8441-87e8e04f2a7c-kube-api-access-s9gz7\") pod \"maas-api-revoke-sub-22ec26b7-9rqqj\" (UID: \"69cdf024-85ab-4f82-8441-87e8e04f2a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ec26b7-9rqqj" Nov 03 20:01:02.225524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.225500 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69cdf024-85ab-4f82-8441-87e8e04f2a7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22ec26b7-9rqqj\" (UID: \"69cdf024-85ab-4f82-8441-87e8e04f2a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ec26b7-9rqqj" Nov 03 20:01:02.225712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.225692 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69cdf024-85ab-4f82-8441-87e8e04f2a7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22ec26b7-9rqqj\" (UID: \"69cdf024-85ab-4f82-8441-87e8e04f2a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ec26b7-9rqqj" Nov 03 20:01:02.228230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.228197 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69cdf024-85ab-4f82-8441-87e8e04f2a7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22ec26b7-9rqqj\" (UID: \"69cdf024-85ab-4f82-8441-87e8e04f2a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ec26b7-9rqqj" Nov 03 20:01:02.238566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.238522 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9gz7\" (UniqueName: \"kubernetes.io/projected/69cdf024-85ab-4f82-8441-87e8e04f2a7c-kube-api-access-s9gz7\") pod \"maas-api-revoke-sub-22ec26b7-9rqqj\" (UID: \"69cdf024-85ab-4f82-8441-87e8e04f2a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ec26b7-9rqqj" Nov 03 20:01:02.433093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.433019 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ec26b7-9rqqj" Nov 03 20:01:02.720829 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:01:02.720796 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69cdf024_85ab_4f82_8441_87e8e04f2a7c.slice/crio-0c134b678d7303c0f5afc4c3a813f11b38de6655320f49e2b8d42e6bc60d5951 WatchSource:0}: Error finding container 0c134b678d7303c0f5afc4c3a813f11b38de6655320f49e2b8d42e6bc60d5951: Status 404 returned error can't find the container with id 0c134b678d7303c0f5afc4c3a813f11b38de6655320f49e2b8d42e6bc60d5951 Nov 03 20:01:02.725266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.725237 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22ec26b7-9rqqj"] Nov 03 20:01:02.732135 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.732079 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 20:01:02.833533 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.833503 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-credential-socket\") pod \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " Nov 03 20:01:02.833740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.833559 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-token\" (UniqueName: \"kubernetes.io/projected/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-token\") pod \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " Nov 03 20:01:02.833740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.833591 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-kube-api-access-zg9b6\" (UniqueName: \"kubernetes.io/projected/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-kube-api-access-zg9b6\") pod \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " Nov 03 20:01:02.833740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.833647 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-workload-socket\") pod \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " Nov 03 20:01:02.833945 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.833845 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-credential-socket" pod "c7ae3583-b9fa-4fe3-9621-fe95f254bce7" (UID: "c7ae3583-b9fa-4fe3-9621-fe95f254bce7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:02.833945 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.833855 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-envoy\") pod \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " Nov 03 20:01:02.833945 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.833934 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istiod-ca-cert\") pod \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " Nov 03 20:01:02.834106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.834046 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-podinfo\") pod \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " Nov 03 20:01:02.834170 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.834102 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-workload-certs\") pod \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " Nov 03 20:01:02.834231 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.834176 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-data\") pod \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\" (UID: \"c7ae3583-b9fa-4fe3-9621-fe95f254bce7\") " Nov 03 20:01:02.834286 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.834216 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-workload-socket" pod "c7ae3583-b9fa-4fe3-9621-fe95f254bce7" (UID: "c7ae3583-b9fa-4fe3-9621-fe95f254bce7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:02.834517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.834468 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-workload-certs" pod "c7ae3583-b9fa-4fe3-9621-fe95f254bce7" (UID: "c7ae3583-b9fa-4fe3-9621-fe95f254bce7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:02.834517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.834507 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istiod-ca-cert" pod "c7ae3583-b9fa-4fe3-9621-fe95f254bce7" (UID: "c7ae3583-b9fa-4fe3-9621-fe95f254bce7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:01:02.834917 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.834887 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-data" pod "c7ae3583-b9fa-4fe3-9621-fe95f254bce7" (UID: "c7ae3583-b9fa-4fe3-9621-fe95f254bce7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:02.835027 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.834922 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:02.835027 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.834946 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:02.835027 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.834960 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:02.835027 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.834977 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:02.836286 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.836259 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-podinfo" pod "c7ae3583-b9fa-4fe3-9621-fe95f254bce7" (UID: "c7ae3583-b9fa-4fe3-9621-fe95f254bce7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:01:02.836399 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.836320 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-kube-api-access-zg9b6" pod "c7ae3583-b9fa-4fe3-9621-fe95f254bce7" (UID: "c7ae3583-b9fa-4fe3-9621-fe95f254bce7"). InnerVolumeSpecName "kube-api-access-zg9b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:02.836399 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.836342 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-envoy" pod "c7ae3583-b9fa-4fe3-9621-fe95f254bce7" (UID: "c7ae3583-b9fa-4fe3-9621-fe95f254bce7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:02.836399 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.836362 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-token" pod "c7ae3583-b9fa-4fe3-9621-fe95f254bce7" (UID: "c7ae3583-b9fa-4fe3-9621-fe95f254bce7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:02.937037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.936997 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:02.937037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.937027 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:02.937037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.937038 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:02.937473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.937052 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:02.937473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:02.937061 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg9b6\" (UniqueName: \"kubernetes.io/projected/c7ae3583-b9fa-4fe3-9621-fe95f254bce7-kube-api-access-zg9b6\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:03.151562 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:03.151509 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ec26b7-9rqqj" event={"ID":"69cdf024-85ab-4f82-8441-87e8e04f2a7c","Type":"ContainerStarted","Data":"be4e474aec01d226b253dfff7b360cf1f37548ad866e35c071ff0bfd36f30588"} Nov 03 20:01:03.151814 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:03.151585 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ec26b7-9rqqj" event={"ID":"69cdf024-85ab-4f82-8441-87e8e04f2a7c","Type":"ContainerStarted","Data":"0c134b678d7303c0f5afc4c3a813f11b38de6655320f49e2b8d42e6bc60d5951"} Nov 03 20:01:03.160739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:03.160708 5059 generic.go:356] "Generic (PLEG): container finished" podID="c7ae3583-b9fa-4fe3-9621-fe95f254bce7" containerID="7c09c7574e1818265f6c829187102752ced557d8a23f1942190802c9afb945e6" exitCode=0 Nov 03 20:01:03.160884 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:03.160782 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" event={"ID":"c7ae3583-b9fa-4fe3-9621-fe95f254bce7","Type":"ContainerDied","Data":"7c09c7574e1818265f6c829187102752ced557d8a23f1942190802c9afb945e6"} Nov 03 20:01:03.160884 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:03.160844 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" event={"ID":"c7ae3583-b9fa-4fe3-9621-fe95f254bce7","Type":"ContainerDied","Data":"10723e3dc7ea42b9274094c99ef94f6eecbceaf7f6506e81bfe854a1077de76f"} Nov 03 20:01:03.160884 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:03.160870 5059 scope.go:122] "RemoveContainer" containerID="7c09c7574e1818265f6c829187102752ced557d8a23f1942190802c9afb945e6" Nov 03 20:01:03.161156 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:03.161111 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh" Nov 03 20:01:03.171959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:03.171945 5059 scope.go:122] "RemoveContainer" containerID="7c09c7574e1818265f6c829187102752ced557d8a23f1942190802c9afb945e6" Nov 03 20:01:03.172208 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:01:03.172187 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c09c7574e1818265f6c829187102752ced557d8a23f1942190802c9afb945e6\": container with ID starting with 7c09c7574e1818265f6c829187102752ced557d8a23f1942190802c9afb945e6 not found: ID does not exist" containerID="7c09c7574e1818265f6c829187102752ced557d8a23f1942190802c9afb945e6" Nov 03 20:01:03.172266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:03.172216 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c09c7574e1818265f6c829187102752ced557d8a23f1942190802c9afb945e6"} err="failed to get container status \"7c09c7574e1818265f6c829187102752ced557d8a23f1942190802c9afb945e6\": rpc error: code = NotFound desc = could not find container \"7c09c7574e1818265f6c829187102752ced557d8a23f1942190802c9afb945e6\": container with ID starting with 7c09c7574e1818265f6c829187102752ced557d8a23f1942190802c9afb945e6 not found: ID does not exist" Nov 03 20:01:03.176908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:03.176867 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ec26b7-9rqqj" podStartSLOduration=2.176856275 podStartE2EDuration="2.176856275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:03.173433659 +0000 UTC m=+1457.431230928" watchObservedRunningTime="2026-11-03 20:01:03.176856275 +0000 UTC m=+1457.434653531" Nov 03 20:01:03.190426 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:03.190406 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh"] Nov 03 20:01:03.201356 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:03.201337 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-pv6zh"] Nov 03 20:01:03.859958 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:03.859923 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7ae3583-b9fa-4fe3-9621-fe95f254bce7" path="/var/lib/kubelet/pods/c7ae3583-b9fa-4fe3-9621-fe95f254bce7/volumes" Nov 03 20:01:04.170079 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:04.170048 5059 generic.go:356] "Generic (PLEG): container finished" podID="69cdf024-85ab-4f82-8441-87e8e04f2a7c" containerID="be4e474aec01d226b253dfff7b360cf1f37548ad866e35c071ff0bfd36f30588" exitCode=0 Nov 03 20:01:04.170517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:04.170155 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ec26b7-9rqqj" event={"ID":"69cdf024-85ab-4f82-8441-87e8e04f2a7c","Type":"ContainerDied","Data":"be4e474aec01d226b253dfff7b360cf1f37548ad866e35c071ff0bfd36f30588"} Nov 03 20:01:05.340974 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:05.340949 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ec26b7-9rqqj" Nov 03 20:01:05.486277 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:05.486152 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69cdf024-85ab-4f82-8441-87e8e04f2a7c-kube-api-access-s9gz7\" (UniqueName: \"kubernetes.io/projected/69cdf024-85ab-4f82-8441-87e8e04f2a7c-kube-api-access-s9gz7\") pod \"69cdf024-85ab-4f82-8441-87e8e04f2a7c\" (UID: \"69cdf024-85ab-4f82-8441-87e8e04f2a7c\") " Nov 03 20:01:05.486485 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:05.486364 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69cdf024-85ab-4f82-8441-87e8e04f2a7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69cdf024-85ab-4f82-8441-87e8e04f2a7c-maas-api-xks-ca\") pod \"69cdf024-85ab-4f82-8441-87e8e04f2a7c\" (UID: \"69cdf024-85ab-4f82-8441-87e8e04f2a7c\") " Nov 03 20:01:05.490689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:05.489151 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69cdf024-85ab-4f82-8441-87e8e04f2a7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69cdf024-85ab-4f82-8441-87e8e04f2a7c-maas-api-openshift-ca\") pod \"69cdf024-85ab-4f82-8441-87e8e04f2a7c\" (UID: \"69cdf024-85ab-4f82-8441-87e8e04f2a7c\") " Nov 03 20:01:05.490689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:05.490118 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69cdf024-85ab-4f82-8441-87e8e04f2a7c-maas-api-openshift-ca" pod "69cdf024-85ab-4f82-8441-87e8e04f2a7c" (UID: "69cdf024-85ab-4f82-8441-87e8e04f2a7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:01:05.494271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:05.494242 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69cdf024-85ab-4f82-8441-87e8e04f2a7c-kube-api-access-s9gz7" pod "69cdf024-85ab-4f82-8441-87e8e04f2a7c" (UID: "69cdf024-85ab-4f82-8441-87e8e04f2a7c"). InnerVolumeSpecName "kube-api-access-s9gz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:05.494505 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:05.494482 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69cdf024-85ab-4f82-8441-87e8e04f2a7c-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:05.494597 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:05.494511 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9gz7\" (UniqueName: \"kubernetes.io/projected/69cdf024-85ab-4f82-8441-87e8e04f2a7c-kube-api-access-s9gz7\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:05.499640 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:05.499598 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69cdf024-85ab-4f82-8441-87e8e04f2a7c-maas-api-xks-ca" pod "69cdf024-85ab-4f82-8441-87e8e04f2a7c" (UID: "69cdf024-85ab-4f82-8441-87e8e04f2a7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:01:05.596406 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:05.596373 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69cdf024-85ab-4f82-8441-87e8e04f2a7c-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:06.187223 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:06.187128 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ec26b7-9rqqj" event={"ID":"69cdf024-85ab-4f82-8441-87e8e04f2a7c","Type":"ContainerDied","Data":"0c134b678d7303c0f5afc4c3a813f11b38de6655320f49e2b8d42e6bc60d5951"} Nov 03 20:01:06.187223 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:06.187185 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c134b678d7303c0f5afc4c3a813f11b38de6655320f49e2b8d42e6bc60d5951" Nov 03 20:01:06.187223 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:06.187137 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ec26b7-9rqqj" Nov 03 20:01:07.466503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:07.466469 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bca3de8-8q2pm"] Nov 03 20:01:07.467159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:07.467144 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c7ae3583-b9fa-4fe3-9621-fe95f254bce7" containerName="istio-proxy" Nov 03 20:01:07.467199 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:07.467160 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ae3583-b9fa-4fe3-9621-fe95f254bce7" containerName="istio-proxy" Nov 03 20:01:07.467249 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:07.467205 5059 cpu_manager.go:414] "containerMap: removing container" podUID="69cdf024-85ab-4f82-8441-87e8e04f2a7c" containerName="revoke-keys" Nov 03 20:01:07.467249 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:07.467210 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69cdf024-85ab-4f82-8441-87e8e04f2a7c" containerName="revoke-keys" Nov 03 20:01:07.467307 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:07.467299 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="69cdf024-85ab-4f82-8441-87e8e04f2a7c" containerName="revoke-keys" Nov 03 20:01:07.467343 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:07.467311 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ae3583-b9fa-4fe3-9621-fe95f254bce7" containerName="istio-proxy" Nov 03 20:01:07.470945 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:07.470929 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bca3de8-8q2pm" Nov 03 20:01:07.476120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:07.475942 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:01:07.476120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:07.476098 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:01:07.483955 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:07.480105 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bca3de8-8q2pm"] Nov 03 20:01:07.628155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:07.628128 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e3083b-94d7-4655-9e85-dc27bc209f44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bca3de8-8q2pm\" (UID: \"16e3083b-94d7-4655-9e85-dc27bc209f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bca3de8-8q2pm" Nov 03 20:01:07.628303 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:07.628167 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e3083b-94d7-4655-9e85-dc27bc209f44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bca3de8-8q2pm\" (UID: \"16e3083b-94d7-4655-9e85-dc27bc209f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bca3de8-8q2pm" Nov 03 20:01:07.628303 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:07.628294 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plhb9\" (UniqueName: \"kubernetes.io/projected/16e3083b-94d7-4655-9e85-dc27bc209f44-kube-api-access-plhb9\") pod \"maas-api-revoke-sub-6bca3de8-8q2pm\" (UID: \"16e3083b-94d7-4655-9e85-dc27bc209f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bca3de8-8q2pm" Nov 03 20:01:07.730009 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:07.729924 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e3083b-94d7-4655-9e85-dc27bc209f44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bca3de8-8q2pm\" (UID: \"16e3083b-94d7-4655-9e85-dc27bc209f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bca3de8-8q2pm" Nov 03 20:01:07.730009 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:07.729988 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e3083b-94d7-4655-9e85-dc27bc209f44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bca3de8-8q2pm\" (UID: \"16e3083b-94d7-4655-9e85-dc27bc209f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bca3de8-8q2pm" Nov 03 20:01:07.730249 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:07.730119 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plhb9\" (UniqueName: \"kubernetes.io/projected/16e3083b-94d7-4655-9e85-dc27bc209f44-kube-api-access-plhb9\") pod \"maas-api-revoke-sub-6bca3de8-8q2pm\" (UID: \"16e3083b-94d7-4655-9e85-dc27bc209f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bca3de8-8q2pm" Nov 03 20:01:07.732085 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:07.732052 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e3083b-94d7-4655-9e85-dc27bc209f44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bca3de8-8q2pm\" (UID: \"16e3083b-94d7-4655-9e85-dc27bc209f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bca3de8-8q2pm" Nov 03 20:01:07.734535 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:07.734509 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e3083b-94d7-4655-9e85-dc27bc209f44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bca3de8-8q2pm\" (UID: \"16e3083b-94d7-4655-9e85-dc27bc209f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bca3de8-8q2pm" Nov 03 20:01:07.744775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:07.744697 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plhb9\" (UniqueName: \"kubernetes.io/projected/16e3083b-94d7-4655-9e85-dc27bc209f44-kube-api-access-plhb9\") pod \"maas-api-revoke-sub-6bca3de8-8q2pm\" (UID: \"16e3083b-94d7-4655-9e85-dc27bc209f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bca3de8-8q2pm" Nov 03 20:01:07.794156 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:07.794124 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2"] Nov 03 20:01:07.794437 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:07.794396 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2" podUID="539fad2d-5ad7-4007-8b7f-94c325a943db" containerName="payload-processing" containerID="cri-o://5372c3fa517a24caedc3959e64453282ab3ba4fbe715df9d40382c0ee3686dae" gracePeriod=30 Nov 03 20:01:07.809877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:07.809846 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bca3de8-8q2pm" Nov 03 20:01:08.162786 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:01:08.161744 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e3083b_94d7_4655_9e85_dc27bc209f44.slice/crio-d981c8a419f0c4aad9bb7aa57b088b3d1b7ae9568ba14ca6b704b10d03e3a55d WatchSource:0}: Error finding container d981c8a419f0c4aad9bb7aa57b088b3d1b7ae9568ba14ca6b704b10d03e3a55d: Status 404 returned error can't find the container with id d981c8a419f0c4aad9bb7aa57b088b3d1b7ae9568ba14ca6b704b10d03e3a55d Nov 03 20:01:08.172967 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:08.170933 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bca3de8-8q2pm"] Nov 03 20:01:08.212328 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:08.212288 5059 generic.go:356] "Generic (PLEG): container finished" podID="539fad2d-5ad7-4007-8b7f-94c325a943db" containerID="5372c3fa517a24caedc3959e64453282ab3ba4fbe715df9d40382c0ee3686dae" exitCode=0 Nov 03 20:01:08.212492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:08.212422 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2" event={"ID":"539fad2d-5ad7-4007-8b7f-94c325a943db","Type":"ContainerDied","Data":"5372c3fa517a24caedc3959e64453282ab3ba4fbe715df9d40382c0ee3686dae"} Nov 03 20:01:08.214404 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:08.214368 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bca3de8-8q2pm" event={"ID":"16e3083b-94d7-4655-9e85-dc27bc209f44","Type":"ContainerStarted","Data":"d981c8a419f0c4aad9bb7aa57b088b3d1b7ae9568ba14ca6b704b10d03e3a55d"} Nov 03 20:01:08.380323 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:08.380293 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2" Nov 03 20:01:08.460035 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:08.459951 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/539fad2d-5ad7-4007-8b7f-94c325a943db-kube-api-access-sj2kk\" (UniqueName: \"kubernetes.io/projected/539fad2d-5ad7-4007-8b7f-94c325a943db-kube-api-access-sj2kk\") pod \"539fad2d-5ad7-4007-8b7f-94c325a943db\" (UID: \"539fad2d-5ad7-4007-8b7f-94c325a943db\") " Nov 03 20:01:08.460301 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:08.460279 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/539fad2d-5ad7-4007-8b7f-94c325a943db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/539fad2d-5ad7-4007-8b7f-94c325a943db-plugins-config-volume\") pod \"539fad2d-5ad7-4007-8b7f-94c325a943db\" (UID: \"539fad2d-5ad7-4007-8b7f-94c325a943db\") " Nov 03 20:01:08.460845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:08.460816 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/539fad2d-5ad7-4007-8b7f-94c325a943db-plugins-config-volume" pod "539fad2d-5ad7-4007-8b7f-94c325a943db" (UID: "539fad2d-5ad7-4007-8b7f-94c325a943db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:01:08.477382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:08.477337 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/539fad2d-5ad7-4007-8b7f-94c325a943db-kube-api-access-sj2kk" pod "539fad2d-5ad7-4007-8b7f-94c325a943db" (UID: "539fad2d-5ad7-4007-8b7f-94c325a943db"). InnerVolumeSpecName "kube-api-access-sj2kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:08.562354 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:08.562319 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj2kk\" (UniqueName: \"kubernetes.io/projected/539fad2d-5ad7-4007-8b7f-94c325a943db-kube-api-access-sj2kk\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:08.562354 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:08.562350 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/539fad2d-5ad7-4007-8b7f-94c325a943db-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:09.223230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:09.223201 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2" Nov 03 20:01:09.223415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:09.223234 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2" event={"ID":"539fad2d-5ad7-4007-8b7f-94c325a943db","Type":"ContainerDied","Data":"ed33dcd767378db33e642cb1ed6ec23e81ebd964e1fbe02a07793e13b53aa59d"} Nov 03 20:01:09.223415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:09.223300 5059 scope.go:122] "RemoveContainer" containerID="5372c3fa517a24caedc3959e64453282ab3ba4fbe715df9d40382c0ee3686dae" Nov 03 20:01:09.235919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:09.235881 5059 generic.go:356] "Generic (PLEG): container finished" podID="16e3083b-94d7-4655-9e85-dc27bc209f44" containerID="a2b48b8f2850952268bc5107611ef5320329e8cc68de71e1aa5ec8b933039f0d" exitCode=0 Nov 03 20:01:09.236039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:09.236002 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bca3de8-8q2pm" event={"ID":"16e3083b-94d7-4655-9e85-dc27bc209f44","Type":"ContainerDied","Data":"a2b48b8f2850952268bc5107611ef5320329e8cc68de71e1aa5ec8b933039f0d"} Nov 03 20:01:09.270876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:09.270846 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2"] Nov 03 20:01:09.275243 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:09.275219 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6092e-548467b6bd-7gkz2"] Nov 03 20:01:09.863160 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:09.863110 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="539fad2d-5ad7-4007-8b7f-94c325a943db" path="/var/lib/kubelet/pods/539fad2d-5ad7-4007-8b7f-94c325a943db/volumes" Nov 03 20:01:10.419694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:10.419653 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bca3de8-8q2pm" Nov 03 20:01:10.506388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:10.506314 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e3083b-94d7-4655-9e85-dc27bc209f44-kube-api-access-plhb9\" (UniqueName: \"kubernetes.io/projected/16e3083b-94d7-4655-9e85-dc27bc209f44-kube-api-access-plhb9\") pod \"16e3083b-94d7-4655-9e85-dc27bc209f44\" (UID: \"16e3083b-94d7-4655-9e85-dc27bc209f44\") " Nov 03 20:01:10.506539 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:10.506422 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e3083b-94d7-4655-9e85-dc27bc209f44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e3083b-94d7-4655-9e85-dc27bc209f44-maas-api-openshift-ca\") pod \"16e3083b-94d7-4655-9e85-dc27bc209f44\" (UID: \"16e3083b-94d7-4655-9e85-dc27bc209f44\") " Nov 03 20:01:10.506539 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:10.506488 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16e3083b-94d7-4655-9e85-dc27bc209f44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e3083b-94d7-4655-9e85-dc27bc209f44-maas-api-xks-ca\") pod \"16e3083b-94d7-4655-9e85-dc27bc209f44\" (UID: \"16e3083b-94d7-4655-9e85-dc27bc209f44\") " Nov 03 20:01:10.506850 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:10.506826 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e3083b-94d7-4655-9e85-dc27bc209f44-maas-api-openshift-ca" pod "16e3083b-94d7-4655-9e85-dc27bc209f44" (UID: "16e3083b-94d7-4655-9e85-dc27bc209f44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:01:10.507062 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:10.507039 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e3083b-94d7-4655-9e85-dc27bc209f44-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:10.508597 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:10.508571 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e3083b-94d7-4655-9e85-dc27bc209f44-maas-api-xks-ca" pod "16e3083b-94d7-4655-9e85-dc27bc209f44" (UID: "16e3083b-94d7-4655-9e85-dc27bc209f44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:01:10.508755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:10.508705 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e3083b-94d7-4655-9e85-dc27bc209f44-kube-api-access-plhb9" pod "16e3083b-94d7-4655-9e85-dc27bc209f44" (UID: "16e3083b-94d7-4655-9e85-dc27bc209f44"). InnerVolumeSpecName "kube-api-access-plhb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:10.608506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:10.608465 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plhb9\" (UniqueName: \"kubernetes.io/projected/16e3083b-94d7-4655-9e85-dc27bc209f44-kube-api-access-plhb9\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:10.608506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:10.608509 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e3083b-94d7-4655-9e85-dc27bc209f44-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:10.613383 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:01:10.613347 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:01:11.253611 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:11.253577 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bca3de8-8q2pm" event={"ID":"16e3083b-94d7-4655-9e85-dc27bc209f44","Type":"ContainerDied","Data":"d981c8a419f0c4aad9bb7aa57b088b3d1b7ae9568ba14ca6b704b10d03e3a55d"} Nov 03 20:01:11.253611 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:11.253597 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bca3de8-8q2pm" Nov 03 20:01:11.254190 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:11.253634 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d981c8a419f0c4aad9bb7aa57b088b3d1b7ae9568ba14ca6b704b10d03e3a55d" Nov 03 20:01:14.095976 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.095945 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55629980-6qsbv"] Nov 03 20:01:14.096562 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.096548 5059 cpu_manager.go:414] "containerMap: removing container" podUID="539fad2d-5ad7-4007-8b7f-94c325a943db" containerName="payload-processing" Nov 03 20:01:14.096602 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.096564 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="539fad2d-5ad7-4007-8b7f-94c325a943db" containerName="payload-processing" Nov 03 20:01:14.096650 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.096610 5059 cpu_manager.go:414] "containerMap: removing container" podUID="16e3083b-94d7-4655-9e85-dc27bc209f44" containerName="revoke-keys" Nov 03 20:01:14.096650 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.096615 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e3083b-94d7-4655-9e85-dc27bc209f44" containerName="revoke-keys" Nov 03 20:01:14.096748 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.096711 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e3083b-94d7-4655-9e85-dc27bc209f44" containerName="revoke-keys" Nov 03 20:01:14.096748 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.096721 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="539fad2d-5ad7-4007-8b7f-94c325a943db" containerName="payload-processing" Nov 03 20:01:14.103170 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.103134 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55629980-6qsbv" Nov 03 20:01:14.110216 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.110192 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:01:14.110360 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.110220 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:01:14.114584 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.114561 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55629980-6qsbv"] Nov 03 20:01:14.153894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.153866 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70630500-5f32-4c09-8bd1-01b8e5835671-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55629980-6qsbv\" (UID: \"70630500-5f32-4c09-8bd1-01b8e5835671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55629980-6qsbv" Nov 03 20:01:14.154014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.154001 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70630500-5f32-4c09-8bd1-01b8e5835671-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55629980-6qsbv\" (UID: \"70630500-5f32-4c09-8bd1-01b8e5835671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55629980-6qsbv" Nov 03 20:01:14.154180 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.154155 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkzph\" (UniqueName: \"kubernetes.io/projected/70630500-5f32-4c09-8bd1-01b8e5835671-kube-api-access-mkzph\") pod \"maas-api-revoke-sub-55629980-6qsbv\" (UID: \"70630500-5f32-4c09-8bd1-01b8e5835671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55629980-6qsbv" Nov 03 20:01:14.255567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.255537 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70630500-5f32-4c09-8bd1-01b8e5835671-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55629980-6qsbv\" (UID: \"70630500-5f32-4c09-8bd1-01b8e5835671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55629980-6qsbv" Nov 03 20:01:14.255716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.255645 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70630500-5f32-4c09-8bd1-01b8e5835671-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55629980-6qsbv\" (UID: \"70630500-5f32-4c09-8bd1-01b8e5835671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55629980-6qsbv" Nov 03 20:01:14.255808 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.255790 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkzph\" (UniqueName: \"kubernetes.io/projected/70630500-5f32-4c09-8bd1-01b8e5835671-kube-api-access-mkzph\") pod \"maas-api-revoke-sub-55629980-6qsbv\" (UID: \"70630500-5f32-4c09-8bd1-01b8e5835671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55629980-6qsbv" Nov 03 20:01:14.256289 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.256269 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70630500-5f32-4c09-8bd1-01b8e5835671-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55629980-6qsbv\" (UID: \"70630500-5f32-4c09-8bd1-01b8e5835671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55629980-6qsbv" Nov 03 20:01:14.257960 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.257941 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70630500-5f32-4c09-8bd1-01b8e5835671-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55629980-6qsbv\" (UID: \"70630500-5f32-4c09-8bd1-01b8e5835671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55629980-6qsbv" Nov 03 20:01:14.268109 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.268088 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkzph\" (UniqueName: \"kubernetes.io/projected/70630500-5f32-4c09-8bd1-01b8e5835671-kube-api-access-mkzph\") pod \"maas-api-revoke-sub-55629980-6qsbv\" (UID: \"70630500-5f32-4c09-8bd1-01b8e5835671\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55629980-6qsbv" Nov 03 20:01:14.422890 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.422815 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55629980-6qsbv" Nov 03 20:01:14.719015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.718968 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55629980-6qsbv"] Nov 03 20:01:14.719151 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:01:14.719041 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70630500_5f32_4c09_8bd1_01b8e5835671.slice/crio-6577b4fa8438da4c98982f3329ade36b0cbeef6828e282655e156127f80ee3ff WatchSource:0}: Error finding container 6577b4fa8438da4c98982f3329ade36b0cbeef6828e282655e156127f80ee3ff: Status 404 returned error can't find the container with id 6577b4fa8438da4c98982f3329ade36b0cbeef6828e282655e156127f80ee3ff Nov 03 20:01:14.884468 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.884440 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e9e11b-dvsxb"] Nov 03 20:01:14.890074 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.890053 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e9e11b-dvsxb" Nov 03 20:01:14.902749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.902726 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e9e11b-dvsxb"] Nov 03 20:01:14.966839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.966779 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de62ad5-f378-4eba-9a18-53f085e6c5c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91e9e11b-dvsxb\" (UID: \"5de62ad5-f378-4eba-9a18-53f085e6c5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e9e11b-dvsxb" Nov 03 20:01:14.967037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.966919 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de62ad5-f378-4eba-9a18-53f085e6c5c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91e9e11b-dvsxb\" (UID: \"5de62ad5-f378-4eba-9a18-53f085e6c5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e9e11b-dvsxb" Nov 03 20:01:14.967037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:14.966967 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsdk4\" (UniqueName: \"kubernetes.io/projected/5de62ad5-f378-4eba-9a18-53f085e6c5c1-kube-api-access-nsdk4\") pod \"maas-api-revoke-sub-91e9e11b-dvsxb\" (UID: \"5de62ad5-f378-4eba-9a18-53f085e6c5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e9e11b-dvsxb" Nov 03 20:01:15.067985 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:15.067959 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de62ad5-f378-4eba-9a18-53f085e6c5c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91e9e11b-dvsxb\" (UID: \"5de62ad5-f378-4eba-9a18-53f085e6c5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e9e11b-dvsxb" Nov 03 20:01:15.068111 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:15.068077 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de62ad5-f378-4eba-9a18-53f085e6c5c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91e9e11b-dvsxb\" (UID: \"5de62ad5-f378-4eba-9a18-53f085e6c5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e9e11b-dvsxb" Nov 03 20:01:15.068164 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:15.068111 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsdk4\" (UniqueName: \"kubernetes.io/projected/5de62ad5-f378-4eba-9a18-53f085e6c5c1-kube-api-access-nsdk4\") pod \"maas-api-revoke-sub-91e9e11b-dvsxb\" (UID: \"5de62ad5-f378-4eba-9a18-53f085e6c5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e9e11b-dvsxb" Nov 03 20:01:15.068773 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:15.068751 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de62ad5-f378-4eba-9a18-53f085e6c5c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91e9e11b-dvsxb\" (UID: \"5de62ad5-f378-4eba-9a18-53f085e6c5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e9e11b-dvsxb" Nov 03 20:01:15.070243 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:15.070218 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de62ad5-f378-4eba-9a18-53f085e6c5c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91e9e11b-dvsxb\" (UID: \"5de62ad5-f378-4eba-9a18-53f085e6c5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e9e11b-dvsxb" Nov 03 20:01:15.080876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:15.080852 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsdk4\" (UniqueName: \"kubernetes.io/projected/5de62ad5-f378-4eba-9a18-53f085e6c5c1-kube-api-access-nsdk4\") pod \"maas-api-revoke-sub-91e9e11b-dvsxb\" (UID: \"5de62ad5-f378-4eba-9a18-53f085e6c5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e9e11b-dvsxb" Nov 03 20:01:15.205232 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:15.205207 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e9e11b-dvsxb" Nov 03 20:01:15.327910 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:15.327715 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55629980-6qsbv" event={"ID":"70630500-5f32-4c09-8bd1-01b8e5835671","Type":"ContainerStarted","Data":"a180a1113880b44329da78f6d64a3b3c2de991dfd69ae30fe205df26c1a938f5"} Nov 03 20:01:15.327910 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:15.327818 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55629980-6qsbv" event={"ID":"70630500-5f32-4c09-8bd1-01b8e5835671","Type":"ContainerStarted","Data":"6577b4fa8438da4c98982f3329ade36b0cbeef6828e282655e156127f80ee3ff"} Nov 03 20:01:15.349656 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:15.349606 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55629980-6qsbv" podStartSLOduration=1.349590246 podStartE2EDuration="1.349590246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:15.345916993 +0000 UTC m=+1469.603714251" watchObservedRunningTime="2026-11-03 20:01:15.349590246 +0000 UTC m=+1469.607387502" Nov 03 20:01:15.494322 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:01:15.494291 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de62ad5_f378_4eba_9a18_53f085e6c5c1.slice/crio-0d8a302dd522999c456f21e0ee41c336afc3e915b512c4cca0de5776fbc791b6 WatchSource:0}: Error finding container 0d8a302dd522999c456f21e0ee41c336afc3e915b512c4cca0de5776fbc791b6: Status 404 returned error can't find the container with id 0d8a302dd522999c456f21e0ee41c336afc3e915b512c4cca0de5776fbc791b6 Nov 03 20:01:15.494599 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:15.494576 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e9e11b-dvsxb"] Nov 03 20:01:16.337830 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:16.337794 5059 generic.go:356] "Generic (PLEG): container finished" podID="70630500-5f32-4c09-8bd1-01b8e5835671" containerID="a180a1113880b44329da78f6d64a3b3c2de991dfd69ae30fe205df26c1a938f5" exitCode=0 Nov 03 20:01:16.338240 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:16.337881 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55629980-6qsbv" event={"ID":"70630500-5f32-4c09-8bd1-01b8e5835671","Type":"ContainerDied","Data":"a180a1113880b44329da78f6d64a3b3c2de991dfd69ae30fe205df26c1a938f5"} Nov 03 20:01:16.339526 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:16.339503 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e9e11b-dvsxb" event={"ID":"5de62ad5-f378-4eba-9a18-53f085e6c5c1","Type":"ContainerStarted","Data":"97c073b05ba2c3446c585c4d7017383f339c42f9af2a0154eead946d75919100"} Nov 03 20:01:16.339599 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:16.339543 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e9e11b-dvsxb" event={"ID":"5de62ad5-f378-4eba-9a18-53f085e6c5c1","Type":"ContainerStarted","Data":"0d8a302dd522999c456f21e0ee41c336afc3e915b512c4cca0de5776fbc791b6"} Nov 03 20:01:17.349324 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:17.349291 5059 generic.go:356] "Generic (PLEG): container finished" podID="5de62ad5-f378-4eba-9a18-53f085e6c5c1" containerID="97c073b05ba2c3446c585c4d7017383f339c42f9af2a0154eead946d75919100" exitCode=0 Nov 03 20:01:17.349725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:17.349381 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e9e11b-dvsxb" event={"ID":"5de62ad5-f378-4eba-9a18-53f085e6c5c1","Type":"ContainerDied","Data":"97c073b05ba2c3446c585c4d7017383f339c42f9af2a0154eead946d75919100"} Nov 03 20:01:17.522204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:17.522175 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55629980-6qsbv" Nov 03 20:01:17.606917 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:17.606836 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70630500-5f32-4c09-8bd1-01b8e5835671-kube-api-access-mkzph\" (UniqueName: \"kubernetes.io/projected/70630500-5f32-4c09-8bd1-01b8e5835671-kube-api-access-mkzph\") pod \"70630500-5f32-4c09-8bd1-01b8e5835671\" (UID: \"70630500-5f32-4c09-8bd1-01b8e5835671\") " Nov 03 20:01:17.607083 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:17.606916 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70630500-5f32-4c09-8bd1-01b8e5835671-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70630500-5f32-4c09-8bd1-01b8e5835671-maas-api-xks-ca\") pod \"70630500-5f32-4c09-8bd1-01b8e5835671\" (UID: \"70630500-5f32-4c09-8bd1-01b8e5835671\") " Nov 03 20:01:17.607083 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:17.606991 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70630500-5f32-4c09-8bd1-01b8e5835671-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70630500-5f32-4c09-8bd1-01b8e5835671-maas-api-openshift-ca\") pod \"70630500-5f32-4c09-8bd1-01b8e5835671\" (UID: \"70630500-5f32-4c09-8bd1-01b8e5835671\") " Nov 03 20:01:17.607762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:17.607710 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70630500-5f32-4c09-8bd1-01b8e5835671-maas-api-openshift-ca" pod "70630500-5f32-4c09-8bd1-01b8e5835671" (UID: "70630500-5f32-4c09-8bd1-01b8e5835671"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:01:17.609194 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:17.609167 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70630500-5f32-4c09-8bd1-01b8e5835671-maas-api-xks-ca" pod "70630500-5f32-4c09-8bd1-01b8e5835671" (UID: "70630500-5f32-4c09-8bd1-01b8e5835671"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:01:17.609335 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:17.609302 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70630500-5f32-4c09-8bd1-01b8e5835671-kube-api-access-mkzph" pod "70630500-5f32-4c09-8bd1-01b8e5835671" (UID: "70630500-5f32-4c09-8bd1-01b8e5835671"). InnerVolumeSpecName "kube-api-access-mkzph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:17.708541 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:17.708514 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkzph\" (UniqueName: \"kubernetes.io/projected/70630500-5f32-4c09-8bd1-01b8e5835671-kube-api-access-mkzph\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:17.708698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:17.708541 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70630500-5f32-4c09-8bd1-01b8e5835671-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:17.708698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:17.708562 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70630500-5f32-4c09-8bd1-01b8e5835671-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:18.357997 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:18.357968 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55629980-6qsbv" Nov 03 20:01:18.358399 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:18.358003 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55629980-6qsbv" event={"ID":"70630500-5f32-4c09-8bd1-01b8e5835671","Type":"ContainerDied","Data":"6577b4fa8438da4c98982f3329ade36b0cbeef6828e282655e156127f80ee3ff"} Nov 03 20:01:18.358399 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:18.358037 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6577b4fa8438da4c98982f3329ade36b0cbeef6828e282655e156127f80ee3ff" Nov 03 20:01:18.528609 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:18.528584 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e9e11b-dvsxb" Nov 03 20:01:18.621682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:18.620580 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de62ad5-f378-4eba-9a18-53f085e6c5c1-kube-api-access-nsdk4\" (UniqueName: \"kubernetes.io/projected/5de62ad5-f378-4eba-9a18-53f085e6c5c1-kube-api-access-nsdk4\") pod \"5de62ad5-f378-4eba-9a18-53f085e6c5c1\" (UID: \"5de62ad5-f378-4eba-9a18-53f085e6c5c1\") " Nov 03 20:01:18.621682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:18.620776 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5de62ad5-f378-4eba-9a18-53f085e6c5c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de62ad5-f378-4eba-9a18-53f085e6c5c1-maas-api-xks-ca\") pod \"5de62ad5-f378-4eba-9a18-53f085e6c5c1\" (UID: \"5de62ad5-f378-4eba-9a18-53f085e6c5c1\") " Nov 03 20:01:18.621682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:18.620911 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5de62ad5-f378-4eba-9a18-53f085e6c5c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de62ad5-f378-4eba-9a18-53f085e6c5c1-maas-api-openshift-ca\") pod \"5de62ad5-f378-4eba-9a18-53f085e6c5c1\" (UID: \"5de62ad5-f378-4eba-9a18-53f085e6c5c1\") " Nov 03 20:01:18.623959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:18.623096 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5de62ad5-f378-4eba-9a18-53f085e6c5c1-maas-api-openshift-ca" pod "5de62ad5-f378-4eba-9a18-53f085e6c5c1" (UID: "5de62ad5-f378-4eba-9a18-53f085e6c5c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:01:18.626219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:18.626159 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5de62ad5-f378-4eba-9a18-53f085e6c5c1-maas-api-xks-ca" pod "5de62ad5-f378-4eba-9a18-53f085e6c5c1" (UID: "5de62ad5-f378-4eba-9a18-53f085e6c5c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:01:18.628952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:18.628929 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de62ad5-f378-4eba-9a18-53f085e6c5c1-kube-api-access-nsdk4" pod "5de62ad5-f378-4eba-9a18-53f085e6c5c1" (UID: "5de62ad5-f378-4eba-9a18-53f085e6c5c1"). InnerVolumeSpecName "kube-api-access-nsdk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:18.723312 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:18.723280 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de62ad5-f378-4eba-9a18-53f085e6c5c1-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:18.723312 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:18.723314 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de62ad5-f378-4eba-9a18-53f085e6c5c1-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:18.723504 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:18.723327 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsdk4\" (UniqueName: \"kubernetes.io/projected/5de62ad5-f378-4eba-9a18-53f085e6c5c1-kube-api-access-nsdk4\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:19.370827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:19.370794 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e9e11b-dvsxb" Nov 03 20:01:19.371221 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:19.370817 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e9e11b-dvsxb" event={"ID":"5de62ad5-f378-4eba-9a18-53f085e6c5c1","Type":"ContainerDied","Data":"0d8a302dd522999c456f21e0ee41c336afc3e915b512c4cca0de5776fbc791b6"} Nov 03 20:01:19.371221 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:19.370876 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d8a302dd522999c456f21e0ee41c336afc3e915b512c4cca0de5776fbc791b6" Nov 03 20:01:20.606993 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.606959 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc"] Nov 03 20:01:20.607777 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.607756 5059 cpu_manager.go:414] "containerMap: removing container" podUID="5de62ad5-f378-4eba-9a18-53f085e6c5c1" containerName="revoke-keys" Nov 03 20:01:20.607887 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.607779 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5de62ad5-f378-4eba-9a18-53f085e6c5c1" containerName="revoke-keys" Nov 03 20:01:20.607887 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.607816 5059 cpu_manager.go:414] "containerMap: removing container" podUID="70630500-5f32-4c09-8bd1-01b8e5835671" containerName="revoke-keys" Nov 03 20:01:20.607887 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.607824 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70630500-5f32-4c09-8bd1-01b8e5835671" containerName="revoke-keys" Nov 03 20:01:20.608034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.607971 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="70630500-5f32-4c09-8bd1-01b8e5835671" containerName="revoke-keys" Nov 03 20:01:20.608034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.607998 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5de62ad5-f378-4eba-9a18-53f085e6c5c1" containerName="revoke-keys" Nov 03 20:01:20.617129 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.617106 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.626274 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.626244 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc"] Nov 03 20:01:20.753944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.753909 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.753944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.753946 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e91375b-123b-4c6b-b589-d1a6716be6df-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.754183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.753981 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.754183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.754085 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.754183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.754148 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.754353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.754197 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69k5k\" (UniqueName: \"kubernetes.io/projected/4e91375b-123b-4c6b-b589-d1a6716be6df-kube-api-access-69k5k\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.754416 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.754346 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.754476 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.754443 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.754562 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.754546 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.856038 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.856005 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.856208 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.856074 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.856208 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.856102 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e91375b-123b-4c6b-b589-d1a6716be6df-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.856208 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.856148 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.856349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.856260 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.856349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.856335 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.856450 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.856396 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69k5k\" (UniqueName: \"kubernetes.io/projected/4e91375b-123b-4c6b-b589-d1a6716be6df-kube-api-access-69k5k\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.856512 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.856500 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.856596 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.856564 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.856685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.856572 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.856685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.856499 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.856933 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.856911 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.857066 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.857009 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e91375b-123b-4c6b-b589-d1a6716be6df-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.857066 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.857013 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.858756 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.858736 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.859117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.859099 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.911101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.911032 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.931696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.924422 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69k5k\" (UniqueName: \"kubernetes.io/projected/4e91375b-123b-4c6b-b589-d1a6716be6df-kube-api-access-69k5k\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:20.947316 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:20.946989 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:21.330440 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:01:21.330409 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e91375b_123b_4c6b_b589_d1a6716be6df.slice/crio-ca17ae84cdac2bb8ef9c9278d7a1ba99c554b1bf7d4c7ac5d92af1a7bc5d3ec4 WatchSource:0}: Error finding container ca17ae84cdac2bb8ef9c9278d7a1ba99c554b1bf7d4c7ac5d92af1a7bc5d3ec4: Status 404 returned error can't find the container with id ca17ae84cdac2bb8ef9c9278d7a1ba99c554b1bf7d4c7ac5d92af1a7bc5d3ec4 Nov 03 20:01:21.330702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:21.330676 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc"] Nov 03 20:01:21.333182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:21.333148 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:01:21.333289 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:21.333224 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:01:21.333289 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:21.333272 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:01:21.386470 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:21.386444 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" event={"ID":"4e91375b-123b-4c6b-b589-d1a6716be6df","Type":"ContainerStarted","Data":"ca17ae84cdac2bb8ef9c9278d7a1ba99c554b1bf7d4c7ac5d92af1a7bc5d3ec4"} Nov 03 20:01:22.347950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:22.347914 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e5e5ae4-phxs9"] Nov 03 20:01:22.353419 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:22.353390 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5e5ae4-phxs9" Nov 03 20:01:22.353597 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:22.353573 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e5e5ae4-phxs9"] Nov 03 20:01:22.384161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:22.383846 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:01:22.384161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:22.384126 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:01:22.406875 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:22.406841 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" event={"ID":"4e91375b-123b-4c6b-b589-d1a6716be6df","Type":"ContainerStarted","Data":"883d52d005f1c301ec66a2cc37ef13c20cf63f3a44d00d881c47f3045f54d03c"} Nov 03 20:01:22.429963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:22.429909 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" podStartSLOduration=2.429891853 podStartE2EDuration="2.429891853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:22.429049978 +0000 UTC m=+1476.686847236" watchObservedRunningTime="2026-11-03 20:01:22.429891853 +0000 UTC m=+1476.687689110" Nov 03 20:01:22.503790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:22.503753 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbpfk\" (UniqueName: \"kubernetes.io/projected/cee28a4b-6a4e-43f3-9905-eaedd2068dad-kube-api-access-qbpfk\") pod \"maas-api-revoke-sub-2e5e5ae4-phxs9\" (UID: \"cee28a4b-6a4e-43f3-9905-eaedd2068dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5e5ae4-phxs9" Nov 03 20:01:22.504005 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:22.503828 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee28a4b-6a4e-43f3-9905-eaedd2068dad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e5e5ae4-phxs9\" (UID: \"cee28a4b-6a4e-43f3-9905-eaedd2068dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5e5ae4-phxs9" Nov 03 20:01:22.504183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:22.504162 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee28a4b-6a4e-43f3-9905-eaedd2068dad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e5e5ae4-phxs9\" (UID: \"cee28a4b-6a4e-43f3-9905-eaedd2068dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5e5ae4-phxs9" Nov 03 20:01:22.605542 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:22.605447 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee28a4b-6a4e-43f3-9905-eaedd2068dad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e5e5ae4-phxs9\" (UID: \"cee28a4b-6a4e-43f3-9905-eaedd2068dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5e5ae4-phxs9" Nov 03 20:01:22.605735 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:22.605604 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbpfk\" (UniqueName: \"kubernetes.io/projected/cee28a4b-6a4e-43f3-9905-eaedd2068dad-kube-api-access-qbpfk\") pod \"maas-api-revoke-sub-2e5e5ae4-phxs9\" (UID: \"cee28a4b-6a4e-43f3-9905-eaedd2068dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5e5ae4-phxs9" Nov 03 20:01:22.605735 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:22.605707 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee28a4b-6a4e-43f3-9905-eaedd2068dad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e5e5ae4-phxs9\" (UID: \"cee28a4b-6a4e-43f3-9905-eaedd2068dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5e5ae4-phxs9" Nov 03 20:01:22.606268 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:22.606245 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee28a4b-6a4e-43f3-9905-eaedd2068dad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e5e5ae4-phxs9\" (UID: \"cee28a4b-6a4e-43f3-9905-eaedd2068dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5e5ae4-phxs9" Nov 03 20:01:22.608605 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:22.608582 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee28a4b-6a4e-43f3-9905-eaedd2068dad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e5e5ae4-phxs9\" (UID: \"cee28a4b-6a4e-43f3-9905-eaedd2068dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5e5ae4-phxs9" Nov 03 20:01:22.618591 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:22.618568 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbpfk\" (UniqueName: \"kubernetes.io/projected/cee28a4b-6a4e-43f3-9905-eaedd2068dad-kube-api-access-qbpfk\") pod \"maas-api-revoke-sub-2e5e5ae4-phxs9\" (UID: \"cee28a4b-6a4e-43f3-9905-eaedd2068dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5e5ae4-phxs9" Nov 03 20:01:22.705603 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:22.705571 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5e5ae4-phxs9" Nov 03 20:01:22.947855 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:22.947764 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:22.949228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:22.949206 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 03 20:01:22.949303 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:22.949253 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" podUID="4e91375b-123b-4c6b-b589-d1a6716be6df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 03 20:01:23.030020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:23.029245 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e5e5ae4-phxs9"] Nov 03 20:01:23.345710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:23.345652 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz"] Nov 03 20:01:23.345937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:23.345910 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" podUID="8531b4f3-5179-4948-86ce-52e13576ea11" containerName="istio-proxy" containerID="cri-o://214a53ef694c00238d5e914252f452ff7a8677228ee8c69a81ba92d007afde19" gracePeriod=30 Nov 03 20:01:23.415615 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:23.415580 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5e5ae4-phxs9" event={"ID":"cee28a4b-6a4e-43f3-9905-eaedd2068dad","Type":"ContainerStarted","Data":"eaffe3ca02cd5dfc60b0d3f1d2d8cdb28357ad77766ff36a5012c46cc65492bc"} Nov 03 20:01:23.952988 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:23.952135 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 03 20:01:23.952988 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:23.952212 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" podUID="4e91375b-123b-4c6b-b589-d1a6716be6df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 03 20:01:24.424464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:24.424422 5059 generic.go:356] "Generic (PLEG): container finished" podID="cee28a4b-6a4e-43f3-9905-eaedd2068dad" containerID="d99b7e96ffe84d1e65f27f9474ec64f38ac1a3fa15074b34cda21681c7d56b1d" exitCode=0 Nov 03 20:01:24.425054 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:24.424508 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5e5ae4-phxs9" event={"ID":"cee28a4b-6a4e-43f3-9905-eaedd2068dad","Type":"ContainerDied","Data":"d99b7e96ffe84d1e65f27f9474ec64f38ac1a3fa15074b34cda21681c7d56b1d"} Nov 03 20:01:24.948697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:24.948647 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 03 20:01:24.948915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:24.948734 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" podUID="4e91375b-123b-4c6b-b589-d1a6716be6df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 03 20:01:24.984907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:24.984870 5059 patch_prober.go:29] interesting pod/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:01:24.985064 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:24.984930 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" podUID="8531b4f3-5179-4948-86ce-52e13576ea11" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:01:25.102849 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:25.102790 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e75eb6-g4rv5"] Nov 03 20:01:25.108888 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:25.108861 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e75eb6-g4rv5" Nov 03 20:01:25.218786 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:25.182786 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szbgj\" (UniqueName: \"kubernetes.io/projected/3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9-kube-api-access-szbgj\") pod \"maas-api-revoke-sub-65e75eb6-g4rv5\" (UID: \"3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e75eb6-g4rv5" Nov 03 20:01:25.218786 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:25.182941 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65e75eb6-g4rv5\" (UID: \"3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e75eb6-g4rv5" Nov 03 20:01:25.218786 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:25.182974 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65e75eb6-g4rv5\" (UID: \"3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e75eb6-g4rv5" Nov 03 20:01:25.300686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:25.294874 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szbgj\" (UniqueName: \"kubernetes.io/projected/3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9-kube-api-access-szbgj\") pod \"maas-api-revoke-sub-65e75eb6-g4rv5\" (UID: \"3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e75eb6-g4rv5" Nov 03 20:01:25.300686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:25.295068 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65e75eb6-g4rv5\" (UID: \"3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e75eb6-g4rv5" Nov 03 20:01:25.300686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:25.295107 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65e75eb6-g4rv5\" (UID: \"3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e75eb6-g4rv5" Nov 03 20:01:25.300686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:25.298470 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65e75eb6-g4rv5\" (UID: \"3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e75eb6-g4rv5" Nov 03 20:01:25.352727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:25.336011 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e75eb6-g4rv5"] Nov 03 20:01:25.357358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:25.357321 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65e75eb6-g4rv5\" (UID: \"3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e75eb6-g4rv5" Nov 03 20:01:25.441693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:25.419422 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szbgj\" (UniqueName: \"kubernetes.io/projected/3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9-kube-api-access-szbgj\") pod \"maas-api-revoke-sub-65e75eb6-g4rv5\" (UID: \"3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e75eb6-g4rv5" Nov 03 20:01:25.641485 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:25.623905 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e75eb6-g4rv5" Nov 03 20:01:25.982308 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:25.982224 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 03 20:01:25.982544 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:25.982512 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" podUID="4e91375b-123b-4c6b-b589-d1a6716be6df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 03 20:01:26.630899 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:26.613140 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:26.680610 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:26.678707 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:26.783271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:26.783227 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:01:26.836324 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:26.836228 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5e5ae4-phxs9" event={"ID":"cee28a4b-6a4e-43f3-9905-eaedd2068dad","Type":"ContainerDied","Data":"eaffe3ca02cd5dfc60b0d3f1d2d8cdb28357ad77766ff36a5012c46cc65492bc"} Nov 03 20:01:26.836324 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:26.836294 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eaffe3ca02cd5dfc60b0d3f1d2d8cdb28357ad77766ff36a5012c46cc65492bc" Nov 03 20:01:27.001798 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:26.975083 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 03 20:01:27.001798 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:26.995797 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 03 20:01:27.001798 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:26.995864 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" podUID="4e91375b-123b-4c6b-b589-d1a6716be6df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 03 20:01:27.002560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:27.002268 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5e5ae4-phxs9" Nov 03 20:01:27.114689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:27.105146 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee28a4b-6a4e-43f3-9905-eaedd2068dad-kube-api-access-qbpfk\" (UniqueName: \"kubernetes.io/projected/cee28a4b-6a4e-43f3-9905-eaedd2068dad-kube-api-access-qbpfk\") pod \"cee28a4b-6a4e-43f3-9905-eaedd2068dad\" (UID: \"cee28a4b-6a4e-43f3-9905-eaedd2068dad\") " Nov 03 20:01:27.114689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:27.105221 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee28a4b-6a4e-43f3-9905-eaedd2068dad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee28a4b-6a4e-43f3-9905-eaedd2068dad-maas-api-openshift-ca\") pod \"cee28a4b-6a4e-43f3-9905-eaedd2068dad\" (UID: \"cee28a4b-6a4e-43f3-9905-eaedd2068dad\") " Nov 03 20:01:27.114689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:27.105325 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cee28a4b-6a4e-43f3-9905-eaedd2068dad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee28a4b-6a4e-43f3-9905-eaedd2068dad-maas-api-xks-ca\") pod \"cee28a4b-6a4e-43f3-9905-eaedd2068dad\" (UID: \"cee28a4b-6a4e-43f3-9905-eaedd2068dad\") " Nov 03 20:01:27.252484 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:27.211238 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:27.252484 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:27.217146 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:27.252484 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:27.240856 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee28a4b-6a4e-43f3-9905-eaedd2068dad-maas-api-openshift-ca" pod "cee28a4b-6a4e-43f3-9905-eaedd2068dad" (UID: "cee28a4b-6a4e-43f3-9905-eaedd2068dad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:01:27.362972 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:27.348390 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee28a4b-6a4e-43f3-9905-eaedd2068dad-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:27.408684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:27.395868 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cee28a4b-6a4e-43f3-9905-eaedd2068dad-maas-api-xks-ca" pod "cee28a4b-6a4e-43f3-9905-eaedd2068dad" (UID: "cee28a4b-6a4e-43f3-9905-eaedd2068dad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:01:27.453252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:27.450147 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee28a4b-6a4e-43f3-9905-eaedd2068dad-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:27.636398 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:27.632192 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:27.636398 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:27.632906 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 20:01:27.686443 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:27.644501 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee28a4b-6a4e-43f3-9905-eaedd2068dad-kube-api-access-qbpfk" pod "cee28a4b-6a4e-43f3-9905-eaedd2068dad" (UID: "cee28a4b-6a4e-43f3-9905-eaedd2068dad"). InnerVolumeSpecName "kube-api-access-qbpfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:27.686443 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:27.664373 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbpfk\" (UniqueName: \"kubernetes.io/projected/cee28a4b-6a4e-43f3-9905-eaedd2068dad-kube-api-access-qbpfk\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:27.686443 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:27.685323 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:27.686443 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:27.685482 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 20:01:27.725640 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:27.725583 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e75eb6-g4rv5"] Nov 03 20:01:27.820745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:27.816765 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:01:27.845784 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:27.845736 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" podUID="8f5fe53c-7085-4890-99ad-0c4e9381a2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:27.881685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:27.846253 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:01:27.982697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:27.981986 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 03 20:01:28.270014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:28.232256 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:28.563819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:28.320171 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 03 20:01:28.563819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:28.563117 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" podUID="4e91375b-123b-4c6b-b589-d1a6716be6df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 03 20:01:28.588685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:28.586796 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:01:28.588685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:28.586860 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" podUID="474b2e1d-832b-4edd-8765-31e0bc77cc2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:28.639093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:28.639048 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:01:28.716471 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:28.716222 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:01:28.716471 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:28.716292 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" podUID="7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:28.734608 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:28.734524 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:29.154369 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:29.154328 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 03 20:01:29.154690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:29.154626 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" podUID="4e91375b-123b-4c6b-b589-d1a6716be6df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 03 20:01:29.350066 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:29.350015 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:29.506709 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:01:29.506678 5059 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.648s" Nov 03 20:01:29.507271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:29.507245 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 20:01:29.622644 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:29.622562 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:01:29.736177 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:29.697741 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e5e5ae4-phxs9" Nov 03 20:01:29.738479 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:29.736400 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 20:01:29.738479 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:29.737965 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 20:01:29.738479 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:29.738010 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 20:01:29.738479 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:29.738060 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 20:01:29.738479 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:29.738086 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e75eb6-g4rv5" event={"ID":"3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9","Type":"ContainerStarted","Data":"e2ba82c34644082651aec5024e3ce5c5ae2e09fa741ecc4c6bb6689e3be13fa0"} Nov 03 20:01:29.738479 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:29.738144 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 20:01:29.738479 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:29.738163 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:01:29.948422 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:29.948328 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 03 20:01:29.948422 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:29.948397 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" podUID="4e91375b-123b-4c6b-b589-d1a6716be6df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 03 20:01:29.994801 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:29.994776 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 20:01:30.161689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.157192 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-istio-envoy\") pod \"8531b4f3-5179-4948-86ce-52e13576ea11\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " Nov 03 20:01:30.161689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.157915 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-istio-data\") pod \"8531b4f3-5179-4948-86ce-52e13576ea11\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " Nov 03 20:01:30.161689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.157967 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-workload-certs\") pod \"8531b4f3-5179-4948-86ce-52e13576ea11\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " Nov 03 20:01:30.161689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.158114 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-credential-socket\") pod \"8531b4f3-5179-4948-86ce-52e13576ea11\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " Nov 03 20:01:30.161689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.158236 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8531b4f3-5179-4948-86ce-52e13576ea11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8531b4f3-5179-4948-86ce-52e13576ea11-istio-podinfo\") pod \"8531b4f3-5179-4948-86ce-52e13576ea11\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " Nov 03 20:01:30.161689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.158290 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8531b4f3-5179-4948-86ce-52e13576ea11-istio-token\" (UniqueName: \"kubernetes.io/projected/8531b4f3-5179-4948-86ce-52e13576ea11-istio-token\") pod \"8531b4f3-5179-4948-86ce-52e13576ea11\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " Nov 03 20:01:30.161689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.158365 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8531b4f3-5179-4948-86ce-52e13576ea11-kube-api-access-f4mfn\" (UniqueName: \"kubernetes.io/projected/8531b4f3-5179-4948-86ce-52e13576ea11-kube-api-access-f4mfn\") pod \"8531b4f3-5179-4948-86ce-52e13576ea11\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " Nov 03 20:01:30.161689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.158496 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-workload-socket\") pod \"8531b4f3-5179-4948-86ce-52e13576ea11\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " Nov 03 20:01:30.161689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.158526 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8531b4f3-5179-4948-86ce-52e13576ea11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8531b4f3-5179-4948-86ce-52e13576ea11-istiod-ca-cert\") pod \"8531b4f3-5179-4948-86ce-52e13576ea11\" (UID: \"8531b4f3-5179-4948-86ce-52e13576ea11\") " Nov 03 20:01:30.168534 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.166075 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-workload-certs" pod "8531b4f3-5179-4948-86ce-52e13576ea11" (UID: "8531b4f3-5179-4948-86ce-52e13576ea11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:30.168534 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.167997 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-credential-socket" pod "8531b4f3-5179-4948-86ce-52e13576ea11" (UID: "8531b4f3-5179-4948-86ce-52e13576ea11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:30.168534 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.168227 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-workload-socket" pod "8531b4f3-5179-4948-86ce-52e13576ea11" (UID: "8531b4f3-5179-4948-86ce-52e13576ea11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:30.172724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.170854 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8531b4f3-5179-4948-86ce-52e13576ea11-istiod-ca-cert" pod "8531b4f3-5179-4948-86ce-52e13576ea11" (UID: "8531b4f3-5179-4948-86ce-52e13576ea11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:01:30.176382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.176313 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-istio-data" pod "8531b4f3-5179-4948-86ce-52e13576ea11" (UID: "8531b4f3-5179-4948-86ce-52e13576ea11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:30.177714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.177685 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8531b4f3-5179-4948-86ce-52e13576ea11-kube-api-access-f4mfn" pod "8531b4f3-5179-4948-86ce-52e13576ea11" (UID: "8531b4f3-5179-4948-86ce-52e13576ea11"). InnerVolumeSpecName "kube-api-access-f4mfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:30.177981 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.177953 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8531b4f3-5179-4948-86ce-52e13576ea11-istio-token" pod "8531b4f3-5179-4948-86ce-52e13576ea11" (UID: "8531b4f3-5179-4948-86ce-52e13576ea11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:30.178152 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.178135 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8531b4f3-5179-4948-86ce-52e13576ea11-istio-podinfo" pod "8531b4f3-5179-4948-86ce-52e13576ea11" (UID: "8531b4f3-5179-4948-86ce-52e13576ea11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:01:30.180155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.180103 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-istio-envoy" pod "8531b4f3-5179-4948-86ce-52e13576ea11" (UID: "8531b4f3-5179-4948-86ce-52e13576ea11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:30.264647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.264545 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4mfn\" (UniqueName: \"kubernetes.io/projected/8531b4f3-5179-4948-86ce-52e13576ea11-kube-api-access-f4mfn\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:30.264647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.264582 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:30.264647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.264599 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8531b4f3-5179-4948-86ce-52e13576ea11-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:30.264647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.264611 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:30.264647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.264624 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:30.264647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.264637 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:30.264647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.264650 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8531b4f3-5179-4948-86ce-52e13576ea11-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:30.265031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.264677 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8531b4f3-5179-4948-86ce-52e13576ea11-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:30.265031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.264691 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8531b4f3-5179-4948-86ce-52e13576ea11-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:30.710483 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.710443 5059 generic.go:356] "Generic (PLEG): container finished" podID="8531b4f3-5179-4948-86ce-52e13576ea11" containerID="214a53ef694c00238d5e914252f452ff7a8677228ee8c69a81ba92d007afde19" exitCode=0 Nov 03 20:01:30.710708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.710517 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" Nov 03 20:01:30.710708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.710640 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" event={"ID":"8531b4f3-5179-4948-86ce-52e13576ea11","Type":"ContainerDied","Data":"214a53ef694c00238d5e914252f452ff7a8677228ee8c69a81ba92d007afde19"} Nov 03 20:01:30.710850 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.710713 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz" event={"ID":"8531b4f3-5179-4948-86ce-52e13576ea11","Type":"ContainerDied","Data":"9950b053e8368a1e4a2fc23f9001b946a63062f035c882f354fab5963f763b71"} Nov 03 20:01:30.710850 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.710745 5059 scope.go:122] "RemoveContainer" containerID="214a53ef694c00238d5e914252f452ff7a8677228ee8c69a81ba92d007afde19" Nov 03 20:01:30.713168 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.713143 5059 generic.go:356] "Generic (PLEG): container finished" podID="3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9" containerID="5eb5bb4ea69a8f03657f822ef8d3b6b1314a329897845d5d392bf127f67c5ae9" exitCode=0 Nov 03 20:01:30.713934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.713909 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e75eb6-g4rv5" event={"ID":"3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9","Type":"ContainerDied","Data":"5eb5bb4ea69a8f03657f822ef8d3b6b1314a329897845d5d392bf127f67c5ae9"} Nov 03 20:01:30.734165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.734138 5059 scope.go:122] "RemoveContainer" containerID="75e02563ce0318f1dbc83f69e4cc5d059fe289723fa324d9d3379736d1b57fdf" Nov 03 20:01:30.747224 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.747204 5059 scope.go:122] "RemoveContainer" containerID="214a53ef694c00238d5e914252f452ff7a8677228ee8c69a81ba92d007afde19" Nov 03 20:01:30.747560 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:01:30.747512 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"214a53ef694c00238d5e914252f452ff7a8677228ee8c69a81ba92d007afde19\": container with ID starting with 214a53ef694c00238d5e914252f452ff7a8677228ee8c69a81ba92d007afde19 not found: ID does not exist" containerID="214a53ef694c00238d5e914252f452ff7a8677228ee8c69a81ba92d007afde19" Nov 03 20:01:30.747560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.747543 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"214a53ef694c00238d5e914252f452ff7a8677228ee8c69a81ba92d007afde19"} err="failed to get container status \"214a53ef694c00238d5e914252f452ff7a8677228ee8c69a81ba92d007afde19\": rpc error: code = NotFound desc = could not find container \"214a53ef694c00238d5e914252f452ff7a8677228ee8c69a81ba92d007afde19\": container with ID starting with 214a53ef694c00238d5e914252f452ff7a8677228ee8c69a81ba92d007afde19 not found: ID does not exist" Nov 03 20:01:30.747735 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.747561 5059 scope.go:122] "RemoveContainer" containerID="75e02563ce0318f1dbc83f69e4cc5d059fe289723fa324d9d3379736d1b57fdf" Nov 03 20:01:30.747873 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:01:30.747841 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75e02563ce0318f1dbc83f69e4cc5d059fe289723fa324d9d3379736d1b57fdf\": container with ID starting with 75e02563ce0318f1dbc83f69e4cc5d059fe289723fa324d9d3379736d1b57fdf not found: ID does not exist" containerID="75e02563ce0318f1dbc83f69e4cc5d059fe289723fa324d9d3379736d1b57fdf" Nov 03 20:01:30.747940 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.747885 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75e02563ce0318f1dbc83f69e4cc5d059fe289723fa324d9d3379736d1b57fdf"} err="failed to get container status \"75e02563ce0318f1dbc83f69e4cc5d059fe289723fa324d9d3379736d1b57fdf\": rpc error: code = NotFound desc = could not find container \"75e02563ce0318f1dbc83f69e4cc5d059fe289723fa324d9d3379736d1b57fdf\": container with ID starting with 75e02563ce0318f1dbc83f69e4cc5d059fe289723fa324d9d3379736d1b57fdf not found: ID does not exist" Nov 03 20:01:30.768321 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.768282 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz"] Nov 03 20:01:30.779058 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.779033 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c6092e-openshift-default-6d8ccdf74f-vtnkz"] Nov 03 20:01:30.793640 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.793615 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv"] Nov 03 20:01:30.794468 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.794445 5059 cpu_manager.go:414] "containerMap: removing container" podUID="8531b4f3-5179-4948-86ce-52e13576ea11" containerName="istio-proxy" Nov 03 20:01:30.794612 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.794485 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8531b4f3-5179-4948-86ce-52e13576ea11" containerName="istio-proxy" Nov 03 20:01:30.794612 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.794532 5059 cpu_manager.go:414] "containerMap: removing container" podUID="cee28a4b-6a4e-43f3-9905-eaedd2068dad" containerName="revoke-keys" Nov 03 20:01:30.794612 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.794559 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee28a4b-6a4e-43f3-9905-eaedd2068dad" containerName="revoke-keys" Nov 03 20:01:30.794612 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.794575 5059 cpu_manager.go:414] "containerMap: removing container" podUID="8531b4f3-5179-4948-86ce-52e13576ea11" containerName="istio-proxy" Nov 03 20:01:30.794612 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.794584 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8531b4f3-5179-4948-86ce-52e13576ea11" containerName="istio-proxy" Nov 03 20:01:30.795039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.794880 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8531b4f3-5179-4948-86ce-52e13576ea11" containerName="istio-proxy" Nov 03 20:01:30.795039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.794912 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee28a4b-6a4e-43f3-9905-eaedd2068dad" containerName="revoke-keys" Nov 03 20:01:30.795039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.794922 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8531b4f3-5179-4948-86ce-52e13576ea11" containerName="istio-proxy" Nov 03 20:01:30.800419 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.800395 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:30.810140 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.810110 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv"] Nov 03 20:01:30.978744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.976902 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:30.978744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.977094 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp7gv\" (UniqueName: \"kubernetes.io/projected/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-kube-api-access-zp7gv\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:30.978744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.977140 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:30.978744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.977237 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:30.978744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.977289 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:30.978744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.977430 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:30.978744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.977503 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:30.978744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.977614 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:30.978744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:30.977756 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:31.079491 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.079409 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:31.079742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.079518 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:31.079742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.079593 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:31.079742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.079676 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:31.079970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.079742 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:31.079970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.079810 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp7gv\" (UniqueName: \"kubernetes.io/projected/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-kube-api-access-zp7gv\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:31.079970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.079841 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:31.079970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.079898 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:31.079970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.079924 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:31.080256 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.080084 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:31.080417 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.080371 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:31.080417 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.080388 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:31.080417 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.080396 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:31.080633 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.080429 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:31.082107 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.082079 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:31.082871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.082849 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:31.092409 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.092383 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:31.092953 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.092928 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp7gv\" (UniqueName: \"kubernetes.io/projected/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-kube-api-access-zp7gv\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:31.119067 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.119029 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:31.422127 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.422094 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv"] Nov 03 20:01:31.428499 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.428444 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:01:31.428680 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.428586 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:01:31.428777 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.428696 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:01:31.462526 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.462503 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:31.462915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.462884 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:31.463902 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.463880 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:01:31.726731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.726613 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" event={"ID":"fd6f9e73-1c5a-4151-a720-96f136b7b3a4","Type":"ContainerStarted","Data":"3879c9e81754313a4cb696508516f2230f2f9c2c215c51e2eb7ec23ec4b34aa3"} Nov 03 20:01:31.726731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.726722 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" event={"ID":"fd6f9e73-1c5a-4151-a720-96f136b7b3a4","Type":"ContainerStarted","Data":"dbccc6a151c723ededb6e3105f57f7e135e704ff03418168552d033cbd4a28eb"} Nov 03 20:01:31.762152 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.762097 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" podStartSLOduration=1.7620830079999998 podStartE2EDuration="1.762083008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:31.756088913 +0000 UTC m=+1486.013886170" watchObservedRunningTime="2026-11-03 20:01:31.762083008 +0000 UTC m=+1486.019880259" Nov 03 20:01:31.866414 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.866373 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8531b4f3-5179-4948-86ce-52e13576ea11" path="/var/lib/kubelet/pods/8531b4f3-5179-4948-86ce-52e13576ea11/volumes" Nov 03 20:01:31.888296 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.888265 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e75eb6-g4rv5" Nov 03 20:01:31.993351 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.993277 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9-maas-api-xks-ca\") pod \"3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9\" (UID: \"3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9\") " Nov 03 20:01:31.993351 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.993344 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9-maas-api-openshift-ca\") pod \"3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9\" (UID: \"3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9\") " Nov 03 20:01:31.993538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.993474 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9-kube-api-access-szbgj\" (UniqueName: \"kubernetes.io/projected/3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9-kube-api-access-szbgj\") pod \"3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9\" (UID: \"3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9\") " Nov 03 20:01:31.993759 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.993733 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9-maas-api-openshift-ca" pod "3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9" (UID: "3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:01:31.993993 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.993969 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:31.995494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.995472 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9-kube-api-access-szbgj" pod "3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9" (UID: "3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9"). InnerVolumeSpecName "kube-api-access-szbgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:31.995494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:31.995481 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9-maas-api-xks-ca" pod "3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9" (UID: "3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:01:32.096439 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:32.096392 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szbgj\" (UniqueName: \"kubernetes.io/projected/3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9-kube-api-access-szbgj\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:32.096439 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:32.096439 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:32.119202 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:32.119180 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:32.120983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:32.120953 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 20:01:32.121105 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:32.121002 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" podUID="fd6f9e73-1c5a-4151-a720-96f136b7b3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 20:01:32.738821 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:32.738790 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e75eb6-g4rv5" Nov 03 20:01:32.738821 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:32.738817 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e75eb6-g4rv5" event={"ID":"3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9","Type":"ContainerDied","Data":"e2ba82c34644082651aec5024e3ce5c5ae2e09fa741ecc4c6bb6689e3be13fa0"} Nov 03 20:01:32.739059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:32.738879 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2ba82c34644082651aec5024e3ce5c5ae2e09fa741ecc4c6bb6689e3be13fa0" Nov 03 20:01:33.120084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:33.120042 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 20:01:33.120479 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:33.120105 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" podUID="fd6f9e73-1c5a-4151-a720-96f136b7b3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 20:01:34.119654 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:34.119621 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 20:01:34.119862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:34.119718 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" podUID="fd6f9e73-1c5a-4151-a720-96f136b7b3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 20:01:34.412939 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:34.412519 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4"] Nov 03 20:01:34.414685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:34.413607 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4" podUID="76e4398b-91ee-4b1f-b498-4b54d7322193" containerName="payload-pre-processing" containerID="cri-o://8cffdefedb54423d797c3eae4993cb83799f831bb5e05c62ff386dd41339cf5f" gracePeriod=30 Nov 03 20:01:34.430492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:34.430456 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv"] Nov 03 20:01:34.430798 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:34.430749 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv" podUID="fbe92210-726b-458d-9766-452f49c237d7" containerName="payload-processing" containerID="cri-o://a49ed5cb02983f39d30db29b7ff938dd01ab1f494991a6df6e616c84808342bd" gracePeriod=30 Nov 03 20:01:34.847030 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:34.846991 5059 generic.go:356] "Generic (PLEG): container finished" podID="76e4398b-91ee-4b1f-b498-4b54d7322193" containerID="8cffdefedb54423d797c3eae4993cb83799f831bb5e05c62ff386dd41339cf5f" exitCode=0 Nov 03 20:01:34.848152 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:34.847267 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4" event={"ID":"76e4398b-91ee-4b1f-b498-4b54d7322193","Type":"ContainerDied","Data":"8cffdefedb54423d797c3eae4993cb83799f831bb5e05c62ff386dd41339cf5f"} Nov 03 20:01:34.853099 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:34.851688 5059 generic.go:356] "Generic (PLEG): container finished" podID="fbe92210-726b-458d-9766-452f49c237d7" containerID="a49ed5cb02983f39d30db29b7ff938dd01ab1f494991a6df6e616c84808342bd" exitCode=0 Nov 03 20:01:34.853099 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:34.851743 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv" event={"ID":"fbe92210-726b-458d-9766-452f49c237d7","Type":"ContainerDied","Data":"a49ed5cb02983f39d30db29b7ff938dd01ab1f494991a6df6e616c84808342bd"} Nov 03 20:01:35.008122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.006938 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv" Nov 03 20:01:35.131977 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.131480 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 20:01:35.131977 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.131543 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" podUID="fd6f9e73-1c5a-4151-a720-96f136b7b3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 20:01:35.134503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.133571 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbe92210-726b-458d-9766-452f49c237d7-kube-api-access-9zr4h\" (UniqueName: \"kubernetes.io/projected/fbe92210-726b-458d-9766-452f49c237d7-kube-api-access-9zr4h\") pod \"fbe92210-726b-458d-9766-452f49c237d7\" (UID: \"fbe92210-726b-458d-9766-452f49c237d7\") " Nov 03 20:01:35.134503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.133626 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbe92210-726b-458d-9766-452f49c237d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe92210-726b-458d-9766-452f49c237d7-plugins-config-volume\") pod \"fbe92210-726b-458d-9766-452f49c237d7\" (UID: \"fbe92210-726b-458d-9766-452f49c237d7\") " Nov 03 20:01:35.134881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.134859 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe92210-726b-458d-9766-452f49c237d7-plugins-config-volume" pod "fbe92210-726b-458d-9766-452f49c237d7" (UID: "fbe92210-726b-458d-9766-452f49c237d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:01:35.139354 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.139326 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe92210-726b-458d-9766-452f49c237d7-kube-api-access-9zr4h" pod "fbe92210-726b-458d-9766-452f49c237d7" (UID: "fbe92210-726b-458d-9766-452f49c237d7"). InnerVolumeSpecName "kube-api-access-9zr4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:35.149051 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.149011 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4" Nov 03 20:01:35.234959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.234926 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76e4398b-91ee-4b1f-b498-4b54d7322193-kube-api-access-sk4vs\" (UniqueName: \"kubernetes.io/projected/76e4398b-91ee-4b1f-b498-4b54d7322193-kube-api-access-sk4vs\") pod \"76e4398b-91ee-4b1f-b498-4b54d7322193\" (UID: \"76e4398b-91ee-4b1f-b498-4b54d7322193\") " Nov 03 20:01:35.235217 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.235185 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76e4398b-91ee-4b1f-b498-4b54d7322193-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76e4398b-91ee-4b1f-b498-4b54d7322193-plugins-config-volume\") pod \"76e4398b-91ee-4b1f-b498-4b54d7322193\" (UID: \"76e4398b-91ee-4b1f-b498-4b54d7322193\") " Nov 03 20:01:35.235597 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.235565 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76e4398b-91ee-4b1f-b498-4b54d7322193-plugins-config-volume" pod "76e4398b-91ee-4b1f-b498-4b54d7322193" (UID: "76e4398b-91ee-4b1f-b498-4b54d7322193"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:01:35.236282 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.236257 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zr4h\" (UniqueName: \"kubernetes.io/projected/fbe92210-726b-458d-9766-452f49c237d7-kube-api-access-9zr4h\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:35.236444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.236427 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe92210-726b-458d-9766-452f49c237d7-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:35.236577 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.236563 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76e4398b-91ee-4b1f-b498-4b54d7322193-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:35.237276 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.237248 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e4398b-91ee-4b1f-b498-4b54d7322193-kube-api-access-sk4vs" pod "76e4398b-91ee-4b1f-b498-4b54d7322193" (UID: "76e4398b-91ee-4b1f-b498-4b54d7322193"). InnerVolumeSpecName "kube-api-access-sk4vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:35.338303 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.338218 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk4vs\" (UniqueName: \"kubernetes.io/projected/76e4398b-91ee-4b1f-b498-4b54d7322193-kube-api-access-sk4vs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:35.912906 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.912867 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4" event={"ID":"76e4398b-91ee-4b1f-b498-4b54d7322193","Type":"ContainerDied","Data":"68e8cbaf9052dfbc9df05dabe07bc13299af3bed2569860d93a8ed1df441f586"} Nov 03 20:01:35.913455 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.913433 5059 scope.go:122] "RemoveContainer" containerID="8cffdefedb54423d797c3eae4993cb83799f831bb5e05c62ff386dd41339cf5f" Nov 03 20:01:35.915421 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.913064 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4" Nov 03 20:01:35.920767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.920697 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv" event={"ID":"fbe92210-726b-458d-9766-452f49c237d7","Type":"ContainerDied","Data":"ff553bd533cdaea8146bc61ab282c16199de2b94c80d808a66ff7471abd0eb31"} Nov 03 20:01:35.920767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.920735 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv" Nov 03 20:01:35.940114 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.940096 5059 scope.go:122] "RemoveContainer" containerID="a49ed5cb02983f39d30db29b7ff938dd01ab1f494991a6df6e616c84808342bd" Nov 03 20:01:35.979686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.963812 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4"] Nov 03 20:01:35.979686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.973339 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-8445c77dc-wpmx4"] Nov 03 20:01:35.997879 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:35.997852 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv"] Nov 03 20:01:36.003760 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:36.003736 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8b3020-7477b6db4b-rphdv"] Nov 03 20:01:36.121245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:36.120788 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 20:01:36.121245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:36.120848 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" podUID="fd6f9e73-1c5a-4151-a720-96f136b7b3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 20:01:37.119804 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:37.119772 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 20:01:37.120217 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:37.119825 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" podUID="fd6f9e73-1c5a-4151-a720-96f136b7b3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 20:01:37.889194 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:37.889153 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76e4398b-91ee-4b1f-b498-4b54d7322193" path="/var/lib/kubelet/pods/76e4398b-91ee-4b1f-b498-4b54d7322193/volumes" Nov 03 20:01:37.890157 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:37.890122 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbe92210-726b-458d-9766-452f49c237d7" path="/var/lib/kubelet/pods/fbe92210-726b-458d-9766-452f49c237d7/volumes" Nov 03 20:01:38.119728 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.119696 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 20:01:38.119898 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.119761 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" podUID="fd6f9e73-1c5a-4151-a720-96f136b7b3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 20:01:38.161445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.161351 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq"] Nov 03 20:01:38.161632 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.161607 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" podUID="b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f" containerName="istio-proxy" containerID="cri-o://5ef15580b65fdc48c306f5da5fe11dc6b97f20f1719efaf5b37bb08947787e4b" gracePeriod=30 Nov 03 20:01:38.816260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.816221 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-834054e8-kq9d2"] Nov 03 20:01:38.816905 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.816886 5059 cpu_manager.go:414] "containerMap: removing container" podUID="76e4398b-91ee-4b1f-b498-4b54d7322193" containerName="payload-pre-processing" Nov 03 20:01:38.816905 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.816905 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76e4398b-91ee-4b1f-b498-4b54d7322193" containerName="payload-pre-processing" Nov 03 20:01:38.817094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.816926 5059 cpu_manager.go:414] "containerMap: removing container" podUID="3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9" containerName="revoke-keys" Nov 03 20:01:38.817094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.816932 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9" containerName="revoke-keys" Nov 03 20:01:38.817094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.816952 5059 cpu_manager.go:414] "containerMap: removing container" podUID="fbe92210-726b-458d-9766-452f49c237d7" containerName="payload-processing" Nov 03 20:01:38.817094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.816957 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe92210-726b-458d-9766-452f49c237d7" containerName="payload-processing" Nov 03 20:01:38.817094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.817077 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="76e4398b-91ee-4b1f-b498-4b54d7322193" containerName="payload-pre-processing" Nov 03 20:01:38.817094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.817090 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9" containerName="revoke-keys" Nov 03 20:01:38.817412 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.817100 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe92210-726b-458d-9766-452f49c237d7" containerName="payload-processing" Nov 03 20:01:38.822336 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.822312 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-834054e8-kq9d2" Nov 03 20:01:38.827139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.827113 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:01:38.829763 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.828491 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:01:38.834169 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.834136 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-834054e8-kq9d2"] Nov 03 20:01:38.866445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.866417 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d42779d9-caf1-41e2-8596-955a7a6bf405-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-834054e8-kq9d2\" (UID: \"d42779d9-caf1-41e2-8596-955a7a6bf405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-834054e8-kq9d2" Nov 03 20:01:38.866598 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.866476 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d42779d9-caf1-41e2-8596-955a7a6bf405-maas-api-xks-ca\") pod \"maas-api-revoke-sub-834054e8-kq9d2\" (UID: \"d42779d9-caf1-41e2-8596-955a7a6bf405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-834054e8-kq9d2" Nov 03 20:01:38.866710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.866652 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l94d\" (UniqueName: \"kubernetes.io/projected/d42779d9-caf1-41e2-8596-955a7a6bf405-kube-api-access-2l94d\") pod \"maas-api-revoke-sub-834054e8-kq9d2\" (UID: \"d42779d9-caf1-41e2-8596-955a7a6bf405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-834054e8-kq9d2" Nov 03 20:01:38.967889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.967852 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l94d\" (UniqueName: \"kubernetes.io/projected/d42779d9-caf1-41e2-8596-955a7a6bf405-kube-api-access-2l94d\") pod \"maas-api-revoke-sub-834054e8-kq9d2\" (UID: \"d42779d9-caf1-41e2-8596-955a7a6bf405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-834054e8-kq9d2" Nov 03 20:01:38.968042 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.967982 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d42779d9-caf1-41e2-8596-955a7a6bf405-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-834054e8-kq9d2\" (UID: \"d42779d9-caf1-41e2-8596-955a7a6bf405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-834054e8-kq9d2" Nov 03 20:01:38.968090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.968039 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d42779d9-caf1-41e2-8596-955a7a6bf405-maas-api-xks-ca\") pod \"maas-api-revoke-sub-834054e8-kq9d2\" (UID: \"d42779d9-caf1-41e2-8596-955a7a6bf405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-834054e8-kq9d2" Nov 03 20:01:38.968631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.968608 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d42779d9-caf1-41e2-8596-955a7a6bf405-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-834054e8-kq9d2\" (UID: \"d42779d9-caf1-41e2-8596-955a7a6bf405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-834054e8-kq9d2" Nov 03 20:01:38.970985 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.970958 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d42779d9-caf1-41e2-8596-955a7a6bf405-maas-api-xks-ca\") pod \"maas-api-revoke-sub-834054e8-kq9d2\" (UID: \"d42779d9-caf1-41e2-8596-955a7a6bf405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-834054e8-kq9d2" Nov 03 20:01:38.980425 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:38.980394 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l94d\" (UniqueName: \"kubernetes.io/projected/d42779d9-caf1-41e2-8596-955a7a6bf405-kube-api-access-2l94d\") pod \"maas-api-revoke-sub-834054e8-kq9d2\" (UID: \"d42779d9-caf1-41e2-8596-955a7a6bf405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-834054e8-kq9d2" Nov 03 20:01:39.147448 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:39.147412 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-834054e8-kq9d2" Nov 03 20:01:39.372163 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:39.372130 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:39.373735 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:39.373699 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:39.375514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:39.375491 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:01:39.462182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:39.462148 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-834054e8-kq9d2"] Nov 03 20:01:39.463964 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:01:39.463929 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42779d9_caf1_41e2_8596_955a7a6bf405.slice/crio-aee44563f71b86a91f76b9290c17dda02d6e2c3b51642edf1513bfdf18650c62 WatchSource:0}: Error finding container aee44563f71b86a91f76b9290c17dda02d6e2c3b51642edf1513bfdf18650c62: Status 404 returned error can't find the container with id aee44563f71b86a91f76b9290c17dda02d6e2c3b51642edf1513bfdf18650c62 Nov 03 20:01:39.969314 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:39.969229 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-834054e8-kq9d2" event={"ID":"d42779d9-caf1-41e2-8596-955a7a6bf405","Type":"ContainerStarted","Data":"aee44563f71b86a91f76b9290c17dda02d6e2c3b51642edf1513bfdf18650c62"} Nov 03 20:01:40.604574 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:01:40.604541 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:01:40.978994 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:40.978958 5059 generic.go:356] "Generic (PLEG): container finished" podID="d42779d9-caf1-41e2-8596-955a7a6bf405" containerID="0b2c5ee5f116b488344166b030a8c8c15d4a7739e51fe991225a449493d671ad" exitCode=0 Nov 03 20:01:40.979196 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:40.979041 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-834054e8-kq9d2" event={"ID":"d42779d9-caf1-41e2-8596-955a7a6bf405","Type":"ContainerDied","Data":"0b2c5ee5f116b488344166b030a8c8c15d4a7739e51fe991225a449493d671ad"} Nov 03 20:01:42.370581 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:42.370553 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-834054e8-kq9d2" Nov 03 20:01:42.424738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:42.424709 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d42779d9-caf1-41e2-8596-955a7a6bf405-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d42779d9-caf1-41e2-8596-955a7a6bf405-maas-api-openshift-ca\") pod \"d42779d9-caf1-41e2-8596-955a7a6bf405\" (UID: \"d42779d9-caf1-41e2-8596-955a7a6bf405\") " Nov 03 20:01:42.424877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:42.424828 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d42779d9-caf1-41e2-8596-955a7a6bf405-kube-api-access-2l94d\" (UniqueName: \"kubernetes.io/projected/d42779d9-caf1-41e2-8596-955a7a6bf405-kube-api-access-2l94d\") pod \"d42779d9-caf1-41e2-8596-955a7a6bf405\" (UID: \"d42779d9-caf1-41e2-8596-955a7a6bf405\") " Nov 03 20:01:42.424941 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:42.424888 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d42779d9-caf1-41e2-8596-955a7a6bf405-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d42779d9-caf1-41e2-8596-955a7a6bf405-maas-api-xks-ca\") pod \"d42779d9-caf1-41e2-8596-955a7a6bf405\" (UID: \"d42779d9-caf1-41e2-8596-955a7a6bf405\") " Nov 03 20:01:42.425126 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:42.425097 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d42779d9-caf1-41e2-8596-955a7a6bf405-maas-api-openshift-ca" pod "d42779d9-caf1-41e2-8596-955a7a6bf405" (UID: "d42779d9-caf1-41e2-8596-955a7a6bf405"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:01:42.425496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:42.425469 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d42779d9-caf1-41e2-8596-955a7a6bf405-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:42.427115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:42.427082 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d42779d9-caf1-41e2-8596-955a7a6bf405-maas-api-xks-ca" pod "d42779d9-caf1-41e2-8596-955a7a6bf405" (UID: "d42779d9-caf1-41e2-8596-955a7a6bf405"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:01:42.427230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:42.427145 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42779d9-caf1-41e2-8596-955a7a6bf405-kube-api-access-2l94d" pod "d42779d9-caf1-41e2-8596-955a7a6bf405" (UID: "d42779d9-caf1-41e2-8596-955a7a6bf405"). InnerVolumeSpecName "kube-api-access-2l94d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:42.527071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:42.527027 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l94d\" (UniqueName: \"kubernetes.io/projected/d42779d9-caf1-41e2-8596-955a7a6bf405-kube-api-access-2l94d\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:42.527071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:42.527067 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d42779d9-caf1-41e2-8596-955a7a6bf405-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:42.995357 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:42.995329 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-834054e8-kq9d2" Nov 03 20:01:42.995529 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:42.995367 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-834054e8-kq9d2" event={"ID":"d42779d9-caf1-41e2-8596-955a7a6bf405","Type":"ContainerDied","Data":"aee44563f71b86a91f76b9290c17dda02d6e2c3b51642edf1513bfdf18650c62"} Nov 03 20:01:42.995529 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:42.995413 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aee44563f71b86a91f76b9290c17dda02d6e2c3b51642edf1513bfdf18650c62" Nov 03 20:01:43.465857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.465827 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 20:01:43.516540 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.516508 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6c3944f-kk8bm"] Nov 03 20:01:43.517477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.517455 5059 cpu_manager.go:414] "containerMap: removing container" podUID="b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f" containerName="istio-proxy" Nov 03 20:01:43.517542 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.517482 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f" containerName="istio-proxy" Nov 03 20:01:43.517542 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.517517 5059 cpu_manager.go:414] "containerMap: removing container" podUID="d42779d9-caf1-41e2-8596-955a7a6bf405" containerName="revoke-keys" Nov 03 20:01:43.517542 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.517525 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d42779d9-caf1-41e2-8596-955a7a6bf405" containerName="revoke-keys" Nov 03 20:01:43.517756 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.517741 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f" containerName="istio-proxy" Nov 03 20:01:43.517792 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.517766 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d42779d9-caf1-41e2-8596-955a7a6bf405" containerName="revoke-keys" Nov 03 20:01:43.521943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.521924 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c3944f-kk8bm" Nov 03 20:01:43.527938 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.527911 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:01:43.528527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.528505 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:01:43.529718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.529693 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6c3944f-kk8bm"] Nov 03 20:01:43.540381 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.540360 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-podinfo\") pod \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " Nov 03 20:01:43.540500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.540410 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istiod-ca-cert\") pod \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " Nov 03 20:01:43.540563 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.540537 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-workload-certs\") pod \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " Nov 03 20:01:43.540638 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.540620 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-token\" (UniqueName: \"kubernetes.io/projected/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-token\") pod \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " Nov 03 20:01:43.540707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.540686 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-envoy\") pod \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " Nov 03 20:01:43.540809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.540738 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-credential-socket\") pod \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " Nov 03 20:01:43.540910 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.540826 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-workload-certs" pod "b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f" (UID: "b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:43.540910 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.540840 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-kube-api-access-r2bwx\" (UniqueName: \"kubernetes.io/projected/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-kube-api-access-r2bwx\") pod \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " Nov 03 20:01:43.540910 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.540854 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istiod-ca-cert" pod "b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f" (UID: "b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:01:43.541148 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.540921 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-workload-socket\") pod \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " Nov 03 20:01:43.541234 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.541145 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-credential-socket" pod "b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f" (UID: "b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:43.541234 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.541177 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-data\") pod \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\" (UID: \"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f\") " Nov 03 20:01:43.541389 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.541340 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-workload-socket" pod "b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f" (UID: "b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:43.541698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.541641 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-data" pod "b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f" (UID: "b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:43.542044 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.542014 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e43a9baa-e726-44d2-98a9-cbd3b27871ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6c3944f-kk8bm\" (UID: \"e43a9baa-e726-44d2-98a9-cbd3b27871ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c3944f-kk8bm" Nov 03 20:01:43.542434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.542298 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpg2k\" (UniqueName: \"kubernetes.io/projected/e43a9baa-e726-44d2-98a9-cbd3b27871ea-kube-api-access-tpg2k\") pod \"maas-api-revoke-sub-f6c3944f-kk8bm\" (UID: \"e43a9baa-e726-44d2-98a9-cbd3b27871ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c3944f-kk8bm" Nov 03 20:01:43.542761 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.542706 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e43a9baa-e726-44d2-98a9-cbd3b27871ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6c3944f-kk8bm\" (UID: \"e43a9baa-e726-44d2-98a9-cbd3b27871ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c3944f-kk8bm" Nov 03 20:01:43.542881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.542856 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:43.542995 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.542884 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:43.542995 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.542901 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:43.543401 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.543216 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:43.543401 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.543293 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:43.547631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.547606 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-envoy" pod "b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f" (UID: "b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:43.556052 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.556027 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-kube-api-access-r2bwx" pod "b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f" (UID: "b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f"). InnerVolumeSpecName "kube-api-access-r2bwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:43.565345 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.565319 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-token" pod "b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f" (UID: "b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:43.566745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.565571 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-podinfo" pod "b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f" (UID: "b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:01:43.644269 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.644238 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpg2k\" (UniqueName: \"kubernetes.io/projected/e43a9baa-e726-44d2-98a9-cbd3b27871ea-kube-api-access-tpg2k\") pod \"maas-api-revoke-sub-f6c3944f-kk8bm\" (UID: \"e43a9baa-e726-44d2-98a9-cbd3b27871ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c3944f-kk8bm" Nov 03 20:01:43.644383 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.644339 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e43a9baa-e726-44d2-98a9-cbd3b27871ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6c3944f-kk8bm\" (UID: \"e43a9baa-e726-44d2-98a9-cbd3b27871ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c3944f-kk8bm" Nov 03 20:01:43.644440 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.644407 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e43a9baa-e726-44d2-98a9-cbd3b27871ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6c3944f-kk8bm\" (UID: \"e43a9baa-e726-44d2-98a9-cbd3b27871ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c3944f-kk8bm" Nov 03 20:01:43.644610 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.644585 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:43.644712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.644611 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:43.644712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.644626 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:43.644712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.644641 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2bwx\" (UniqueName: \"kubernetes.io/projected/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f-kube-api-access-r2bwx\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:43.645078 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.645055 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e43a9baa-e726-44d2-98a9-cbd3b27871ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6c3944f-kk8bm\" (UID: \"e43a9baa-e726-44d2-98a9-cbd3b27871ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c3944f-kk8bm" Nov 03 20:01:43.646805 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.646778 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e43a9baa-e726-44d2-98a9-cbd3b27871ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6c3944f-kk8bm\" (UID: \"e43a9baa-e726-44d2-98a9-cbd3b27871ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c3944f-kk8bm" Nov 03 20:01:43.674059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.672017 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpg2k\" (UniqueName: \"kubernetes.io/projected/e43a9baa-e726-44d2-98a9-cbd3b27871ea-kube-api-access-tpg2k\") pod \"maas-api-revoke-sub-f6c3944f-kk8bm\" (UID: \"e43a9baa-e726-44d2-98a9-cbd3b27871ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c3944f-kk8bm" Nov 03 20:01:43.842768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:43.842726 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c3944f-kk8bm" Nov 03 20:01:44.004007 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:44.003974 5059 generic.go:356] "Generic (PLEG): container finished" podID="b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f" containerID="5ef15580b65fdc48c306f5da5fe11dc6b97f20f1719efaf5b37bb08947787e4b" exitCode=0 Nov 03 20:01:44.004148 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:44.004038 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" Nov 03 20:01:44.004148 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:44.004056 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" event={"ID":"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f","Type":"ContainerDied","Data":"5ef15580b65fdc48c306f5da5fe11dc6b97f20f1719efaf5b37bb08947787e4b"} Nov 03 20:01:44.004148 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:44.004114 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq" event={"ID":"b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f","Type":"ContainerDied","Data":"6305ec56316a207af186bbc8d932641ea1e2d368e68880100b7c59997f11b353"} Nov 03 20:01:44.004148 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:44.004138 5059 scope.go:122] "RemoveContainer" containerID="5ef15580b65fdc48c306f5da5fe11dc6b97f20f1719efaf5b37bb08947787e4b" Nov 03 20:01:44.018115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:44.018093 5059 scope.go:122] "RemoveContainer" containerID="5ef15580b65fdc48c306f5da5fe11dc6b97f20f1719efaf5b37bb08947787e4b" Nov 03 20:01:44.018390 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:01:44.018367 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ef15580b65fdc48c306f5da5fe11dc6b97f20f1719efaf5b37bb08947787e4b\": container with ID starting with 5ef15580b65fdc48c306f5da5fe11dc6b97f20f1719efaf5b37bb08947787e4b not found: ID does not exist" containerID="5ef15580b65fdc48c306f5da5fe11dc6b97f20f1719efaf5b37bb08947787e4b" Nov 03 20:01:44.018490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:44.018401 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ef15580b65fdc48c306f5da5fe11dc6b97f20f1719efaf5b37bb08947787e4b"} err="failed to get container status \"5ef15580b65fdc48c306f5da5fe11dc6b97f20f1719efaf5b37bb08947787e4b\": rpc error: code = NotFound desc = could not find container \"5ef15580b65fdc48c306f5da5fe11dc6b97f20f1719efaf5b37bb08947787e4b\": container with ID starting with 5ef15580b65fdc48c306f5da5fe11dc6b97f20f1719efaf5b37bb08947787e4b not found: ID does not exist" Nov 03 20:01:44.025989 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:44.025959 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq"] Nov 03 20:01:44.029755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:44.029732 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-mhprq"] Nov 03 20:01:44.127335 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:44.127310 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6c3944f-kk8bm"] Nov 03 20:01:44.127614 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:01:44.127594 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43a9baa_e726_44d2_98a9_cbd3b27871ea.slice/crio-038be96ca6e1cebd80035efbec3eab38a72ec191f7ba8360eb8427249dca4ad8 WatchSource:0}: Error finding container 038be96ca6e1cebd80035efbec3eab38a72ec191f7ba8360eb8427249dca4ad8: Status 404 returned error can't find the container with id 038be96ca6e1cebd80035efbec3eab38a72ec191f7ba8360eb8427249dca4ad8 Nov 03 20:01:45.014636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:45.014608 5059 generic.go:356] "Generic (PLEG): container finished" podID="e43a9baa-e726-44d2-98a9-cbd3b27871ea" containerID="376181b768f9f5bd054faa8e9c391626a9b7e07d8f21b87327bbb318eaaaf524" exitCode=0 Nov 03 20:01:45.015041 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:45.014695 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c3944f-kk8bm" event={"ID":"e43a9baa-e726-44d2-98a9-cbd3b27871ea","Type":"ContainerDied","Data":"376181b768f9f5bd054faa8e9c391626a9b7e07d8f21b87327bbb318eaaaf524"} Nov 03 20:01:45.015041 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:45.014746 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c3944f-kk8bm" event={"ID":"e43a9baa-e726-44d2-98a9-cbd3b27871ea","Type":"ContainerStarted","Data":"038be96ca6e1cebd80035efbec3eab38a72ec191f7ba8360eb8427249dca4ad8"} Nov 03 20:01:45.858760 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:45.858722 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f" path="/var/lib/kubelet/pods/b01f88a8-c8c2-44ba-ab1d-bd69d2b4084f/volumes" Nov 03 20:01:46.187115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:46.187093 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c3944f-kk8bm" Nov 03 20:01:46.283818 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:46.283787 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e43a9baa-e726-44d2-98a9-cbd3b27871ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e43a9baa-e726-44d2-98a9-cbd3b27871ea-maas-api-xks-ca\") pod \"e43a9baa-e726-44d2-98a9-cbd3b27871ea\" (UID: \"e43a9baa-e726-44d2-98a9-cbd3b27871ea\") " Nov 03 20:01:46.284100 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:46.283831 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e43a9baa-e726-44d2-98a9-cbd3b27871ea-kube-api-access-tpg2k\" (UniqueName: \"kubernetes.io/projected/e43a9baa-e726-44d2-98a9-cbd3b27871ea-kube-api-access-tpg2k\") pod \"e43a9baa-e726-44d2-98a9-cbd3b27871ea\" (UID: \"e43a9baa-e726-44d2-98a9-cbd3b27871ea\") " Nov 03 20:01:46.284100 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:46.283869 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e43a9baa-e726-44d2-98a9-cbd3b27871ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e43a9baa-e726-44d2-98a9-cbd3b27871ea-maas-api-openshift-ca\") pod \"e43a9baa-e726-44d2-98a9-cbd3b27871ea\" (UID: \"e43a9baa-e726-44d2-98a9-cbd3b27871ea\") " Nov 03 20:01:46.284331 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:46.284308 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e43a9baa-e726-44d2-98a9-cbd3b27871ea-maas-api-openshift-ca" pod "e43a9baa-e726-44d2-98a9-cbd3b27871ea" (UID: "e43a9baa-e726-44d2-98a9-cbd3b27871ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:01:46.286762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:46.286534 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43a9baa-e726-44d2-98a9-cbd3b27871ea-kube-api-access-tpg2k" pod "e43a9baa-e726-44d2-98a9-cbd3b27871ea" (UID: "e43a9baa-e726-44d2-98a9-cbd3b27871ea"). InnerVolumeSpecName "kube-api-access-tpg2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:46.286762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:46.286548 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e43a9baa-e726-44d2-98a9-cbd3b27871ea-maas-api-xks-ca" pod "e43a9baa-e726-44d2-98a9-cbd3b27871ea" (UID: "e43a9baa-e726-44d2-98a9-cbd3b27871ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:01:46.385533 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:46.385499 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e43a9baa-e726-44d2-98a9-cbd3b27871ea-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:46.385710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:46.385561 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpg2k\" (UniqueName: \"kubernetes.io/projected/e43a9baa-e726-44d2-98a9-cbd3b27871ea-kube-api-access-tpg2k\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:46.385710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:46.385580 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e43a9baa-e726-44d2-98a9-cbd3b27871ea-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:46.590873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:46.590847 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 20:01:46.592325 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:46.592302 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 20:01:47.033710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:47.033561 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c3944f-kk8bm" event={"ID":"e43a9baa-e726-44d2-98a9-cbd3b27871ea","Type":"ContainerDied","Data":"038be96ca6e1cebd80035efbec3eab38a72ec191f7ba8360eb8427249dca4ad8"} Nov 03 20:01:47.033710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:47.033637 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="038be96ca6e1cebd80035efbec3eab38a72ec191f7ba8360eb8427249dca4ad8" Nov 03 20:01:47.033710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:47.033683 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c3944f-kk8bm" Nov 03 20:01:47.932228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:47.932196 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj"] Nov 03 20:01:47.932962 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:47.932942 5059 cpu_manager.go:414] "containerMap: removing container" podUID="e43a9baa-e726-44d2-98a9-cbd3b27871ea" containerName="revoke-keys" Nov 03 20:01:47.933022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:47.932964 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e43a9baa-e726-44d2-98a9-cbd3b27871ea" containerName="revoke-keys" Nov 03 20:01:47.933089 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:47.933079 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e43a9baa-e726-44d2-98a9-cbd3b27871ea" containerName="revoke-keys" Nov 03 20:01:47.936853 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:47.936835 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:47.950253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:47.950211 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj"] Nov 03 20:01:48.007292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.007262 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.007444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.007308 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.007527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.007503 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.007598 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.007579 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.007702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.007682 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.007839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.007819 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.007912 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.007874 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-594f4\" (UniqueName: \"kubernetes.io/projected/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-kube-api-access-594f4\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.007967 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.007954 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.008012 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.007983 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.109536 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.109504 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.109714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.109555 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-594f4\" (UniqueName: \"kubernetes.io/projected/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-kube-api-access-594f4\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.109714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.109601 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.109714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.109624 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.109714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.109699 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.109953 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.109726 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.109953 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.109800 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.109953 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.109839 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.110118 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.109958 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.110118 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.110033 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.110242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.110154 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.110242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.110190 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.110374 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.110253 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.110503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.110482 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.111941 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.111916 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.112313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.112297 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.121844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.121821 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.121948 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.121932 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-594f4\" (UniqueName: \"kubernetes.io/projected/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-kube-api-access-594f4\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.258921 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.258847 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:48.552461 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.552418 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj"] Nov 03 20:01:48.554509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.554471 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:01:48.554609 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.554557 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:01:48.554609 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:48.554597 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:01:49.080483 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:49.080447 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" event={"ID":"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8","Type":"ContainerStarted","Data":"30b686d20aa315b3d217ef4f6c8035860614ce718ff61acc1dbc7cd8fcb7308e"} Nov 03 20:01:49.081489 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:49.081463 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" event={"ID":"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8","Type":"ContainerStarted","Data":"3e529567addfc699af7196420c9d58e20b2a82cae52ab801a7af30e5f4001334"} Nov 03 20:01:49.131405 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:49.131343 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" podStartSLOduration=2.13132392 podStartE2EDuration="2.13132392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:49.131029902 +0000 UTC m=+1503.388827171" watchObservedRunningTime="2026-11-03 20:01:49.13132392 +0000 UTC m=+1503.389121179" Nov 03 20:01:49.267033 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:49.267002 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:49.267793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:49.267764 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 20:01:49.267991 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:49.267965 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" podUID="0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 20:01:49.748801 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:49.748751 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:50.090945 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:50.090897 5059 scope.go:122] "RemoveContainer" containerID="93a88f6eefe518149493f36ad88ef3160f1d400b6ac95b64287b395c5fb27ef4" Nov 03 20:01:50.303705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:50.278639 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 20:01:50.304028 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:50.303988 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" podUID="0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 20:01:50.349178 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:50.349144 5059 scope.go:122] "RemoveContainer" containerID="0ea193a090b91d09c90e23c8abd032827496598ffbb839e0505022e1784dd565" Nov 03 20:01:50.606095 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:50.596506 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:50.888984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:50.888761 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:51.100989 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:50.993843 5059 scope.go:122] "RemoveContainer" containerID="2012fe05eff12aba66893f4a0842bdb7c135cb9a7413da3da728d65095b721ec" Nov 03 20:01:51.180138 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:51.180020 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 20:01:51.317718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:51.266718 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba2227d-jl6q8"] Nov 03 20:01:51.335063 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:51.335024 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba2227d-jl6q8" Nov 03 20:01:51.404688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:51.341550 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:01:51.405227 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:51.356249 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 20:01:51.405387 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:51.405361 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" podUID="0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 20:01:51.412787 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:51.356301 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:01:51.491769 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:51.491731 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba2227d-jl6q8"] Nov 03 20:01:51.615960 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:51.533789 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dba2227d-jl6q8\" (UID: \"d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba2227d-jl6q8" Nov 03 20:01:51.615960 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:51.533886 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h86t\" (UniqueName: \"kubernetes.io/projected/d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c-kube-api-access-9h86t\") pod \"maas-api-revoke-sub-dba2227d-jl6q8\" (UID: \"d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba2227d-jl6q8" Nov 03 20:01:51.615960 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:51.533969 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dba2227d-jl6q8\" (UID: \"d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba2227d-jl6q8" Nov 03 20:01:51.679734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:51.649694 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dba2227d-jl6q8\" (UID: \"d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba2227d-jl6q8" Nov 03 20:01:51.679734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:51.649807 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h86t\" (UniqueName: \"kubernetes.io/projected/d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c-kube-api-access-9h86t\") pod \"maas-api-revoke-sub-dba2227d-jl6q8\" (UID: \"d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba2227d-jl6q8" Nov 03 20:01:51.679734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:51.649893 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dba2227d-jl6q8\" (UID: \"d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba2227d-jl6q8" Nov 03 20:01:51.679734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:51.667525 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dba2227d-jl6q8\" (UID: \"d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba2227d-jl6q8" Nov 03 20:01:51.787752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:51.772825 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:51.787752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:51.774810 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 20:01:51.787752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:51.776138 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:51.787752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:51.778549 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:01:51.851720 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:51.845697 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:51.897541 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:51.896175 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:51.924904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:51.924866 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:01:52.183051 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:52.182998 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:52.332687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:52.328904 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 20:01:52.332687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:52.329017 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" podUID="0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 20:01:52.368647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:52.368611 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 20:01:52.415962 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:52.415844 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dba2227d-jl6q8\" (UID: \"d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba2227d-jl6q8" Nov 03 20:01:52.462062 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:52.458803 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:01:52.607468 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:52.575635 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h86t\" (UniqueName: \"kubernetes.io/projected/d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c-kube-api-access-9h86t\") pod \"maas-api-revoke-sub-dba2227d-jl6q8\" (UID: \"d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba2227d-jl6q8" Nov 03 20:01:52.687003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:52.686844 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba2227d-jl6q8" Nov 03 20:01:52.773829 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:52.773739 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:52.850008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:52.795628 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:52.924847 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:52.874526 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 20:01:52.924847 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:52.809021 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:01:52.924847 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:52.908175 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 20:01:53.168685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:53.167997 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": context deadline exceeded" Nov 03 20:01:53.379694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:53.349527 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:53.450737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:53.447860 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 20:01:53.450737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:53.447926 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" podUID="0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 20:01:53.450737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:53.449317 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 20:01:53.458876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:53.458835 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:01:53.538716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:53.536492 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 20:01:53.542292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:53.542269 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 20:01:53.628258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:53.628232 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 20:01:53.806812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:53.806349 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba2227d-jl6q8"] Nov 03 20:01:53.808823 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:01:53.808791 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b95e13_e1c1_4ae7_bddf_9c05687f3f5c.slice/crio-7e99b8690d56c3de07338520ee3a4fb1234056c0802784d93edbe6477d74a604 WatchSource:0}: Error finding container 7e99b8690d56c3de07338520ee3a4fb1234056c0802784d93edbe6477d74a604: Status 404 returned error can't find the container with id 7e99b8690d56c3de07338520ee3a4fb1234056c0802784d93edbe6477d74a604 Nov 03 20:01:54.260576 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:54.260034 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 20:01:54.260576 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:54.260111 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" podUID="0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 20:01:54.504362 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:54.504273 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba2227d-jl6q8" event={"ID":"d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c","Type":"ContainerStarted","Data":"5c3b5c41fcd1e2c40dd974c0b7f58f69911d7723950cc20f06b1ae6b464a85f0"} Nov 03 20:01:54.504362 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:54.504331 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba2227d-jl6q8" event={"ID":"d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c","Type":"ContainerStarted","Data":"7e99b8690d56c3de07338520ee3a4fb1234056c0802784d93edbe6477d74a604"} Nov 03 20:01:54.525998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:54.525952 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba2227d-jl6q8" podStartSLOduration=3.525938118 podStartE2EDuration="3.525938118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:54.522147329 +0000 UTC m=+1508.779944598" watchObservedRunningTime="2026-11-03 20:01:54.525938118 +0000 UTC m=+1508.783735374" Nov 03 20:01:55.269741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:55.262979 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 20:01:55.269741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:55.263052 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" podUID="0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 20:01:55.657392 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:55.655102 5059 generic.go:356] "Generic (PLEG): container finished" podID="d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c" containerID="5c3b5c41fcd1e2c40dd974c0b7f58f69911d7723950cc20f06b1ae6b464a85f0" exitCode=0 Nov 03 20:01:55.657392 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:55.655345 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba2227d-jl6q8" event={"ID":"d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c","Type":"ContainerDied","Data":"5c3b5c41fcd1e2c40dd974c0b7f58f69911d7723950cc20f06b1ae6b464a85f0"} Nov 03 20:01:56.262688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:56.262037 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 20:01:56.262688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:56.262102 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" podUID="0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 20:01:57.085804 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:57.085359 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba2227d-jl6q8" Nov 03 20:01:57.210411 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:57.210377 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c-maas-api-xks-ca\") pod \"d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c\" (UID: \"d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c\") " Nov 03 20:01:57.210590 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:57.210461 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c-kube-api-access-9h86t\" (UniqueName: \"kubernetes.io/projected/d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c-kube-api-access-9h86t\") pod \"d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c\" (UID: \"d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c\") " Nov 03 20:01:57.210645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:57.210625 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c-maas-api-openshift-ca\") pod \"d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c\" (UID: \"d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c\") " Nov 03 20:01:57.210998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:57.210968 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c-maas-api-openshift-ca" pod "d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c" (UID: "d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:01:57.211244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:57.211226 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:57.212600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:57.212570 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c-maas-api-xks-ca" pod "d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c" (UID: "d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:01:57.212707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:57.212629 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c-kube-api-access-9h86t" pod "d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c" (UID: "d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c"). InnerVolumeSpecName "kube-api-access-9h86t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:57.282707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:57.266998 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 20:01:57.282707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:57.267055 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" podUID="0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 20:01:57.312065 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:57.312000 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:57.312065 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:57.312024 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h86t\" (UniqueName: \"kubernetes.io/projected/d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c-kube-api-access-9h86t\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:57.703253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:57.703176 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba2227d-jl6q8" event={"ID":"d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c","Type":"ContainerDied","Data":"7e99b8690d56c3de07338520ee3a4fb1234056c0802784d93edbe6477d74a604"} Nov 03 20:01:57.703253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:57.703230 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e99b8690d56c3de07338520ee3a4fb1234056c0802784d93edbe6477d74a604" Nov 03 20:01:57.703477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:57.703466 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba2227d-jl6q8" Nov 03 20:01:58.260069 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:58.260036 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 20:01:58.260486 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:58.260089 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" podUID="0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 20:01:58.816341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:58.816305 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s"] Nov 03 20:01:58.816807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:58.816594 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" podUID="7ad642be-b0b8-4a68-9392-ec28c87264a3" containerName="istio-proxy" containerID="cri-o://f57104699dec2690120dbe503676167d6c6ecc3dab6a2e4344002cc2e3cd3d5a" gracePeriod=30 Nov 03 20:01:59.605858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:59.605818 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:59.606390 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:59.606371 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:01:59.607190 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:01:59.607168 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:02:01.178970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:01.178932 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-861c2614-thcbf"] Nov 03 20:02:01.182294 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:01.182267 5059 cpu_manager.go:414] "containerMap: removing container" podUID="d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c" containerName="revoke-keys" Nov 03 20:02:01.182489 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:01.182464 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c" containerName="revoke-keys" Nov 03 20:02:01.183269 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:01.182785 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c" containerName="revoke-keys" Nov 03 20:02:01.188146 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:01.188118 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861c2614-thcbf" Nov 03 20:02:01.193503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:01.193421 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-861c2614-thcbf"] Nov 03 20:02:01.198127 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:01.196700 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:02:01.198127 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:01.196737 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:02:01.300690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:01.297835 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/973e14b6-7688-4f59-83b0-f353eb44a382-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-861c2614-thcbf\" (UID: \"973e14b6-7688-4f59-83b0-f353eb44a382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861c2614-thcbf" Nov 03 20:02:01.300690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:01.297928 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/973e14b6-7688-4f59-83b0-f353eb44a382-maas-api-xks-ca\") pod \"maas-api-revoke-sub-861c2614-thcbf\" (UID: \"973e14b6-7688-4f59-83b0-f353eb44a382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861c2614-thcbf" Nov 03 20:02:01.300690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:01.298124 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcwz7\" (UniqueName: \"kubernetes.io/projected/973e14b6-7688-4f59-83b0-f353eb44a382-kube-api-access-bcwz7\") pod \"maas-api-revoke-sub-861c2614-thcbf\" (UID: \"973e14b6-7688-4f59-83b0-f353eb44a382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861c2614-thcbf" Nov 03 20:02:01.399603 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:01.399571 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/973e14b6-7688-4f59-83b0-f353eb44a382-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-861c2614-thcbf\" (UID: \"973e14b6-7688-4f59-83b0-f353eb44a382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861c2614-thcbf" Nov 03 20:02:01.399815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:01.399616 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/973e14b6-7688-4f59-83b0-f353eb44a382-maas-api-xks-ca\") pod \"maas-api-revoke-sub-861c2614-thcbf\" (UID: \"973e14b6-7688-4f59-83b0-f353eb44a382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861c2614-thcbf" Nov 03 20:02:01.399815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:01.399767 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcwz7\" (UniqueName: \"kubernetes.io/projected/973e14b6-7688-4f59-83b0-f353eb44a382-kube-api-access-bcwz7\") pod \"maas-api-revoke-sub-861c2614-thcbf\" (UID: \"973e14b6-7688-4f59-83b0-f353eb44a382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861c2614-thcbf" Nov 03 20:02:01.400265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:01.400241 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/973e14b6-7688-4f59-83b0-f353eb44a382-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-861c2614-thcbf\" (UID: \"973e14b6-7688-4f59-83b0-f353eb44a382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861c2614-thcbf" Nov 03 20:02:01.402043 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:01.402020 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/973e14b6-7688-4f59-83b0-f353eb44a382-maas-api-xks-ca\") pod \"maas-api-revoke-sub-861c2614-thcbf\" (UID: \"973e14b6-7688-4f59-83b0-f353eb44a382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861c2614-thcbf" Nov 03 20:02:01.412426 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:01.412395 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcwz7\" (UniqueName: \"kubernetes.io/projected/973e14b6-7688-4f59-83b0-f353eb44a382-kube-api-access-bcwz7\") pod \"maas-api-revoke-sub-861c2614-thcbf\" (UID: \"973e14b6-7688-4f59-83b0-f353eb44a382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861c2614-thcbf" Nov 03 20:02:01.509487 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:01.509419 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861c2614-thcbf" Nov 03 20:02:01.794288 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:01.794256 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-861c2614-thcbf"] Nov 03 20:02:01.794434 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:02:01.794404 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod973e14b6_7688_4f59_83b0_f353eb44a382.slice/crio-1c98359b53aa312bdc072f594205464ebfc5d8b11091d88d1e79c69f2e466a63 WatchSource:0}: Error finding container 1c98359b53aa312bdc072f594205464ebfc5d8b11091d88d1e79c69f2e466a63: Status 404 returned error can't find the container with id 1c98359b53aa312bdc072f594205464ebfc5d8b11091d88d1e79c69f2e466a63 Nov 03 20:02:02.656814 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:02.653429 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-262d413b-jgbwg"] Nov 03 20:02:02.665964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:02.665937 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-262d413b-jgbwg" Nov 03 20:02:02.677785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:02.677750 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-262d413b-jgbwg"] Nov 03 20:02:02.725887 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:02.725853 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v22d\" (UniqueName: \"kubernetes.io/projected/a93f5c75-f751-4643-9908-04c1bd453e20-kube-api-access-6v22d\") pod \"maas-api-revoke-sub-262d413b-jgbwg\" (UID: \"a93f5c75-f751-4643-9908-04c1bd453e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-262d413b-jgbwg" Nov 03 20:02:02.726045 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:02.725994 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93f5c75-f751-4643-9908-04c1bd453e20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-262d413b-jgbwg\" (UID: \"a93f5c75-f751-4643-9908-04c1bd453e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-262d413b-jgbwg" Nov 03 20:02:02.726140 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:02.726070 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93f5c75-f751-4643-9908-04c1bd453e20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-262d413b-jgbwg\" (UID: \"a93f5c75-f751-4643-9908-04c1bd453e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-262d413b-jgbwg" Nov 03 20:02:02.746059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:02.746030 5059 generic.go:356] "Generic (PLEG): container finished" podID="973e14b6-7688-4f59-83b0-f353eb44a382" containerID="89ff664338bb927f838ffab3dc041ca737f5a8bdac000bec04e282765d3cd95c" exitCode=0 Nov 03 20:02:02.746219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:02.746115 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861c2614-thcbf" event={"ID":"973e14b6-7688-4f59-83b0-f353eb44a382","Type":"ContainerDied","Data":"89ff664338bb927f838ffab3dc041ca737f5a8bdac000bec04e282765d3cd95c"} Nov 03 20:02:02.746219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:02.746186 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861c2614-thcbf" event={"ID":"973e14b6-7688-4f59-83b0-f353eb44a382","Type":"ContainerStarted","Data":"1c98359b53aa312bdc072f594205464ebfc5d8b11091d88d1e79c69f2e466a63"} Nov 03 20:02:02.827367 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:02.827336 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v22d\" (UniqueName: \"kubernetes.io/projected/a93f5c75-f751-4643-9908-04c1bd453e20-kube-api-access-6v22d\") pod \"maas-api-revoke-sub-262d413b-jgbwg\" (UID: \"a93f5c75-f751-4643-9908-04c1bd453e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-262d413b-jgbwg" Nov 03 20:02:02.827511 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:02.827449 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93f5c75-f751-4643-9908-04c1bd453e20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-262d413b-jgbwg\" (UID: \"a93f5c75-f751-4643-9908-04c1bd453e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-262d413b-jgbwg" Nov 03 20:02:02.827511 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:02.827503 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93f5c75-f751-4643-9908-04c1bd453e20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-262d413b-jgbwg\" (UID: \"a93f5c75-f751-4643-9908-04c1bd453e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-262d413b-jgbwg" Nov 03 20:02:02.828230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:02.828205 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93f5c75-f751-4643-9908-04c1bd453e20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-262d413b-jgbwg\" (UID: \"a93f5c75-f751-4643-9908-04c1bd453e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-262d413b-jgbwg" Nov 03 20:02:02.829966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:02.829937 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93f5c75-f751-4643-9908-04c1bd453e20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-262d413b-jgbwg\" (UID: \"a93f5c75-f751-4643-9908-04c1bd453e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-262d413b-jgbwg" Nov 03 20:02:02.839622 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:02.839597 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v22d\" (UniqueName: \"kubernetes.io/projected/a93f5c75-f751-4643-9908-04c1bd453e20-kube-api-access-6v22d\") pod \"maas-api-revoke-sub-262d413b-jgbwg\" (UID: \"a93f5c75-f751-4643-9908-04c1bd453e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-262d413b-jgbwg" Nov 03 20:02:03.028657 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:03.028581 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-262d413b-jgbwg" Nov 03 20:02:03.358686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:03.358036 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-262d413b-jgbwg"] Nov 03 20:02:03.368567 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:02:03.368535 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93f5c75_f751_4643_9908_04c1bd453e20.slice/crio-10cdd25af50ab22454fdb1e708ba2b8340e5d1a474bad579cf43b33abebac9e1 WatchSource:0}: Error finding container 10cdd25af50ab22454fdb1e708ba2b8340e5d1a474bad579cf43b33abebac9e1: Status 404 returned error can't find the container with id 10cdd25af50ab22454fdb1e708ba2b8340e5d1a474bad579cf43b33abebac9e1 Nov 03 20:02:03.754266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:03.754233 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-262d413b-jgbwg" event={"ID":"a93f5c75-f751-4643-9908-04c1bd453e20","Type":"ContainerStarted","Data":"1e8b5ec575cf8860e615ae6e140ec0e37a39068e5a9285a1544505d7b1b326e6"} Nov 03 20:02:03.754749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:03.754285 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-262d413b-jgbwg" event={"ID":"a93f5c75-f751-4643-9908-04c1bd453e20","Type":"ContainerStarted","Data":"10cdd25af50ab22454fdb1e708ba2b8340e5d1a474bad579cf43b33abebac9e1"} Nov 03 20:02:03.774593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:03.774547 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-262d413b-jgbwg" podStartSLOduration=1.774532459 podStartE2EDuration="1.774532459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:03.772836856 +0000 UTC m=+1518.030634113" watchObservedRunningTime="2026-11-03 20:02:03.774532459 +0000 UTC m=+1518.032329715" Nov 03 20:02:03.947246 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:03.947215 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861c2614-thcbf" Nov 03 20:02:03.947699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:03.947673 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/973e14b6-7688-4f59-83b0-f353eb44a382-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/973e14b6-7688-4f59-83b0-f353eb44a382-maas-api-openshift-ca\") pod \"973e14b6-7688-4f59-83b0-f353eb44a382\" (UID: \"973e14b6-7688-4f59-83b0-f353eb44a382\") " Nov 03 20:02:03.947803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:03.947733 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/973e14b6-7688-4f59-83b0-f353eb44a382-kube-api-access-bcwz7\" (UniqueName: \"kubernetes.io/projected/973e14b6-7688-4f59-83b0-f353eb44a382-kube-api-access-bcwz7\") pod \"973e14b6-7688-4f59-83b0-f353eb44a382\" (UID: \"973e14b6-7688-4f59-83b0-f353eb44a382\") " Nov 03 20:02:03.948067 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:03.948029 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/973e14b6-7688-4f59-83b0-f353eb44a382-maas-api-openshift-ca" pod "973e14b6-7688-4f59-83b0-f353eb44a382" (UID: "973e14b6-7688-4f59-83b0-f353eb44a382"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:03.948197 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:03.948181 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/973e14b6-7688-4f59-83b0-f353eb44a382-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:03.949905 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:03.949875 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/973e14b6-7688-4f59-83b0-f353eb44a382-kube-api-access-bcwz7" pod "973e14b6-7688-4f59-83b0-f353eb44a382" (UID: "973e14b6-7688-4f59-83b0-f353eb44a382"). InnerVolumeSpecName "kube-api-access-bcwz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:04.063291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.063256 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/973e14b6-7688-4f59-83b0-f353eb44a382-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/973e14b6-7688-4f59-83b0-f353eb44a382-maas-api-xks-ca\") pod \"973e14b6-7688-4f59-83b0-f353eb44a382\" (UID: \"973e14b6-7688-4f59-83b0-f353eb44a382\") " Nov 03 20:02:04.064293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.064264 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcwz7\" (UniqueName: \"kubernetes.io/projected/973e14b6-7688-4f59-83b0-f353eb44a382-kube-api-access-bcwz7\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:04.066464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.066191 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/973e14b6-7688-4f59-83b0-f353eb44a382-maas-api-xks-ca" pod "973e14b6-7688-4f59-83b0-f353eb44a382" (UID: "973e14b6-7688-4f59-83b0-f353eb44a382"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:04.149752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.149725 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 20:02:04.165838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.165814 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ad642be-b0b8-4a68-9392-ec28c87264a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad642be-b0b8-4a68-9392-ec28c87264a3-istiod-ca-cert\") pod \"7ad642be-b0b8-4a68-9392-ec28c87264a3\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " Nov 03 20:02:04.165973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.165867 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-token\" (UniqueName: \"kubernetes.io/projected/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-token\") pod \"7ad642be-b0b8-4a68-9392-ec28c87264a3\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " Nov 03 20:02:04.165973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.165965 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-credential-socket\") pod \"7ad642be-b0b8-4a68-9392-ec28c87264a3\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " Nov 03 20:02:04.166055 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.166028 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad642be-b0b8-4a68-9392-ec28c87264a3-kube-api-access-lfn4t\" (UniqueName: \"kubernetes.io/projected/7ad642be-b0b8-4a68-9392-ec28c87264a3-kube-api-access-lfn4t\") pod \"7ad642be-b0b8-4a68-9392-ec28c87264a3\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " Nov 03 20:02:04.166139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.166126 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-podinfo\") pod \"7ad642be-b0b8-4a68-9392-ec28c87264a3\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " Nov 03 20:02:04.166215 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.166163 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-workload-certs\") pod \"7ad642be-b0b8-4a68-9392-ec28c87264a3\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " Nov 03 20:02:04.166276 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.166212 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-data\") pod \"7ad642be-b0b8-4a68-9392-ec28c87264a3\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " Nov 03 20:02:04.166276 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.166255 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-credential-socket" pod "7ad642be-b0b8-4a68-9392-ec28c87264a3" (UID: "7ad642be-b0b8-4a68-9392-ec28c87264a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:04.166386 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.166275 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad642be-b0b8-4a68-9392-ec28c87264a3-istiod-ca-cert" pod "7ad642be-b0b8-4a68-9392-ec28c87264a3" (UID: "7ad642be-b0b8-4a68-9392-ec28c87264a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:04.166386 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.166315 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-workload-socket\") pod \"7ad642be-b0b8-4a68-9392-ec28c87264a3\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " Nov 03 20:02:04.166386 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.166346 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-envoy\") pod \"7ad642be-b0b8-4a68-9392-ec28c87264a3\" (UID: \"7ad642be-b0b8-4a68-9392-ec28c87264a3\") " Nov 03 20:02:04.166552 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.166509 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-workload-certs" pod "7ad642be-b0b8-4a68-9392-ec28c87264a3" (UID: "7ad642be-b0b8-4a68-9392-ec28c87264a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:04.166552 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.166537 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-data" pod "7ad642be-b0b8-4a68-9392-ec28c87264a3" (UID: "7ad642be-b0b8-4a68-9392-ec28c87264a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:04.166677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.166555 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-workload-socket" pod "7ad642be-b0b8-4a68-9392-ec28c87264a3" (UID: "7ad642be-b0b8-4a68-9392-ec28c87264a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:04.166962 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.166937 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:04.167049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.166968 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/973e14b6-7688-4f59-83b0-f353eb44a382-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:04.167049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.166983 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:04.167049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.166997 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:04.167049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.167012 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:04.167049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.167026 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad642be-b0b8-4a68-9392-ec28c87264a3-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:04.168110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.168085 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-token" pod "7ad642be-b0b8-4a68-9392-ec28c87264a3" (UID: "7ad642be-b0b8-4a68-9392-ec28c87264a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:04.168360 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.168341 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad642be-b0b8-4a68-9392-ec28c87264a3-kube-api-access-lfn4t" pod "7ad642be-b0b8-4a68-9392-ec28c87264a3" (UID: "7ad642be-b0b8-4a68-9392-ec28c87264a3"). InnerVolumeSpecName "kube-api-access-lfn4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:04.168429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.168414 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-podinfo" pod "7ad642be-b0b8-4a68-9392-ec28c87264a3" (UID: "7ad642be-b0b8-4a68-9392-ec28c87264a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:02:04.168589 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.168538 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-envoy" pod "7ad642be-b0b8-4a68-9392-ec28c87264a3" (UID: "7ad642be-b0b8-4a68-9392-ec28c87264a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:04.268631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.268603 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:04.268631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.268628 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:04.268863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.268638 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfn4t\" (UniqueName: \"kubernetes.io/projected/7ad642be-b0b8-4a68-9392-ec28c87264a3-kube-api-access-lfn4t\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:04.268863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.268648 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad642be-b0b8-4a68-9392-ec28c87264a3-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:04.772820 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.772777 5059 generic.go:356] "Generic (PLEG): container finished" podID="7ad642be-b0b8-4a68-9392-ec28c87264a3" containerID="f57104699dec2690120dbe503676167d6c6ecc3dab6a2e4344002cc2e3cd3d5a" exitCode=0 Nov 03 20:02:04.773312 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.772836 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" Nov 03 20:02:04.773312 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.772867 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" event={"ID":"7ad642be-b0b8-4a68-9392-ec28c87264a3","Type":"ContainerDied","Data":"f57104699dec2690120dbe503676167d6c6ecc3dab6a2e4344002cc2e3cd3d5a"} Nov 03 20:02:04.773312 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.772961 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s" event={"ID":"7ad642be-b0b8-4a68-9392-ec28c87264a3","Type":"ContainerDied","Data":"96d996aaa01afe9809b705054a293f987e93df8eca20bda1eed713607b99fc7b"} Nov 03 20:02:04.773312 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.772990 5059 scope.go:122] "RemoveContainer" containerID="f57104699dec2690120dbe503676167d6c6ecc3dab6a2e4344002cc2e3cd3d5a" Nov 03 20:02:04.774990 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.774956 5059 generic.go:356] "Generic (PLEG): container finished" podID="a93f5c75-f751-4643-9908-04c1bd453e20" containerID="1e8b5ec575cf8860e615ae6e140ec0e37a39068e5a9285a1544505d7b1b326e6" exitCode=0 Nov 03 20:02:04.775201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.775174 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-262d413b-jgbwg" event={"ID":"a93f5c75-f751-4643-9908-04c1bd453e20","Type":"ContainerDied","Data":"1e8b5ec575cf8860e615ae6e140ec0e37a39068e5a9285a1544505d7b1b326e6"} Nov 03 20:02:04.776551 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.776529 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861c2614-thcbf" event={"ID":"973e14b6-7688-4f59-83b0-f353eb44a382","Type":"ContainerDied","Data":"1c98359b53aa312bdc072f594205464ebfc5d8b11091d88d1e79c69f2e466a63"} Nov 03 20:02:04.776644 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.776572 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c98359b53aa312bdc072f594205464ebfc5d8b11091d88d1e79c69f2e466a63" Nov 03 20:02:04.776644 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.776616 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861c2614-thcbf" Nov 03 20:02:04.788305 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.788284 5059 scope.go:122] "RemoveContainer" containerID="70e22e24b3a0416be1693328143270603f0b640b05256d39bbdc39f123cbedc9" Nov 03 20:02:04.805348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.805324 5059 scope.go:122] "RemoveContainer" containerID="f57104699dec2690120dbe503676167d6c6ecc3dab6a2e4344002cc2e3cd3d5a" Nov 03 20:02:04.805696 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:02:04.805656 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f57104699dec2690120dbe503676167d6c6ecc3dab6a2e4344002cc2e3cd3d5a\": container with ID starting with f57104699dec2690120dbe503676167d6c6ecc3dab6a2e4344002cc2e3cd3d5a not found: ID does not exist" containerID="f57104699dec2690120dbe503676167d6c6ecc3dab6a2e4344002cc2e3cd3d5a" Nov 03 20:02:04.805771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.805711 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f57104699dec2690120dbe503676167d6c6ecc3dab6a2e4344002cc2e3cd3d5a"} err="failed to get container status \"f57104699dec2690120dbe503676167d6c6ecc3dab6a2e4344002cc2e3cd3d5a\": rpc error: code = NotFound desc = could not find container \"f57104699dec2690120dbe503676167d6c6ecc3dab6a2e4344002cc2e3cd3d5a\": container with ID starting with f57104699dec2690120dbe503676167d6c6ecc3dab6a2e4344002cc2e3cd3d5a not found: ID does not exist" Nov 03 20:02:04.805771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.805729 5059 scope.go:122] "RemoveContainer" containerID="70e22e24b3a0416be1693328143270603f0b640b05256d39bbdc39f123cbedc9" Nov 03 20:02:04.806024 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:02:04.806004 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70e22e24b3a0416be1693328143270603f0b640b05256d39bbdc39f123cbedc9\": container with ID starting with 70e22e24b3a0416be1693328143270603f0b640b05256d39bbdc39f123cbedc9 not found: ID does not exist" containerID="70e22e24b3a0416be1693328143270603f0b640b05256d39bbdc39f123cbedc9" Nov 03 20:02:04.806088 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.806029 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70e22e24b3a0416be1693328143270603f0b640b05256d39bbdc39f123cbedc9"} err="failed to get container status \"70e22e24b3a0416be1693328143270603f0b640b05256d39bbdc39f123cbedc9\": rpc error: code = NotFound desc = could not find container \"70e22e24b3a0416be1693328143270603f0b640b05256d39bbdc39f123cbedc9\": container with ID starting with 70e22e24b3a0416be1693328143270603f0b640b05256d39bbdc39f123cbedc9 not found: ID does not exist" Nov 03 20:02:04.826395 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.826366 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s"] Nov 03 20:02:04.830419 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:04.830395 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-f493f7-openshift-default-f8bdc57b9-bwd5s"] Nov 03 20:02:05.481032 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.481001 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3a5d746-4h8f4"] Nov 03 20:02:05.481686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.481653 5059 cpu_manager.go:414] "containerMap: removing container" podUID="7ad642be-b0b8-4a68-9392-ec28c87264a3" containerName="istio-proxy" Nov 03 20:02:05.481739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.481686 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ad642be-b0b8-4a68-9392-ec28c87264a3" containerName="istio-proxy" Nov 03 20:02:05.481739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.481725 5059 cpu_manager.go:414] "containerMap: removing container" podUID="7ad642be-b0b8-4a68-9392-ec28c87264a3" containerName="istio-proxy" Nov 03 20:02:05.481739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.481732 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ad642be-b0b8-4a68-9392-ec28c87264a3" containerName="istio-proxy" Nov 03 20:02:05.481832 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.481769 5059 cpu_manager.go:414] "containerMap: removing container" podUID="973e14b6-7688-4f59-83b0-f353eb44a382" containerName="revoke-keys" Nov 03 20:02:05.481832 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.481776 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="973e14b6-7688-4f59-83b0-f353eb44a382" containerName="revoke-keys" Nov 03 20:02:05.481934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.481923 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="973e14b6-7688-4f59-83b0-f353eb44a382" containerName="revoke-keys" Nov 03 20:02:05.481967 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.481943 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ad642be-b0b8-4a68-9392-ec28c87264a3" containerName="istio-proxy" Nov 03 20:02:05.481967 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.481956 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ad642be-b0b8-4a68-9392-ec28c87264a3" containerName="istio-proxy" Nov 03 20:02:05.487105 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.487087 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a5d746-4h8f4" Nov 03 20:02:05.493701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.493643 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3a5d746-4h8f4"] Nov 03 20:02:05.583949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.583914 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e469f89a-632b-43ab-883a-645ab4576243-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3a5d746-4h8f4\" (UID: \"e469f89a-632b-43ab-883a-645ab4576243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a5d746-4h8f4" Nov 03 20:02:05.584138 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.584045 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc6lh\" (UniqueName: \"kubernetes.io/projected/e469f89a-632b-43ab-883a-645ab4576243-kube-api-access-rc6lh\") pod \"maas-api-revoke-sub-c3a5d746-4h8f4\" (UID: \"e469f89a-632b-43ab-883a-645ab4576243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a5d746-4h8f4" Nov 03 20:02:05.584138 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.584123 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e469f89a-632b-43ab-883a-645ab4576243-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3a5d746-4h8f4\" (UID: \"e469f89a-632b-43ab-883a-645ab4576243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a5d746-4h8f4" Nov 03 20:02:05.685423 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.685388 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e469f89a-632b-43ab-883a-645ab4576243-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3a5d746-4h8f4\" (UID: \"e469f89a-632b-43ab-883a-645ab4576243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a5d746-4h8f4" Nov 03 20:02:05.685590 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.685517 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e469f89a-632b-43ab-883a-645ab4576243-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3a5d746-4h8f4\" (UID: \"e469f89a-632b-43ab-883a-645ab4576243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a5d746-4h8f4" Nov 03 20:02:05.685590 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.685580 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc6lh\" (UniqueName: \"kubernetes.io/projected/e469f89a-632b-43ab-883a-645ab4576243-kube-api-access-rc6lh\") pod \"maas-api-revoke-sub-c3a5d746-4h8f4\" (UID: \"e469f89a-632b-43ab-883a-645ab4576243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a5d746-4h8f4" Nov 03 20:02:05.686129 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.686106 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e469f89a-632b-43ab-883a-645ab4576243-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3a5d746-4h8f4\" (UID: \"e469f89a-632b-43ab-883a-645ab4576243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a5d746-4h8f4" Nov 03 20:02:05.687919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.687896 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e469f89a-632b-43ab-883a-645ab4576243-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3a5d746-4h8f4\" (UID: \"e469f89a-632b-43ab-883a-645ab4576243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a5d746-4h8f4" Nov 03 20:02:05.700774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.700743 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc6lh\" (UniqueName: \"kubernetes.io/projected/e469f89a-632b-43ab-883a-645ab4576243-kube-api-access-rc6lh\") pod \"maas-api-revoke-sub-c3a5d746-4h8f4\" (UID: \"e469f89a-632b-43ab-883a-645ab4576243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a5d746-4h8f4" Nov 03 20:02:05.806245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.806218 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a5d746-4h8f4" Nov 03 20:02:05.892301 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.892268 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ad642be-b0b8-4a68-9392-ec28c87264a3" path="/var/lib/kubelet/pods/7ad642be-b0b8-4a68-9392-ec28c87264a3/volumes" Nov 03 20:02:05.945036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.945008 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-262d413b-jgbwg" Nov 03 20:02:05.989927 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.989894 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a93f5c75-f751-4643-9908-04c1bd453e20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93f5c75-f751-4643-9908-04c1bd453e20-maas-api-openshift-ca\") pod \"a93f5c75-f751-4643-9908-04c1bd453e20\" (UID: \"a93f5c75-f751-4643-9908-04c1bd453e20\") " Nov 03 20:02:05.990081 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.990062 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a93f5c75-f751-4643-9908-04c1bd453e20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93f5c75-f751-4643-9908-04c1bd453e20-maas-api-xks-ca\") pod \"a93f5c75-f751-4643-9908-04c1bd453e20\" (UID: \"a93f5c75-f751-4643-9908-04c1bd453e20\") " Nov 03 20:02:05.990156 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.990133 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a93f5c75-f751-4643-9908-04c1bd453e20-kube-api-access-6v22d\" (UniqueName: \"kubernetes.io/projected/a93f5c75-f751-4643-9908-04c1bd453e20-kube-api-access-6v22d\") pod \"a93f5c75-f751-4643-9908-04c1bd453e20\" (UID: \"a93f5c75-f751-4643-9908-04c1bd453e20\") " Nov 03 20:02:05.990316 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.990294 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a93f5c75-f751-4643-9908-04c1bd453e20-maas-api-openshift-ca" pod "a93f5c75-f751-4643-9908-04c1bd453e20" (UID: "a93f5c75-f751-4643-9908-04c1bd453e20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:05.990546 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.990528 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93f5c75-f751-4643-9908-04c1bd453e20-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.992297 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.992271 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a93f5c75-f751-4643-9908-04c1bd453e20-maas-api-xks-ca" pod "a93f5c75-f751-4643-9908-04c1bd453e20" (UID: "a93f5c75-f751-4643-9908-04c1bd453e20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:05.992375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:05.992354 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93f5c75-f751-4643-9908-04c1bd453e20-kube-api-access-6v22d" pod "a93f5c75-f751-4643-9908-04c1bd453e20" (UID: "a93f5c75-f751-4643-9908-04c1bd453e20"). InnerVolumeSpecName "kube-api-access-6v22d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:06.092369 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:06.092343 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v22d\" (UniqueName: \"kubernetes.io/projected/a93f5c75-f751-4643-9908-04c1bd453e20-kube-api-access-6v22d\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:06.092511 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:06.092374 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93f5c75-f751-4643-9908-04c1bd453e20-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:06.102240 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:02:06.102215 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode469f89a_632b_43ab_883a_645ab4576243.slice/crio-6201e433e8c8d4ef55b31c6f929832ec9fc26ef064ec67784d74042de6d0f5f9 WatchSource:0}: Error finding container 6201e433e8c8d4ef55b31c6f929832ec9fc26ef064ec67784d74042de6d0f5f9: Status 404 returned error can't find the container with id 6201e433e8c8d4ef55b31c6f929832ec9fc26ef064ec67784d74042de6d0f5f9 Nov 03 20:02:06.102543 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:06.102520 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3a5d746-4h8f4"] Nov 03 20:02:06.793178 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:06.793133 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-262d413b-jgbwg" event={"ID":"a93f5c75-f751-4643-9908-04c1bd453e20","Type":"ContainerDied","Data":"10cdd25af50ab22454fdb1e708ba2b8340e5d1a474bad579cf43b33abebac9e1"} Nov 03 20:02:06.793359 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:06.793185 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-262d413b-jgbwg" Nov 03 20:02:06.793359 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:06.793223 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10cdd25af50ab22454fdb1e708ba2b8340e5d1a474bad579cf43b33abebac9e1" Nov 03 20:02:06.795027 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:06.795000 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a5d746-4h8f4" event={"ID":"e469f89a-632b-43ab-883a-645ab4576243","Type":"ContainerStarted","Data":"d564999636dfe02ab63256d324b4ff599b8034e06ec7444d1ea0dab9ae115e83"} Nov 03 20:02:06.795141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:06.795041 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a5d746-4h8f4" event={"ID":"e469f89a-632b-43ab-883a-645ab4576243","Type":"ContainerStarted","Data":"6201e433e8c8d4ef55b31c6f929832ec9fc26ef064ec67784d74042de6d0f5f9"} Nov 03 20:02:06.818255 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:06.818208 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a5d746-4h8f4" podStartSLOduration=1.818196555 podStartE2EDuration="1.818196555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:06.81603755 +0000 UTC m=+1521.073834807" watchObservedRunningTime="2026-11-03 20:02:06.818196555 +0000 UTC m=+1521.075993810" Nov 03 20:02:07.805256 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:07.805221 5059 generic.go:356] "Generic (PLEG): container finished" podID="e469f89a-632b-43ab-883a-645ab4576243" containerID="d564999636dfe02ab63256d324b4ff599b8034e06ec7444d1ea0dab9ae115e83" exitCode=0 Nov 03 20:02:07.805441 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:07.805305 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a5d746-4h8f4" event={"ID":"e469f89a-632b-43ab-883a-645ab4576243","Type":"ContainerDied","Data":"d564999636dfe02ab63256d324b4ff599b8034e06ec7444d1ea0dab9ae115e83"} Nov 03 20:02:09.758706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:09.758298 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a5d746-4h8f4" Nov 03 20:02:09.901778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:09.900549 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e469f89a-632b-43ab-883a-645ab4576243-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e469f89a-632b-43ab-883a-645ab4576243-maas-api-xks-ca\") pod \"e469f89a-632b-43ab-883a-645ab4576243\" (UID: \"e469f89a-632b-43ab-883a-645ab4576243\") " Nov 03 20:02:09.901778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:09.900682 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e469f89a-632b-43ab-883a-645ab4576243-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e469f89a-632b-43ab-883a-645ab4576243-maas-api-openshift-ca\") pod \"e469f89a-632b-43ab-883a-645ab4576243\" (UID: \"e469f89a-632b-43ab-883a-645ab4576243\") " Nov 03 20:02:09.901778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:09.900802 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e469f89a-632b-43ab-883a-645ab4576243-kube-api-access-rc6lh\" (UniqueName: \"kubernetes.io/projected/e469f89a-632b-43ab-883a-645ab4576243-kube-api-access-rc6lh\") pod \"e469f89a-632b-43ab-883a-645ab4576243\" (UID: \"e469f89a-632b-43ab-883a-645ab4576243\") " Nov 03 20:02:09.939065 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:09.939020 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e469f89a-632b-43ab-883a-645ab4576243-maas-api-openshift-ca" pod "e469f89a-632b-43ab-883a-645ab4576243" (UID: "e469f89a-632b-43ab-883a-645ab4576243"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:10.032754 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:10.012585 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e469f89a-632b-43ab-883a-645ab4576243-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:10.142262 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:10.142193 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e469f89a-632b-43ab-883a-645ab4576243-maas-api-xks-ca" pod "e469f89a-632b-43ab-883a-645ab4576243" (UID: "e469f89a-632b-43ab-883a-645ab4576243"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:10.165217 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:10.165178 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a5d746-4h8f4" event={"ID":"e469f89a-632b-43ab-883a-645ab4576243","Type":"ContainerDied","Data":"6201e433e8c8d4ef55b31c6f929832ec9fc26ef064ec67784d74042de6d0f5f9"} Nov 03 20:02:10.214431 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:10.214397 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6201e433e8c8d4ef55b31c6f929832ec9fc26ef064ec67784d74042de6d0f5f9" Nov 03 20:02:10.215885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:10.171990 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3a5d746-4h8f4" Nov 03 20:02:10.216153 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:10.215820 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e469f89a-632b-43ab-883a-645ab4576243-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:10.254207 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:10.254139 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e469f89a-632b-43ab-883a-645ab4576243-kube-api-access-rc6lh" pod "e469f89a-632b-43ab-883a-645ab4576243" (UID: "e469f89a-632b-43ab-883a-645ab4576243"). InnerVolumeSpecName "kube-api-access-rc6lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:10.356253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:10.356217 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc6lh\" (UniqueName: \"kubernetes.io/projected/e469f89a-632b-43ab-883a-645ab4576243-kube-api-access-rc6lh\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:10.464865 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:10.464786 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:10.614286 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:02:10.614179 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:02:12.908843 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:12.908811 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a932901-tlzmp"] Nov 03 20:02:12.909448 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:12.909434 5059 cpu_manager.go:414] "containerMap: removing container" podUID="a93f5c75-f751-4643-9908-04c1bd453e20" containerName="revoke-keys" Nov 03 20:02:12.909499 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:12.909449 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a93f5c75-f751-4643-9908-04c1bd453e20" containerName="revoke-keys" Nov 03 20:02:12.909499 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:12.909474 5059 cpu_manager.go:414] "containerMap: removing container" podUID="e469f89a-632b-43ab-883a-645ab4576243" containerName="revoke-keys" Nov 03 20:02:12.909499 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:12.909479 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e469f89a-632b-43ab-883a-645ab4576243" containerName="revoke-keys" Nov 03 20:02:12.909598 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:12.909579 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a93f5c75-f751-4643-9908-04c1bd453e20" containerName="revoke-keys" Nov 03 20:02:12.909598 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:12.909592 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e469f89a-632b-43ab-883a-645ab4576243" containerName="revoke-keys" Nov 03 20:02:12.913496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:12.913479 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a932901-tlzmp" Nov 03 20:02:12.918389 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:12.918364 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:02:12.918570 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:12.918425 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:02:12.923800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:12.923781 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a932901-tlzmp"] Nov 03 20:02:13.000150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:13.000122 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97ed753-ba37-4f08-ae9b-f8044a64b349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a932901-tlzmp\" (UID: \"b97ed753-ba37-4f08-ae9b-f8044a64b349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a932901-tlzmp" Nov 03 20:02:13.000294 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:13.000175 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nk8g\" (UniqueName: \"kubernetes.io/projected/b97ed753-ba37-4f08-ae9b-f8044a64b349-kube-api-access-9nk8g\") pod \"maas-api-revoke-sub-0a932901-tlzmp\" (UID: \"b97ed753-ba37-4f08-ae9b-f8044a64b349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a932901-tlzmp" Nov 03 20:02:13.000338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:13.000321 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97ed753-ba37-4f08-ae9b-f8044a64b349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a932901-tlzmp\" (UID: \"b97ed753-ba37-4f08-ae9b-f8044a64b349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a932901-tlzmp" Nov 03 20:02:13.101506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:13.101476 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97ed753-ba37-4f08-ae9b-f8044a64b349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a932901-tlzmp\" (UID: \"b97ed753-ba37-4f08-ae9b-f8044a64b349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a932901-tlzmp" Nov 03 20:02:13.101620 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:13.101523 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nk8g\" (UniqueName: \"kubernetes.io/projected/b97ed753-ba37-4f08-ae9b-f8044a64b349-kube-api-access-9nk8g\") pod \"maas-api-revoke-sub-0a932901-tlzmp\" (UID: \"b97ed753-ba37-4f08-ae9b-f8044a64b349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a932901-tlzmp" Nov 03 20:02:13.101620 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:13.101594 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97ed753-ba37-4f08-ae9b-f8044a64b349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a932901-tlzmp\" (UID: \"b97ed753-ba37-4f08-ae9b-f8044a64b349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a932901-tlzmp" Nov 03 20:02:13.102280 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:13.102259 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97ed753-ba37-4f08-ae9b-f8044a64b349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a932901-tlzmp\" (UID: \"b97ed753-ba37-4f08-ae9b-f8044a64b349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a932901-tlzmp" Nov 03 20:02:13.103895 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:13.103875 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97ed753-ba37-4f08-ae9b-f8044a64b349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a932901-tlzmp\" (UID: \"b97ed753-ba37-4f08-ae9b-f8044a64b349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a932901-tlzmp" Nov 03 20:02:13.113729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:13.113705 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nk8g\" (UniqueName: \"kubernetes.io/projected/b97ed753-ba37-4f08-ae9b-f8044a64b349-kube-api-access-9nk8g\") pod \"maas-api-revoke-sub-0a932901-tlzmp\" (UID: \"b97ed753-ba37-4f08-ae9b-f8044a64b349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a932901-tlzmp" Nov 03 20:02:13.232831 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:13.232757 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a932901-tlzmp" Nov 03 20:02:13.564424 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:13.564392 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a932901-tlzmp"] Nov 03 20:02:13.564554 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:02:13.564528 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97ed753_ba37_4f08_ae9b_f8044a64b349.slice/crio-9542b67c9bbe7c8bd4b21186fd88101dff02c5008b905abdf72ee41aa1764b5d WatchSource:0}: Error finding container 9542b67c9bbe7c8bd4b21186fd88101dff02c5008b905abdf72ee41aa1764b5d: Status 404 returned error can't find the container with id 9542b67c9bbe7c8bd4b21186fd88101dff02c5008b905abdf72ee41aa1764b5d Nov 03 20:02:14.196910 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:14.196871 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a932901-tlzmp" event={"ID":"b97ed753-ba37-4f08-ae9b-f8044a64b349","Type":"ContainerStarted","Data":"0fa8daa2bb513452f6e11d8859bcd5159b4218aad834fd86aebad08807b0b170"} Nov 03 20:02:14.197315 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:14.196936 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a932901-tlzmp" event={"ID":"b97ed753-ba37-4f08-ae9b-f8044a64b349","Type":"ContainerStarted","Data":"9542b67c9bbe7c8bd4b21186fd88101dff02c5008b905abdf72ee41aa1764b5d"} Nov 03 20:02:14.220772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:14.220723 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a932901-tlzmp" podStartSLOduration=2.220711563 podStartE2EDuration="2.220711563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:14.216453257 +0000 UTC m=+1528.474250513" watchObservedRunningTime="2026-11-03 20:02:14.220711563 +0000 UTC m=+1528.478508831" Nov 03 20:02:15.206686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:15.206622 5059 generic.go:356] "Generic (PLEG): container finished" podID="b97ed753-ba37-4f08-ae9b-f8044a64b349" containerID="0fa8daa2bb513452f6e11d8859bcd5159b4218aad834fd86aebad08807b0b170" exitCode=0 Nov 03 20:02:15.207077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:15.206712 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a932901-tlzmp" event={"ID":"b97ed753-ba37-4f08-ae9b-f8044a64b349","Type":"ContainerDied","Data":"0fa8daa2bb513452f6e11d8859bcd5159b4218aad834fd86aebad08807b0b170"} Nov 03 20:02:16.387756 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:16.387730 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a932901-tlzmp" Nov 03 20:02:16.455023 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:16.454987 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b97ed753-ba37-4f08-ae9b-f8044a64b349-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97ed753-ba37-4f08-ae9b-f8044a64b349-maas-api-openshift-ca\") pod \"b97ed753-ba37-4f08-ae9b-f8044a64b349\" (UID: \"b97ed753-ba37-4f08-ae9b-f8044a64b349\") " Nov 03 20:02:16.455190 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:16.455163 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b97ed753-ba37-4f08-ae9b-f8044a64b349-kube-api-access-9nk8g\" (UniqueName: \"kubernetes.io/projected/b97ed753-ba37-4f08-ae9b-f8044a64b349-kube-api-access-9nk8g\") pod \"b97ed753-ba37-4f08-ae9b-f8044a64b349\" (UID: \"b97ed753-ba37-4f08-ae9b-f8044a64b349\") " Nov 03 20:02:16.455308 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:16.455291 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b97ed753-ba37-4f08-ae9b-f8044a64b349-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97ed753-ba37-4f08-ae9b-f8044a64b349-maas-api-xks-ca\") pod \"b97ed753-ba37-4f08-ae9b-f8044a64b349\" (UID: \"b97ed753-ba37-4f08-ae9b-f8044a64b349\") " Nov 03 20:02:16.455442 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:16.455393 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97ed753-ba37-4f08-ae9b-f8044a64b349-maas-api-openshift-ca" pod "b97ed753-ba37-4f08-ae9b-f8044a64b349" (UID: "b97ed753-ba37-4f08-ae9b-f8044a64b349"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:16.455973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:16.455948 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97ed753-ba37-4f08-ae9b-f8044a64b349-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:16.457933 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:16.457902 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97ed753-ba37-4f08-ae9b-f8044a64b349-kube-api-access-9nk8g" pod "b97ed753-ba37-4f08-ae9b-f8044a64b349" (UID: "b97ed753-ba37-4f08-ae9b-f8044a64b349"). InnerVolumeSpecName "kube-api-access-9nk8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:16.458103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:16.458064 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b97ed753-ba37-4f08-ae9b-f8044a64b349-maas-api-xks-ca" pod "b97ed753-ba37-4f08-ae9b-f8044a64b349" (UID: "b97ed753-ba37-4f08-ae9b-f8044a64b349"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:16.572501 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:16.563902 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nk8g\" (UniqueName: \"kubernetes.io/projected/b97ed753-ba37-4f08-ae9b-f8044a64b349-kube-api-access-9nk8g\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:16.572501 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:16.563978 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97ed753-ba37-4f08-ae9b-f8044a64b349-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:17.109561 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.109521 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw"] Nov 03 20:02:17.110548 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.110527 5059 cpu_manager.go:414] "containerMap: removing container" podUID="b97ed753-ba37-4f08-ae9b-f8044a64b349" containerName="revoke-keys" Nov 03 20:02:17.110633 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.110551 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b97ed753-ba37-4f08-ae9b-f8044a64b349" containerName="revoke-keys" Nov 03 20:02:17.110736 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.110721 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b97ed753-ba37-4f08-ae9b-f8044a64b349" containerName="revoke-keys" Nov 03 20:02:17.115119 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.115096 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.120562 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.120541 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-c131fa-openshift-default-dockercfg-8qr2b\"" Nov 03 20:02:17.131602 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.131580 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw"] Nov 03 20:02:17.230198 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.230157 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a932901-tlzmp" Nov 03 20:02:17.230198 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.230185 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a932901-tlzmp" event={"ID":"b97ed753-ba37-4f08-ae9b-f8044a64b349","Type":"ContainerDied","Data":"9542b67c9bbe7c8bd4b21186fd88101dff02c5008b905abdf72ee41aa1764b5d"} Nov 03 20:02:17.230433 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.230241 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9542b67c9bbe7c8bd4b21186fd88101dff02c5008b905abdf72ee41aa1764b5d" Nov 03 20:02:17.272297 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.272257 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613a180b-7079-4633-8060-c4adb54db7f8-istio-podinfo\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.272460 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.272327 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-workload-certs\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.272460 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.272440 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-credential-socket\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.272567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.272488 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-istio-data\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.272610 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.272588 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-istio-envoy\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.272651 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.272636 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613a180b-7079-4633-8060-c4adb54db7f8-istiod-ca-cert\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.272810 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.272797 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/613a180b-7079-4633-8060-c4adb54db7f8-istio-token\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.272846 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.272821 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z6z4\" (UniqueName: \"kubernetes.io/projected/613a180b-7079-4633-8060-c4adb54db7f8-kube-api-access-8z6z4\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.272920 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.272908 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-workload-socket\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.379852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.377998 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-istio-envoy\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.379852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.378061 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613a180b-7079-4633-8060-c4adb54db7f8-istiod-ca-cert\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.379852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.378155 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/613a180b-7079-4633-8060-c4adb54db7f8-istio-token\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.379852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.378189 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z6z4\" (UniqueName: \"kubernetes.io/projected/613a180b-7079-4633-8060-c4adb54db7f8-kube-api-access-8z6z4\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.379852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.378263 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-workload-socket\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.379852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.378295 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613a180b-7079-4633-8060-c4adb54db7f8-istio-podinfo\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.379852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.378349 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-workload-certs\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.379852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.378436 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-credential-socket\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.379852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.378487 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-istio-data\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.403727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.395041 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-workload-socket\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.403727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.395319 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-workload-certs\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.403727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.395558 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-credential-socket\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.403727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.402896 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-istio-envoy\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.408683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.405392 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613a180b-7079-4633-8060-c4adb54db7f8-istiod-ca-cert\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.409453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.409429 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-istio-data\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.445689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.432325 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z6z4\" (UniqueName: \"kubernetes.io/projected/613a180b-7079-4633-8060-c4adb54db7f8-kube-api-access-8z6z4\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.446509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.446446 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/613a180b-7079-4633-8060-c4adb54db7f8-istio-token\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.470693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.468144 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613a180b-7079-4633-8060-c4adb54db7f8-istio-podinfo\") pod \"e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:17.735926 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:17.735842 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:18.034955 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:18.034913 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw"] Nov 03 20:02:18.035074 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:02:18.034932 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613a180b_7079_4633_8060_c4adb54db7f8.slice/crio-95dea34409cbd7d6df10caad28c5365ec8cd543ccffb944e62b0b559307dac2c WatchSource:0}: Error finding container 95dea34409cbd7d6df10caad28c5365ec8cd543ccffb944e62b0b559307dac2c: Status 404 returned error can't find the container with id 95dea34409cbd7d6df10caad28c5365ec8cd543ccffb944e62b0b559307dac2c Nov 03 20:02:18.037382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:18.037348 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:02:18.037469 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:18.037412 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:02:18.037469 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:18.037441 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:02:18.240496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:18.240456 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" event={"ID":"613a180b-7079-4633-8060-c4adb54db7f8","Type":"ContainerStarted","Data":"f1bfc585a41bbd3e04faf680f0d866afc990a6d0cc1f757a0ad49c4504dcfd97"} Nov 03 20:02:18.240676 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:18.240519 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" event={"ID":"613a180b-7079-4633-8060-c4adb54db7f8","Type":"ContainerStarted","Data":"95dea34409cbd7d6df10caad28c5365ec8cd543ccffb944e62b0b559307dac2c"} Nov 03 20:02:18.267949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:18.267897 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" podStartSLOduration=1.267883405 podStartE2EDuration="1.267883405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:18.263473607 +0000 UTC m=+1532.521270864" watchObservedRunningTime="2026-11-03 20:02:18.267883405 +0000 UTC m=+1532.525680697" Nov 03 20:02:18.736254 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:18.736223 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:19.004878 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:19.004766 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3968d0fb-lgt62"] Nov 03 20:02:19.011795 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:19.011757 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3968d0fb-lgt62" Nov 03 20:02:19.015403 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:19.015377 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3968d0fb-lgt62"] Nov 03 20:02:19.016062 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:19.016038 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:02:19.017203 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:19.017180 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:02:19.112920 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:19.112889 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6aaa4d2-402f-4605-a0f6-3af867f7fbf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3968d0fb-lgt62\" (UID: \"c6aaa4d2-402f-4605-a0f6-3af867f7fbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3968d0fb-lgt62" Nov 03 20:02:19.113098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:19.112954 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4spg6\" (UniqueName: \"kubernetes.io/projected/c6aaa4d2-402f-4605-a0f6-3af867f7fbf9-kube-api-access-4spg6\") pod \"maas-api-revoke-sub-3968d0fb-lgt62\" (UID: \"c6aaa4d2-402f-4605-a0f6-3af867f7fbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3968d0fb-lgt62" Nov 03 20:02:19.113098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:19.113064 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6aaa4d2-402f-4605-a0f6-3af867f7fbf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3968d0fb-lgt62\" (UID: \"c6aaa4d2-402f-4605-a0f6-3af867f7fbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3968d0fb-lgt62" Nov 03 20:02:19.214934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:19.214899 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6aaa4d2-402f-4605-a0f6-3af867f7fbf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3968d0fb-lgt62\" (UID: \"c6aaa4d2-402f-4605-a0f6-3af867f7fbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3968d0fb-lgt62" Nov 03 20:02:19.215134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:19.214958 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4spg6\" (UniqueName: \"kubernetes.io/projected/c6aaa4d2-402f-4605-a0f6-3af867f7fbf9-kube-api-access-4spg6\") pod \"maas-api-revoke-sub-3968d0fb-lgt62\" (UID: \"c6aaa4d2-402f-4605-a0f6-3af867f7fbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3968d0fb-lgt62" Nov 03 20:02:19.215134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:19.214991 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6aaa4d2-402f-4605-a0f6-3af867f7fbf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3968d0fb-lgt62\" (UID: \"c6aaa4d2-402f-4605-a0f6-3af867f7fbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3968d0fb-lgt62" Nov 03 20:02:19.215763 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:19.215741 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6aaa4d2-402f-4605-a0f6-3af867f7fbf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3968d0fb-lgt62\" (UID: \"c6aaa4d2-402f-4605-a0f6-3af867f7fbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3968d0fb-lgt62" Nov 03 20:02:19.217188 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:19.217163 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6aaa4d2-402f-4605-a0f6-3af867f7fbf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3968d0fb-lgt62\" (UID: \"c6aaa4d2-402f-4605-a0f6-3af867f7fbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3968d0fb-lgt62" Nov 03 20:02:19.228141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:19.228117 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4spg6\" (UniqueName: \"kubernetes.io/projected/c6aaa4d2-402f-4605-a0f6-3af867f7fbf9-kube-api-access-4spg6\") pod \"maas-api-revoke-sub-3968d0fb-lgt62\" (UID: \"c6aaa4d2-402f-4605-a0f6-3af867f7fbf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3968d0fb-lgt62" Nov 03 20:02:19.334513 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:19.334481 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3968d0fb-lgt62" Nov 03 20:02:19.614728 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:02:19.614700 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6aaa4d2_402f_4605_a0f6_3af867f7fbf9.slice/crio-7e8a0854fcda51e9a20f03aad08d440e43779d90a78f52d7c5710d180a9ce4ba WatchSource:0}: Error finding container 7e8a0854fcda51e9a20f03aad08d440e43779d90a78f52d7c5710d180a9ce4ba: Status 404 returned error can't find the container with id 7e8a0854fcda51e9a20f03aad08d440e43779d90a78f52d7c5710d180a9ce4ba Nov 03 20:02:19.616877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:19.615758 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3968d0fb-lgt62"] Nov 03 20:02:19.742038 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:19.742011 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:20.256278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:20.256197 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3968d0fb-lgt62" event={"ID":"c6aaa4d2-402f-4605-a0f6-3af867f7fbf9","Type":"ContainerStarted","Data":"20731930d7c6361891df9a9b39fdb761ac072612c83e6f102d5a6aea384aae6e"} Nov 03 20:02:20.256278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:20.256258 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3968d0fb-lgt62" event={"ID":"c6aaa4d2-402f-4605-a0f6-3af867f7fbf9","Type":"ContainerStarted","Data":"7e8a0854fcda51e9a20f03aad08d440e43779d90a78f52d7c5710d180a9ce4ba"} Nov 03 20:02:20.256731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:20.256704 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:20.257738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:20.257719 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:02:20.278724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:20.278683 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3968d0fb-lgt62" podStartSLOduration=2.278656713 podStartE2EDuration="2.278656713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:20.274886029 +0000 UTC m=+1534.532683286" watchObservedRunningTime="2026-11-03 20:02:20.278656713 +0000 UTC m=+1534.536453970" Nov 03 20:02:21.265377 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:21.265348 5059 generic.go:356] "Generic (PLEG): container finished" podID="c6aaa4d2-402f-4605-a0f6-3af867f7fbf9" containerID="20731930d7c6361891df9a9b39fdb761ac072612c83e6f102d5a6aea384aae6e" exitCode=0 Nov 03 20:02:21.265793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:21.265433 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3968d0fb-lgt62" event={"ID":"c6aaa4d2-402f-4605-a0f6-3af867f7fbf9","Type":"ContainerDied","Data":"20731930d7c6361891df9a9b39fdb761ac072612c83e6f102d5a6aea384aae6e"} Nov 03 20:02:22.431253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:22.431228 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3968d0fb-lgt62" Nov 03 20:02:22.585193 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:22.585165 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6aaa4d2-402f-4605-a0f6-3af867f7fbf9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6aaa4d2-402f-4605-a0f6-3af867f7fbf9-maas-api-xks-ca\") pod \"c6aaa4d2-402f-4605-a0f6-3af867f7fbf9\" (UID: \"c6aaa4d2-402f-4605-a0f6-3af867f7fbf9\") " Nov 03 20:02:22.585388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:22.585373 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6aaa4d2-402f-4605-a0f6-3af867f7fbf9-kube-api-access-4spg6\" (UniqueName: \"kubernetes.io/projected/c6aaa4d2-402f-4605-a0f6-3af867f7fbf9-kube-api-access-4spg6\") pod \"c6aaa4d2-402f-4605-a0f6-3af867f7fbf9\" (UID: \"c6aaa4d2-402f-4605-a0f6-3af867f7fbf9\") " Nov 03 20:02:22.585437 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:22.585414 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6aaa4d2-402f-4605-a0f6-3af867f7fbf9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6aaa4d2-402f-4605-a0f6-3af867f7fbf9-maas-api-openshift-ca\") pod \"c6aaa4d2-402f-4605-a0f6-3af867f7fbf9\" (UID: \"c6aaa4d2-402f-4605-a0f6-3af867f7fbf9\") " Nov 03 20:02:22.585862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:22.585833 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6aaa4d2-402f-4605-a0f6-3af867f7fbf9-maas-api-openshift-ca" pod "c6aaa4d2-402f-4605-a0f6-3af867f7fbf9" (UID: "c6aaa4d2-402f-4605-a0f6-3af867f7fbf9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:22.586000 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:22.585987 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6aaa4d2-402f-4605-a0f6-3af867f7fbf9-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:22.587269 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:22.587238 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6aaa4d2-402f-4605-a0f6-3af867f7fbf9-maas-api-xks-ca" pod "c6aaa4d2-402f-4605-a0f6-3af867f7fbf9" (UID: "c6aaa4d2-402f-4605-a0f6-3af867f7fbf9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:22.587377 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:22.587330 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6aaa4d2-402f-4605-a0f6-3af867f7fbf9-kube-api-access-4spg6" pod "c6aaa4d2-402f-4605-a0f6-3af867f7fbf9" (UID: "c6aaa4d2-402f-4605-a0f6-3af867f7fbf9"). InnerVolumeSpecName "kube-api-access-4spg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:22.687181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:22.687140 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4spg6\" (UniqueName: \"kubernetes.io/projected/c6aaa4d2-402f-4605-a0f6-3af867f7fbf9-kube-api-access-4spg6\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:22.687181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:22.687174 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6aaa4d2-402f-4605-a0f6-3af867f7fbf9-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:23.281175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:23.281146 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3968d0fb-lgt62" Nov 03 20:02:23.281175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:23.281167 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3968d0fb-lgt62" event={"ID":"c6aaa4d2-402f-4605-a0f6-3af867f7fbf9","Type":"ContainerDied","Data":"7e8a0854fcda51e9a20f03aad08d440e43779d90a78f52d7c5710d180a9ce4ba"} Nov 03 20:02:23.281428 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:23.281211 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e8a0854fcda51e9a20f03aad08d440e43779d90a78f52d7c5710d180a9ce4ba" Nov 03 20:02:25.705514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:25.705478 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr"] Nov 03 20:02:25.706002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:25.705713 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" podUID="1b1f91b8-973a-4345-acd7-ea3325faf638" containerName="istio-proxy" containerID="cri-o://0957e98fb98eca19314edc21a5a36bb3335ce419f10eaf36789a50068a116b53" gracePeriod=30 Nov 03 20:02:28.950687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:28.883277 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:29.485189 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:29.485147 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:02:29.485874 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:29.485459 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:29.549786 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:29.549737 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:29.574776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:29.574733 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 20:02:29.962687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:29.924956 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:30.266736 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:30.264391 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 20:02:30.695686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:30.659039 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:31.188699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:31.175766 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 20:02:31.188699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:30.741613 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:02:31.188699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:31.176745 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" podUID="d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:31.188699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:30.948766 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:31.188699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:30.923570 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:02:31.230558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:30.923739 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:31.230558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:31.004740 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:02:31.230558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:31.202502 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" podUID="0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:31.230558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:31.020303 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:31.230558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:31.203208 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 20:02:31.230558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:31.203270 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:02:31.230558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:31.203307 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 20:02:31.828349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:31.828313 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 20:02:32.050744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.036720 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded" Nov 03 20:02:32.050744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.040124 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 20:02:32.050744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.040179 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 20:02:32.050744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.040210 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 20:02:32.050744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.040239 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:02:32.353912 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:02:32.353881 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1f91b8_973a_4345_acd7_ea3325faf638.slice/crio-0957e98fb98eca19314edc21a5a36bb3335ce419f10eaf36789a50068a116b53.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1f91b8_973a_4345_acd7_ea3325faf638.slice/crio-conmon-0957e98fb98eca19314edc21a5a36bb3335ce419f10eaf36789a50068a116b53.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:02:32.356398 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.356352 5059 generic.go:356] "Generic (PLEG): container finished" podID="1b1f91b8-973a-4345-acd7-ea3325faf638" containerID="0957e98fb98eca19314edc21a5a36bb3335ce419f10eaf36789a50068a116b53" exitCode=0 Nov 03 20:02:32.358279 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.358248 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" event={"ID":"1b1f91b8-973a-4345-acd7-ea3325faf638","Type":"ContainerDied","Data":"0957e98fb98eca19314edc21a5a36bb3335ce419f10eaf36789a50068a116b53"} Nov 03 20:02:32.458326 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.458300 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 20:02:32.510892 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.510860 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b18bc4e-g98qk"] Nov 03 20:02:32.511516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.511497 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c6aaa4d2-402f-4605-a0f6-3af867f7fbf9" containerName="revoke-keys" Nov 03 20:02:32.511516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.511512 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6aaa4d2-402f-4605-a0f6-3af867f7fbf9" containerName="revoke-keys" Nov 03 20:02:32.511729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.511537 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1b1f91b8-973a-4345-acd7-ea3325faf638" containerName="istio-proxy" Nov 03 20:02:32.511729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.511543 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b1f91b8-973a-4345-acd7-ea3325faf638" containerName="istio-proxy" Nov 03 20:02:32.511729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.511645 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6aaa4d2-402f-4605-a0f6-3af867f7fbf9" containerName="revoke-keys" Nov 03 20:02:32.511729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.511674 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b1f91b8-973a-4345-acd7-ea3325faf638" containerName="istio-proxy" Nov 03 20:02:32.515186 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.515166 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b18bc4e-g98qk" Nov 03 20:02:32.526189 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.526159 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b18bc4e-g98qk"] Nov 03 20:02:32.530103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.530079 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b1f91b8-973a-4345-acd7-ea3325faf638-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1f91b8-973a-4345-acd7-ea3325faf638-istiod-ca-cert\") pod \"1b1f91b8-973a-4345-acd7-ea3325faf638\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " Nov 03 20:02:32.530231 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.530121 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-envoy\") pod \"1b1f91b8-973a-4345-acd7-ea3325faf638\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " Nov 03 20:02:32.530231 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.530169 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-workload-certs\") pod \"1b1f91b8-973a-4345-acd7-ea3325faf638\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " Nov 03 20:02:32.530368 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.530303 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-workload-socket\") pod \"1b1f91b8-973a-4345-acd7-ea3325faf638\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " Nov 03 20:02:32.530368 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.530340 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-data\") pod \"1b1f91b8-973a-4345-acd7-ea3325faf638\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " Nov 03 20:02:32.530496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.530446 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-token\" (UniqueName: \"kubernetes.io/projected/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-token\") pod \"1b1f91b8-973a-4345-acd7-ea3325faf638\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " Nov 03 20:02:32.530496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.530462 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b1f91b8-973a-4345-acd7-ea3325faf638-istiod-ca-cert" pod "1b1f91b8-973a-4345-acd7-ea3325faf638" (UID: "1b1f91b8-973a-4345-acd7-ea3325faf638"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:32.530496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.530479 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1f91b8-973a-4345-acd7-ea3325faf638-kube-api-access-brprv\" (UniqueName: \"kubernetes.io/projected/1b1f91b8-973a-4345-acd7-ea3325faf638-kube-api-access-brprv\") pod \"1b1f91b8-973a-4345-acd7-ea3325faf638\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " Nov 03 20:02:32.530655 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.530554 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-credential-socket\") pod \"1b1f91b8-973a-4345-acd7-ea3325faf638\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " Nov 03 20:02:32.530655 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.530624 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-podinfo\") pod \"1b1f91b8-973a-4345-acd7-ea3325faf638\" (UID: \"1b1f91b8-973a-4345-acd7-ea3325faf638\") " Nov 03 20:02:32.530803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.530749 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-workload-socket" pod "1b1f91b8-973a-4345-acd7-ea3325faf638" (UID: "1b1f91b8-973a-4345-acd7-ea3325faf638"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:32.530928 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.530885 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4636458d-1dfd-404f-ac36-e21f7b91d48e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b18bc4e-g98qk\" (UID: \"4636458d-1dfd-404f-ac36-e21f7b91d48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b18bc4e-g98qk" Nov 03 20:02:32.531225 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.531199 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-workload-certs" pod "1b1f91b8-973a-4345-acd7-ea3325faf638" (UID: "1b1f91b8-973a-4345-acd7-ea3325faf638"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:32.531320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.531227 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-data" pod "1b1f91b8-973a-4345-acd7-ea3325faf638" (UID: "1b1f91b8-973a-4345-acd7-ea3325faf638"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:32.531461 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.531439 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-credential-socket" pod "1b1f91b8-973a-4345-acd7-ea3325faf638" (UID: "1b1f91b8-973a-4345-acd7-ea3325faf638"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:32.531550 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.530989 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4636458d-1dfd-404f-ac36-e21f7b91d48e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b18bc4e-g98qk\" (UID: \"4636458d-1dfd-404f-ac36-e21f7b91d48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b18bc4e-g98qk" Nov 03 20:02:32.531810 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.531786 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm78h\" (UniqueName: \"kubernetes.io/projected/4636458d-1dfd-404f-ac36-e21f7b91d48e-kube-api-access-xm78h\") pod \"maas-api-revoke-sub-8b18bc4e-g98qk\" (UID: \"4636458d-1dfd-404f-ac36-e21f7b91d48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b18bc4e-g98qk" Nov 03 20:02:32.532272 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.532144 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.532272 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.532201 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.532272 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.532217 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.532272 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.532233 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1f91b8-973a-4345-acd7-ea3325faf638-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.532272 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.532264 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.534471 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.534436 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1f91b8-973a-4345-acd7-ea3325faf638-kube-api-access-brprv" pod "1b1f91b8-973a-4345-acd7-ea3325faf638" (UID: "1b1f91b8-973a-4345-acd7-ea3325faf638"). InnerVolumeSpecName "kube-api-access-brprv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:32.534641 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.534602 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-token" pod "1b1f91b8-973a-4345-acd7-ea3325faf638" (UID: "1b1f91b8-973a-4345-acd7-ea3325faf638"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:32.534760 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.534734 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-podinfo" pod "1b1f91b8-973a-4345-acd7-ea3325faf638" (UID: "1b1f91b8-973a-4345-acd7-ea3325faf638"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:02:32.534983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.534963 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-envoy" pod "1b1f91b8-973a-4345-acd7-ea3325faf638" (UID: "1b1f91b8-973a-4345-acd7-ea3325faf638"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:32.539699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.539638 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:02:32.539802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.539731 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:02:32.633419 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.633338 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm78h\" (UniqueName: \"kubernetes.io/projected/4636458d-1dfd-404f-ac36-e21f7b91d48e-kube-api-access-xm78h\") pod \"maas-api-revoke-sub-8b18bc4e-g98qk\" (UID: \"4636458d-1dfd-404f-ac36-e21f7b91d48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b18bc4e-g98qk" Nov 03 20:02:32.633562 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.633495 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4636458d-1dfd-404f-ac36-e21f7b91d48e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b18bc4e-g98qk\" (UID: \"4636458d-1dfd-404f-ac36-e21f7b91d48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b18bc4e-g98qk" Nov 03 20:02:32.633611 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.633568 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4636458d-1dfd-404f-ac36-e21f7b91d48e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b18bc4e-g98qk\" (UID: \"4636458d-1dfd-404f-ac36-e21f7b91d48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b18bc4e-g98qk" Nov 03 20:02:32.633727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.633712 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.633777 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.633734 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brprv\" (UniqueName: \"kubernetes.io/projected/1b1f91b8-973a-4345-acd7-ea3325faf638-kube-api-access-brprv\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.633777 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.633749 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.633777 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.633759 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1f91b8-973a-4345-acd7-ea3325faf638-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.634128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.634107 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4636458d-1dfd-404f-ac36-e21f7b91d48e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b18bc4e-g98qk\" (UID: \"4636458d-1dfd-404f-ac36-e21f7b91d48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b18bc4e-g98qk" Nov 03 20:02:32.636073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.636050 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4636458d-1dfd-404f-ac36-e21f7b91d48e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b18bc4e-g98qk\" (UID: \"4636458d-1dfd-404f-ac36-e21f7b91d48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b18bc4e-g98qk" Nov 03 20:02:32.646062 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.646033 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm78h\" (UniqueName: \"kubernetes.io/projected/4636458d-1dfd-404f-ac36-e21f7b91d48e-kube-api-access-xm78h\") pod \"maas-api-revoke-sub-8b18bc4e-g98qk\" (UID: \"4636458d-1dfd-404f-ac36-e21f7b91d48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b18bc4e-g98qk" Nov 03 20:02:32.833799 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:32.833773 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b18bc4e-g98qk" Nov 03 20:02:33.115766 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:02:33.115738 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4636458d_1dfd_404f_ac36_e21f7b91d48e.slice/crio-e83a3bda5e5cef92f714cd7c45ab0ad476a4f2dd59850c4a1e141e9b5e907448 WatchSource:0}: Error finding container e83a3bda5e5cef92f714cd7c45ab0ad476a4f2dd59850c4a1e141e9b5e907448: Status 404 returned error can't find the container with id e83a3bda5e5cef92f714cd7c45ab0ad476a4f2dd59850c4a1e141e9b5e907448 Nov 03 20:02:33.116829 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:33.116805 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b18bc4e-g98qk"] Nov 03 20:02:33.377393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:33.377364 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" Nov 03 20:02:33.377900 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:33.377390 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr" event={"ID":"1b1f91b8-973a-4345-acd7-ea3325faf638","Type":"ContainerDied","Data":"e4d63cbbad213bec6820394b5b1cd23984fdd2c9fd732c27187a960835ddce84"} Nov 03 20:02:33.377900 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:33.377459 5059 scope.go:122] "RemoveContainer" containerID="0957e98fb98eca19314edc21a5a36bb3335ce419f10eaf36789a50068a116b53" Nov 03 20:02:33.380450 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:33.380420 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b18bc4e-g98qk" event={"ID":"4636458d-1dfd-404f-ac36-e21f7b91d48e","Type":"ContainerStarted","Data":"e83a3bda5e5cef92f714cd7c45ab0ad476a4f2dd59850c4a1e141e9b5e907448"} Nov 03 20:02:33.405016 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:33.404959 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr"] Nov 03 20:02:33.412953 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:33.412931 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cnppr"] Nov 03 20:02:33.627266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:33.627231 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0f6f815-vfxm5"] Nov 03 20:02:33.632815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:33.632793 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f6f815-vfxm5" Nov 03 20:02:33.641753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:33.641717 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0f6f815-vfxm5"] Nov 03 20:02:33.657341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:33.657276 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168586b3-99f0-47df-b549-1cbd409efaa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0f6f815-vfxm5\" (UID: \"168586b3-99f0-47df-b549-1cbd409efaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f6f815-vfxm5" Nov 03 20:02:33.657593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:33.657568 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbn8l\" (UniqueName: \"kubernetes.io/projected/168586b3-99f0-47df-b549-1cbd409efaa6-kube-api-access-gbn8l\") pod \"maas-api-revoke-sub-c0f6f815-vfxm5\" (UID: \"168586b3-99f0-47df-b549-1cbd409efaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f6f815-vfxm5" Nov 03 20:02:33.657752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:33.657733 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168586b3-99f0-47df-b549-1cbd409efaa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0f6f815-vfxm5\" (UID: \"168586b3-99f0-47df-b549-1cbd409efaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f6f815-vfxm5" Nov 03 20:02:33.759760 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:33.759710 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbn8l\" (UniqueName: \"kubernetes.io/projected/168586b3-99f0-47df-b549-1cbd409efaa6-kube-api-access-gbn8l\") pod \"maas-api-revoke-sub-c0f6f815-vfxm5\" (UID: \"168586b3-99f0-47df-b549-1cbd409efaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f6f815-vfxm5" Nov 03 20:02:33.759931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:33.759819 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168586b3-99f0-47df-b549-1cbd409efaa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0f6f815-vfxm5\" (UID: \"168586b3-99f0-47df-b549-1cbd409efaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f6f815-vfxm5" Nov 03 20:02:33.759931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:33.759915 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168586b3-99f0-47df-b549-1cbd409efaa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0f6f815-vfxm5\" (UID: \"168586b3-99f0-47df-b549-1cbd409efaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f6f815-vfxm5" Nov 03 20:02:33.760561 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:33.760533 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168586b3-99f0-47df-b549-1cbd409efaa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0f6f815-vfxm5\" (UID: \"168586b3-99f0-47df-b549-1cbd409efaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f6f815-vfxm5" Nov 03 20:02:33.762238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:33.762216 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168586b3-99f0-47df-b549-1cbd409efaa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0f6f815-vfxm5\" (UID: \"168586b3-99f0-47df-b549-1cbd409efaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f6f815-vfxm5" Nov 03 20:02:33.772612 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:33.772582 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbn8l\" (UniqueName: \"kubernetes.io/projected/168586b3-99f0-47df-b549-1cbd409efaa6-kube-api-access-gbn8l\") pod \"maas-api-revoke-sub-c0f6f815-vfxm5\" (UID: \"168586b3-99f0-47df-b549-1cbd409efaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f6f815-vfxm5" Nov 03 20:02:33.858884 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:33.858841 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b1f91b8-973a-4345-acd7-ea3325faf638" path="/var/lib/kubelet/pods/1b1f91b8-973a-4345-acd7-ea3325faf638/volumes" Nov 03 20:02:33.949924 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:33.949848 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f6f815-vfxm5" Nov 03 20:02:34.235054 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:02:34.235027 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168586b3_99f0_47df_b549_1cbd409efaa6.slice/crio-cfad4b9b38e9f5128b6fb14079974cf831d6c189fe409dd7086c4004629f607b WatchSource:0}: Error finding container cfad4b9b38e9f5128b6fb14079974cf831d6c189fe409dd7086c4004629f607b: Status 404 returned error can't find the container with id cfad4b9b38e9f5128b6fb14079974cf831d6c189fe409dd7086c4004629f607b Nov 03 20:02:34.235489 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:34.235467 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0f6f815-vfxm5"] Nov 03 20:02:34.390102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:34.390065 5059 generic.go:356] "Generic (PLEG): container finished" podID="4636458d-1dfd-404f-ac36-e21f7b91d48e" containerID="f45a5ba6d0a6597f36d533287eea3dacb5392b6373db535d57f78e7b1c48721c" exitCode=0 Nov 03 20:02:34.390521 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:34.390159 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b18bc4e-g98qk" event={"ID":"4636458d-1dfd-404f-ac36-e21f7b91d48e","Type":"ContainerDied","Data":"f45a5ba6d0a6597f36d533287eea3dacb5392b6373db535d57f78e7b1c48721c"} Nov 03 20:02:34.391643 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:34.391618 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f6f815-vfxm5" event={"ID":"168586b3-99f0-47df-b549-1cbd409efaa6","Type":"ContainerStarted","Data":"cfad4b9b38e9f5128b6fb14079974cf831d6c189fe409dd7086c4004629f607b"} Nov 03 20:02:35.401332 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:35.401295 5059 generic.go:356] "Generic (PLEG): container finished" podID="168586b3-99f0-47df-b549-1cbd409efaa6" containerID="41824370ba2ad3319dab6d9bbfa7a3f63744067163de4417cf5bbc2373257cdd" exitCode=0 Nov 03 20:02:35.401835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:35.401380 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f6f815-vfxm5" event={"ID":"168586b3-99f0-47df-b549-1cbd409efaa6","Type":"ContainerDied","Data":"41824370ba2ad3319dab6d9bbfa7a3f63744067163de4417cf5bbc2373257cdd"} Nov 03 20:02:35.564059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:35.564036 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b18bc4e-g98qk" Nov 03 20:02:35.688108 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:35.688077 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4636458d-1dfd-404f-ac36-e21f7b91d48e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4636458d-1dfd-404f-ac36-e21f7b91d48e-maas-api-openshift-ca\") pod \"4636458d-1dfd-404f-ac36-e21f7b91d48e\" (UID: \"4636458d-1dfd-404f-ac36-e21f7b91d48e\") " Nov 03 20:02:35.688268 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:35.688116 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4636458d-1dfd-404f-ac36-e21f7b91d48e-kube-api-access-xm78h\" (UniqueName: \"kubernetes.io/projected/4636458d-1dfd-404f-ac36-e21f7b91d48e-kube-api-access-xm78h\") pod \"4636458d-1dfd-404f-ac36-e21f7b91d48e\" (UID: \"4636458d-1dfd-404f-ac36-e21f7b91d48e\") " Nov 03 20:02:35.688268 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:35.688146 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4636458d-1dfd-404f-ac36-e21f7b91d48e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4636458d-1dfd-404f-ac36-e21f7b91d48e-maas-api-xks-ca\") pod \"4636458d-1dfd-404f-ac36-e21f7b91d48e\" (UID: \"4636458d-1dfd-404f-ac36-e21f7b91d48e\") " Nov 03 20:02:35.688508 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:35.688483 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4636458d-1dfd-404f-ac36-e21f7b91d48e-maas-api-openshift-ca" pod "4636458d-1dfd-404f-ac36-e21f7b91d48e" (UID: "4636458d-1dfd-404f-ac36-e21f7b91d48e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:35.688741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:35.688719 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4636458d-1dfd-404f-ac36-e21f7b91d48e-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:35.690222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:35.690201 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4636458d-1dfd-404f-ac36-e21f7b91d48e-maas-api-xks-ca" pod "4636458d-1dfd-404f-ac36-e21f7b91d48e" (UID: "4636458d-1dfd-404f-ac36-e21f7b91d48e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:35.690368 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:35.690345 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4636458d-1dfd-404f-ac36-e21f7b91d48e-kube-api-access-xm78h" pod "4636458d-1dfd-404f-ac36-e21f7b91d48e" (UID: "4636458d-1dfd-404f-ac36-e21f7b91d48e"). InnerVolumeSpecName "kube-api-access-xm78h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:35.790431 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:35.790355 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm78h\" (UniqueName: \"kubernetes.io/projected/4636458d-1dfd-404f-ac36-e21f7b91d48e-kube-api-access-xm78h\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:35.790431 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:35.790386 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4636458d-1dfd-404f-ac36-e21f7b91d48e-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:36.410161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.410126 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b18bc4e-g98qk" Nov 03 20:02:36.410593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.410130 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b18bc4e-g98qk" event={"ID":"4636458d-1dfd-404f-ac36-e21f7b91d48e","Type":"ContainerDied","Data":"e83a3bda5e5cef92f714cd7c45ab0ad476a4f2dd59850c4a1e141e9b5e907448"} Nov 03 20:02:36.410593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.410273 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e83a3bda5e5cef92f714cd7c45ab0ad476a4f2dd59850c4a1e141e9b5e907448" Nov 03 20:02:36.596135 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.596104 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f6f815-vfxm5" Nov 03 20:02:36.646112 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.646081 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-174f2038-q6k6v"] Nov 03 20:02:36.647139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.647112 5059 cpu_manager.go:414] "containerMap: removing container" podUID="4636458d-1dfd-404f-ac36-e21f7b91d48e" containerName="revoke-keys" Nov 03 20:02:36.647211 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.647138 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4636458d-1dfd-404f-ac36-e21f7b91d48e" containerName="revoke-keys" Nov 03 20:02:36.647211 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.647191 5059 cpu_manager.go:414] "containerMap: removing container" podUID="168586b3-99f0-47df-b549-1cbd409efaa6" containerName="revoke-keys" Nov 03 20:02:36.647211 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.647199 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168586b3-99f0-47df-b549-1cbd409efaa6" containerName="revoke-keys" Nov 03 20:02:36.647364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.647350 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="168586b3-99f0-47df-b549-1cbd409efaa6" containerName="revoke-keys" Nov 03 20:02:36.647402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.647380 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4636458d-1dfd-404f-ac36-e21f7b91d48e" containerName="revoke-keys" Nov 03 20:02:36.651583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.651557 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f2038-q6k6v" Nov 03 20:02:36.659312 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.658328 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-174f2038-q6k6v"] Nov 03 20:02:36.700323 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.700285 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/168586b3-99f0-47df-b549-1cbd409efaa6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168586b3-99f0-47df-b549-1cbd409efaa6-maas-api-xks-ca\") pod \"168586b3-99f0-47df-b549-1cbd409efaa6\" (UID: \"168586b3-99f0-47df-b549-1cbd409efaa6\") " Nov 03 20:02:36.700557 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.700537 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168586b3-99f0-47df-b549-1cbd409efaa6-kube-api-access-gbn8l\" (UniqueName: \"kubernetes.io/projected/168586b3-99f0-47df-b549-1cbd409efaa6-kube-api-access-gbn8l\") pod \"168586b3-99f0-47df-b549-1cbd409efaa6\" (UID: \"168586b3-99f0-47df-b549-1cbd409efaa6\") " Nov 03 20:02:36.700642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.700628 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/168586b3-99f0-47df-b549-1cbd409efaa6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168586b3-99f0-47df-b549-1cbd409efaa6-maas-api-openshift-ca\") pod \"168586b3-99f0-47df-b549-1cbd409efaa6\" (UID: \"168586b3-99f0-47df-b549-1cbd409efaa6\") " Nov 03 20:02:36.700979 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.700955 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/168586b3-99f0-47df-b549-1cbd409efaa6-maas-api-openshift-ca" pod "168586b3-99f0-47df-b549-1cbd409efaa6" (UID: "168586b3-99f0-47df-b549-1cbd409efaa6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:36.701077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.701051 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82192848-35ee-4ca0-88e7-3f477f3d3bbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-174f2038-q6k6v\" (UID: \"82192848-35ee-4ca0-88e7-3f477f3d3bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f2038-q6k6v" Nov 03 20:02:36.701241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.701216 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88rxf\" (UniqueName: \"kubernetes.io/projected/82192848-35ee-4ca0-88e7-3f477f3d3bbf-kube-api-access-88rxf\") pod \"maas-api-revoke-sub-174f2038-q6k6v\" (UID: \"82192848-35ee-4ca0-88e7-3f477f3d3bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f2038-q6k6v" Nov 03 20:02:36.701333 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.701289 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82192848-35ee-4ca0-88e7-3f477f3d3bbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-174f2038-q6k6v\" (UID: \"82192848-35ee-4ca0-88e7-3f477f3d3bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f2038-q6k6v" Nov 03 20:02:36.701545 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.701525 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168586b3-99f0-47df-b549-1cbd409efaa6-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:36.703058 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.703035 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/168586b3-99f0-47df-b549-1cbd409efaa6-maas-api-xks-ca" pod "168586b3-99f0-47df-b549-1cbd409efaa6" (UID: "168586b3-99f0-47df-b549-1cbd409efaa6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:36.703321 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.703296 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168586b3-99f0-47df-b549-1cbd409efaa6-kube-api-access-gbn8l" pod "168586b3-99f0-47df-b549-1cbd409efaa6" (UID: "168586b3-99f0-47df-b549-1cbd409efaa6"). InnerVolumeSpecName "kube-api-access-gbn8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:36.802635 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.802601 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88rxf\" (UniqueName: \"kubernetes.io/projected/82192848-35ee-4ca0-88e7-3f477f3d3bbf-kube-api-access-88rxf\") pod \"maas-api-revoke-sub-174f2038-q6k6v\" (UID: \"82192848-35ee-4ca0-88e7-3f477f3d3bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f2038-q6k6v" Nov 03 20:02:36.802856 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.802682 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82192848-35ee-4ca0-88e7-3f477f3d3bbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-174f2038-q6k6v\" (UID: \"82192848-35ee-4ca0-88e7-3f477f3d3bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f2038-q6k6v" Nov 03 20:02:36.802926 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.802888 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82192848-35ee-4ca0-88e7-3f477f3d3bbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-174f2038-q6k6v\" (UID: \"82192848-35ee-4ca0-88e7-3f477f3d3bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f2038-q6k6v" Nov 03 20:02:36.802989 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.802978 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168586b3-99f0-47df-b549-1cbd409efaa6-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:36.803038 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.802999 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbn8l\" (UniqueName: \"kubernetes.io/projected/168586b3-99f0-47df-b549-1cbd409efaa6-kube-api-access-gbn8l\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:36.803526 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.803505 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82192848-35ee-4ca0-88e7-3f477f3d3bbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-174f2038-q6k6v\" (UID: \"82192848-35ee-4ca0-88e7-3f477f3d3bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f2038-q6k6v" Nov 03 20:02:36.805105 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.805080 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82192848-35ee-4ca0-88e7-3f477f3d3bbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-174f2038-q6k6v\" (UID: \"82192848-35ee-4ca0-88e7-3f477f3d3bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f2038-q6k6v" Nov 03 20:02:36.836352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.836194 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88rxf\" (UniqueName: \"kubernetes.io/projected/82192848-35ee-4ca0-88e7-3f477f3d3bbf-kube-api-access-88rxf\") pod \"maas-api-revoke-sub-174f2038-q6k6v\" (UID: \"82192848-35ee-4ca0-88e7-3f477f3d3bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f2038-q6k6v" Nov 03 20:02:36.974509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:36.974428 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f2038-q6k6v" Nov 03 20:02:37.256572 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:02:37.256545 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82192848_35ee_4ca0_88e7_3f477f3d3bbf.slice/crio-5c8ce71ddeac37f17468149be4e3db09ecefa7506341fa371b3dbcf06a6f13d9 WatchSource:0}: Error finding container 5c8ce71ddeac37f17468149be4e3db09ecefa7506341fa371b3dbcf06a6f13d9: Status 404 returned error can't find the container with id 5c8ce71ddeac37f17468149be4e3db09ecefa7506341fa371b3dbcf06a6f13d9 Nov 03 20:02:37.256794 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:37.256766 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-174f2038-q6k6v"] Nov 03 20:02:37.418575 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:37.418541 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f6f815-vfxm5" Nov 03 20:02:37.419001 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:37.418544 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0f6f815-vfxm5" event={"ID":"168586b3-99f0-47df-b549-1cbd409efaa6","Type":"ContainerDied","Data":"cfad4b9b38e9f5128b6fb14079974cf831d6c189fe409dd7086c4004629f607b"} Nov 03 20:02:37.419001 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:37.418677 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfad4b9b38e9f5128b6fb14079974cf831d6c189fe409dd7086c4004629f607b" Nov 03 20:02:37.420142 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:37.420119 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f2038-q6k6v" event={"ID":"82192848-35ee-4ca0-88e7-3f477f3d3bbf","Type":"ContainerStarted","Data":"5c8ce71ddeac37f17468149be4e3db09ecefa7506341fa371b3dbcf06a6f13d9"} Nov 03 20:02:38.429198 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:38.429117 5059 generic.go:356] "Generic (PLEG): container finished" podID="82192848-35ee-4ca0-88e7-3f477f3d3bbf" containerID="6fb909a8b978acddb86a4265052ea2982fb88641f65a989e41d09c720c2c93fc" exitCode=0 Nov 03 20:02:38.429567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:38.429200 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f2038-q6k6v" event={"ID":"82192848-35ee-4ca0-88e7-3f477f3d3bbf","Type":"ContainerDied","Data":"6fb909a8b978acddb86a4265052ea2982fb88641f65a989e41d09c720c2c93fc"} Nov 03 20:02:38.962708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:38.962676 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb1d7137-wcw5n"] Nov 03 20:02:38.966903 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:38.966885 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d7137-wcw5n" Nov 03 20:02:38.976347 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:38.976324 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb1d7137-wcw5n"] Nov 03 20:02:39.048946 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:39.048912 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fg6z\" (UniqueName: \"kubernetes.io/projected/3313416d-c233-439a-ad7d-f52125194720-kube-api-access-4fg6z\") pod \"maas-api-revoke-sub-cb1d7137-wcw5n\" (UID: \"3313416d-c233-439a-ad7d-f52125194720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d7137-wcw5n" Nov 03 20:02:39.048946 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:39.048951 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3313416d-c233-439a-ad7d-f52125194720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb1d7137-wcw5n\" (UID: \"3313416d-c233-439a-ad7d-f52125194720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d7137-wcw5n" Nov 03 20:02:39.049204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:39.049152 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3313416d-c233-439a-ad7d-f52125194720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb1d7137-wcw5n\" (UID: \"3313416d-c233-439a-ad7d-f52125194720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d7137-wcw5n" Nov 03 20:02:39.150323 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:39.150294 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fg6z\" (UniqueName: \"kubernetes.io/projected/3313416d-c233-439a-ad7d-f52125194720-kube-api-access-4fg6z\") pod \"maas-api-revoke-sub-cb1d7137-wcw5n\" (UID: \"3313416d-c233-439a-ad7d-f52125194720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d7137-wcw5n" Nov 03 20:02:39.150481 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:39.150331 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3313416d-c233-439a-ad7d-f52125194720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb1d7137-wcw5n\" (UID: \"3313416d-c233-439a-ad7d-f52125194720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d7137-wcw5n" Nov 03 20:02:39.150481 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:39.150447 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3313416d-c233-439a-ad7d-f52125194720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb1d7137-wcw5n\" (UID: \"3313416d-c233-439a-ad7d-f52125194720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d7137-wcw5n" Nov 03 20:02:39.151043 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:39.151014 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3313416d-c233-439a-ad7d-f52125194720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb1d7137-wcw5n\" (UID: \"3313416d-c233-439a-ad7d-f52125194720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d7137-wcw5n" Nov 03 20:02:39.152956 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:39.152931 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3313416d-c233-439a-ad7d-f52125194720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb1d7137-wcw5n\" (UID: \"3313416d-c233-439a-ad7d-f52125194720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d7137-wcw5n" Nov 03 20:02:39.162274 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:39.162250 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fg6z\" (UniqueName: \"kubernetes.io/projected/3313416d-c233-439a-ad7d-f52125194720-kube-api-access-4fg6z\") pod \"maas-api-revoke-sub-cb1d7137-wcw5n\" (UID: \"3313416d-c233-439a-ad7d-f52125194720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d7137-wcw5n" Nov 03 20:02:39.283287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:39.283211 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d7137-wcw5n" Nov 03 20:02:39.570473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:39.570442 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb1d7137-wcw5n"] Nov 03 20:02:39.588732 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:02:39.575456 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3313416d_c233_439a_ad7d_f52125194720.slice/crio-8ea0400e767cac2fd20e17a87b5993961659722bec7636a81fdf01ec4ae69c8c WatchSource:0}: Error finding container 8ea0400e767cac2fd20e17a87b5993961659722bec7636a81fdf01ec4ae69c8c: Status 404 returned error can't find the container with id 8ea0400e767cac2fd20e17a87b5993961659722bec7636a81fdf01ec4ae69c8c Nov 03 20:02:39.617501 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:39.617473 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f2038-q6k6v" Nov 03 20:02:39.761041 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:39.761008 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82192848-35ee-4ca0-88e7-3f477f3d3bbf-kube-api-access-88rxf\" (UniqueName: \"kubernetes.io/projected/82192848-35ee-4ca0-88e7-3f477f3d3bbf-kube-api-access-88rxf\") pod \"82192848-35ee-4ca0-88e7-3f477f3d3bbf\" (UID: \"82192848-35ee-4ca0-88e7-3f477f3d3bbf\") " Nov 03 20:02:39.761218 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:39.761091 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82192848-35ee-4ca0-88e7-3f477f3d3bbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82192848-35ee-4ca0-88e7-3f477f3d3bbf-maas-api-openshift-ca\") pod \"82192848-35ee-4ca0-88e7-3f477f3d3bbf\" (UID: \"82192848-35ee-4ca0-88e7-3f477f3d3bbf\") " Nov 03 20:02:39.761308 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:39.761289 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82192848-35ee-4ca0-88e7-3f477f3d3bbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82192848-35ee-4ca0-88e7-3f477f3d3bbf-maas-api-xks-ca\") pod \"82192848-35ee-4ca0-88e7-3f477f3d3bbf\" (UID: \"82192848-35ee-4ca0-88e7-3f477f3d3bbf\") " Nov 03 20:02:39.761530 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:39.761501 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82192848-35ee-4ca0-88e7-3f477f3d3bbf-maas-api-openshift-ca" pod "82192848-35ee-4ca0-88e7-3f477f3d3bbf" (UID: "82192848-35ee-4ca0-88e7-3f477f3d3bbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:39.761865 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:39.761831 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82192848-35ee-4ca0-88e7-3f477f3d3bbf-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:39.763700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:39.763640 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82192848-35ee-4ca0-88e7-3f477f3d3bbf-maas-api-xks-ca" pod "82192848-35ee-4ca0-88e7-3f477f3d3bbf" (UID: "82192848-35ee-4ca0-88e7-3f477f3d3bbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:39.763789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:39.763691 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82192848-35ee-4ca0-88e7-3f477f3d3bbf-kube-api-access-88rxf" pod "82192848-35ee-4ca0-88e7-3f477f3d3bbf" (UID: "82192848-35ee-4ca0-88e7-3f477f3d3bbf"). InnerVolumeSpecName "kube-api-access-88rxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:39.862482 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:39.862455 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88rxf\" (UniqueName: \"kubernetes.io/projected/82192848-35ee-4ca0-88e7-3f477f3d3bbf-kube-api-access-88rxf\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:39.862482 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:39.862481 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82192848-35ee-4ca0-88e7-3f477f3d3bbf-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:40.468406 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:40.468361 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d7137-wcw5n" event={"ID":"3313416d-c233-439a-ad7d-f52125194720","Type":"ContainerStarted","Data":"1221bae475bd9e70dc904bfc4a69edde256ca351a86211ae77cfecb90da7c033"} Nov 03 20:02:40.468574 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:40.468431 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d7137-wcw5n" event={"ID":"3313416d-c233-439a-ad7d-f52125194720","Type":"ContainerStarted","Data":"8ea0400e767cac2fd20e17a87b5993961659722bec7636a81fdf01ec4ae69c8c"} Nov 03 20:02:40.469989 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:40.469964 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f2038-q6k6v" Nov 03 20:02:40.470104 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:40.470009 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f2038-q6k6v" event={"ID":"82192848-35ee-4ca0-88e7-3f477f3d3bbf","Type":"ContainerDied","Data":"5c8ce71ddeac37f17468149be4e3db09ecefa7506341fa371b3dbcf06a6f13d9"} Nov 03 20:02:40.470104 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:40.470059 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c8ce71ddeac37f17468149be4e3db09ecefa7506341fa371b3dbcf06a6f13d9" Nov 03 20:02:40.604867 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:02:40.604837 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:02:41.479268 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:41.479233 5059 generic.go:356] "Generic (PLEG): container finished" podID="3313416d-c233-439a-ad7d-f52125194720" containerID="1221bae475bd9e70dc904bfc4a69edde256ca351a86211ae77cfecb90da7c033" exitCode=0 Nov 03 20:02:41.479443 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:41.479315 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d7137-wcw5n" event={"ID":"3313416d-c233-439a-ad7d-f52125194720","Type":"ContainerDied","Data":"1221bae475bd9e70dc904bfc4a69edde256ca351a86211ae77cfecb90da7c033"} Nov 03 20:02:41.640871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:41.640846 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d7137-wcw5n" Nov 03 20:02:41.789818 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:41.789706 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3313416d-c233-439a-ad7d-f52125194720-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3313416d-c233-439a-ad7d-f52125194720-maas-api-openshift-ca\") pod \"3313416d-c233-439a-ad7d-f52125194720\" (UID: \"3313416d-c233-439a-ad7d-f52125194720\") " Nov 03 20:02:41.789818 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:41.789806 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3313416d-c233-439a-ad7d-f52125194720-kube-api-access-4fg6z\" (UniqueName: \"kubernetes.io/projected/3313416d-c233-439a-ad7d-f52125194720-kube-api-access-4fg6z\") pod \"3313416d-c233-439a-ad7d-f52125194720\" (UID: \"3313416d-c233-439a-ad7d-f52125194720\") " Nov 03 20:02:41.790058 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:41.789959 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3313416d-c233-439a-ad7d-f52125194720-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3313416d-c233-439a-ad7d-f52125194720-maas-api-xks-ca\") pod \"3313416d-c233-439a-ad7d-f52125194720\" (UID: \"3313416d-c233-439a-ad7d-f52125194720\") " Nov 03 20:02:41.791104 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:41.791064 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3313416d-c233-439a-ad7d-f52125194720-maas-api-openshift-ca" pod "3313416d-c233-439a-ad7d-f52125194720" (UID: "3313416d-c233-439a-ad7d-f52125194720"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:41.792757 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:41.792724 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3313416d-c233-439a-ad7d-f52125194720-kube-api-access-4fg6z" pod "3313416d-c233-439a-ad7d-f52125194720" (UID: "3313416d-c233-439a-ad7d-f52125194720"). InnerVolumeSpecName "kube-api-access-4fg6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:41.793086 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:41.793062 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3313416d-c233-439a-ad7d-f52125194720-maas-api-xks-ca" pod "3313416d-c233-439a-ad7d-f52125194720" (UID: "3313416d-c233-439a-ad7d-f52125194720"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:41.891501 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:41.891471 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3313416d-c233-439a-ad7d-f52125194720-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:41.891672 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:41.891504 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fg6z\" (UniqueName: \"kubernetes.io/projected/3313416d-c233-439a-ad7d-f52125194720-kube-api-access-4fg6z\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:41.891672 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:41.891522 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3313416d-c233-439a-ad7d-f52125194720-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:42.489075 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:42.489046 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d7137-wcw5n" Nov 03 20:02:42.489264 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:42.489047 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d7137-wcw5n" event={"ID":"3313416d-c233-439a-ad7d-f52125194720","Type":"ContainerDied","Data":"8ea0400e767cac2fd20e17a87b5993961659722bec7636a81fdf01ec4ae69c8c"} Nov 03 20:02:42.489264 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:42.489173 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ea0400e767cac2fd20e17a87b5993961659722bec7636a81fdf01ec4ae69c8c" Nov 03 20:02:43.862925 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:43.862888 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eda4db2e-s8gjr"] Nov 03 20:02:43.863484 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:43.863468 5059 cpu_manager.go:414] "containerMap: removing container" podUID="82192848-35ee-4ca0-88e7-3f477f3d3bbf" containerName="revoke-keys" Nov 03 20:02:43.863484 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:43.863484 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82192848-35ee-4ca0-88e7-3f477f3d3bbf" containerName="revoke-keys" Nov 03 20:02:43.863591 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:43.863499 5059 cpu_manager.go:414] "containerMap: removing container" podUID="3313416d-c233-439a-ad7d-f52125194720" containerName="revoke-keys" Nov 03 20:02:43.863591 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:43.863504 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3313416d-c233-439a-ad7d-f52125194720" containerName="revoke-keys" Nov 03 20:02:43.863707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:43.863611 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="82192848-35ee-4ca0-88e7-3f477f3d3bbf" containerName="revoke-keys" Nov 03 20:02:43.863707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:43.863622 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3313416d-c233-439a-ad7d-f52125194720" containerName="revoke-keys" Nov 03 20:02:43.867371 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:43.867355 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda4db2e-s8gjr" Nov 03 20:02:43.872017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:43.871629 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:02:43.872864 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:43.872838 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:02:43.873812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:43.873787 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eda4db2e-s8gjr"] Nov 03 20:02:44.018698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:44.018638 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33b17a4-a052-439f-a02a-fb3cad0ae4fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eda4db2e-s8gjr\" (UID: \"a33b17a4-a052-439f-a02a-fb3cad0ae4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda4db2e-s8gjr" Nov 03 20:02:44.018847 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:44.018776 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33b17a4-a052-439f-a02a-fb3cad0ae4fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eda4db2e-s8gjr\" (UID: \"a33b17a4-a052-439f-a02a-fb3cad0ae4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda4db2e-s8gjr" Nov 03 20:02:44.018889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:44.018854 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68h8s\" (UniqueName: \"kubernetes.io/projected/a33b17a4-a052-439f-a02a-fb3cad0ae4fb-kube-api-access-68h8s\") pod \"maas-api-revoke-sub-eda4db2e-s8gjr\" (UID: \"a33b17a4-a052-439f-a02a-fb3cad0ae4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda4db2e-s8gjr" Nov 03 20:02:44.120113 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:44.119980 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33b17a4-a052-439f-a02a-fb3cad0ae4fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eda4db2e-s8gjr\" (UID: \"a33b17a4-a052-439f-a02a-fb3cad0ae4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda4db2e-s8gjr" Nov 03 20:02:44.120113 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:44.120100 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68h8s\" (UniqueName: \"kubernetes.io/projected/a33b17a4-a052-439f-a02a-fb3cad0ae4fb-kube-api-access-68h8s\") pod \"maas-api-revoke-sub-eda4db2e-s8gjr\" (UID: \"a33b17a4-a052-439f-a02a-fb3cad0ae4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda4db2e-s8gjr" Nov 03 20:02:44.120365 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:44.120343 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33b17a4-a052-439f-a02a-fb3cad0ae4fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eda4db2e-s8gjr\" (UID: \"a33b17a4-a052-439f-a02a-fb3cad0ae4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda4db2e-s8gjr" Nov 03 20:02:44.120720 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:44.120703 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33b17a4-a052-439f-a02a-fb3cad0ae4fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eda4db2e-s8gjr\" (UID: \"a33b17a4-a052-439f-a02a-fb3cad0ae4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda4db2e-s8gjr" Nov 03 20:02:44.122927 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:44.122905 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33b17a4-a052-439f-a02a-fb3cad0ae4fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eda4db2e-s8gjr\" (UID: \"a33b17a4-a052-439f-a02a-fb3cad0ae4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda4db2e-s8gjr" Nov 03 20:02:44.132521 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:44.132490 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68h8s\" (UniqueName: \"kubernetes.io/projected/a33b17a4-a052-439f-a02a-fb3cad0ae4fb-kube-api-access-68h8s\") pod \"maas-api-revoke-sub-eda4db2e-s8gjr\" (UID: \"a33b17a4-a052-439f-a02a-fb3cad0ae4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda4db2e-s8gjr" Nov 03 20:02:44.182994 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:44.182956 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda4db2e-s8gjr" Nov 03 20:02:44.471324 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:02:44.471297 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33b17a4_a052_439f_a02a_fb3cad0ae4fb.slice/crio-5268cb871dfcba87bbcabed12c8b21e85762f1acebc6ea5db2ff028e006ba305 WatchSource:0}: Error finding container 5268cb871dfcba87bbcabed12c8b21e85762f1acebc6ea5db2ff028e006ba305: Status 404 returned error can't find the container with id 5268cb871dfcba87bbcabed12c8b21e85762f1acebc6ea5db2ff028e006ba305 Nov 03 20:02:44.471468 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:44.471445 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eda4db2e-s8gjr"] Nov 03 20:02:44.504870 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:44.504844 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda4db2e-s8gjr" event={"ID":"a33b17a4-a052-439f-a02a-fb3cad0ae4fb","Type":"ContainerStarted","Data":"5268cb871dfcba87bbcabed12c8b21e85762f1acebc6ea5db2ff028e006ba305"} Nov 03 20:02:45.566744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:45.557090 5059 generic.go:356] "Generic (PLEG): container finished" podID="a33b17a4-a052-439f-a02a-fb3cad0ae4fb" containerID="673b24a7ca1f1d3180a9a878a642de7a77773958213ccd7f39ee93062ba8d629" exitCode=0 Nov 03 20:02:45.566744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:45.557158 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda4db2e-s8gjr" event={"ID":"a33b17a4-a052-439f-a02a-fb3cad0ae4fb","Type":"ContainerDied","Data":"673b24a7ca1f1d3180a9a878a642de7a77773958213ccd7f39ee93062ba8d629"} Nov 03 20:02:47.058307 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:47.058271 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda4db2e-s8gjr" Nov 03 20:02:47.307064 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:47.298455 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a33b17a4-a052-439f-a02a-fb3cad0ae4fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33b17a4-a052-439f-a02a-fb3cad0ae4fb-maas-api-openshift-ca\") pod \"a33b17a4-a052-439f-a02a-fb3cad0ae4fb\" (UID: \"a33b17a4-a052-439f-a02a-fb3cad0ae4fb\") " Nov 03 20:02:47.324443 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:47.324416 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a33b17a4-a052-439f-a02a-fb3cad0ae4fb-kube-api-access-68h8s\" (UniqueName: \"kubernetes.io/projected/a33b17a4-a052-439f-a02a-fb3cad0ae4fb-kube-api-access-68h8s\") pod \"a33b17a4-a052-439f-a02a-fb3cad0ae4fb\" (UID: \"a33b17a4-a052-439f-a02a-fb3cad0ae4fb\") " Nov 03 20:02:47.324734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:47.324329 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33b17a4-a052-439f-a02a-fb3cad0ae4fb-maas-api-openshift-ca" pod "a33b17a4-a052-439f-a02a-fb3cad0ae4fb" (UID: "a33b17a4-a052-439f-a02a-fb3cad0ae4fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:47.346686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:47.324935 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a33b17a4-a052-439f-a02a-fb3cad0ae4fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33b17a4-a052-439f-a02a-fb3cad0ae4fb-maas-api-xks-ca\") pod \"a33b17a4-a052-439f-a02a-fb3cad0ae4fb\" (UID: \"a33b17a4-a052-439f-a02a-fb3cad0ae4fb\") " Nov 03 20:02:47.346686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:47.325573 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33b17a4-a052-439f-a02a-fb3cad0ae4fb-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:47.398876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:47.398814 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a33b17a4-a052-439f-a02a-fb3cad0ae4fb-maas-api-xks-ca" pod "a33b17a4-a052-439f-a02a-fb3cad0ae4fb" (UID: "a33b17a4-a052-439f-a02a-fb3cad0ae4fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:47.426685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:47.426040 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33b17a4-a052-439f-a02a-fb3cad0ae4fb-kube-api-access-68h8s" pod "a33b17a4-a052-439f-a02a-fb3cad0ae4fb" (UID: "a33b17a4-a052-439f-a02a-fb3cad0ae4fb"). InnerVolumeSpecName "kube-api-access-68h8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:47.430699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:47.427346 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68h8s\" (UniqueName: \"kubernetes.io/projected/a33b17a4-a052-439f-a02a-fb3cad0ae4fb-kube-api-access-68h8s\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:47.430699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:47.427389 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33b17a4-a052-439f-a02a-fb3cad0ae4fb-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:47.865687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:47.863325 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda4db2e-s8gjr" Nov 03 20:02:48.111780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:48.111731 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 20:02:48.201521 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:48.201429 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda4db2e-s8gjr" event={"ID":"a33b17a4-a052-439f-a02a-fb3cad0ae4fb","Type":"ContainerDied","Data":"5268cb871dfcba87bbcabed12c8b21e85762f1acebc6ea5db2ff028e006ba305"} Nov 03 20:02:48.237702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:48.237673 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5268cb871dfcba87bbcabed12c8b21e85762f1acebc6ea5db2ff028e006ba305" Nov 03 20:02:48.251837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:48.251794 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:48.306686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:48.305757 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:49.057694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:49.049427 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:49.279076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:49.279033 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:02:49.299464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:49.299431 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 20:02:49.353745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:49.307743 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:49.387688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:49.323150 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:49.387688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:49.360533 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 20:02:49.387688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:49.360600 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 20:02:49.845808 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:49.845745 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:02:50.068809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:50.068755 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:50.219855 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:50.187629 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:02:50.307694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:50.300568 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:02:50.371793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:50.371745 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 20:02:50.443729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:50.377728 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:02:50.859011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:50.858965 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:50.865196 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:50.865169 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 20:02:51.075690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:51.073594 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:51.328692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:51.327008 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:51.540686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:51.540603 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:02:51.558686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:51.541538 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 20:02:51.584423 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:51.584321 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:51.597687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:51.595291 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:02:51.597687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:51.597357 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 20:02:51.632264 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:51.623918 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 20:02:51.646952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:51.646923 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 20:02:51.914088 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:51.913999 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r"] Nov 03 20:02:51.914788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:51.914765 5059 cpu_manager.go:414] "containerMap: removing container" podUID="a33b17a4-a052-439f-a02a-fb3cad0ae4fb" containerName="revoke-keys" Nov 03 20:02:51.914788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:51.914787 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a33b17a4-a052-439f-a02a-fb3cad0ae4fb" containerName="revoke-keys" Nov 03 20:02:51.914984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:51.914969 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a33b17a4-a052-439f-a02a-fb3cad0ae4fb" containerName="revoke-keys" Nov 03 20:02:51.925111 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:51.925085 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r"] Nov 03 20:02:51.929103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:51.925991 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:51.935048 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:51.935023 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-942930-openshift-default-dockercfg-vwmg4\"" Nov 03 20:02:52.048450 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.040188 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwcjn\" (UniqueName: \"kubernetes.io/projected/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-kube-api-access-vwcjn\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.048903 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.048862 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-token\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.049172 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.049150 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-credential-socket\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.049520 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.049441 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-workload-socket\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.049731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.049698 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-podinfo\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.049897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.049880 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-workload-certs\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.050040 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.050024 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istiod-ca-cert\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.050293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.050276 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-envoy\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.050452 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.050438 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-data\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.153688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.153601 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istiod-ca-cert\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.153893 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.153770 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-envoy\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.153893 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.153820 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-data\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.153893 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.153873 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwcjn\" (UniqueName: \"kubernetes.io/projected/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-kube-api-access-vwcjn\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.154087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.153926 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-token\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.154087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.154005 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-credential-socket\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.154216 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.154111 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-workload-socket\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.154216 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.154143 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-podinfo\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.154216 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.154200 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-workload-certs\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.181694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.181592 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istiod-ca-cert\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.183160 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.183131 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-data\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.183649 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.183601 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-credential-socket\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.184166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.184145 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-workload-socket\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.185106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.185084 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-workload-certs\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.199262 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.199096 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwcjn\" (UniqueName: \"kubernetes.io/projected/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-kube-api-access-vwcjn\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.203840 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.203779 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-envoy\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.207705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.207641 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-token\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.232145 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.232079 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-podinfo\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.443929 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.443837 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:52.805894 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:02:52.805861 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade5b9f4_d822_4418_a31b_f0c0a21ddb0c.slice/crio-4de41ea151f3956a8493e5e18a1b5d1167462b47993743ad2f7cd53a11edc886 WatchSource:0}: Error finding container 4de41ea151f3956a8493e5e18a1b5d1167462b47993743ad2f7cd53a11edc886: Status 404 returned error can't find the container with id 4de41ea151f3956a8493e5e18a1b5d1167462b47993743ad2f7cd53a11edc886 Nov 03 20:02:52.806744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.806717 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r"] Nov 03 20:02:52.808530 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.808500 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:02:52.808619 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.808559 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:02:52.808619 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:52.808592 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:02:53.718389 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:53.718356 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" event={"ID":"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c","Type":"ContainerStarted","Data":"083613d8d2a66dbf18be20acf40c9c2e55284f3701c7490d658626a11a1dff74"} Nov 03 20:02:53.718913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:53.718409 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" event={"ID":"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c","Type":"ContainerStarted","Data":"4de41ea151f3956a8493e5e18a1b5d1167462b47993743ad2f7cd53a11edc886"} Nov 03 20:02:53.747188 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:53.747135 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" podStartSLOduration=2.747120786 podStartE2EDuration="2.747120786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:53.742872701 +0000 UTC m=+1568.000669970" watchObservedRunningTime="2026-11-03 20:02:53.747120786 +0000 UTC m=+1568.004918042" Nov 03 20:02:54.446012 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:54.445959 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:54.453904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:54.453872 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:54.725672 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:54.725582 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:54.726618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:54.726595 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:02:59.621610 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:59.621578 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3abefc2-9ff2n"] Nov 03 20:02:59.626123 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:59.626098 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3abefc2-9ff2n" Nov 03 20:02:59.631028 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:59.631006 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:02:59.631757 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:59.631736 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:02:59.637409 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:59.637382 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3abefc2-9ff2n"] Nov 03 20:02:59.711440 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:59.711406 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3abefc2-9ff2n\" (UID: \"ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3abefc2-9ff2n" Nov 03 20:02:59.711608 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:59.711520 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zck6\" (UniqueName: \"kubernetes.io/projected/ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1-kube-api-access-6zck6\") pod \"maas-api-revoke-sub-e3abefc2-9ff2n\" (UID: \"ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3abefc2-9ff2n" Nov 03 20:02:59.711747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:59.711727 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3abefc2-9ff2n\" (UID: \"ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3abefc2-9ff2n" Nov 03 20:02:59.808707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:59.808655 5059 generic.go:356] "Generic (PLEG): container finished" podID="613a180b-7079-4633-8060-c4adb54db7f8" containerID="f1bfc585a41bbd3e04faf680f0d866afc990a6d0cc1f757a0ad49c4504dcfd97" exitCode=0 Nov 03 20:02:59.808877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:59.808710 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" event={"ID":"613a180b-7079-4633-8060-c4adb54db7f8","Type":"ContainerDied","Data":"f1bfc585a41bbd3e04faf680f0d866afc990a6d0cc1f757a0ad49c4504dcfd97"} Nov 03 20:02:59.809184 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:59.809166 5059 scope.go:122] "RemoveContainer" containerID="f1bfc585a41bbd3e04faf680f0d866afc990a6d0cc1f757a0ad49c4504dcfd97" Nov 03 20:02:59.810645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:59.810616 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:02:59.810738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:59.810721 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:02:59.810781 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:59.810762 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:02:59.813075 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:59.813030 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3abefc2-9ff2n\" (UID: \"ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3abefc2-9ff2n" Nov 03 20:02:59.813166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:59.813151 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3abefc2-9ff2n\" (UID: \"ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3abefc2-9ff2n" Nov 03 20:02:59.813213 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:59.813192 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zck6\" (UniqueName: \"kubernetes.io/projected/ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1-kube-api-access-6zck6\") pod \"maas-api-revoke-sub-e3abefc2-9ff2n\" (UID: \"ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3abefc2-9ff2n" Nov 03 20:02:59.813838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:59.813817 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3abefc2-9ff2n\" (UID: \"ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3abefc2-9ff2n" Nov 03 20:02:59.815469 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:59.815448 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3abefc2-9ff2n\" (UID: \"ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3abefc2-9ff2n" Nov 03 20:02:59.825426 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:59.825407 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zck6\" (UniqueName: \"kubernetes.io/projected/ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1-kube-api-access-6zck6\") pod \"maas-api-revoke-sub-e3abefc2-9ff2n\" (UID: \"ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3abefc2-9ff2n" Nov 03 20:02:59.968656 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:02:59.958901 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3abefc2-9ff2n" Nov 03 20:03:00.308713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:00.308679 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3abefc2-9ff2n"] Nov 03 20:03:00.309841 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:03:00.309813 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac60a0ea_6cbc_441e_9d6b_986d5eb0c6a1.slice/crio-db34d919707bed39a7d0c88163a6ccf9fff92db04c1703835a85649042c7da84 WatchSource:0}: Error finding container db34d919707bed39a7d0c88163a6ccf9fff92db04c1703835a85649042c7da84: Status 404 returned error can't find the container with id db34d919707bed39a7d0c88163a6ccf9fff92db04c1703835a85649042c7da84 Nov 03 20:03:00.547123 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:00.547090 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584f7a67-tqzsq"] Nov 03 20:03:00.552843 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:00.552823 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584f7a67-tqzsq" Nov 03 20:03:00.597905 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:00.597819 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584f7a67-tqzsq"] Nov 03 20:03:00.625896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:00.625865 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c7c194-490b-4fcf-b043-249e7aa0aa15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-584f7a67-tqzsq\" (UID: \"24c7c194-490b-4fcf-b043-249e7aa0aa15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584f7a67-tqzsq" Nov 03 20:03:00.626388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:00.625952 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkhpm\" (UniqueName: \"kubernetes.io/projected/24c7c194-490b-4fcf-b043-249e7aa0aa15-kube-api-access-nkhpm\") pod \"maas-api-revoke-sub-584f7a67-tqzsq\" (UID: \"24c7c194-490b-4fcf-b043-249e7aa0aa15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584f7a67-tqzsq" Nov 03 20:03:00.626388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:00.626085 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c7c194-490b-4fcf-b043-249e7aa0aa15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-584f7a67-tqzsq\" (UID: \"24c7c194-490b-4fcf-b043-249e7aa0aa15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584f7a67-tqzsq" Nov 03 20:03:00.727645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:00.727606 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c7c194-490b-4fcf-b043-249e7aa0aa15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-584f7a67-tqzsq\" (UID: \"24c7c194-490b-4fcf-b043-249e7aa0aa15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584f7a67-tqzsq" Nov 03 20:03:00.727846 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:00.727725 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkhpm\" (UniqueName: \"kubernetes.io/projected/24c7c194-490b-4fcf-b043-249e7aa0aa15-kube-api-access-nkhpm\") pod \"maas-api-revoke-sub-584f7a67-tqzsq\" (UID: \"24c7c194-490b-4fcf-b043-249e7aa0aa15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584f7a67-tqzsq" Nov 03 20:03:00.727846 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:00.727775 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c7c194-490b-4fcf-b043-249e7aa0aa15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-584f7a67-tqzsq\" (UID: \"24c7c194-490b-4fcf-b043-249e7aa0aa15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584f7a67-tqzsq" Nov 03 20:03:00.728367 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:00.728344 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c7c194-490b-4fcf-b043-249e7aa0aa15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-584f7a67-tqzsq\" (UID: \"24c7c194-490b-4fcf-b043-249e7aa0aa15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584f7a67-tqzsq" Nov 03 20:03:00.730182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:00.730151 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c7c194-490b-4fcf-b043-249e7aa0aa15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-584f7a67-tqzsq\" (UID: \"24c7c194-490b-4fcf-b043-249e7aa0aa15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584f7a67-tqzsq" Nov 03 20:03:00.739957 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:00.739936 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkhpm\" (UniqueName: \"kubernetes.io/projected/24c7c194-490b-4fcf-b043-249e7aa0aa15-kube-api-access-nkhpm\") pod \"maas-api-revoke-sub-584f7a67-tqzsq\" (UID: \"24c7c194-490b-4fcf-b043-249e7aa0aa15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584f7a67-tqzsq" Nov 03 20:03:00.818154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:00.818117 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" event={"ID":"613a180b-7079-4633-8060-c4adb54db7f8","Type":"ContainerStarted","Data":"11e9a10d4054b2c7a58b27f8c6b92f3989b82e42bd63266b82cd4e670d6b18fc"} Nov 03 20:03:00.820025 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:00.820001 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3abefc2-9ff2n" event={"ID":"ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1","Type":"ContainerStarted","Data":"33b2fa04c0c89641a426819ffbf53835935a93b8ed77c9e64765b51cf4eaa1e3"} Nov 03 20:03:00.820156 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:00.820044 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3abefc2-9ff2n" event={"ID":"ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1","Type":"ContainerStarted","Data":"db34d919707bed39a7d0c88163a6ccf9fff92db04c1703835a85649042c7da84"} Nov 03 20:03:00.878744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:00.868938 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3abefc2-9ff2n" podStartSLOduration=1.868919454 podStartE2EDuration="1.868919454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:00.867892066 +0000 UTC m=+1575.125689323" watchObservedRunningTime="2026-11-03 20:03:00.868919454 +0000 UTC m=+1575.126716712" Nov 03 20:03:00.900104 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:00.900066 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584f7a67-tqzsq" Nov 03 20:03:01.192110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:01.192078 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584f7a67-tqzsq"] Nov 03 20:03:01.194464 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:03:01.193987 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c7c194_490b_4fcf_b043_249e7aa0aa15.slice/crio-2ef0dccf74b71c8e27e5d9d799dc6168fafad69ba954a64746ab53a197d508f0 WatchSource:0}: Error finding container 2ef0dccf74b71c8e27e5d9d799dc6168fafad69ba954a64746ab53a197d508f0: Status 404 returned error can't find the container with id 2ef0dccf74b71c8e27e5d9d799dc6168fafad69ba954a64746ab53a197d508f0 Nov 03 20:03:01.736574 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:01.736538 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:03:01.737811 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:01.737785 5059 patch_prober.go:29] interesting pod/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 20:03:01.737901 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:01.737843 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" podUID="613a180b-7079-4633-8060-c4adb54db7f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 20:03:01.829230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:01.829146 5059 generic.go:356] "Generic (PLEG): container finished" podID="ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1" containerID="33b2fa04c0c89641a426819ffbf53835935a93b8ed77c9e64765b51cf4eaa1e3" exitCode=0 Nov 03 20:03:01.829380 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:01.829233 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3abefc2-9ff2n" event={"ID":"ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1","Type":"ContainerDied","Data":"33b2fa04c0c89641a426819ffbf53835935a93b8ed77c9e64765b51cf4eaa1e3"} Nov 03 20:03:01.831027 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:01.831001 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584f7a67-tqzsq" event={"ID":"24c7c194-490b-4fcf-b043-249e7aa0aa15","Type":"ContainerStarted","Data":"ac82175b4c92dd7084229550834cdd419f955a76a25aacbc757f7a5944d129fe"} Nov 03 20:03:01.831158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:01.831044 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584f7a67-tqzsq" event={"ID":"24c7c194-490b-4fcf-b043-249e7aa0aa15","Type":"ContainerStarted","Data":"2ef0dccf74b71c8e27e5d9d799dc6168fafad69ba954a64746ab53a197d508f0"} Nov 03 20:03:01.905558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:01.905508 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584f7a67-tqzsq" podStartSLOduration=1.905492827 podStartE2EDuration="1.905492827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:01.903472217 +0000 UTC m=+1576.161269490" watchObservedRunningTime="2026-11-03 20:03:01.905492827 +0000 UTC m=+1576.163290219" Nov 03 20:03:02.737707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:02.736394 5059 patch_prober.go:29] interesting pod/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 20:03:02.737707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:02.736455 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" podUID="613a180b-7079-4633-8060-c4adb54db7f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 20:03:02.840494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:02.840455 5059 generic.go:356] "Generic (PLEG): container finished" podID="24c7c194-490b-4fcf-b043-249e7aa0aa15" containerID="ac82175b4c92dd7084229550834cdd419f955a76a25aacbc757f7a5944d129fe" exitCode=0 Nov 03 20:03:02.840693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:02.840551 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584f7a67-tqzsq" event={"ID":"24c7c194-490b-4fcf-b043-249e7aa0aa15","Type":"ContainerDied","Data":"ac82175b4c92dd7084229550834cdd419f955a76a25aacbc757f7a5944d129fe"} Nov 03 20:03:03.296711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:03.296680 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3abefc2-9ff2n" Nov 03 20:03:03.435101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:03.435065 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1-maas-api-openshift-ca\") pod \"ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1\" (UID: \"ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1\") " Nov 03 20:03:03.435303 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:03.435285 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1-maas-api-xks-ca\") pod \"ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1\" (UID: \"ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1\") " Nov 03 20:03:03.435366 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:03.435354 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1-kube-api-access-6zck6\" (UniqueName: \"kubernetes.io/projected/ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1-kube-api-access-6zck6\") pod \"ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1\" (UID: \"ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1\") " Nov 03 20:03:03.435427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:03.435389 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1-maas-api-openshift-ca" pod "ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1" (UID: "ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:03.435811 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:03.435790 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:03.437339 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:03.437320 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1-maas-api-xks-ca" pod "ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1" (UID: "ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:03:03.437657 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:03.437641 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1-kube-api-access-6zck6" pod "ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1" (UID: "ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1"). InnerVolumeSpecName "kube-api-access-6zck6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:03.552743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:03.549498 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:03.552743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:03.549554 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zck6\" (UniqueName: \"kubernetes.io/projected/ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1-kube-api-access-6zck6\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:03.736777 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:03.736741 5059 patch_prober.go:29] interesting pod/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 20:03:03.736949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:03.736802 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" podUID="613a180b-7079-4633-8060-c4adb54db7f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 20:03:03.849500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:03.849471 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3abefc2-9ff2n" Nov 03 20:03:03.860984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:03.860110 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3abefc2-9ff2n" event={"ID":"ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1","Type":"ContainerDied","Data":"db34d919707bed39a7d0c88163a6ccf9fff92db04c1703835a85649042c7da84"} Nov 03 20:03:03.860984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:03.860195 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db34d919707bed39a7d0c88163a6ccf9fff92db04c1703835a85649042c7da84" Nov 03 20:03:04.171733 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:04.171511 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584f7a67-tqzsq" Nov 03 20:03:04.274745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:04.273183 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24c7c194-490b-4fcf-b043-249e7aa0aa15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c7c194-490b-4fcf-b043-249e7aa0aa15-maas-api-xks-ca\") pod \"24c7c194-490b-4fcf-b043-249e7aa0aa15\" (UID: \"24c7c194-490b-4fcf-b043-249e7aa0aa15\") " Nov 03 20:03:04.274745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:04.273420 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24c7c194-490b-4fcf-b043-249e7aa0aa15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c7c194-490b-4fcf-b043-249e7aa0aa15-maas-api-openshift-ca\") pod \"24c7c194-490b-4fcf-b043-249e7aa0aa15\" (UID: \"24c7c194-490b-4fcf-b043-249e7aa0aa15\") " Nov 03 20:03:04.274745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:04.273451 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c7c194-490b-4fcf-b043-249e7aa0aa15-kube-api-access-nkhpm\" (UniqueName: \"kubernetes.io/projected/24c7c194-490b-4fcf-b043-249e7aa0aa15-kube-api-access-nkhpm\") pod \"24c7c194-490b-4fcf-b043-249e7aa0aa15\" (UID: \"24c7c194-490b-4fcf-b043-249e7aa0aa15\") " Nov 03 20:03:04.291688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:04.278886 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c7c194-490b-4fcf-b043-249e7aa0aa15-maas-api-openshift-ca" pod "24c7c194-490b-4fcf-b043-249e7aa0aa15" (UID: "24c7c194-490b-4fcf-b043-249e7aa0aa15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:04.310911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:04.302828 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24c7c194-490b-4fcf-b043-249e7aa0aa15-maas-api-xks-ca" pod "24c7c194-490b-4fcf-b043-249e7aa0aa15" (UID: "24c7c194-490b-4fcf-b043-249e7aa0aa15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:03:04.310911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:04.310220 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c7c194-490b-4fcf-b043-249e7aa0aa15-kube-api-access-nkhpm" pod "24c7c194-490b-4fcf-b043-249e7aa0aa15" (UID: "24c7c194-490b-4fcf-b043-249e7aa0aa15"). InnerVolumeSpecName "kube-api-access-nkhpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:04.391305 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:04.390813 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c7c194-490b-4fcf-b043-249e7aa0aa15-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:04.391305 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:04.390845 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c7c194-490b-4fcf-b043-249e7aa0aa15-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:04.391305 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:04.390863 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkhpm\" (UniqueName: \"kubernetes.io/projected/24c7c194-490b-4fcf-b043-249e7aa0aa15-kube-api-access-nkhpm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:04.739697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:04.738223 5059 patch_prober.go:29] interesting pod/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 20:03:04.739697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:04.738297 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" podUID="613a180b-7079-4633-8060-c4adb54db7f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 20:03:04.872685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:04.859518 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584f7a67-tqzsq" event={"ID":"24c7c194-490b-4fcf-b043-249e7aa0aa15","Type":"ContainerDied","Data":"2ef0dccf74b71c8e27e5d9d799dc6168fafad69ba954a64746ab53a197d508f0"} Nov 03 20:03:04.872685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:04.859574 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ef0dccf74b71c8e27e5d9d799dc6168fafad69ba954a64746ab53a197d508f0" Nov 03 20:03:04.872685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:04.859636 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584f7a67-tqzsq" Nov 03 20:03:05.256961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.256927 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:03:05.390685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.390224 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l"] Nov 03 20:03:05.391749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.391395 5059 cpu_manager.go:414] "containerMap: removing container" podUID="24c7c194-490b-4fcf-b043-249e7aa0aa15" containerName="revoke-keys" Nov 03 20:03:05.391749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.391425 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c7c194-490b-4fcf-b043-249e7aa0aa15" containerName="revoke-keys" Nov 03 20:03:05.391749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.391460 5059 cpu_manager.go:414] "containerMap: removing container" podUID="ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1" containerName="revoke-keys" Nov 03 20:03:05.391749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.391469 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1" containerName="revoke-keys" Nov 03 20:03:05.391749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.391686 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c7c194-490b-4fcf-b043-249e7aa0aa15" containerName="revoke-keys" Nov 03 20:03:05.391749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.391705 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1" containerName="revoke-keys" Nov 03 20:03:05.396700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.396088 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l"] Nov 03 20:03:05.396700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.396200 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.414069 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.411998 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p"] Nov 03 20:03:05.419685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.416963 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.431068 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.428059 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p"] Nov 03 20:03:05.549790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.543590 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.549790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.543675 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp4jg\" (UniqueName: \"kubernetes.io/projected/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-kube-api-access-mp4jg\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.549790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.543710 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.549790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.543751 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.549790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.543800 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.549790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.543875 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.549790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.543939 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.549790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.543967 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.549790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.544011 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.549790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.544042 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.549790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.544112 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60b061f-60f0-48bb-8668-fd828a372a8a-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.549790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.544187 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.549790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.544277 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.549790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.544316 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxct6\" (UniqueName: \"kubernetes.io/projected/e60b061f-60f0-48bb-8668-fd828a372a8a-kube-api-access-jxct6\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.549790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.544364 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.549790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.544411 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.549790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.544444 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.549790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.544482 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.646148 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.646116 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp4jg\" (UniqueName: \"kubernetes.io/projected/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-kube-api-access-mp4jg\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.646359 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.646155 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.646359 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.646193 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.646359 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.646237 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.646359 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.646299 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.646359 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.646353 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.646564 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.646378 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.646564 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.646417 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.646564 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.646447 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.646564 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.646512 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60b061f-60f0-48bb-8668-fd828a372a8a-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.646716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.646580 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.646716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.646587 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.647046 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.647017 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.647209 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.647184 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.647848 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.647826 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxct6\" (UniqueName: \"kubernetes.io/projected/e60b061f-60f0-48bb-8668-fd828a372a8a-kube-api-access-jxct6\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.647954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.647880 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.647954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.647925 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.648062 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.647958 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.648062 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.647960 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60b061f-60f0-48bb-8668-fd828a372a8a-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.648062 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.647995 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.648062 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.648039 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.648281 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.647254 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.648281 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.647556 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.648281 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.647316 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.648464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.648383 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.648529 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.648466 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.648529 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.647776 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.648997 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.648939 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.649852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.649696 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.650451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.650433 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.650899 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.650880 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.651436 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.651413 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.682682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.682192 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.686272 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.685520 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp4jg\" (UniqueName: \"kubernetes.io/projected/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-kube-api-access-mp4jg\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.686804 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.686775 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.687316 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.687292 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxct6\" (UniqueName: \"kubernetes.io/projected/e60b061f-60f0-48bb-8668-fd828a372a8a-kube-api-access-jxct6\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:05.734126 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.734096 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:05.736314 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.736292 5059 patch_prober.go:29] interesting pod/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 20:03:05.736398 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.736335 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" podUID="613a180b-7079-4633-8060-c4adb54db7f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 20:03:05.752120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:05.752098 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:06.063366 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:06.063331 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l"] Nov 03 20:03:06.065148 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:03:06.065107 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf601d7_8af9_4beb_b64e_f2e9c6b00711.slice/crio-f2543d77f2decc8ee1bea967fea0369c0aeed38031f2a8770cb5da2b14b75e92 WatchSource:0}: Error finding container f2543d77f2decc8ee1bea967fea0369c0aeed38031f2a8770cb5da2b14b75e92: Status 404 returned error can't find the container with id f2543d77f2decc8ee1bea967fea0369c0aeed38031f2a8770cb5da2b14b75e92 Nov 03 20:03:06.068307 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:06.068271 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:06.068435 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:06.068356 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:06.068435 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:06.068401 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:06.088491 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:03:06.088448 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60b061f_60f0_48bb_8668_fd828a372a8a.slice/crio-567231dcb6e8df2363f355c4ac1cc809d01976747c63de08d98a3ae8b6e76df8 WatchSource:0}: Error finding container 567231dcb6e8df2363f355c4ac1cc809d01976747c63de08d98a3ae8b6e76df8: Status 404 returned error can't find the container with id 567231dcb6e8df2363f355c4ac1cc809d01976747c63de08d98a3ae8b6e76df8 Nov 03 20:03:06.090438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:06.090329 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p"] Nov 03 20:03:06.091399 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:06.091359 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:06.091510 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:06.091453 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:06.091510 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:06.091490 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:06.744836 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:06.744802 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:03:06.746131 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:06.746103 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:03:06.931811 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:06.931771 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" event={"ID":"2cf601d7-8af9-4beb-b64e-f2e9c6b00711","Type":"ContainerStarted","Data":"84938316ee8fac845a7ea3c47c48e02b03c836b41aeb68977686718055dab32e"} Nov 03 20:03:06.932050 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:06.931850 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" event={"ID":"2cf601d7-8af9-4beb-b64e-f2e9c6b00711","Type":"ContainerStarted","Data":"f2543d77f2decc8ee1bea967fea0369c0aeed38031f2a8770cb5da2b14b75e92"} Nov 03 20:03:06.933731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:06.933701 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" event={"ID":"e60b061f-60f0-48bb-8668-fd828a372a8a","Type":"ContainerStarted","Data":"2ba5e6c6035d1030506e79f3a951df6422700eaabd987c4ba622506c7cd704e7"} Nov 03 20:03:06.933869 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:06.933782 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" event={"ID":"e60b061f-60f0-48bb-8668-fd828a372a8a","Type":"ContainerStarted","Data":"567231dcb6e8df2363f355c4ac1cc809d01976747c63de08d98a3ae8b6e76df8"} Nov 03 20:03:07.059556 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:07.059426 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" podStartSLOduration=2.059404926 podStartE2EDuration="2.059404926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:06.95526551 +0000 UTC m=+1581.213062767" watchObservedRunningTime="2026-11-03 20:03:07.059404926 +0000 UTC m=+1581.317202187" Nov 03 20:03:07.117220 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:07.117082 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" podStartSLOduration=2.11705716 podStartE2EDuration="2.11705716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:07.00236543 +0000 UTC m=+1581.260162688" watchObservedRunningTime="2026-11-03 20:03:07.11705716 +0000 UTC m=+1581.374854419" Nov 03 20:03:07.734953 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:07.734824 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:07.741860 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:07.741811 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 20:03:07.742159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:07.742101 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" podUID="2cf601d7-8af9-4beb-b64e-f2e9c6b00711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 20:03:07.760697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:07.754955 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:07.760697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:07.755121 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 20:03:07.760697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:07.755202 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" podUID="e60b061f-60f0-48bb-8668-fd828a372a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 20:03:08.744682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:08.741217 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 20:03:08.744682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:08.741303 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" podUID="2cf601d7-8af9-4beb-b64e-f2e9c6b00711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 20:03:08.756684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:08.756558 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 20:03:08.756684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:08.756628 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" podUID="e60b061f-60f0-48bb-8668-fd828a372a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 20:03:09.607697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:09.604741 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:09.632696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:09.628063 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:03:09.657692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:09.655198 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:09.738688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:09.736419 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 20:03:09.738688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:09.736501 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" podUID="2cf601d7-8af9-4beb-b64e-f2e9c6b00711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 20:03:09.757695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:09.757280 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 20:03:09.757695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:09.757336 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" podUID="e60b061f-60f0-48bb-8668-fd828a372a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 20:03:10.291705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:10.283910 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:10.376124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:10.375965 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:10.627683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:10.624812 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:03:10.651688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:10.629262 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 20:03:10.709937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:10.675839 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:10.709937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:10.676168 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:03:10.709937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:10.686433 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 20:03:10.709937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:10.686494 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 20:03:10.790738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:10.788861 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 20:03:10.790738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:10.788916 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" podUID="2cf601d7-8af9-4beb-b64e-f2e9c6b00711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 20:03:10.921745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:10.919139 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 20:03:10.921745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:10.919198 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" podUID="e60b061f-60f0-48bb-8668-fd828a372a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 20:03:11.140808 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:03:11.130719 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:03:11.321132 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:11.304750 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:11.321132 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:11.305354 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:03:11.396685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:11.390127 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:11.396685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:11.391078 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 20:03:11.641241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:11.641143 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:03:11.725686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:11.691881 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:03:11.725686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:11.691921 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 20:03:11.758684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:11.752800 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 20:03:11.862806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:11.862750 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" podUID="2cf601d7-8af9-4beb-b64e-f2e9c6b00711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 20:03:11.907743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:11.764096 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 20:03:11.909792 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:11.909729 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" podUID="e60b061f-60f0-48bb-8668-fd828a372a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 20:03:11.910316 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:11.909410 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:12.475361 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:12.475312 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:12.538788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:12.520855 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:12.675634 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:12.675510 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x"] Nov 03 20:03:12.762785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:12.748054 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:03:12.762785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:12.749912 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 20:03:12.762785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:12.749952 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" podUID="2cf601d7-8af9-4beb-b64e-f2e9c6b00711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 20:03:12.800229 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:12.788761 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:12.919158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:12.912887 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:12.995657 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:12.992793 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 20:03:13.150877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.104296 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.150877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.108317 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-workload-certs\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.150877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.108393 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-workload-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.150877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.108487 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f043a660-eddb-4de5-88a7-cf0b1fda6512-istiod-ca-cert\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.150877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.108572 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-podinfo\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.150877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.108617 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-credential-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.150877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.108753 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-token\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.150877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.108808 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-data\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.150877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.108847 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-envoy\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.150877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.108906 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh69z\" (UniqueName: \"kubernetes.io/projected/f043a660-eddb-4de5-88a7-cf0b1fda6512-kube-api-access-vh69z\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.150877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.150756 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 20:03:13.150877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.150814 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" podUID="e60b061f-60f0-48bb-8668-fd828a372a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 20:03:13.151689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.151254 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:13.227713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.152589 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-bfc240-openshift-default-dockercfg-dxqxq\"" Nov 03 20:03:13.227713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.218820 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh69z\" (UniqueName: \"kubernetes.io/projected/f043a660-eddb-4de5-88a7-cf0b1fda6512-kube-api-access-vh69z\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.227713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.218914 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-workload-certs\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.227713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.218965 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-workload-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.227713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.219082 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f043a660-eddb-4de5-88a7-cf0b1fda6512-istiod-ca-cert\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.227713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.219199 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-podinfo\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.227713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.219244 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-credential-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.227713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.219374 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-token\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.227713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.219433 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-data\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.227713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.219462 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-envoy\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.250825 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.250791 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-workload-certs\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.251330 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.251305 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-workload-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.438825 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.252047 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f043a660-eddb-4de5-88a7-cf0b1fda6512-istiod-ca-cert\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.439045 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.265120 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-credential-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.445349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.284872 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-data\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.462759 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.458392 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-podinfo\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.521372 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.521327 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:13.708483 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.708396 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x"] Nov 03 20:03:13.748870 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.748402 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:13.750368 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.750330 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 20:03:13.756685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.751007 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" podUID="2cf601d7-8af9-4beb-b64e-f2e9c6b00711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 20:03:13.797732 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.757116 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 20:03:13.798030 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.797995 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" podUID="e60b061f-60f0-48bb-8668-fd828a372a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 20:03:13.798485 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.798463 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 20:03:13.835839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.831211 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh69z\" (UniqueName: \"kubernetes.io/projected/f043a660-eddb-4de5-88a7-cf0b1fda6512-kube-api-access-vh69z\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:13.941775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.941724 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:13.942711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:13.942682 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 20:03:14.047776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.044975 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-envoy\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:14.134546 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.134510 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-token\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:14.147999 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.147970 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:14.216045 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.216011 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 20:03:14.219703 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.216308 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 20:03:14.219970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.219951 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:03:14.329738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.288127 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff"] Nov 03 20:03:14.336452 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.336419 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff"] Nov 03 20:03:14.336628 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.336518 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.341545 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.341517 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-f154ac-openshift-default-dockercfg-xndgl\"" Nov 03 20:03:14.452406 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.451704 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b9d0847-df69-4b6a-b346-2cd206e42339-istiod-ca-cert\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.452406 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.451810 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-workload-certs\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.452406 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.451935 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-credential-socket\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.452406 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.451971 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-podinfo\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.452406 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.452018 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-data\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.452406 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.452126 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-envoy\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.452406 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.452240 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zzxr\" (UniqueName: \"kubernetes.io/projected/3b9d0847-df69-4b6a-b346-2cd206e42339-kube-api-access-6zzxr\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.452406 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.452341 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-token\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.452406 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.452411 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-workload-socket\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.574686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.565341 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-envoy\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.574686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.565449 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zzxr\" (UniqueName: \"kubernetes.io/projected/3b9d0847-df69-4b6a-b346-2cd206e42339-kube-api-access-6zzxr\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.574686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.565537 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-token\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.574686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.565604 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-workload-socket\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.574686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.565707 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b9d0847-df69-4b6a-b346-2cd206e42339-istiod-ca-cert\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.574686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.565774 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-workload-certs\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.574686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.565862 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-credential-socket\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.574686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.565897 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-podinfo\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.574686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.565924 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-data\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.574686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.566428 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-data\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.671294 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.629343 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b9d0847-df69-4b6a-b346-2cd206e42339-istiod-ca-cert\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.714857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.679112 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-credential-socket\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.715124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.690058 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-workload-socket\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.715260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.690500 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-workload-certs\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.715355 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.694345 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-envoy\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.780434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.763032 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-token\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.780434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.779421 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-podinfo\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.780434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.779916 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 20:03:14.780434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.779964 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" podUID="2cf601d7-8af9-4beb-b64e-f2e9c6b00711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 20:03:14.780434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.780332 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 20:03:14.780434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.780373 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" podUID="e60b061f-60f0-48bb-8668-fd828a372a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 20:03:14.811377 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.811346 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zzxr\" (UniqueName: \"kubernetes.io/projected/3b9d0847-df69-4b6a-b346-2cd206e42339-kube-api-access-6zzxr\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:14.828741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.817502 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:14.828970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.828939 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:14.829134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.829114 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:14.833122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:14.833092 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x"] Nov 03 20:03:15.030284 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:15.030264 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:15.493795 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:15.493610 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" event={"ID":"f043a660-eddb-4de5-88a7-cf0b1fda6512","Type":"ContainerStarted","Data":"873a0ffe9f526eaa4ece120b96a6e7e29de1a52daefeff130e5e76d7773021ad"} Nov 03 20:03:15.493795 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:15.493704 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" event={"ID":"f043a660-eddb-4de5-88a7-cf0b1fda6512","Type":"ContainerStarted","Data":"4f71494403b009093856f2e7fa63f4a666e4cba8027d0b0e586e9f6a5daa59fc"} Nov 03 20:03:15.638108 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:15.636838 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs"] Nov 03 20:03:15.638108 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:15.637074 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" podUID="5992c515-22ba-4191-b5f6-8181c094340c" containerName="istio-proxy" containerID="cri-o://6624d12d15032ac6bbfd8aebd5287d8b324d73fb942cc55f988eead98a46a289" gracePeriod=30 Nov 03 20:03:15.640955 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:15.640879 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" podStartSLOduration=5.64086141 podStartE2EDuration="5.64086141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:15.557656711 +0000 UTC m=+1589.815453973" watchObservedRunningTime="2026-11-03 20:03:15.64086141 +0000 UTC m=+1589.898658665" Nov 03 20:03:15.656722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:15.653437 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff"] Nov 03 20:03:15.817088 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:15.816989 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 20:03:15.817346 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:15.817314 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" podUID="2cf601d7-8af9-4beb-b64e-f2e9c6b00711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 20:03:15.817912 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:15.817888 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 20:03:15.835535 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:15.835497 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" podUID="e60b061f-60f0-48bb-8668-fd828a372a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 20:03:15.835780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:15.820230 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:15.836017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:15.835987 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:15.836184 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:15.836162 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:16.152241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:16.152207 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:16.162276 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:16.161946 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:16.506742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:16.506647 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" event={"ID":"3b9d0847-df69-4b6a-b346-2cd206e42339","Type":"ContainerStarted","Data":"f40938fdfd8d12fe6529e191274b54750315e326f7aeec32fac2e699b64c43a1"} Nov 03 20:03:16.506742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:16.506727 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" event={"ID":"3b9d0847-df69-4b6a-b346-2cd206e42339","Type":"ContainerStarted","Data":"b129ab4014dd05316589eb057a3b903e1c83703402f17902eb6f41cc97ab87a3"} Nov 03 20:03:16.507107 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:16.507084 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:16.508354 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:16.508335 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:03:16.547113 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:16.547059 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" podStartSLOduration=3.547042291 podStartE2EDuration="3.547042291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:16.544428646 +0000 UTC m=+1590.802225904" watchObservedRunningTime="2026-11-03 20:03:16.547042291 +0000 UTC m=+1590.804839549" Nov 03 20:03:16.741880 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:16.741843 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 20:03:16.742198 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:16.742154 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" podUID="2cf601d7-8af9-4beb-b64e-f2e9c6b00711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 20:03:16.754515 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:16.754482 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 20:03:16.754697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:16.754530 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" podUID="e60b061f-60f0-48bb-8668-fd828a372a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 20:03:17.032122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:17.031346 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:17.038982 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:17.038836 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:17.517208 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:17.517175 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:17.518801 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:17.518774 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:03:18.052362 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:18.052326 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:18.052950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:18.052923 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:18.054019 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:18.053992 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:03:18.427459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:18.427431 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:18.427951 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:18.427928 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:18.428868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:18.428848 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:03:21.017455 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.017396 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 20:03:21.206752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.206708 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-workload-certs\") pod \"5992c515-22ba-4191-b5f6-8181c094340c\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " Nov 03 20:03:21.206914 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.206778 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5992c515-22ba-4191-b5f6-8181c094340c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5992c515-22ba-4191-b5f6-8181c094340c-istio-podinfo\") pod \"5992c515-22ba-4191-b5f6-8181c094340c\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " Nov 03 20:03:21.206914 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.206889 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-workload-socket\") pod \"5992c515-22ba-4191-b5f6-8181c094340c\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " Nov 03 20:03:21.207039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.207021 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5992c515-22ba-4191-b5f6-8181c094340c-kube-api-access-db29t\" (UniqueName: \"kubernetes.io/projected/5992c515-22ba-4191-b5f6-8181c094340c-kube-api-access-db29t\") pod \"5992c515-22ba-4191-b5f6-8181c094340c\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " Nov 03 20:03:21.207088 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.207058 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-workload-certs" pod "5992c515-22ba-4191-b5f6-8181c094340c" (UID: "5992c515-22ba-4191-b5f6-8181c094340c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:21.207138 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.207083 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5992c515-22ba-4191-b5f6-8181c094340c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5992c515-22ba-4191-b5f6-8181c094340c-istiod-ca-cert\") pod \"5992c515-22ba-4191-b5f6-8181c094340c\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " Nov 03 20:03:21.207198 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.207135 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-istio-data\") pod \"5992c515-22ba-4191-b5f6-8181c094340c\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " Nov 03 20:03:21.207198 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.207188 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-credential-socket\") pod \"5992c515-22ba-4191-b5f6-8181c094340c\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " Nov 03 20:03:21.207315 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.207255 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5992c515-22ba-4191-b5f6-8181c094340c-istio-token\" (UniqueName: \"kubernetes.io/projected/5992c515-22ba-4191-b5f6-8181c094340c-istio-token\") pod \"5992c515-22ba-4191-b5f6-8181c094340c\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " Nov 03 20:03:21.207376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.207342 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-istio-envoy\") pod \"5992c515-22ba-4191-b5f6-8181c094340c\" (UID: \"5992c515-22ba-4191-b5f6-8181c094340c\") " Nov 03 20:03:21.207437 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.207381 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-workload-socket" pod "5992c515-22ba-4191-b5f6-8181c094340c" (UID: "5992c515-22ba-4191-b5f6-8181c094340c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:21.207519 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.207498 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5992c515-22ba-4191-b5f6-8181c094340c-istiod-ca-cert" pod "5992c515-22ba-4191-b5f6-8181c094340c" (UID: "5992c515-22ba-4191-b5f6-8181c094340c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:21.207611 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.207593 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-credential-socket" pod "5992c515-22ba-4191-b5f6-8181c094340c" (UID: "5992c515-22ba-4191-b5f6-8181c094340c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:21.207833 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.207810 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-istio-data" pod "5992c515-22ba-4191-b5f6-8181c094340c" (UID: "5992c515-22ba-4191-b5f6-8181c094340c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:21.208090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.208068 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:21.208241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.208225 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:21.208351 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.208338 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5992c515-22ba-4191-b5f6-8181c094340c-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:21.208442 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.208430 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:21.208538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.208525 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:21.210410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.210381 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5992c515-22ba-4191-b5f6-8181c094340c-kube-api-access-db29t" pod "5992c515-22ba-4191-b5f6-8181c094340c" (UID: "5992c515-22ba-4191-b5f6-8181c094340c"). InnerVolumeSpecName "kube-api-access-db29t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:21.210515 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.210410 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5992c515-22ba-4191-b5f6-8181c094340c-istio-podinfo" pod "5992c515-22ba-4191-b5f6-8181c094340c" (UID: "5992c515-22ba-4191-b5f6-8181c094340c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:03:21.210515 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.210500 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-istio-envoy" pod "5992c515-22ba-4191-b5f6-8181c094340c" (UID: "5992c515-22ba-4191-b5f6-8181c094340c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:21.210600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.210586 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5992c515-22ba-4191-b5f6-8181c094340c-istio-token" pod "5992c515-22ba-4191-b5f6-8181c094340c" (UID: "5992c515-22ba-4191-b5f6-8181c094340c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:21.310361 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.310292 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5992c515-22ba-4191-b5f6-8181c094340c-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:21.310361 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.310320 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5992c515-22ba-4191-b5f6-8181c094340c-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:21.310361 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.310330 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5992c515-22ba-4191-b5f6-8181c094340c-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:21.310361 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.310340 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db29t\" (UniqueName: \"kubernetes.io/projected/5992c515-22ba-4191-b5f6-8181c094340c-kube-api-access-db29t\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:21.568120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.568021 5059 generic.go:356] "Generic (PLEG): container finished" podID="5992c515-22ba-4191-b5f6-8181c094340c" containerID="6624d12d15032ac6bbfd8aebd5287d8b324d73fb942cc55f988eead98a46a289" exitCode=0 Nov 03 20:03:21.568341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.568320 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" event={"ID":"5992c515-22ba-4191-b5f6-8181c094340c","Type":"ContainerDied","Data":"6624d12d15032ac6bbfd8aebd5287d8b324d73fb942cc55f988eead98a46a289"} Nov 03 20:03:21.568413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.568395 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" event={"ID":"5992c515-22ba-4191-b5f6-8181c094340c","Type":"ContainerDied","Data":"ff98738e80a507ddeec44b24ad2a5cc42928be4ad60b38c4d107ac52a6785c37"} Nov 03 20:03:21.568494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.568450 5059 scope.go:122] "RemoveContainer" containerID="6624d12d15032ac6bbfd8aebd5287d8b324d73fb942cc55f988eead98a46a289" Nov 03 20:03:21.568714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.568697 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs" Nov 03 20:03:21.591346 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.591319 5059 scope.go:122] "RemoveContainer" containerID="6624d12d15032ac6bbfd8aebd5287d8b324d73fb942cc55f988eead98a46a289" Nov 03 20:03:21.591719 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:03:21.591648 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6624d12d15032ac6bbfd8aebd5287d8b324d73fb942cc55f988eead98a46a289\": container with ID starting with 6624d12d15032ac6bbfd8aebd5287d8b324d73fb942cc55f988eead98a46a289 not found: ID does not exist" containerID="6624d12d15032ac6bbfd8aebd5287d8b324d73fb942cc55f988eead98a46a289" Nov 03 20:03:21.591828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.591717 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6624d12d15032ac6bbfd8aebd5287d8b324d73fb942cc55f988eead98a46a289"} err="failed to get container status \"6624d12d15032ac6bbfd8aebd5287d8b324d73fb942cc55f988eead98a46a289\": rpc error: code = NotFound desc = could not find container \"6624d12d15032ac6bbfd8aebd5287d8b324d73fb942cc55f988eead98a46a289\": container with ID starting with 6624d12d15032ac6bbfd8aebd5287d8b324d73fb942cc55f988eead98a46a289 not found: ID does not exist" Nov 03 20:03:21.604825 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.604792 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs"] Nov 03 20:03:21.610001 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.609973 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gvdgs"] Nov 03 20:03:21.859405 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:21.859369 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5992c515-22ba-4191-b5f6-8181c094340c" path="/var/lib/kubelet/pods/5992c515-22ba-4191-b5f6-8181c094340c/volumes" Nov 03 20:03:27.564247 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.564209 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw"] Nov 03 20:03:27.565167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.565107 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" podUID="613a180b-7079-4633-8060-c4adb54db7f8" containerName="istio-proxy" containerID="cri-o://11e9a10d4054b2c7a58b27f8c6b92f3989b82e42bd63266b82cd4e670d6b18fc" gracePeriod=30 Nov 03 20:03:27.613960 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.613927 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl"] Nov 03 20:03:27.615258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.615228 5059 cpu_manager.go:414] "containerMap: removing container" podUID="5992c515-22ba-4191-b5f6-8181c094340c" containerName="istio-proxy" Nov 03 20:03:27.615258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.615259 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5992c515-22ba-4191-b5f6-8181c094340c" containerName="istio-proxy" Nov 03 20:03:27.615474 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.615460 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5992c515-22ba-4191-b5f6-8181c094340c" containerName="istio-proxy" Nov 03 20:03:27.621744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.621717 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.642755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.632342 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl"] Nov 03 20:03:27.682509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.682481 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-81a5ae-openshift-default-dockercfg-pvl2f\"" Nov 03 20:03:27.694557 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.694522 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-data\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.694747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.694611 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-workload-socket\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.694747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.694640 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istiod-ca-cert\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.694747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.694715 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-envoy\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.694913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.694753 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-token\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.694913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.694879 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-workload-certs\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.694913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.694905 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-podinfo\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.695027 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.694956 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-credential-socket\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.695149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.695113 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mbqm\" (UniqueName: \"kubernetes.io/projected/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-kube-api-access-7mbqm\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.795913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.795882 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-data\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.796097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.795948 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-workload-socket\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.796097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.795975 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istiod-ca-cert\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.796097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.796013 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-envoy\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.796097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.796050 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-token\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.796341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.796107 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-workload-certs\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.796341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.796130 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-podinfo\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.796341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.796190 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-credential-socket\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.796341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.796261 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mbqm\" (UniqueName: \"kubernetes.io/projected/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-kube-api-access-7mbqm\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.796549 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.796476 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-workload-socket\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.796613 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.796587 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-workload-certs\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.796695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.796657 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-credential-socket\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.796781 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.796761 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-data\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.796853 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.796805 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istiod-ca-cert\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.798416 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.798397 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-envoy\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.798802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.798783 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-podinfo\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.809301 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.809279 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mbqm\" (UniqueName: \"kubernetes.io/projected/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-kube-api-access-7mbqm\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.811024 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.810998 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-token\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:27.975766 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:27.953237 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:28.356113 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:28.353183 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl"] Nov 03 20:03:28.356290 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:28.356244 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:28.356446 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:28.356333 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:28.356446 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:28.356376 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:28.639636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:28.639555 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" event={"ID":"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3","Type":"ContainerStarted","Data":"aafac57bb6098e1fd609b713edc7da1b412af807beeafa6e9063b066a1c006b1"} Nov 03 20:03:28.639636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:28.639621 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" event={"ID":"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3","Type":"ContainerStarted","Data":"d978f9fb5358f9acf3231902c969f661ecdcd235d6162a4233dd2e7376cf7eb7"} Nov 03 20:03:28.667191 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:28.667136 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" podStartSLOduration=1.667119376 podStartE2EDuration="1.667119376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:28.664247801 +0000 UTC m=+1602.922045059" watchObservedRunningTime="2026-11-03 20:03:28.667119376 +0000 UTC m=+1602.924916634" Nov 03 20:03:28.959746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:28.959654 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:29.965632 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:29.965604 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:30.655537 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:30.655507 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:30.656460 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:30.656438 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:03:32.186970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.185386 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8"] Nov 03 20:03:32.192279 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.191277 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.204308 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.203154 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-88a18e-openshift-default-dockercfg-bs4qg\"" Nov 03 20:03:32.221605 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.221573 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8"] Nov 03 20:03:32.278076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.278040 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c609469-d7d7-4530-8d74-e34aa56dda8f-istiod-ca-cert\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.278254 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.278144 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-token\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.278254 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.278215 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-podinfo\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.278383 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.278252 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-workload-socket\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.278383 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.278276 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wskdx\" (UniqueName: \"kubernetes.io/projected/4c609469-d7d7-4530-8d74-e34aa56dda8f-kube-api-access-wskdx\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.278491 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.278397 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-envoy\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.278491 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.278438 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-data\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.278595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.278511 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-workload-certs\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.278595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.278544 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-credential-socket\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.379739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.379697 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-envoy\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.379932 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.379764 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-data\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.379932 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.379834 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-workload-certs\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.379932 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.379865 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-credential-socket\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.380180 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.380016 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c609469-d7d7-4530-8d74-e34aa56dda8f-istiod-ca-cert\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.380180 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.380097 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-token\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.380180 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.380158 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-podinfo\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.380351 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.380192 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-workload-socket\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.380351 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.380218 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wskdx\" (UniqueName: \"kubernetes.io/projected/4c609469-d7d7-4530-8d74-e34aa56dda8f-kube-api-access-wskdx\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.381400 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.381372 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-workload-certs\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.381776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.381748 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c609469-d7d7-4530-8d74-e34aa56dda8f-istiod-ca-cert\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.382572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.381814 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-data\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.382572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.382057 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-credential-socket\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.382787 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.382297 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-workload-socket\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.384701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.384677 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-envoy\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.385711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.385641 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-podinfo\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.392592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.392569 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wskdx\" (UniqueName: \"kubernetes.io/projected/4c609469-d7d7-4530-8d74-e34aa56dda8f-kube-api-access-wskdx\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.395499 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.395295 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-token\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.517368 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.517289 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:32.928695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:32.927485 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8"] Nov 03 20:03:33.329706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:33.326535 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:33.329706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:33.326636 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:33.329706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:33.326701 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:34.170691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:34.170073 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:03:34.170691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:34.170143 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" podUID="eac98813-7b4c-4c4a-8bf9-444bba2fa644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:34.276145 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:34.259461 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 03 20:03:34.486696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:34.484748 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:34.579959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:34.579921 5059 generic.go:356] "Generic (PLEG): container finished" podID="613a180b-7079-4633-8060-c4adb54db7f8" containerID="11e9a10d4054b2c7a58b27f8c6b92f3989b82e42bd63266b82cd4e670d6b18fc" exitCode=0 Nov 03 20:03:34.738771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:34.738722 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" event={"ID":"613a180b-7079-4633-8060-c4adb54db7f8","Type":"ContainerDied","Data":"11e9a10d4054b2c7a58b27f8c6b92f3989b82e42bd63266b82cd4e670d6b18fc"} Nov 03 20:03:34.742790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:34.742761 5059 scope.go:122] "RemoveContainer" containerID="f1bfc585a41bbd3e04faf680f0d866afc990a6d0cc1f757a0ad49c4504dcfd97" Nov 03 20:03:34.759881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:34.759747 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:34.784713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:34.784019 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" event={"ID":"4c609469-d7d7-4530-8d74-e34aa56dda8f","Type":"ContainerStarted","Data":"e1c46745a7ea7a6fa41f53b3c9f518f32c768160d6591f0ccc7d8b4fc14c110d"} Nov 03 20:03:34.846038 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:34.845991 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:03:34.934090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:34.934048 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/613a180b-7079-4633-8060-c4adb54db7f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613a180b-7079-4633-8060-c4adb54db7f8-istio-podinfo\") pod \"613a180b-7079-4633-8060-c4adb54db7f8\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " Nov 03 20:03:35.005251 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.005222 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-istio-data\") pod \"613a180b-7079-4633-8060-c4adb54db7f8\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " Nov 03 20:03:35.005935 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.005910 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613a180b-7079-4633-8060-c4adb54db7f8-kube-api-access-8z6z4\" (UniqueName: \"kubernetes.io/projected/613a180b-7079-4633-8060-c4adb54db7f8-kube-api-access-8z6z4\") pod \"613a180b-7079-4633-8060-c4adb54db7f8\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " Nov 03 20:03:35.006577 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.006553 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-istio-envoy\") pod \"613a180b-7079-4633-8060-c4adb54db7f8\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " Nov 03 20:03:35.033071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.007126 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613a180b-7079-4633-8060-c4adb54db7f8-istio-token\" (UniqueName: \"kubernetes.io/projected/613a180b-7079-4633-8060-c4adb54db7f8-istio-token\") pod \"613a180b-7079-4633-8060-c4adb54db7f8\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " Nov 03 20:03:35.035135 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.035106 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/613a180b-7079-4633-8060-c4adb54db7f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613a180b-7079-4633-8060-c4adb54db7f8-istiod-ca-cert\") pod \"613a180b-7079-4633-8060-c4adb54db7f8\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " Nov 03 20:03:35.035783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.035761 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-workload-certs\") pod \"613a180b-7079-4633-8060-c4adb54db7f8\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " Nov 03 20:03:35.036029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.036010 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-workload-socket\") pod \"613a180b-7079-4633-8060-c4adb54db7f8\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " Nov 03 20:03:35.036193 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.036175 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-credential-socket\") pod \"613a180b-7079-4633-8060-c4adb54db7f8\" (UID: \"613a180b-7079-4633-8060-c4adb54db7f8\") " Nov 03 20:03:35.053082 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.003914 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:03:35.053499 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.007054 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-istio-data" pod "613a180b-7079-4633-8060-c4adb54db7f8" (UID: "613a180b-7079-4633-8060-c4adb54db7f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:35.053653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.035626 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613a180b-7079-4633-8060-c4adb54db7f8-istiod-ca-cert" pod "613a180b-7079-4633-8060-c4adb54db7f8" (UID: "613a180b-7079-4633-8060-c4adb54db7f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:35.063821 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.063784 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-workload-certs" pod "613a180b-7079-4633-8060-c4adb54db7f8" (UID: "613a180b-7079-4633-8060-c4adb54db7f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:35.081804 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.081761 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-workload-socket" pod "613a180b-7079-4633-8060-c4adb54db7f8" (UID: "613a180b-7079-4633-8060-c4adb54db7f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:35.082006 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:34.934736 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:35.119165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.119076 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/613a180b-7079-4633-8060-c4adb54db7f8-istio-podinfo" pod "613a180b-7079-4633-8060-c4adb54db7f8" (UID: "613a180b-7079-4633-8060-c4adb54db7f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:03:35.163011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.136256 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613a180b-7079-4633-8060-c4adb54db7f8-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:35.221897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.215614 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:35.221897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.215639 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613a180b-7079-4633-8060-c4adb54db7f8-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:35.221897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.215656 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:35.221897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.215697 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:35.221897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.215559 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-credential-socket" pod "613a180b-7079-4633-8060-c4adb54db7f8" (UID: "613a180b-7079-4633-8060-c4adb54db7f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:35.221897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.200950 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:35.241448 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.241383 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-istio-envoy" pod "613a180b-7079-4633-8060-c4adb54db7f8" (UID: "613a180b-7079-4633-8060-c4adb54db7f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:35.268703 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.241868 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613a180b-7079-4633-8060-c4adb54db7f8-kube-api-access-8z6z4" pod "613a180b-7079-4633-8060-c4adb54db7f8" (UID: "613a180b-7079-4633-8060-c4adb54db7f8"). InnerVolumeSpecName "kube-api-access-8z6z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:35.297102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.294814 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:35.297102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.295544 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:03:35.429783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.415981 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z6z4\" (UniqueName: \"kubernetes.io/projected/613a180b-7079-4633-8060-c4adb54db7f8-kube-api-access-8z6z4\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:35.429783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.416022 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:35.429783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.416037 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613a180b-7079-4633-8060-c4adb54db7f8-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:35.545385 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.545225 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613a180b-7079-4633-8060-c4adb54db7f8-istio-token" pod "613a180b-7079-4633-8060-c4adb54db7f8" (UID: "613a180b-7079-4633-8060-c4adb54db7f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:35.657095 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.651827 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:03:35.657095 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.652743 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/613a180b-7079-4633-8060-c4adb54db7f8-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:35.885887 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.885853 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" Nov 03 20:03:35.914382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.914350 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw" event={"ID":"613a180b-7079-4633-8060-c4adb54db7f8","Type":"ContainerDied","Data":"95dea34409cbd7d6df10caad28c5365ec8cd543ccffb944e62b0b559307dac2c"} Nov 03 20:03:35.914653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.914631 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" event={"ID":"4c609469-d7d7-4530-8d74-e34aa56dda8f","Type":"ContainerStarted","Data":"01046b5c62acd48be6ebabf851d13a922cd16c44da94bb90f68ca3cae9c5ee8f"} Nov 03 20:03:35.914809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.914795 5059 scope.go:122] "RemoveContainer" containerID="11e9a10d4054b2c7a58b27f8c6b92f3989b82e42bd63266b82cd4e670d6b18fc" Nov 03 20:03:35.926695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.926613 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" podStartSLOduration=3.9265972270000002 podStartE2EDuration="3.926597227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:35.91962772 +0000 UTC m=+1610.177424978" watchObservedRunningTime="2026-11-03 20:03:35.926597227 +0000 UTC m=+1610.184394484" Nov 03 20:03:35.979736 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.970836 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw"] Nov 03 20:03:35.979902 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:35.979731 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-c131fa-openshift-default-5b9cdbc9d5-sjwtw"] Nov 03 20:03:36.518162 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:36.518129 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:36.523425 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:36.523399 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:36.925063 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:36.925027 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:36.927796 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:36.927768 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:03:37.885902 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:37.885863 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="613a180b-7079-4633-8060-c4adb54db7f8" path="/var/lib/kubelet/pods/613a180b-7079-4633-8060-c4adb54db7f8/volumes" Nov 03 20:03:40.622934 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:03:40.622879 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:03:43.984429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:43.984390 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg"] Nov 03 20:03:43.985589 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:43.985559 5059 cpu_manager.go:414] "containerMap: removing container" podUID="613a180b-7079-4633-8060-c4adb54db7f8" containerName="istio-proxy" Nov 03 20:03:43.985589 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:43.985588 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613a180b-7079-4633-8060-c4adb54db7f8" containerName="istio-proxy" Nov 03 20:03:43.985851 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:43.985813 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="613a180b-7079-4633-8060-c4adb54db7f8" containerName="istio-proxy" Nov 03 20:03:43.985851 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:43.985830 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="613a180b-7079-4633-8060-c4adb54db7f8" containerName="istio-proxy" Nov 03 20:03:43.994468 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:43.994443 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.000183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.000158 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg"] Nov 03 20:03:44.005565 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.005537 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-bcfde8-openshift-default-dockercfg-6xbwb\"" Nov 03 20:03:44.103180 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.103143 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-credential-socket\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.103180 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.103180 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-istio-envoy\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.103386 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.103329 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-istio-data\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.103436 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.103368 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-workload-certs\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.103492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.103453 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-workload-socket\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.103553 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.103538 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqlb8\" (UniqueName: \"kubernetes.io/projected/518169ac-2f8d-4645-ae47-210b67ca080a-kube-api-access-fqlb8\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.103597 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.103565 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/518169ac-2f8d-4645-ae47-210b67ca080a-istio-token\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.103639 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.103609 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/518169ac-2f8d-4645-ae47-210b67ca080a-istio-podinfo\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.103760 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.103738 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/518169ac-2f8d-4645-ae47-210b67ca080a-istiod-ca-cert\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.223712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.216640 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqlb8\" (UniqueName: \"kubernetes.io/projected/518169ac-2f8d-4645-ae47-210b67ca080a-kube-api-access-fqlb8\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.223712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.216729 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/518169ac-2f8d-4645-ae47-210b67ca080a-istio-token\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.223712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.216802 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/518169ac-2f8d-4645-ae47-210b67ca080a-istio-podinfo\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.223712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.216843 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/518169ac-2f8d-4645-ae47-210b67ca080a-istiod-ca-cert\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.223712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.217023 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-credential-socket\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.223712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.217050 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-istio-envoy\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.223712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.217136 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-istio-data\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.223712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.217165 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-workload-certs\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.223712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.217224 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-workload-socket\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.223712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.217763 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-workload-socket\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.233198 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.232397 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/518169ac-2f8d-4645-ae47-210b67ca080a-istiod-ca-cert\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.233198 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.232656 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-istio-data\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.233198 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.232914 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-credential-socket\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.233198 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.233145 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-workload-certs\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.247738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.244319 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/518169ac-2f8d-4645-ae47-210b67ca080a-istio-podinfo\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.262073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.261985 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-istio-envoy\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.280867 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.280795 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqlb8\" (UniqueName: \"kubernetes.io/projected/518169ac-2f8d-4645-ae47-210b67ca080a-kube-api-access-fqlb8\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.299726 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.281467 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/518169ac-2f8d-4645-ae47-210b67ca080a-istio-token\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.322293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.321951 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:44.677896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.677871 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg"] Nov 03 20:03:44.695838 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:03:44.695808 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod518169ac_2f8d_4645_ae47_210b67ca080a.slice/crio-6ffdd1a6aa4c61b216d8c84eba61449574d8e149612f72a033532ef642609540 WatchSource:0}: Error finding container 6ffdd1a6aa4c61b216d8c84eba61449574d8e149612f72a033532ef642609540: Status 404 returned error can't find the container with id 6ffdd1a6aa4c61b216d8c84eba61449574d8e149612f72a033532ef642609540 Nov 03 20:03:44.698496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.698460 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:44.698580 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.698528 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:44.698580 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.698557 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:44.971300 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.971220 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk"] Nov 03 20:03:44.972382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.972353 5059 cpu_manager.go:414] "containerMap: removing container" podUID="613a180b-7079-4633-8060-c4adb54db7f8" containerName="istio-proxy" Nov 03 20:03:44.972382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.972383 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613a180b-7079-4633-8060-c4adb54db7f8" containerName="istio-proxy" Nov 03 20:03:44.977485 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.977460 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:44.979312 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.979284 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk"] Nov 03 20:03:44.983491 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:44.983047 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-43c674-openshift-default-dockercfg-slz8b\"" Nov 03 20:03:45.020620 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.020591 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" event={"ID":"518169ac-2f8d-4645-ae47-210b67ca080a","Type":"ContainerStarted","Data":"45c3ba404ea0796dd18bc58ed44508e09f4ede955131e48779814d8f243781bd"} Nov 03 20:03:45.021068 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.020637 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" event={"ID":"518169ac-2f8d-4645-ae47-210b67ca080a","Type":"ContainerStarted","Data":"6ffdd1a6aa4c61b216d8c84eba61449574d8e149612f72a033532ef642609540"} Nov 03 20:03:45.022409 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.022385 5059 generic.go:356] "Generic (PLEG): container finished" podID="ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" containerID="083613d8d2a66dbf18be20acf40c9c2e55284f3701c7490d658626a11a1dff74" exitCode=0 Nov 03 20:03:45.022528 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.022459 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" event={"ID":"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c","Type":"ContainerDied","Data":"083613d8d2a66dbf18be20acf40c9c2e55284f3701c7490d658626a11a1dff74"} Nov 03 20:03:45.022899 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.022881 5059 scope.go:122] "RemoveContainer" containerID="083613d8d2a66dbf18be20acf40c9c2e55284f3701c7490d658626a11a1dff74" Nov 03 20:03:45.024554 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.024509 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:45.024696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.024571 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:45.024696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.024610 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:45.039475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.039453 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-workload-certs\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.039621 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.039602 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-podinfo\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.039718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.039652 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-workload-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.039789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.039773 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qcvm\" (UniqueName: \"kubernetes.io/projected/8b648c6b-1f7d-42ca-9c3f-a435494bad03-kube-api-access-7qcvm\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.039874 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.039851 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-credential-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.039933 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.039922 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-data\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.040015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.039996 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-token\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.040128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.040031 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istiod-ca-cert\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.040128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.040058 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-envoy\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.049842 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.049803 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" podStartSLOduration=2.049791308 podStartE2EDuration="2.049791308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:45.04434701 +0000 UTC m=+1619.302144267" watchObservedRunningTime="2026-11-03 20:03:45.049791308 +0000 UTC m=+1619.307588565" Nov 03 20:03:45.142966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.142929 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-token\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.143496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.143006 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istiod-ca-cert\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.143496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.143035 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-envoy\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.143496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.143117 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-workload-certs\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.143496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.143285 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-podinfo\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.143496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.143338 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-workload-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.143496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.143444 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qcvm\" (UniqueName: \"kubernetes.io/projected/8b648c6b-1f7d-42ca-9c3f-a435494bad03-kube-api-access-7qcvm\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.143905 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.143499 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-credential-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.143905 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.143628 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-data\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.145073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.145044 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-credential-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.145980 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.145952 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-workload-certs\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.146868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.146844 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-workload-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.147469 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.147426 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-podinfo\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.147599 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.147577 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istiod-ca-cert\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.148006 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.147983 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-data\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.150344 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.150304 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-envoy\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.160216 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.160187 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-token\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.160559 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.160538 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qcvm\" (UniqueName: \"kubernetes.io/projected/8b648c6b-1f7d-42ca-9c3f-a435494bad03-kube-api-access-7qcvm\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.298215 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.296977 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:45.330646 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.328990 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:45.481821 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.480033 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:45.710026 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:03:45.709991 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b648c6b_1f7d_42ca_9c3f_a435494bad03.slice/crio-45c262ea7a7351d836fc55259b94b8ec172b3cdad20f1f7dcfa206941ac825c0 WatchSource:0}: Error finding container 45c262ea7a7351d836fc55259b94b8ec172b3cdad20f1f7dcfa206941ac825c0: Status 404 returned error can't find the container with id 45c262ea7a7351d836fc55259b94b8ec172b3cdad20f1f7dcfa206941ac825c0 Nov 03 20:03:45.713363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.713322 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:45.713490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.713406 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:45.713490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.713449 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:45.714280 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:45.714257 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk"] Nov 03 20:03:46.000716 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:03:46.000602 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf601d7_8af9_4beb_b64e_f2e9c6b00711.slice/crio-f2543d77f2decc8ee1bea967fea0369c0aeed38031f2a8770cb5da2b14b75e92\": RecentStats: unable to find data in memory cache]" Nov 03 20:03:46.035537 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:46.035489 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" event={"ID":"8b648c6b-1f7d-42ca-9c3f-a435494bad03","Type":"ContainerStarted","Data":"e696c1151a11239c9d6d9c393f5e0ca1dfa476058f6f2240ee0c35998ef6a3d3"} Nov 03 20:03:46.036608 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:46.035553 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" event={"ID":"8b648c6b-1f7d-42ca-9c3f-a435494bad03","Type":"ContainerStarted","Data":"45c262ea7a7351d836fc55259b94b8ec172b3cdad20f1f7dcfa206941ac825c0"} Nov 03 20:03:46.039113 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:46.038874 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" event={"ID":"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c","Type":"ContainerStarted","Data":"8880ed12964f51d73ab467040324e0b739812c728f7782757374b9b0f025d69e"} Nov 03 20:03:46.039496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:46.039475 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:46.040450 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:46.040400 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:03:46.100295 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:46.099652 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" podStartSLOduration=2.099637022 podStartE2EDuration="2.099637022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:46.076260469 +0000 UTC m=+1620.334057760" watchObservedRunningTime="2026-11-03 20:03:46.099637022 +0000 UTC m=+1620.357434280" Nov 03 20:03:46.298155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:46.298023 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:46.312166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:46.311803 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Nov 03 20:03:46.312166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:46.311867 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" podUID="8b648c6b-1f7d-42ca-9c3f-a435494bad03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Nov 03 20:03:46.452313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:46.452266 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:03:46.452806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:46.452777 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 03 20:03:46.453031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:46.452851 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" podUID="ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 03 20:03:47.302210 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:47.302181 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:47.447690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:47.444745 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 03 20:03:47.447690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:47.444816 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" podUID="ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 03 20:03:48.108620 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:48.108583 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:48.109454 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:48.109429 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:03:48.444553 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:48.444481 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 03 20:03:48.444553 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:48.444531 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" podUID="ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 03 20:03:49.444207 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:49.444174 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 03 20:03:49.444388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:49.444237 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" podUID="ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 03 20:03:50.457155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:50.457110 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 03 20:03:50.476787 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:50.476740 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" podUID="ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 03 20:03:51.120105 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.120070 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww"] Nov 03 20:03:51.127331 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.127304 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.137114 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.137091 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-78d47a-openshift-default-dockercfg-4ctht\"" Nov 03 20:03:51.169683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.154292 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww"] Nov 03 20:03:51.231252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.231216 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv598\" (UniqueName: \"kubernetes.io/projected/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-kube-api-access-rv598\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.231584 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.231564 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istiod-ca-cert\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.231713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.231694 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-token\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.231797 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.231744 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-podinfo\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.231847 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.231827 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-envoy\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.231940 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.231923 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-workload-certs\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.231993 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.231971 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-credential-socket\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.232077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.232066 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-workload-socket\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.232123 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.232084 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-data\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.333761 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.333730 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-token\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.333957 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.333774 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-podinfo\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.333957 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.333798 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-envoy\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.334092 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.333970 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-workload-certs\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.334092 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.334037 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-credential-socket\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.334198 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.334184 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-workload-socket\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.334269 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.334213 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-data\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.334356 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.334315 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv598\" (UniqueName: \"kubernetes.io/projected/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-kube-api-access-rv598\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.334410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.334354 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-workload-certs\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.334456 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.334415 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-credential-socket\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.334550 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.334519 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istiod-ca-cert\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.334550 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.334529 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-workload-socket\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.335305 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.334806 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-data\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.335305 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.335164 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istiod-ca-cert\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.336130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.336111 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-envoy\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.336985 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.336965 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-podinfo\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.346386 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.346360 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-token\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.346773 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.346753 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv598\" (UniqueName: \"kubernetes.io/projected/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-kube-api-access-rv598\") pod \"e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:51.444409 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.444339 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 03 20:03:51.444409 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.444387 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" podUID="ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 03 20:03:51.510004 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:51.509972 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:52.077654 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:52.077617 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww"] Nov 03 20:03:52.078456 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:03:52.078429 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d64cbd8_7a40_4481_92a4_3ff4fa568da4.slice/crio-87ebbe51ee655f3f10464222e78866bdecfabe19296ef641dddc1f0302de5fb0 WatchSource:0}: Error finding container 87ebbe51ee655f3f10464222e78866bdecfabe19296ef641dddc1f0302de5fb0: Status 404 returned error can't find the container with id 87ebbe51ee655f3f10464222e78866bdecfabe19296ef641dddc1f0302de5fb0 Nov 03 20:03:52.081356 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:52.081313 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:52.081470 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:52.081388 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:52.081470 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:52.081429 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:03:52.232551 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:52.232511 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" event={"ID":"8d64cbd8-7a40-4481-92a4-3ff4fa568da4","Type":"ContainerStarted","Data":"265304de16a1f88550cfea645b333844512146599e8ef5183262e49aadb58abe"} Nov 03 20:03:52.232782 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:52.232573 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" event={"ID":"8d64cbd8-7a40-4481-92a4-3ff4fa568da4","Type":"ContainerStarted","Data":"87ebbe51ee655f3f10464222e78866bdecfabe19296ef641dddc1f0302de5fb0"} Nov 03 20:03:52.260889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:52.260827 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" podStartSLOduration=1.260807166 podStartE2EDuration="1.260807166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:52.255931133 +0000 UTC m=+1626.513728391" watchObservedRunningTime="2026-11-03 20:03:52.260807166 +0000 UTC m=+1626.518604424" Nov 03 20:03:52.449768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:52.449657 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:03:52.451485 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:52.451448 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:03:52.454398 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:52.452624 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:03:52.510712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:52.510357 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:53.516991 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:53.516956 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:54.249621 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:54.249585 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:03:54.250555 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:03:54.250530 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:04:03.832830 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:03.832764 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv"] Nov 03 20:04:03.840996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:03.840839 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:04:03.852760 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:03.852579 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-4fb1d0\"/\"openshift-service-ca.crt\"" Nov 03 20:04:03.852760 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:03.852722 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-4fb1d0\"/\"kube-root-ca.crt\"" Nov 03 20:04:03.861390 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:03.861362 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-4fb1d0\"/\"auth-test-model-4fb1d0-kserve-self-signed-certs\"" Nov 03 20:04:03.861532 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:03.861423 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-4fb1d0\"/\"default-dockercfg-v7k75\"" Nov 03 20:04:03.867642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:03.867617 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv"] Nov 03 20:04:03.976530 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:03.976440 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-model-cache\") pod \"auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:04:03.976714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:03.976642 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a66dcf87-0a9a-4bdd-981c-a529e797ba29-tls-certs\") pod \"auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:04:03.976775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:03.976721 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-home\") pod \"auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:04:03.976873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:03.976850 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-tmp-dir\") pod \"auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:04:03.976985 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:03.976962 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-dshm\") pod \"auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:04:03.977041 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:03.977026 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx5th\" (UniqueName: \"kubernetes.io/projected/a66dcf87-0a9a-4bdd-981c-a529e797ba29-kube-api-access-lx5th\") pod \"auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:04:04.103839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:04.096815 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a66dcf87-0a9a-4bdd-981c-a529e797ba29-tls-certs\") pod \"auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:04:04.103839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:04.096881 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-home\") pod \"auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:04:04.103839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:04.096960 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-tmp-dir\") pod \"auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:04:04.103839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:04.097028 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-dshm\") pod \"auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:04:04.103839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:04.097078 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx5th\" (UniqueName: \"kubernetes.io/projected/a66dcf87-0a9a-4bdd-981c-a529e797ba29-kube-api-access-lx5th\") pod \"auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:04:04.103839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:04.097374 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-model-cache\") pod \"auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:04:04.103839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:04.097543 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-home\") pod \"auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:04:04.103839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:04.097840 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-model-cache\") pod \"auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:04:04.103839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:04.098157 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-tmp-dir\") pod \"auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:04:04.104994 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:04.104965 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-dshm\") pod \"auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:04:04.116385 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:04.114166 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a66dcf87-0a9a-4bdd-981c-a529e797ba29-tls-certs\") pod \"auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:04:04.123517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:04.123339 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx5th\" (UniqueName: \"kubernetes.io/projected/a66dcf87-0a9a-4bdd-981c-a529e797ba29-kube-api-access-lx5th\") pod \"auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:04:04.172276 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:04.171963 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:04:04.515018 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:04:04.514982 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda66dcf87_0a9a_4bdd_981c_a529e797ba29.slice/crio-1e6b0e93b0c7ae5a2fc034f91df3c65c499d697210bb6a110223f4890753337b WatchSource:0}: Error finding container 1e6b0e93b0c7ae5a2fc034f91df3c65c499d697210bb6a110223f4890753337b: Status 404 returned error can't find the container with id 1e6b0e93b0c7ae5a2fc034f91df3c65c499d697210bb6a110223f4890753337b Nov 03 20:04:04.517359 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:04.517327 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv"] Nov 03 20:04:04.767051 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:04.765571 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" event={"ID":"a66dcf87-0a9a-4bdd-981c-a529e797ba29","Type":"ContainerStarted","Data":"7fe9df6e9dc941bdc5c989ff624f231d29d7d890e6e99c5dd70c051a232d47a2"} Nov 03 20:04:04.767051 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:04.765630 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" event={"ID":"a66dcf87-0a9a-4bdd-981c-a529e797ba29","Type":"ContainerStarted","Data":"1e6b0e93b0c7ae5a2fc034f91df3c65c499d697210bb6a110223f4890753337b"} Nov 03 20:04:04.848275 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:04.848208 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" podStartSLOduration=1.8481833810000001 podStartE2EDuration="1.848183381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:04.80394953 +0000 UTC m=+1639.061746788" watchObservedRunningTime="2026-11-03 20:04:04.848183381 +0000 UTC m=+1639.105980640" Nov 03 20:04:05.175613 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:05.175581 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:04:10.608126 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:04:10.608080 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:04:11.241680 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.241584 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r"] Nov 03 20:04:11.266287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.265623 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r"] Nov 03 20:04:11.266287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.265786 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.423837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.423804 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-credential-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.423868 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-istio-envoy\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.423956 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34880975-a897-4ad4-af59-18617a93cdad-istiod-ca-cert\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.424009 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-workload-certs\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.424171 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-istio-data\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.424218 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34880975-a897-4ad4-af59-18617a93cdad-istio-podinfo\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.424274 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34880975-a897-4ad4-af59-18617a93cdad-istio-token\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.424353 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cjvf\" (UniqueName: \"kubernetes.io/projected/34880975-a897-4ad4-af59-18617a93cdad-kube-api-access-5cjvf\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.424403 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-workload-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.526265 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cjvf\" (UniqueName: \"kubernetes.io/projected/34880975-a897-4ad4-af59-18617a93cdad-kube-api-access-5cjvf\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.526307 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-workload-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.526351 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-credential-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.526375 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-istio-envoy\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.526391 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34880975-a897-4ad4-af59-18617a93cdad-istiod-ca-cert\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.526415 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-workload-certs\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.526590 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-istio-data\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.526647 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34880975-a897-4ad4-af59-18617a93cdad-istio-podinfo\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.526709 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34880975-a897-4ad4-af59-18617a93cdad-istio-token\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.526722 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-credential-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.526888 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-workload-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.526930 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-workload-certs\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.526979 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-istio-data\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.527145 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34880975-a897-4ad4-af59-18617a93cdad-istiod-ca-cert\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.528486 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-istio-envoy\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.529036 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34880975-a897-4ad4-af59-18617a93cdad-istio-podinfo\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.542156 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34880975-a897-4ad4-af59-18617a93cdad-istio-token\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.554498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.542423 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cjvf\" (UniqueName: \"kubernetes.io/projected/34880975-a897-4ad4-af59-18617a93cdad-kube-api-access-5cjvf\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.606770 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.606592 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:11.843638 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.843607 5059 generic.go:356] "Generic (PLEG): container finished" podID="f043a660-eddb-4de5-88a7-cf0b1fda6512" containerID="873a0ffe9f526eaa4ece120b96a6e7e29de1a52daefeff130e5e76d7773021ad" exitCode=0 Nov 03 20:04:11.844002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.843711 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" event={"ID":"f043a660-eddb-4de5-88a7-cf0b1fda6512","Type":"ContainerDied","Data":"873a0ffe9f526eaa4ece120b96a6e7e29de1a52daefeff130e5e76d7773021ad"} Nov 03 20:04:11.844150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.844135 5059 scope.go:122] "RemoveContainer" containerID="873a0ffe9f526eaa4ece120b96a6e7e29de1a52daefeff130e5e76d7773021ad" Nov 03 20:04:11.845548 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.845527 5059 generic.go:356] "Generic (PLEG): container finished" podID="b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" containerID="aafac57bb6098e1fd609b713edc7da1b412af807beeafa6e9063b066a1c006b1" exitCode=0 Nov 03 20:04:11.845675 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.845552 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" event={"ID":"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3","Type":"ContainerDied","Data":"aafac57bb6098e1fd609b713edc7da1b412af807beeafa6e9063b066a1c006b1"} Nov 03 20:04:11.845979 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.845953 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:04:11.846055 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.846025 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:04:11.846103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.846061 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:04:11.846620 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.846524 5059 scope.go:122] "RemoveContainer" containerID="aafac57bb6098e1fd609b713edc7da1b412af807beeafa6e9063b066a1c006b1" Nov 03 20:04:11.847479 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.847449 5059 generic.go:356] "Generic (PLEG): container finished" podID="3b9d0847-df69-4b6a-b346-2cd206e42339" containerID="f40938fdfd8d12fe6529e191274b54750315e326f7aeec32fac2e699b64c43a1" exitCode=0 Nov 03 20:04:11.847567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.847473 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" event={"ID":"3b9d0847-df69-4b6a-b346-2cd206e42339","Type":"ContainerDied","Data":"f40938fdfd8d12fe6529e191274b54750315e326f7aeec32fac2e699b64c43a1"} Nov 03 20:04:11.848068 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.848045 5059 scope.go:122] "RemoveContainer" containerID="f40938fdfd8d12fe6529e191274b54750315e326f7aeec32fac2e699b64c43a1" Nov 03 20:04:11.848268 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.848241 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:04:11.848347 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.848321 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:04:11.848408 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.848371 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:04:11.850444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.850413 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:04:11.850518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.850499 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:04:11.850573 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.850557 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:04:11.965032 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.964939 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:04:11.965366 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.965334 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:04:11.965465 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.965446 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:04:11.969441 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:11.969412 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r"] Nov 03 20:04:12.858615 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:12.858575 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" event={"ID":"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3","Type":"ContainerStarted","Data":"083bf41b34c7c17d7a323c71536ca53488c9b5d6024a331fe82c42a4a9894a43"} Nov 03 20:04:12.861451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:12.861427 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" event={"ID":"3b9d0847-df69-4b6a-b346-2cd206e42339","Type":"ContainerStarted","Data":"b5a63189fd2f2d22d3e6f03c32065113a3ccf85221d07eb70f923a647b5b8d38"} Nov 03 20:04:12.863851 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:12.863817 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" event={"ID":"34880975-a897-4ad4-af59-18617a93cdad","Type":"ContainerStarted","Data":"89e013c2e4eeadb512ea4a0824409f67321420e7c404d52597c8ba458a34976b"} Nov 03 20:04:12.863966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:12.863866 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" event={"ID":"34880975-a897-4ad4-af59-18617a93cdad","Type":"ContainerStarted","Data":"2617ba973814cb2eac509072a9b1fdcceb26d9b42e09f34eb13eaf2e3aa6e749"} Nov 03 20:04:12.866217 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:12.866196 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" event={"ID":"f043a660-eddb-4de5-88a7-cf0b1fda6512","Type":"ContainerStarted","Data":"f9f4f7f919667909db546636d5524b9efb53ecdd7bf93966afd963aa9742b298"} Nov 03 20:04:12.929532 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:12.929480 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" podStartSLOduration=1.929462507 podStartE2EDuration="1.929462507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:12.927332903 +0000 UTC m=+1647.185130193" watchObservedRunningTime="2026-11-03 20:04:12.929462507 +0000 UTC m=+1647.187259765" Nov 03 20:04:12.959715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:12.959688 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:04:13.031449 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:13.031418 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:04:13.032307 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:13.032284 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 03 20:04:13.032376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:13.032334 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" podUID="3b9d0847-df69-4b6a-b346-2cd206e42339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 03 20:04:13.152839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:13.152650 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:04:13.154309 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:13.154283 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 03 20:04:13.154448 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:13.154329 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" podUID="f043a660-eddb-4de5-88a7-cf0b1fda6512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 03 20:04:13.618731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:13.617937 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:13.618731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:13.618314 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 20:04:13.618731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:13.618361 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" podUID="34880975-a897-4ad4-af59-18617a93cdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 20:04:13.964873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:13.964755 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 20:04:13.965763 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:13.965433 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" podUID="b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 20:04:14.071686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:14.065124 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 03 20:04:14.071686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:14.065192 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" podUID="3b9d0847-df69-4b6a-b346-2cd206e42339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 03 20:04:14.154685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:14.154044 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 03 20:04:14.154685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:14.154113 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" podUID="f043a660-eddb-4de5-88a7-cf0b1fda6512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 03 20:04:14.193177 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:14.193141 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:04:14.614198 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:14.614146 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 20:04:14.614528 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:14.614479 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" podUID="34880975-a897-4ad4-af59-18617a93cdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 20:04:14.972415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:14.972061 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 20:04:14.972415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:14.972121 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" podUID="b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 20:04:15.047256 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:15.047218 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 03 20:04:15.047535 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:15.047505 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" podUID="3b9d0847-df69-4b6a-b346-2cd206e42339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 03 20:04:15.086992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:15.086960 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:04:15.172697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:15.170987 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 03 20:04:15.172697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:15.171058 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" podUID="f043a660-eddb-4de5-88a7-cf0b1fda6512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 03 20:04:15.618752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:15.611753 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 20:04:15.618752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:15.611813 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" podUID="34880975-a897-4ad4-af59-18617a93cdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 20:04:15.658774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:15.658742 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:04:15.675295 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:15.675252 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:15.764807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:15.764760 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:15.960742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:15.958353 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:15.960742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:15.958802 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:15.960742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:15.960609 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 20:04:15.961553 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:15.961097 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" podUID="b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 20:04:16.135688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:16.123329 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 03 20:04:16.135688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:16.123418 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" podUID="3b9d0847-df69-4b6a-b346-2cd206e42339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 03 20:04:16.177760 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:16.150868 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" podUID="a66dcf87-0a9a-4bdd-981c-a529e797ba29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.215:8000/ready\": context deadline exceeded" Nov 03 20:04:16.178448 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:16.178410 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 03 20:04:16.178570 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:16.178478 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" podUID="f043a660-eddb-4de5-88a7-cf0b1fda6512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 03 20:04:16.507807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:16.507750 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:16.509844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:16.509815 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:04:16.609237 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:16.608061 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 20:04:16.609237 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:16.608121 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" podUID="34880975-a897-4ad4-af59-18617a93cdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 20:04:16.718682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:16.706007 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:16.718682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:16.706590 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:04:16.774736 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:16.771269 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:16.774736 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:16.772921 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 20:04:16.808697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:16.807958 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p"] Nov 03 20:04:16.975774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:16.913822 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:04:16.976741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:16.976714 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-942930\"/\"openshift-service-ca.crt\"" Nov 03 20:04:16.977650 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:16.977628 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-942930\"/\"auth-test-model-942930-kserve-self-signed-certs\"" Nov 03 20:04:17.000724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:16.979816 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:17.002134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:16.979874 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-942930\"/\"kube-root-ca.crt\"" Nov 03 20:04:17.002382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.002365 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 20:04:17.015852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:16.979980 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-942930\"/\"default-dockercfg-lfbb5\"" Nov 03 20:04:17.016017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:16.979743 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:17.017197 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:16.980858 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 20:04:17.017369 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.017345 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" podUID="b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 20:04:17.036764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.018131 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 20:04:17.058142 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.058118 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p"] Nov 03 20:04:17.072801 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.056784 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 03 20:04:17.101782 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.101737 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" podUID="3b9d0847-df69-4b6a-b346-2cd206e42339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 03 20:04:17.102029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.055993 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:17.102137 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.056294 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-tmp-dir\") pod \"auth-test-model-942930-kserve-5859b4bbb9-cjt8p\" (UID: \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\") " pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:04:17.173154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.102461 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-dshm\") pod \"auth-test-model-942930-kserve-5859b4bbb9-cjt8p\" (UID: \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\") " pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:04:17.173154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.122765 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4bxz\" (UniqueName: \"kubernetes.io/projected/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-kube-api-access-v4bxz\") pod \"auth-test-model-942930-kserve-5859b4bbb9-cjt8p\" (UID: \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\") " pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:04:17.173154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.122845 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-home\") pod \"auth-test-model-942930-kserve-5859b4bbb9-cjt8p\" (UID: \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\") " pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:04:17.173154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.122874 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-tls-certs\") pod \"auth-test-model-942930-kserve-5859b4bbb9-cjt8p\" (UID: \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\") " pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:04:17.173154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.122947 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-model-cache\") pod \"auth-test-model-942930-kserve-5859b4bbb9-cjt8p\" (UID: \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\") " pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:04:17.173154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.164963 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 03 20:04:17.173154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.165017 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" podUID="f043a660-eddb-4de5-88a7-cf0b1fda6512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 03 20:04:17.173154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.165090 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" podUID="a66dcf87-0a9a-4bdd-981c-a529e797ba29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.215:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:17.234523 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.234450 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-model-cache\") pod \"auth-test-model-942930-kserve-5859b4bbb9-cjt8p\" (UID: \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\") " pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:04:17.260340 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.260294 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-model-cache\") pod \"auth-test-model-942930-kserve-5859b4bbb9-cjt8p\" (UID: \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\") " pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:04:17.262264 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.262234 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-tmp-dir\") pod \"auth-test-model-942930-kserve-5859b4bbb9-cjt8p\" (UID: \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\") " pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:04:17.262694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.262674 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-dshm\") pod \"auth-test-model-942930-kserve-5859b4bbb9-cjt8p\" (UID: \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\") " pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:04:17.269851 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.269822 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4bxz\" (UniqueName: \"kubernetes.io/projected/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-kube-api-access-v4bxz\") pod \"auth-test-model-942930-kserve-5859b4bbb9-cjt8p\" (UID: \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\") " pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:04:17.270073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.270051 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-home\") pod \"auth-test-model-942930-kserve-5859b4bbb9-cjt8p\" (UID: \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\") " pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:04:17.270215 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.270199 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-tls-certs\") pod \"auth-test-model-942930-kserve-5859b4bbb9-cjt8p\" (UID: \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\") " pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:04:17.279692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.263676 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-tmp-dir\") pod \"auth-test-model-942930-kserve-5859b4bbb9-cjt8p\" (UID: \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\") " pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:04:17.280791 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.280765 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-home\") pod \"auth-test-model-942930-kserve-5859b4bbb9-cjt8p\" (UID: \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\") " pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:04:17.341777 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.341742 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4bxz\" (UniqueName: \"kubernetes.io/projected/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-kube-api-access-v4bxz\") pod \"auth-test-model-942930-kserve-5859b4bbb9-cjt8p\" (UID: \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\") " pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:04:17.416768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.416013 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-tls-certs\") pod \"auth-test-model-942930-kserve-5859b4bbb9-cjt8p\" (UID: \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\") " pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:04:17.460593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.460543 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-dshm\") pod \"auth-test-model-942930-kserve-5859b4bbb9-cjt8p\" (UID: \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\") " pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:04:17.529854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.529814 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:04:17.595784 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.568438 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:17.595784 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.595728 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 20:04:17.664049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.664012 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 20:04:17.664319 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.664285 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" podUID="34880975-a897-4ad4-af59-18617a93cdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 20:04:17.752694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.750976 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:17.778709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.778640 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:17.792133 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:17.792089 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:04:18.020952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:18.020892 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:18.089698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:18.071402 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 20:04:18.089698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:18.071460 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" podUID="b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 20:04:18.110024 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:18.109971 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 03 20:04:18.114271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:18.110253 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" podUID="3b9d0847-df69-4b6a-b346-2cd206e42339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 03 20:04:18.114535 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:18.110131 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:04:18.174762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:18.170813 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" podUID="a66dcf87-0a9a-4bdd-981c-a529e797ba29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.215:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:18.212581 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:18.189871 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 03 20:04:18.212581 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:18.189929 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" podUID="f043a660-eddb-4de5-88a7-cf0b1fda6512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 03 20:04:18.469705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:18.469647 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:04:18.638717 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:18.616117 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:04:18.801401 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:18.801303 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:04:18.821709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:18.806756 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 20:04:18.821709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:18.806810 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" podUID="34880975-a897-4ad4-af59-18617a93cdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 20:04:19.144739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:19.140774 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:19.228689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:19.140873 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 20:04:19.268686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:19.212926 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" podUID="a66dcf87-0a9a-4bdd-981c-a529e797ba29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.215:8000/ready\": context deadline exceeded" Nov 03 20:04:19.280688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:19.257199 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 03 20:04:19.280688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:19.276861 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" podUID="3b9d0847-df69-4b6a-b346-2cd206e42339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 03 20:04:19.321720 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:19.320957 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 03 20:04:19.321720 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:19.321013 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" podUID="f043a660-eddb-4de5-88a7-cf0b1fda6512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 03 20:04:19.422457 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:19.422366 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 20:04:19.422739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:19.422705 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" podUID="b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 20:04:19.454190 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:19.454139 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 03 20:04:19.620140 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:19.620098 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 20:04:19.671691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:19.660297 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" podUID="34880975-a897-4ad4-af59-18617a93cdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 20:04:19.671970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:19.660223 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:04:19.846548 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:19.846504 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:04:19.985243 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:19.985204 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 20:04:19.985516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:19.985486 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" podUID="b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 20:04:20.182147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:20.179350 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 03 20:04:20.182147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:20.179410 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" podUID="f043a660-eddb-4de5-88a7-cf0b1fda6512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 03 20:04:20.182147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:20.180497 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 03 20:04:20.182147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:20.180538 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" podUID="3b9d0847-df69-4b6a-b346-2cd206e42339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 03 20:04:20.306277 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:20.306226 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" podUID="a66dcf87-0a9a-4bdd-981c-a529e797ba29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.215:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:20.326715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:20.321822 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:20.703246 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:20.703198 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:20.751812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:20.751747 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:20.829716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:20.822945 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 20:04:20.829716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:20.823008 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" podUID="34880975-a897-4ad4-af59-18617a93cdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 20:04:20.848163 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:20.848112 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 03 20:04:20.878354 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:20.856443 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:21.140688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:21.124279 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 03 20:04:21.143798 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:21.143753 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" podUID="3b9d0847-df69-4b6a-b346-2cd206e42339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 03 20:04:21.202751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:21.201955 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 03 20:04:21.202751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:21.202029 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" podUID="f043a660-eddb-4de5-88a7-cf0b1fda6512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 03 20:04:21.297689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:21.265947 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 20:04:21.297689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:21.266036 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" podUID="b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 20:04:21.323418 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:21.314268 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" podUID="a66dcf87-0a9a-4bdd-981c-a529e797ba29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.215:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:21.338689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:21.334807 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:21.688525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:21.688481 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 20:04:21.730288 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:21.730240 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" podUID="34880975-a897-4ad4-af59-18617a93cdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 20:04:21.730992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:21.705452 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:21.734491 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:04:21.734438 5059 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.878s" Nov 03 20:04:21.734828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:21.734809 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p"] Nov 03 20:04:21.735031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:21.735018 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:04:21.778948 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:21.778894 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:21.908733 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:21.892048 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:21.908733 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:21.900966 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 03 20:04:21.961002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:21.939816 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded" Nov 03 20:04:21.979861 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:21.979824 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 20:04:21.980153 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:21.980121 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" podUID="b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 20:04:22.192692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:22.182394 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 03 20:04:22.192692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:22.182458 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" podUID="f043a660-eddb-4de5-88a7-cf0b1fda6512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 03 20:04:22.347898 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:22.335792 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" podUID="a66dcf87-0a9a-4bdd-981c-a529e797ba29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.215:8000/ready\": context deadline exceeded" Nov 03 20:04:22.351686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:22.349254 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" event={"ID":"abb3d35a-13d2-4fa9-a21c-0830efce2e7c","Type":"ContainerStarted","Data":"a5f300b3cc6f5f95ad9995a3b30bfd6b00184d7e4001d0eeb3da4e915b5094ca"} Nov 03 20:04:22.351686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:22.349949 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:22.519893 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:22.519787 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 20:04:22.611686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:22.607841 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Nov 03 20:04:22.611686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:22.607902 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" podUID="34880975-a897-4ad4-af59-18617a93cdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Nov 03 20:04:22.666686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:22.662997 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:04:22.683088 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:22.677902 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:04:22.747530 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:22.747479 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:22.916340 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:22.916293 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:23.027645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:23.026287 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:23.048856 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:23.048825 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 20:04:23.049174 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:23.049122 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" podUID="b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 20:04:23.212711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:23.179797 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:23.347688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:23.341534 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" podUID="a66dcf87-0a9a-4bdd-981c-a529e797ba29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.215:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:23.362684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:23.358935 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:04:23.538742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:23.513999 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:04:23.554428 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:23.525957 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 20:04:23.624688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:23.620647 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:04:23.685686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:23.684949 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:23.709692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:23.692098 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:23.709692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:23.693524 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:04:23.752555 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:23.752527 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:04:23.764872 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:23.764837 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:04:23.856702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:23.854724 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 20:04:24.121216 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:24.121123 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:04:24.196619 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:24.151309 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:04:24.220679 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:24.215605 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:04:24.221336 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:24.221311 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:04:24.234476 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:24.234449 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 20:04:24.253680 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:24.253104 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 20:04:24.528514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:24.528409 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" event={"ID":"abb3d35a-13d2-4fa9-a21c-0830efce2e7c","Type":"ContainerStarted","Data":"fb8da14e89cf10f509bfa14c33a07660777ad613c5be9ff76210f29b1180fdf0"} Nov 03 20:04:24.623366 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:24.623291 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" podStartSLOduration=8.623266598 podStartE2EDuration="8.623266598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:24.553774756 +0000 UTC m=+1658.811572014" watchObservedRunningTime="2026-11-03 20:04:24.623266598 +0000 UTC m=+1658.881063857" Nov 03 20:04:24.815739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:24.810785 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:04:27.806595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:27.806558 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:04:27.819251 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:27.819210 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:04:28.733059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:28.733029 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:04:32.079826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:32.079779 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:04:32.080319 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:32.079854 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" podUID="518169ac-2f8d-4645-ae47-210b67ca080a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:33.814747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:33.814711 5059 generic.go:356] "Generic (PLEG): container finished" podID="518169ac-2f8d-4645-ae47-210b67ca080a" containerID="45c3ba404ea0796dd18bc58ed44508e09f4ede955131e48779814d8f243781bd" exitCode=0 Nov 03 20:04:33.815283 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:33.814789 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" event={"ID":"518169ac-2f8d-4645-ae47-210b67ca080a","Type":"ContainerDied","Data":"45c3ba404ea0796dd18bc58ed44508e09f4ede955131e48779814d8f243781bd"} Nov 03 20:04:33.815372 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:33.815352 5059 scope.go:122] "RemoveContainer" containerID="45c3ba404ea0796dd18bc58ed44508e09f4ede955131e48779814d8f243781bd" Nov 03 20:04:33.816724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:33.816696 5059 generic.go:356] "Generic (PLEG): container finished" podID="4c609469-d7d7-4530-8d74-e34aa56dda8f" containerID="01046b5c62acd48be6ebabf851d13a922cd16c44da94bb90f68ca3cae9c5ee8f" exitCode=0 Nov 03 20:04:33.816724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:33.816712 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" event={"ID":"4c609469-d7d7-4530-8d74-e34aa56dda8f","Type":"ContainerDied","Data":"01046b5c62acd48be6ebabf851d13a922cd16c44da94bb90f68ca3cae9c5ee8f"} Nov 03 20:04:33.817126 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:33.817112 5059 scope.go:122] "RemoveContainer" containerID="01046b5c62acd48be6ebabf851d13a922cd16c44da94bb90f68ca3cae9c5ee8f" Nov 03 20:04:33.817255 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:33.817225 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:04:33.817315 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:33.817301 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:04:33.817356 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:33.817336 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:04:33.818733 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:33.818701 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:04:33.818839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:33.818772 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:04:33.818839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:33.818809 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:04:34.877815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:34.877774 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" event={"ID":"518169ac-2f8d-4645-ae47-210b67ca080a","Type":"ContainerStarted","Data":"fcb62eadee37b9a5aa9bdc20f77cd13d99c001430b049cbc4d5006aa5cc74354"} Nov 03 20:04:34.889073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:34.889041 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" event={"ID":"4c609469-d7d7-4530-8d74-e34aa56dda8f","Type":"ContainerStarted","Data":"25b28b428ebc85ad2241009e2e20a1a14a4f98588d071bd9785777b228e75310"} Nov 03 20:04:35.331170 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:35.331133 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:04:35.331604 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:35.331471 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:04:35.331975 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:35.331933 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" podUID="518169ac-2f8d-4645-ae47-210b67ca080a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:04:35.518368 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:35.518335 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:04:35.519600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:35.519573 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 20:04:35.519823 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:35.519794 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" podUID="4c609469-d7d7-4530-8d74-e34aa56dda8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 20:04:36.324113 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:36.324071 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:04:36.325905 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:36.325871 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" podUID="518169ac-2f8d-4645-ae47-210b67ca080a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:04:36.544322 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:36.543512 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 20:04:36.546767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:36.546733 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" podUID="4c609469-d7d7-4530-8d74-e34aa56dda8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 20:04:37.103686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:37.093477 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:04:37.334044 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:37.334000 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:04:37.335455 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:37.335417 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" podUID="518169ac-2f8d-4645-ae47-210b67ca080a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:04:37.555192 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:37.555098 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 20:04:37.555482 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:37.555448 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" podUID="4c609469-d7d7-4530-8d74-e34aa56dda8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 20:04:38.329197 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:38.329157 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:04:38.329512 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:38.329470 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" podUID="518169ac-2f8d-4645-ae47-210b67ca080a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:04:38.522695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:38.522087 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 20:04:38.522695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:38.522149 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" podUID="4c609469-d7d7-4530-8d74-e34aa56dda8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 20:04:39.341858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:39.341813 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:04:39.342037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:39.341894 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" podUID="518169ac-2f8d-4645-ae47-210b67ca080a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:04:39.526686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:39.526109 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 20:04:39.526686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:39.526177 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" podUID="4c609469-d7d7-4530-8d74-e34aa56dda8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 20:04:40.323073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:40.323035 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:04:40.323363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:40.323331 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" podUID="518169ac-2f8d-4645-ae47-210b67ca080a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:04:40.520992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:40.520956 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 20:04:40.521304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:40.521269 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" podUID="4c609469-d7d7-4530-8d74-e34aa56dda8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 20:04:40.851223 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:04:40.851145 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:04:41.326690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:41.326087 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:04:41.326690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:41.326200 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" podUID="518169ac-2f8d-4645-ae47-210b67ca080a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:04:41.518289 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:41.518252 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 20:04:41.518535 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:41.518509 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" podUID="4c609469-d7d7-4530-8d74-e34aa56dda8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 20:04:42.327966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:42.327929 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:04:42.328455 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:42.328422 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:04:42.329162 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:42.329140 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:04:42.522321 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:42.522294 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:04:42.523488 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:42.523466 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:04:47.441036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:47.440946 5059 generic.go:356] "Generic (PLEG): container finished" podID="8d64cbd8-7a40-4481-92a4-3ff4fa568da4" containerID="265304de16a1f88550cfea645b333844512146599e8ef5183262e49aadb58abe" exitCode=0 Nov 03 20:04:47.441556 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:47.441025 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" event={"ID":"8d64cbd8-7a40-4481-92a4-3ff4fa568da4","Type":"ContainerDied","Data":"265304de16a1f88550cfea645b333844512146599e8ef5183262e49aadb58abe"} Nov 03 20:04:47.441642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:47.441559 5059 scope.go:122] "RemoveContainer" containerID="265304de16a1f88550cfea645b333844512146599e8ef5183262e49aadb58abe" Nov 03 20:04:47.443145 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:47.443116 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:04:47.443224 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:47.443181 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:04:47.443224 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:47.443215 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:04:48.450877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:48.450843 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" event={"ID":"8d64cbd8-7a40-4481-92a4-3ff4fa568da4","Type":"ContainerStarted","Data":"0845ec3799e80032191977e24e789e3e84b647204c1b857cdf08fef44de0aa67"} Nov 03 20:04:48.511115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:48.511076 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:04:48.511303 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:48.511283 5059 patch_prober.go:29] interesting pod/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 20:04:48.511366 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:48.511329 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" podUID="8d64cbd8-7a40-4481-92a4-3ff4fa568da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 20:04:49.299119 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.299073 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx"] Nov 03 20:04:49.305187 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.305161 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:04:49.306011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.305980 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx"] Nov 03 20:04:49.313112 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.313082 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-88a18e\"/\"kube-root-ca.crt\"" Nov 03 20:04:49.313247 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.313095 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-88a18e\"/\"default-dockercfg-cblz4\"" Nov 03 20:04:49.313247 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.313141 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-88a18e\"/\"openshift-service-ca.crt\"" Nov 03 20:04:49.313655 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.313500 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-88a18e\"/\"auth-test-model-88a18e-kserve-self-signed-certs\"" Nov 03 20:04:49.440569 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.440524 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-tmp-dir\") pod \"auth-test-model-88a18e-kserve-749fd8c9c-86ccx\" (UID: \"7331481c-0dd0-4acf-8c73-35a38eece3af\") " pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:04:49.440967 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.440941 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm7xb\" (UniqueName: \"kubernetes.io/projected/7331481c-0dd0-4acf-8c73-35a38eece3af-kube-api-access-bm7xb\") pod \"auth-test-model-88a18e-kserve-749fd8c9c-86ccx\" (UID: \"7331481c-0dd0-4acf-8c73-35a38eece3af\") " pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:04:49.456532 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.456490 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-model-cache\") pod \"auth-test-model-88a18e-kserve-749fd8c9c-86ccx\" (UID: \"7331481c-0dd0-4acf-8c73-35a38eece3af\") " pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:04:49.457130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.457097 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-home\") pod \"auth-test-model-88a18e-kserve-749fd8c9c-86ccx\" (UID: \"7331481c-0dd0-4acf-8c73-35a38eece3af\") " pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:04:49.457375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.457356 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7331481c-0dd0-4acf-8c73-35a38eece3af-tls-certs\") pod \"auth-test-model-88a18e-kserve-749fd8c9c-86ccx\" (UID: \"7331481c-0dd0-4acf-8c73-35a38eece3af\") " pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:04:49.457630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.457608 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-dshm\") pod \"auth-test-model-88a18e-kserve-749fd8c9c-86ccx\" (UID: \"7331481c-0dd0-4acf-8c73-35a38eece3af\") " pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:04:49.519681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.514029 5059 patch_prober.go:29] interesting pod/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 20:04:49.519681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.514088 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" podUID="8d64cbd8-7a40-4481-92a4-3ff4fa568da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 20:04:49.563137 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.563017 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-tmp-dir\") pod \"auth-test-model-88a18e-kserve-749fd8c9c-86ccx\" (UID: \"7331481c-0dd0-4acf-8c73-35a38eece3af\") " pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:04:49.563948 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.563817 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-tmp-dir\") pod \"auth-test-model-88a18e-kserve-749fd8c9c-86ccx\" (UID: \"7331481c-0dd0-4acf-8c73-35a38eece3af\") " pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:04:49.564382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.564358 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm7xb\" (UniqueName: \"kubernetes.io/projected/7331481c-0dd0-4acf-8c73-35a38eece3af-kube-api-access-bm7xb\") pod \"auth-test-model-88a18e-kserve-749fd8c9c-86ccx\" (UID: \"7331481c-0dd0-4acf-8c73-35a38eece3af\") " pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:04:49.564992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.564970 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-model-cache\") pod \"auth-test-model-88a18e-kserve-749fd8c9c-86ccx\" (UID: \"7331481c-0dd0-4acf-8c73-35a38eece3af\") " pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:04:49.565471 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.565454 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-home\") pod \"auth-test-model-88a18e-kserve-749fd8c9c-86ccx\" (UID: \"7331481c-0dd0-4acf-8c73-35a38eece3af\") " pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:04:49.565958 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.565934 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7331481c-0dd0-4acf-8c73-35a38eece3af-tls-certs\") pod \"auth-test-model-88a18e-kserve-749fd8c9c-86ccx\" (UID: \"7331481c-0dd0-4acf-8c73-35a38eece3af\") " pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:04:49.566185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.566161 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-dshm\") pod \"auth-test-model-88a18e-kserve-749fd8c9c-86ccx\" (UID: \"7331481c-0dd0-4acf-8c73-35a38eece3af\") " pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:04:49.573705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.565398 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-model-cache\") pod \"auth-test-model-88a18e-kserve-749fd8c9c-86ccx\" (UID: \"7331481c-0dd0-4acf-8c73-35a38eece3af\") " pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:04:49.573897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.565832 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-home\") pod \"auth-test-model-88a18e-kserve-749fd8c9c-86ccx\" (UID: \"7331481c-0dd0-4acf-8c73-35a38eece3af\") " pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:04:49.610268 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.610235 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm7xb\" (UniqueName: \"kubernetes.io/projected/7331481c-0dd0-4acf-8c73-35a38eece3af-kube-api-access-bm7xb\") pod \"auth-test-model-88a18e-kserve-749fd8c9c-86ccx\" (UID: \"7331481c-0dd0-4acf-8c73-35a38eece3af\") " pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:04:49.624848 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.624816 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7331481c-0dd0-4acf-8c73-35a38eece3af-tls-certs\") pod \"auth-test-model-88a18e-kserve-749fd8c9c-86ccx\" (UID: \"7331481c-0dd0-4acf-8c73-35a38eece3af\") " pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:04:49.641686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.634418 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-dshm\") pod \"auth-test-model-88a18e-kserve-749fd8c9c-86ccx\" (UID: \"7331481c-0dd0-4acf-8c73-35a38eece3af\") " pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:04:49.920518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:49.920489 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:04:50.225128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.224520 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss"] Nov 03 20:04:50.231332 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.231299 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss"] Nov 03 20:04:50.231487 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.231407 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:04:50.243734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.237601 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-81a5ae\"/\"auth-test-model-81a5ae-kserve-self-signed-certs\"" Nov 03 20:04:50.243734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.237703 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-81a5ae\"/\"default-dockercfg-ljkfm\"" Nov 03 20:04:50.266232 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:04:50.266183 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7331481c_0dd0_4acf_8c73_35a38eece3af.slice/crio-1357d0cb48541bf119201f22ef60acc15a0272819d3ecd7ad11db6c9cb1c0ef2 WatchSource:0}: Error finding container 1357d0cb48541bf119201f22ef60acc15a0272819d3ecd7ad11db6c9cb1c0ef2: Status 404 returned error can't find the container with id 1357d0cb48541bf119201f22ef60acc15a0272819d3ecd7ad11db6c9cb1c0ef2 Nov 03 20:04:50.272001 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.271976 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx"] Nov 03 20:04:50.295703 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.295421 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-81a5ae\"/\"kube-root-ca.crt\"" Nov 03 20:04:50.295703 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.295555 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-81a5ae\"/\"openshift-service-ca.crt\"" Nov 03 20:04:50.340194 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.340168 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-model-cache\") pod \"auth-test-model-81a5ae-kserve-5666559957-l6wss\" (UID: \"14015184-8b33-4358-b3f2-af083975b689\") " pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:04:50.340302 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.340217 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x26b4\" (UniqueName: \"kubernetes.io/projected/14015184-8b33-4358-b3f2-af083975b689-kube-api-access-x26b4\") pod \"auth-test-model-81a5ae-kserve-5666559957-l6wss\" (UID: \"14015184-8b33-4358-b3f2-af083975b689\") " pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:04:50.340344 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.340303 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-tmp-dir\") pod \"auth-test-model-81a5ae-kserve-5666559957-l6wss\" (UID: \"14015184-8b33-4358-b3f2-af083975b689\") " pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:04:50.340394 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.340363 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14015184-8b33-4358-b3f2-af083975b689-tls-certs\") pod \"auth-test-model-81a5ae-kserve-5666559957-l6wss\" (UID: \"14015184-8b33-4358-b3f2-af083975b689\") " pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:04:50.340519 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.340497 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-home\") pod \"auth-test-model-81a5ae-kserve-5666559957-l6wss\" (UID: \"14015184-8b33-4358-b3f2-af083975b689\") " pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:04:50.340699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.340676 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-dshm\") pod \"auth-test-model-81a5ae-kserve-5666559957-l6wss\" (UID: \"14015184-8b33-4358-b3f2-af083975b689\") " pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:04:50.452685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.445061 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14015184-8b33-4358-b3f2-af083975b689-tls-certs\") pod \"auth-test-model-81a5ae-kserve-5666559957-l6wss\" (UID: \"14015184-8b33-4358-b3f2-af083975b689\") " pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:04:50.452685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.445143 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-home\") pod \"auth-test-model-81a5ae-kserve-5666559957-l6wss\" (UID: \"14015184-8b33-4358-b3f2-af083975b689\") " pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:04:50.452685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.445298 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-dshm\") pod \"auth-test-model-81a5ae-kserve-5666559957-l6wss\" (UID: \"14015184-8b33-4358-b3f2-af083975b689\") " pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:04:50.452685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.445642 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-model-cache\") pod \"auth-test-model-81a5ae-kserve-5666559957-l6wss\" (UID: \"14015184-8b33-4358-b3f2-af083975b689\") " pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:04:50.452685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.445749 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x26b4\" (UniqueName: \"kubernetes.io/projected/14015184-8b33-4358-b3f2-af083975b689-kube-api-access-x26b4\") pod \"auth-test-model-81a5ae-kserve-5666559957-l6wss\" (UID: \"14015184-8b33-4358-b3f2-af083975b689\") " pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:04:50.452685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.445821 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-tmp-dir\") pod \"auth-test-model-81a5ae-kserve-5666559957-l6wss\" (UID: \"14015184-8b33-4358-b3f2-af083975b689\") " pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:04:50.452685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.446385 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-tmp-dir\") pod \"auth-test-model-81a5ae-kserve-5666559957-l6wss\" (UID: \"14015184-8b33-4358-b3f2-af083975b689\") " pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:04:50.463506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.463473 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-model-cache\") pod \"auth-test-model-81a5ae-kserve-5666559957-l6wss\" (UID: \"14015184-8b33-4358-b3f2-af083975b689\") " pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:04:50.481314 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.481232 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-home\") pod \"auth-test-model-81a5ae-kserve-5666559957-l6wss\" (UID: \"14015184-8b33-4358-b3f2-af083975b689\") " pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:04:50.492189 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.491269 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-dshm\") pod \"auth-test-model-81a5ae-kserve-5666559957-l6wss\" (UID: \"14015184-8b33-4358-b3f2-af083975b689\") " pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:04:50.495414 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.495356 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x26b4\" (UniqueName: \"kubernetes.io/projected/14015184-8b33-4358-b3f2-af083975b689-kube-api-access-x26b4\") pod \"auth-test-model-81a5ae-kserve-5666559957-l6wss\" (UID: \"14015184-8b33-4358-b3f2-af083975b689\") " pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:04:50.515954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.513429 5059 patch_prober.go:29] interesting pod/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 20:04:50.515954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.513502 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" podUID="8d64cbd8-7a40-4481-92a4-3ff4fa568da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 20:04:50.531507 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.531460 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14015184-8b33-4358-b3f2-af083975b689-tls-certs\") pod \"auth-test-model-81a5ae-kserve-5666559957-l6wss\" (UID: \"14015184-8b33-4358-b3f2-af083975b689\") " pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:04:50.549075 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.549038 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" event={"ID":"7331481c-0dd0-4acf-8c73-35a38eece3af","Type":"ContainerStarted","Data":"1357d0cb48541bf119201f22ef60acc15a0272819d3ecd7ad11db6c9cb1c0ef2"} Nov 03 20:04:50.604706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.596046 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:04:50.930079 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:04:50.930037 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14015184_8b33_4358_b3f2_af083975b689.slice/crio-ca1e0afd046cd85e49a5fcc7ab5a3f1eaf5b1b792b41b794258ee05ae9a78842 WatchSource:0}: Error finding container ca1e0afd046cd85e49a5fcc7ab5a3f1eaf5b1b792b41b794258ee05ae9a78842: Status 404 returned error can't find the container with id ca1e0afd046cd85e49a5fcc7ab5a3f1eaf5b1b792b41b794258ee05ae9a78842 Nov 03 20:04:50.931930 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:50.931818 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss"] Nov 03 20:04:51.510689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:51.510637 5059 patch_prober.go:29] interesting pod/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 20:04:51.511110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:51.510711 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" podUID="8d64cbd8-7a40-4481-92a4-3ff4fa568da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 20:04:51.584599 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:51.584558 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" event={"ID":"7331481c-0dd0-4acf-8c73-35a38eece3af","Type":"ContainerStarted","Data":"d081881808411dc88c6aff949d2bb38279ffa9fcdfcaf1e1805e834b89a1081c"} Nov 03 20:04:51.587524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:51.587497 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" event={"ID":"14015184-8b33-4358-b3f2-af083975b689","Type":"ContainerStarted","Data":"f1c92f88aca7b16e844bcdf4089de485f2e60dfdddc3a048bf22342485dae7bf"} Nov 03 20:04:51.587701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:51.587532 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" event={"ID":"14015184-8b33-4358-b3f2-af083975b689","Type":"ContainerStarted","Data":"ca1e0afd046cd85e49a5fcc7ab5a3f1eaf5b1b792b41b794258ee05ae9a78842"} Nov 03 20:04:51.605247 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:51.605095 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:04:51.608804 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:51.608752 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" podStartSLOduration=2.60873687 podStartE2EDuration="2.60873687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:51.605727371 +0000 UTC m=+1685.863524630" watchObservedRunningTime="2026-11-03 20:04:51.60873687 +0000 UTC m=+1685.866534128" Nov 03 20:04:51.626186 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:51.626141 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" podStartSLOduration=1.6261294130000001 podStartE2EDuration="1.626129413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:51.625490401 +0000 UTC m=+1685.883287661" watchObservedRunningTime="2026-11-03 20:04:51.626129413 +0000 UTC m=+1685.883926712" Nov 03 20:04:51.921468 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:51.921436 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:04:52.512310 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:52.512096 5059 patch_prober.go:29] interesting pod/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 20:04:52.512310 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:52.512160 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" podUID="8d64cbd8-7a40-4481-92a4-3ff4fa568da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 20:04:53.511237 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:53.511202 5059 patch_prober.go:29] interesting pod/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 20:04:53.511477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:53.511451 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" podUID="8d64cbd8-7a40-4481-92a4-3ff4fa568da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 20:04:54.250358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:54.250326 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:04:54.531062 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:54.530978 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:04:54.749820 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:54.749790 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:04:59.921221 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:59.921188 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:04:59.934308 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:04:59.934284 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:05:00.605739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:00.605705 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:05:00.618555 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:00.618530 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:05:00.812884 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:00.812851 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:05:00.813059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:00.812949 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:05:10.604976 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:05:10.604896 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:05:18.028497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:18.027872 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa494873-tvtzx"] Nov 03 20:05:18.035351 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:18.035310 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa494873-tvtzx"] Nov 03 20:05:19.859834 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:19.859800 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4793e8b-0775-4d3f-ac3c-60197ed96852" path="/var/lib/kubelet/pods/b4793e8b-0775-4d3f-ac3c-60197ed96852/volumes" Nov 03 20:05:36.027205 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:36.027163 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11bdb454-xk9wd"] Nov 03 20:05:36.032241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:36.032215 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11bdb454-xk9wd"] Nov 03 20:05:37.881684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:37.880217 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fea8feed-e6f9-44ca-ab14-d355aea06ed8" path="/var/lib/kubelet/pods/fea8feed-e6f9-44ca-ab14-d355aea06ed8/volumes" Nov 03 20:05:40.616098 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:05:40.616053 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:05:44.074880 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:44.074843 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww"] Nov 03 20:05:44.075423 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:44.075093 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" podUID="8d64cbd8-7a40-4481-92a4-3ff4fa568da4" containerName="istio-proxy" containerID="cri-o://0845ec3799e80032191977e24e789e3e84b647204c1b857cdf08fef44de0aa67" gracePeriod=30 Nov 03 20:05:44.504992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:44.503879 5059 generic.go:356] "Generic (PLEG): container finished" podID="8b648c6b-1f7d-42ca-9c3f-a435494bad03" containerID="e696c1151a11239c9d6d9c393f5e0ca1dfa476058f6f2240ee0c35998ef6a3d3" exitCode=0 Nov 03 20:05:44.504992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:44.503949 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" event={"ID":"8b648c6b-1f7d-42ca-9c3f-a435494bad03","Type":"ContainerDied","Data":"e696c1151a11239c9d6d9c393f5e0ca1dfa476058f6f2240ee0c35998ef6a3d3"} Nov 03 20:05:44.504992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:44.504408 5059 scope.go:122] "RemoveContainer" containerID="e696c1151a11239c9d6d9c393f5e0ca1dfa476058f6f2240ee0c35998ef6a3d3" Nov 03 20:05:44.506618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:44.506163 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:05:44.506618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:44.506247 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:05:44.506618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:44.506283 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:05:45.541807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:45.541770 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" event={"ID":"8b648c6b-1f7d-42ca-9c3f-a435494bad03","Type":"ContainerStarted","Data":"1fe212cabed0fc214cbab7c5d208f7f12d710cdc5a5c86dc76e26a0430ec5af5"} Nov 03 20:05:46.298532 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:46.298487 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:05:46.299872 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:46.299836 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Nov 03 20:05:46.300037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:46.299911 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" podUID="8b648c6b-1f7d-42ca-9c3f-a435494bad03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Nov 03 20:05:47.298207 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:47.298161 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Nov 03 20:05:47.298605 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:47.298229 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" podUID="8b648c6b-1f7d-42ca-9c3f-a435494bad03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Nov 03 20:05:48.109147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:48.109111 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:05:48.298359 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:48.298323 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Nov 03 20:05:48.298791 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:48.298382 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" podUID="8b648c6b-1f7d-42ca-9c3f-a435494bad03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Nov 03 20:05:49.299131 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.299096 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Nov 03 20:05:49.299619 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.299153 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" podUID="8b648c6b-1f7d-42ca-9c3f-a435494bad03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Nov 03 20:05:49.513157 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.513133 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:05:49.614883 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.614846 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-workload-socket\") pod \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " Nov 03 20:05:49.615081 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.614899 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istiod-ca-cert\") pod \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " Nov 03 20:05:49.615081 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.615017 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-workload-certs\") pod \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " Nov 03 20:05:49.615217 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.615114 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-credential-socket\") pod \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " Nov 03 20:05:49.615217 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.615131 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-workload-socket" pod "8d64cbd8-7a40-4481-92a4-3ff4fa568da4" (UID: "8d64cbd8-7a40-4481-92a4-3ff4fa568da4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:49.615217 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.615148 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-envoy\") pod \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " Nov 03 20:05:49.615398 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.615226 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-data\") pod \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " Nov 03 20:05:49.615398 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.615321 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-workload-certs" pod "8d64cbd8-7a40-4481-92a4-3ff4fa568da4" (UID: "8d64cbd8-7a40-4481-92a4-3ff4fa568da4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:49.615398 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.615340 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-kube-api-access-rv598\" (UniqueName: \"kubernetes.io/projected/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-kube-api-access-rv598\") pod \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " Nov 03 20:05:49.615398 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.615371 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-podinfo\") pod \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " Nov 03 20:05:49.615627 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.615489 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-token\" (UniqueName: \"kubernetes.io/projected/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-token\") pod \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\" (UID: \"8d64cbd8-7a40-4481-92a4-3ff4fa568da4\") " Nov 03 20:05:49.615627 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.615368 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istiod-ca-cert" pod "8d64cbd8-7a40-4481-92a4-3ff4fa568da4" (UID: "8d64cbd8-7a40-4481-92a4-3ff4fa568da4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:49.615627 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.615596 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-data" pod "8d64cbd8-7a40-4481-92a4-3ff4fa568da4" (UID: "8d64cbd8-7a40-4481-92a4-3ff4fa568da4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:49.615810 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.615720 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-credential-socket" pod "8d64cbd8-7a40-4481-92a4-3ff4fa568da4" (UID: "8d64cbd8-7a40-4481-92a4-3ff4fa568da4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:49.616245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.616217 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:49.616360 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.616248 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:49.616360 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.616263 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:49.616360 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.616276 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:49.616360 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.616290 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:49.617976 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.617950 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-envoy" pod "8d64cbd8-7a40-4481-92a4-3ff4fa568da4" (UID: "8d64cbd8-7a40-4481-92a4-3ff4fa568da4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:49.618133 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.618056 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-kube-api-access-rv598" pod "8d64cbd8-7a40-4481-92a4-3ff4fa568da4" (UID: "8d64cbd8-7a40-4481-92a4-3ff4fa568da4"). InnerVolumeSpecName "kube-api-access-rv598". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:49.618686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.618640 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-podinfo" pod "8d64cbd8-7a40-4481-92a4-3ff4fa568da4" (UID: "8d64cbd8-7a40-4481-92a4-3ff4fa568da4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:05:49.618769 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.618647 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-token" pod "8d64cbd8-7a40-4481-92a4-3ff4fa568da4" (UID: "8d64cbd8-7a40-4481-92a4-3ff4fa568da4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:49.647878 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.647843 5059 generic.go:356] "Generic (PLEG): container finished" podID="8d64cbd8-7a40-4481-92a4-3ff4fa568da4" containerID="0845ec3799e80032191977e24e789e3e84b647204c1b857cdf08fef44de0aa67" exitCode=0 Nov 03 20:05:49.648124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.648103 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" event={"ID":"8d64cbd8-7a40-4481-92a4-3ff4fa568da4","Type":"ContainerDied","Data":"0845ec3799e80032191977e24e789e3e84b647204c1b857cdf08fef44de0aa67"} Nov 03 20:05:49.648208 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.648194 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" event={"ID":"8d64cbd8-7a40-4481-92a4-3ff4fa568da4","Type":"ContainerDied","Data":"87ebbe51ee655f3f10464222e78866bdecfabe19296ef641dddc1f0302de5fb0"} Nov 03 20:05:49.648265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.648229 5059 scope.go:122] "RemoveContainer" containerID="0845ec3799e80032191977e24e789e3e84b647204c1b857cdf08fef44de0aa67" Nov 03 20:05:49.648397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.648385 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww" Nov 03 20:05:49.689310 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.689287 5059 scope.go:122] "RemoveContainer" containerID="265304de16a1f88550cfea645b333844512146599e8ef5183262e49aadb58abe" Nov 03 20:05:49.700019 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.700000 5059 scope.go:122] "RemoveContainer" containerID="0845ec3799e80032191977e24e789e3e84b647204c1b857cdf08fef44de0aa67" Nov 03 20:05:49.700291 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:05:49.700275 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0845ec3799e80032191977e24e789e3e84b647204c1b857cdf08fef44de0aa67\": container with ID starting with 0845ec3799e80032191977e24e789e3e84b647204c1b857cdf08fef44de0aa67 not found: ID does not exist" containerID="0845ec3799e80032191977e24e789e3e84b647204c1b857cdf08fef44de0aa67" Nov 03 20:05:49.700357 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.700300 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0845ec3799e80032191977e24e789e3e84b647204c1b857cdf08fef44de0aa67"} err="failed to get container status \"0845ec3799e80032191977e24e789e3e84b647204c1b857cdf08fef44de0aa67\": rpc error: code = NotFound desc = could not find container \"0845ec3799e80032191977e24e789e3e84b647204c1b857cdf08fef44de0aa67\": container with ID starting with 0845ec3799e80032191977e24e789e3e84b647204c1b857cdf08fef44de0aa67 not found: ID does not exist" Nov 03 20:05:49.700357 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.700316 5059 scope.go:122] "RemoveContainer" containerID="265304de16a1f88550cfea645b333844512146599e8ef5183262e49aadb58abe" Nov 03 20:05:49.700535 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:05:49.700518 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"265304de16a1f88550cfea645b333844512146599e8ef5183262e49aadb58abe\": container with ID starting with 265304de16a1f88550cfea645b333844512146599e8ef5183262e49aadb58abe not found: ID does not exist" containerID="265304de16a1f88550cfea645b333844512146599e8ef5183262e49aadb58abe" Nov 03 20:05:49.700583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.700539 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"265304de16a1f88550cfea645b333844512146599e8ef5183262e49aadb58abe"} err="failed to get container status \"265304de16a1f88550cfea645b333844512146599e8ef5183262e49aadb58abe\": rpc error: code = NotFound desc = could not find container \"265304de16a1f88550cfea645b333844512146599e8ef5183262e49aadb58abe\": container with ID starting with 265304de16a1f88550cfea645b333844512146599e8ef5183262e49aadb58abe not found: ID does not exist" Nov 03 20:05:49.709486 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.709460 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww"] Nov 03 20:05:49.714368 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.714348 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-78d47a-openshift-default-6fd47b5945-7zzww"] Nov 03 20:05:49.718125 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.718104 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:49.718208 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.718128 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv598\" (UniqueName: \"kubernetes.io/projected/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-kube-api-access-rv598\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:49.718208 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.718140 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:49.718208 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.718148 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d64cbd8-7a40-4481-92a4-3ff4fa568da4-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:49.883149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:49.883113 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d64cbd8-7a40-4481-92a4-3ff4fa568da4" path="/var/lib/kubelet/pods/8d64cbd8-7a40-4481-92a4-3ff4fa568da4/volumes" Nov 03 20:05:50.298426 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:50.298343 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Nov 03 20:05:50.298426 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:50.298397 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" podUID="8b648c6b-1f7d-42ca-9c3f-a435494bad03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Nov 03 20:05:51.298475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:51.298434 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Nov 03 20:05:51.298860 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:51.298493 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" podUID="8b648c6b-1f7d-42ca-9c3f-a435494bad03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Nov 03 20:05:52.303260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:52.303226 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:05:52.304517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:52.304496 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:05:55.077256 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:55.077173 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35852943-4n7m4"] Nov 03 20:05:55.120998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:55.120953 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35852943-4n7m4"] Nov 03 20:05:55.860682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:55.860573 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afe8145b-bfa0-4b87-b570-6fdca730e088" path="/var/lib/kubelet/pods/afe8145b-bfa0-4b87-b570-6fdca730e088/volumes" Nov 03 20:05:59.028225 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:59.028192 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-398e0990-tbplm"] Nov 03 20:05:59.032864 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:59.032838 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-398e0990-tbplm"] Nov 03 20:05:59.862208 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:05:59.862163 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e4c71f5-9a02-4740-b8a8-25a1c4178b23" path="/var/lib/kubelet/pods/0e4c71f5-9a02-4740-b8a8-25a1c4178b23/volumes" Nov 03 20:06:06.028520 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:06.028473 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22ec26b7-9rqqj"] Nov 03 20:06:06.033441 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:06.033410 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22ec26b7-9rqqj"] Nov 03 20:06:07.860119 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:07.860082 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69cdf024-85ab-4f82-8441-87e8e04f2a7c" path="/var/lib/kubelet/pods/69cdf024-85ab-4f82-8441-87e8e04f2a7c/volumes" Nov 03 20:06:10.605911 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:06:10.605876 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:06:11.026782 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:11.026692 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bca3de8-8q2pm"] Nov 03 20:06:11.031454 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:11.031427 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bca3de8-8q2pm"] Nov 03 20:06:11.859530 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:11.859484 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16e3083b-94d7-4655-9e85-dc27bc209f44" path="/var/lib/kubelet/pods/16e3083b-94d7-4655-9e85-dc27bc209f44/volumes" Nov 03 20:06:18.029597 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:18.029187 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55629980-6qsbv"] Nov 03 20:06:18.034524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:18.034489 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55629980-6qsbv"] Nov 03 20:06:19.026548 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:19.026507 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e9e11b-dvsxb"] Nov 03 20:06:19.031285 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:19.031258 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e9e11b-dvsxb"] Nov 03 20:06:19.859378 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:19.859342 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5de62ad5-f378-4eba-9a18-53f085e6c5c1" path="/var/lib/kubelet/pods/5de62ad5-f378-4eba-9a18-53f085e6c5c1/volumes" Nov 03 20:06:19.859874 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:19.859859 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70630500-5f32-4c09-8bd1-01b8e5835671" path="/var/lib/kubelet/pods/70630500-5f32-4c09-8bd1-01b8e5835671/volumes" Nov 03 20:06:28.030929 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:28.030890 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e5e5ae4-phxs9"] Nov 03 20:06:28.035583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:28.035558 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e5e5ae4-phxs9"] Nov 03 20:06:29.920001 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:29.919962 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cee28a4b-6a4e-43f3-9905-eaedd2068dad" path="/var/lib/kubelet/pods/cee28a4b-6a4e-43f3-9905-eaedd2068dad/volumes" Nov 03 20:06:32.042540 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:32.042503 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e75eb6-g4rv5"] Nov 03 20:06:32.053363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:32.053333 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e75eb6-g4rv5"] Nov 03 20:06:33.859082 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:33.859048 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9" path="/var/lib/kubelet/pods/3e1de2b9-0e8d-46fc-a10d-4c738ccc98a9/volumes" Nov 03 20:06:40.605589 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:06:40.605537 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:06:43.063685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:43.063222 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-834054e8-kq9d2"] Nov 03 20:06:43.106898 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:43.103458 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-834054e8-kq9d2"] Nov 03 20:06:43.859807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:43.859768 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d42779d9-caf1-41e2-8596-955a7a6bf405" path="/var/lib/kubelet/pods/d42779d9-caf1-41e2-8596-955a7a6bf405/volumes" Nov 03 20:06:46.740623 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:46.740589 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 20:06:46.744007 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:46.743976 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 20:06:47.024641 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:47.024559 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6c3944f-kk8bm"] Nov 03 20:06:47.029155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:47.029125 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6c3944f-kk8bm"] Nov 03 20:06:47.996319 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:47.996275 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e43a9baa-e726-44d2-98a9-cbd3b27871ea" path="/var/lib/kubelet/pods/e43a9baa-e726-44d2-98a9-cbd3b27871ea/volumes" Nov 03 20:06:53.853029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:53.852992 5059 scope.go:122] "RemoveContainer" containerID="fe8c5965bdb63f76399a237cf281aad4805fa4963b78b83fdd05b1b2865c8f92" Nov 03 20:06:53.865768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:53.865744 5059 scope.go:122] "RemoveContainer" containerID="8841df5b8138c83d4d8639f4f198bfeca78a20e9f540ae8ecce06a600afa7b23" Nov 03 20:06:53.875705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:53.875680 5059 scope.go:122] "RemoveContainer" containerID="cc82e1e6accb53514993ed4694c26d66956bbd1838ba6e3d04decfe587bf5fc1" Nov 03 20:06:58.027460 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:58.027413 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba2227d-jl6q8"] Nov 03 20:06:58.031644 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:58.031623 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba2227d-jl6q8"] Nov 03 20:06:59.859272 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:06:59.859238 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c" path="/var/lib/kubelet/pods/d8b95e13-e1c1-4ae7-bddf-9c05687f3f5c/volumes" Nov 03 20:07:04.028811 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:04.028773 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-861c2614-thcbf"] Nov 03 20:07:04.033451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:04.033422 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-861c2614-thcbf"] Nov 03 20:07:05.859012 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:05.858979 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="973e14b6-7688-4f59-83b0-f353eb44a382" path="/var/lib/kubelet/pods/973e14b6-7688-4f59-83b0-f353eb44a382/volumes" Nov 03 20:07:06.049871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:06.049834 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-262d413b-jgbwg"] Nov 03 20:07:06.061731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:06.058090 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-262d413b-jgbwg"] Nov 03 20:07:07.861272 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:07.861228 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a93f5c75-f751-4643-9908-04c1bd453e20" path="/var/lib/kubelet/pods/a93f5c75-f751-4643-9908-04c1bd453e20/volumes" Nov 03 20:07:09.027378 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:09.027335 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3a5d746-4h8f4"] Nov 03 20:07:09.033785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:09.033758 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3a5d746-4h8f4"] Nov 03 20:07:09.860091 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:09.860049 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e469f89a-632b-43ab-883a-645ab4576243" path="/var/lib/kubelet/pods/e469f89a-632b-43ab-883a-645ab4576243/volumes" Nov 03 20:07:10.844038 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:07:10.844002 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:07:17.027434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.027394 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a932901-tlzmp"] Nov 03 20:07:17.031692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.031652 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a932901-tlzmp"] Nov 03 20:07:17.588019 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.587985 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9"] Nov 03 20:07:17.588773 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.588757 5059 cpu_manager.go:414] "containerMap: removing container" podUID="8d64cbd8-7a40-4481-92a4-3ff4fa568da4" containerName="istio-proxy" Nov 03 20:07:17.588860 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.588773 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d64cbd8-7a40-4481-92a4-3ff4fa568da4" containerName="istio-proxy" Nov 03 20:07:17.588917 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.588884 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d64cbd8-7a40-4481-92a4-3ff4fa568da4" containerName="istio-proxy" Nov 03 20:07:17.588917 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.588893 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d64cbd8-7a40-4481-92a4-3ff4fa568da4" containerName="istio-proxy" Nov 03 20:07:17.592863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.592840 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.688700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.684370 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-660d95-openshift-default-dockercfg-df2mh\"" Nov 03 20:07:17.714719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.690583 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istiod-ca-cert\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.721682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.720528 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-token\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.721682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.720735 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-workload-certs\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.721682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.720862 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-podinfo\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.721682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.720938 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-envoy\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.721682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.721001 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-workload-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.721682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.721143 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-credential-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.721682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.721195 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwpmv\" (UniqueName: \"kubernetes.io/projected/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-kube-api-access-wwpmv\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.721682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.721324 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-data\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.733685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.731004 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9"] Nov 03 20:07:17.843266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.830366 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-workload-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.843266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.830483 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-credential-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.843266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.830517 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwpmv\" (UniqueName: \"kubernetes.io/projected/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-kube-api-access-wwpmv\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.843266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.830600 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-data\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.843266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.830692 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istiod-ca-cert\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.843266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.830760 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-token\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.843266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.830877 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-workload-certs\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.843266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.830969 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-podinfo\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.843266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.831021 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-envoy\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.843266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.832579 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-data\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.843266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.832855 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-workload-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.843266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.833104 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-credential-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.865301 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.845722 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istiod-ca-cert\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.865301 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.845995 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-workload-certs\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.865301 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.864911 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-envoy\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:17.993029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.992979 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b97ed753-ba37-4f08-ae9b-f8044a64b349" path="/var/lib/kubelet/pods/b97ed753-ba37-4f08-ae9b-f8044a64b349/volumes" Nov 03 20:07:17.999253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:17.999194 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-podinfo\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:18.025401 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:18.023549 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-token\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:18.026027 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:18.025972 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwpmv\" (UniqueName: \"kubernetes.io/projected/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-kube-api-access-wwpmv\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:18.207830 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:18.207735 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:18.525986 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:07:18.525954 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b5af1a_f4e4_45a3_a7ee_59037e562b8c.slice/crio-d12731bc11fea58f19d6f8edbc419ac2df5d460c37339f819d94e14eb97ebaf2 WatchSource:0}: Error finding container d12731bc11fea58f19d6f8edbc419ac2df5d460c37339f819d94e14eb97ebaf2: Status 404 returned error can't find the container with id d12731bc11fea58f19d6f8edbc419ac2df5d460c37339f819d94e14eb97ebaf2 Nov 03 20:07:18.526776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:18.526750 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9"] Nov 03 20:07:18.528781 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:18.528749 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:07:18.528848 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:18.528815 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:07:18.528848 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:18.528844 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:07:18.997698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:18.997626 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" event={"ID":"04b5af1a-f4e4-45a3-a7ee-59037e562b8c","Type":"ContainerStarted","Data":"d2ed43f59e564d0a049376f043011d192ceaea1235be3c18c1ae208bbb3f3aa9"} Nov 03 20:07:18.997883 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:18.997706 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" event={"ID":"04b5af1a-f4e4-45a3-a7ee-59037e562b8c","Type":"ContainerStarted","Data":"d12731bc11fea58f19d6f8edbc419ac2df5d460c37339f819d94e14eb97ebaf2"} Nov 03 20:07:19.025121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:19.025076 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" podStartSLOduration=2.025062424 podStartE2EDuration="2.025062424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:19.022485325 +0000 UTC m=+1833.280282584" watchObservedRunningTime="2026-11-03 20:07:19.025062424 +0000 UTC m=+1833.282859681" Nov 03 20:07:19.215684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:19.214225 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:19.233500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:19.233465 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:20.022685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:20.022127 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:20.024585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:20.023392 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:07:23.026899 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:23.026863 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3968d0fb-lgt62"] Nov 03 20:07:23.030932 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:23.030909 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3968d0fb-lgt62"] Nov 03 20:07:23.859007 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:23.858973 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6aaa4d2-402f-4605-a0f6-3af867f7fbf9" path="/var/lib/kubelet/pods/c6aaa4d2-402f-4605-a0f6-3af867f7fbf9/volumes" Nov 03 20:07:36.026411 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:36.026372 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b18bc4e-g98qk"] Nov 03 20:07:36.030516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:36.030497 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b18bc4e-g98qk"] Nov 03 20:07:37.025726 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:37.025687 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0f6f815-vfxm5"] Nov 03 20:07:37.030921 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:37.030892 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0f6f815-vfxm5"] Nov 03 20:07:37.860334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:37.860295 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="168586b3-99f0-47df-b549-1cbd409efaa6" path="/var/lib/kubelet/pods/168586b3-99f0-47df-b549-1cbd409efaa6/volumes" Nov 03 20:07:37.860859 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:37.860841 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4636458d-1dfd-404f-ac36-e21f7b91d48e" path="/var/lib/kubelet/pods/4636458d-1dfd-404f-ac36-e21f7b91d48e/volumes" Nov 03 20:07:40.028529 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:40.028481 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-174f2038-q6k6v"] Nov 03 20:07:40.033158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:40.033126 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-174f2038-q6k6v"] Nov 03 20:07:40.605803 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:07:40.605767 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:07:41.859069 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:41.859031 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82192848-35ee-4ca0-88e7-3f477f3d3bbf" path="/var/lib/kubelet/pods/82192848-35ee-4ca0-88e7-3f477f3d3bbf/volumes" Nov 03 20:07:42.030572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:42.030537 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb1d7137-wcw5n"] Nov 03 20:07:42.035285 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:42.035240 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb1d7137-wcw5n"] Nov 03 20:07:43.859438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:43.859405 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3313416d-c233-439a-ad7d-f52125194720" path="/var/lib/kubelet/pods/3313416d-c233-439a-ad7d-f52125194720/volumes" Nov 03 20:07:48.151224 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:48.151184 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eda4db2e-s8gjr"] Nov 03 20:07:48.155226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:48.155198 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eda4db2e-s8gjr"] Nov 03 20:07:48.536636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:48.536542 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj"] Nov 03 20:07:48.538253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:48.538217 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" podUID="0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" containerName="istio-proxy" containerID="cri-o://30b686d20aa315b3d217ef4f6c8035860614ce718ff61acc1dbc7cd8fcb7308e" gracePeriod=30 Nov 03 20:07:49.859270 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:49.859232 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a33b17a4-a052-439f-a02a-fb3cad0ae4fb" path="/var/lib/kubelet/pods/a33b17a4-a052-439f-a02a-fb3cad0ae4fb/volumes" Nov 03 20:07:53.838283 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.838256 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:07:53.905923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.905886 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-podinfo\") pod \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " Nov 03 20:07:53.906104 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.905931 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-token\" (UniqueName: \"kubernetes.io/projected/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-token\") pod \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " Nov 03 20:07:53.906104 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.906029 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-data\") pod \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " Nov 03 20:07:53.906104 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.906091 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-credential-socket\") pod \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " Nov 03 20:07:53.906293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.906194 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-workload-socket\") pod \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " Nov 03 20:07:53.906293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.906249 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-kube-api-access-594f4\" (UniqueName: \"kubernetes.io/projected/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-kube-api-access-594f4\") pod \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " Nov 03 20:07:53.906399 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.906347 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-envoy\") pod \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " Nov 03 20:07:53.906472 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.906456 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-workload-certs\") pod \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " Nov 03 20:07:53.906539 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.906461 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-data" pod "0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" (UID: "0a00dc11-88ca-4dcd-a387-1ec714a4eeb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:07:53.906595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.906548 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istiod-ca-cert\") pod \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\" (UID: \"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8\") " Nov 03 20:07:53.906710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.906687 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-workload-socket" pod "0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" (UID: "0a00dc11-88ca-4dcd-a387-1ec714a4eeb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:07:53.906919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.906880 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-credential-socket" pod "0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" (UID: "0a00dc11-88ca-4dcd-a387-1ec714a4eeb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:07:53.907337 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.907295 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:53.907337 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.907323 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:53.907337 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.907338 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:53.907635 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.907565 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-workload-certs" pod "0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" (UID: "0a00dc11-88ca-4dcd-a387-1ec714a4eeb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:07:53.907826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.907805 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istiod-ca-cert" pod "0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" (UID: "0a00dc11-88ca-4dcd-a387-1ec714a4eeb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:07:53.910385 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.910354 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-podinfo" pod "0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" (UID: "0a00dc11-88ca-4dcd-a387-1ec714a4eeb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:07:53.910522 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.910504 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-token" pod "0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" (UID: "0a00dc11-88ca-4dcd-a387-1ec714a4eeb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:07:53.910898 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.910871 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-envoy" pod "0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" (UID: "0a00dc11-88ca-4dcd-a387-1ec714a4eeb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:07:53.915162 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.915132 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-kube-api-access-594f4" pod "0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" (UID: "0a00dc11-88ca-4dcd-a387-1ec714a4eeb8"). InnerVolumeSpecName "kube-api-access-594f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:07:53.921384 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.921364 5059 scope.go:122] "RemoveContainer" containerID="30b686d20aa315b3d217ef4f6c8035860614ce718ff61acc1dbc7cd8fcb7308e" Nov 03 20:07:53.944631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.944609 5059 scope.go:122] "RemoveContainer" containerID="376181b768f9f5bd054faa8e9c391626a9b7e07d8f21b87327bbb318eaaaf524" Nov 03 20:07:53.954418 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.954386 5059 scope.go:122] "RemoveContainer" containerID="97c073b05ba2c3446c585c4d7017383f339c42f9af2a0154eead946d75919100" Nov 03 20:07:53.963891 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.963872 5059 scope.go:122] "RemoveContainer" containerID="d99b7e96ffe84d1e65f27f9474ec64f38ac1a3fa15074b34cda21681c7d56b1d" Nov 03 20:07:53.973266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.973245 5059 scope.go:122] "RemoveContainer" containerID="0b2c5ee5f116b488344166b030a8c8c15d4a7739e51fe991225a449493d671ad" Nov 03 20:07:53.996241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:53.996215 5059 scope.go:122] "RemoveContainer" containerID="a180a1113880b44329da78f6d64a3b3c2de991dfd69ae30fe205df26c1a938f5" Nov 03 20:07:54.006842 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.006819 5059 scope.go:122] "RemoveContainer" containerID="be4e474aec01d226b253dfff7b360cf1f37548ad866e35c071ff0bfd36f30588" Nov 03 20:07:54.008486 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.008462 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:54.008585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.008491 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:54.008585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.008505 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:54.008585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.008520 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-594f4\" (UniqueName: \"kubernetes.io/projected/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-kube-api-access-594f4\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:54.008585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.008533 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:54.008585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.008546 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:54.017237 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.017215 5059 scope.go:122] "RemoveContainer" containerID="5c3b5c41fcd1e2c40dd974c0b7f58f69911d7723950cc20f06b1ae6b464a85f0" Nov 03 20:07:54.027366 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.027336 5059 scope.go:122] "RemoveContainer" containerID="c2a6adb3375dc514cd970092ba1eb5db443fef7cb70fb4a6c349f817bfc707d8" Nov 03 20:07:54.040346 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.040323 5059 scope.go:122] "RemoveContainer" containerID="5eb5bb4ea69a8f03657f822ef8d3b6b1314a329897845d5d392bf127f67c5ae9" Nov 03 20:07:54.049488 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.049468 5059 scope.go:122] "RemoveContainer" containerID="a2b48b8f2850952268bc5107611ef5320329e8cc68de71e1aa5ec8b933039f0d" Nov 03 20:07:54.511883 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.511841 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" event={"ID":"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8","Type":"ContainerDied","Data":"30b686d20aa315b3d217ef4f6c8035860614ce718ff61acc1dbc7cd8fcb7308e"} Nov 03 20:07:54.511883 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.511874 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" Nov 03 20:07:54.512139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.511898 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj" event={"ID":"0a00dc11-88ca-4dcd-a387-1ec714a4eeb8","Type":"ContainerDied","Data":"3e529567addfc699af7196420c9d58e20b2a82cae52ab801a7af30e5f4001334"} Nov 03 20:07:54.552911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.552881 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj"] Nov 03 20:07:54.559149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.559123 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-wsnrj"] Nov 03 20:07:54.764753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.764675 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7240-ae2e10"] Nov 03 20:07:54.765400 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.765385 5059 cpu_manager.go:414] "containerMap: removing container" podUID="8d64cbd8-7a40-4481-92a4-3ff4fa568da4" containerName="istio-proxy" Nov 03 20:07:54.765453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.765401 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d64cbd8-7a40-4481-92a4-3ff4fa568da4" containerName="istio-proxy" Nov 03 20:07:54.765453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.765441 5059 cpu_manager.go:414] "containerMap: removing container" podUID="0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" containerName="istio-proxy" Nov 03 20:07:54.765453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.765447 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" containerName="istio-proxy" Nov 03 20:07:54.765564 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.765555 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" containerName="istio-proxy" Nov 03 20:07:54.771565 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.771541 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-ae2e10" Nov 03 20:07:54.776950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.776922 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-ae2e10"] Nov 03 20:07:54.777960 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.777928 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 20:07:54.778163 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.778148 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 20:07:54.778712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.778697 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-tz7np\"" Nov 03 20:07:54.819037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.819001 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd46r\" (UniqueName: \"kubernetes.io/projected/c5761498-dbc0-432e-87b8-cfbe123e74b6-kube-api-access-vd46r\") pod \"test-curl-7240-ae2e10\" (UID: \"c5761498-dbc0-432e-87b8-cfbe123e74b6\") " pod="opendatahub/test-curl-7240-ae2e10" Nov 03 20:07:54.920507 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.920474 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd46r\" (UniqueName: \"kubernetes.io/projected/c5761498-dbc0-432e-87b8-cfbe123e74b6-kube-api-access-vd46r\") pod \"test-curl-7240-ae2e10\" (UID: \"c5761498-dbc0-432e-87b8-cfbe123e74b6\") " pod="opendatahub/test-curl-7240-ae2e10" Nov 03 20:07:54.933801 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:54.933767 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd46r\" (UniqueName: \"kubernetes.io/projected/c5761498-dbc0-432e-87b8-cfbe123e74b6-kube-api-access-vd46r\") pod \"test-curl-7240-ae2e10\" (UID: \"c5761498-dbc0-432e-87b8-cfbe123e74b6\") " pod="opendatahub/test-curl-7240-ae2e10" Nov 03 20:07:55.089808 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:55.089770 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-ae2e10" Nov 03 20:07:55.387530 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:55.387497 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-ae2e10"] Nov 03 20:07:55.387693 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:07:55.387553 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5761498_dbc0_432e_87b8_cfbe123e74b6.slice/crio-4b1da88b7dd74c7ad4662024b8587acbaf073c3bd6de09358670bda6463d50ff WatchSource:0}: Error finding container 4b1da88b7dd74c7ad4662024b8587acbaf073c3bd6de09358670bda6463d50ff: Status 404 returned error can't find the container with id 4b1da88b7dd74c7ad4662024b8587acbaf073c3bd6de09358670bda6463d50ff Nov 03 20:07:55.520510 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:55.520480 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-ae2e10" event={"ID":"c5761498-dbc0-432e-87b8-cfbe123e74b6","Type":"ContainerStarted","Data":"d73bf95c3bcdb0aad962d4f91420182a80a98ef375386c393ad386886db03169"} Nov 03 20:07:55.520648 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:55.520524 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-ae2e10" event={"ID":"c5761498-dbc0-432e-87b8-cfbe123e74b6","Type":"ContainerStarted","Data":"4b1da88b7dd74c7ad4662024b8587acbaf073c3bd6de09358670bda6463d50ff"} Nov 03 20:07:55.546182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:55.546129 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7240-ae2e10" podStartSLOduration=1.5461163930000001 podStartE2EDuration="1.546116393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:55.542161072 +0000 UTC m=+1869.799958328" watchObservedRunningTime="2026-11-03 20:07:55.546116393 +0000 UTC m=+1869.803913650" Nov 03 20:07:55.858803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:55.858770 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a00dc11-88ca-4dcd-a387-1ec714a4eeb8" path="/var/lib/kubelet/pods/0a00dc11-88ca-4dcd-a387-1ec714a4eeb8/volumes" Nov 03 20:07:56.264645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:56.264553 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7240-ae2e10"] Nov 03 20:07:56.269413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:56.269383 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7240-ae2e10"] Nov 03 20:07:56.421351 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:56.421313 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7240-160ed4"] Nov 03 20:07:56.422026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:56.422011 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c5761498-dbc0-432e-87b8-cfbe123e74b6" containerName="test-curl-7240-ae2e10" Nov 03 20:07:56.422081 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:56.422027 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5761498-dbc0-432e-87b8-cfbe123e74b6" containerName="test-curl-7240-ae2e10" Nov 03 20:07:56.422160 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:56.422151 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5761498-dbc0-432e-87b8-cfbe123e74b6" containerName="test-curl-7240-ae2e10" Nov 03 20:07:56.426152 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:56.426132 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-160ed4" Nov 03 20:07:56.433031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:56.433006 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-160ed4"] Nov 03 20:07:56.544052 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:56.543967 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-484z2\" (UniqueName: \"kubernetes.io/projected/de452d33-104d-449f-9fad-31e427a18708-kube-api-access-484z2\") pod \"test-curl-7240-160ed4\" (UID: \"de452d33-104d-449f-9fad-31e427a18708\") " pod="opendatahub/test-curl-7240-160ed4" Nov 03 20:07:56.645652 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:56.645619 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-484z2\" (UniqueName: \"kubernetes.io/projected/de452d33-104d-449f-9fad-31e427a18708-kube-api-access-484z2\") pod \"test-curl-7240-160ed4\" (UID: \"de452d33-104d-449f-9fad-31e427a18708\") " pod="opendatahub/test-curl-7240-160ed4" Nov 03 20:07:56.658251 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:56.658225 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-484z2\" (UniqueName: \"kubernetes.io/projected/de452d33-104d-449f-9fad-31e427a18708-kube-api-access-484z2\") pod \"test-curl-7240-160ed4\" (UID: \"de452d33-104d-449f-9fad-31e427a18708\") " pod="opendatahub/test-curl-7240-160ed4" Nov 03 20:07:56.742777 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:56.742743 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-160ed4" Nov 03 20:07:57.026540 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:07:57.026509 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde452d33_104d_449f_9fad_31e427a18708.slice/crio-d05a5b769bb43d1e98ae32d7ce5487940b6d21a5f8a4c6d44dd673209d6f4c44 WatchSource:0}: Error finding container d05a5b769bb43d1e98ae32d7ce5487940b6d21a5f8a4c6d44dd673209d6f4c44: Status 404 returned error can't find the container with id d05a5b769bb43d1e98ae32d7ce5487940b6d21a5f8a4c6d44dd673209d6f4c44 Nov 03 20:07:57.026936 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:57.026912 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-160ed4"] Nov 03 20:07:57.558092 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:57.557929 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7240-ae2e10" podUID="c5761498-dbc0-432e-87b8-cfbe123e74b6" containerName="test-curl-7240-ae2e10" containerID="cri-o://d73bf95c3bcdb0aad962d4f91420182a80a98ef375386c393ad386886db03169" gracePeriod=2 Nov 03 20:07:57.558948 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:57.558920 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-160ed4" event={"ID":"de452d33-104d-449f-9fad-31e427a18708","Type":"ContainerStarted","Data":"393a7840754f6783e548304d707ede9143426a1c8e3de44b907cd2008d62a95e"} Nov 03 20:07:57.559125 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:57.559111 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-160ed4" event={"ID":"de452d33-104d-449f-9fad-31e427a18708","Type":"ContainerStarted","Data":"d05a5b769bb43d1e98ae32d7ce5487940b6d21a5f8a4c6d44dd673209d6f4c44"} Nov 03 20:07:57.579731 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:07:57.579691 5059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-ae2e10\" is forbidden: User \"system:node:ip-10-0-132-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-83.us-east-2.compute.internal' and this object" podUID="c5761498-dbc0-432e-87b8-cfbe123e74b6" pod="opendatahub/test-curl-7240-ae2e10" Nov 03 20:07:57.579964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:57.579929 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7240-160ed4" podStartSLOduration=1.579916874 podStartE2EDuration="1.579916874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:57.576835831 +0000 UTC m=+1871.834633088" watchObservedRunningTime="2026-11-03 20:07:57.579916874 +0000 UTC m=+1871.837714131" Nov 03 20:07:57.582535 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:07:57.582510 5059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-ae2e10\" is forbidden: User \"system:node:ip-10-0-132-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-83.us-east-2.compute.internal' and this object" podUID="c5761498-dbc0-432e-87b8-cfbe123e74b6" pod="opendatahub/test-curl-7240-ae2e10" Nov 03 20:07:58.293460 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:58.293427 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7240-160ed4"] Nov 03 20:07:58.297572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:58.297533 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7240-160ed4"] Nov 03 20:07:58.454472 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:58.454425 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7240-875e8a"] Nov 03 20:07:58.455168 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:58.455153 5059 cpu_manager.go:414] "containerMap: removing container" podUID="de452d33-104d-449f-9fad-31e427a18708" containerName="test-curl-7240-160ed4" Nov 03 20:07:58.455249 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:58.455169 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de452d33-104d-449f-9fad-31e427a18708" containerName="test-curl-7240-160ed4" Nov 03 20:07:58.455290 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:58.455278 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="de452d33-104d-449f-9fad-31e427a18708" containerName="test-curl-7240-160ed4" Nov 03 20:07:58.459126 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:58.459111 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-875e8a" Nov 03 20:07:58.465221 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:58.465197 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-875e8a"] Nov 03 20:07:58.590076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:58.590044 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnswm\" (UniqueName: \"kubernetes.io/projected/18671ee8-6e47-40a6-9cb5-2dd963a45f60-kube-api-access-tnswm\") pod \"test-curl-7240-875e8a\" (UID: \"18671ee8-6e47-40a6-9cb5-2dd963a45f60\") " pod="opendatahub/test-curl-7240-875e8a" Nov 03 20:07:58.692053 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:58.692020 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnswm\" (UniqueName: \"kubernetes.io/projected/18671ee8-6e47-40a6-9cb5-2dd963a45f60-kube-api-access-tnswm\") pod \"test-curl-7240-875e8a\" (UID: \"18671ee8-6e47-40a6-9cb5-2dd963a45f60\") " pod="opendatahub/test-curl-7240-875e8a" Nov 03 20:07:58.704548 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:58.704519 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnswm\" (UniqueName: \"kubernetes.io/projected/18671ee8-6e47-40a6-9cb5-2dd963a45f60-kube-api-access-tnswm\") pod \"test-curl-7240-875e8a\" (UID: \"18671ee8-6e47-40a6-9cb5-2dd963a45f60\") " pod="opendatahub/test-curl-7240-875e8a" Nov 03 20:07:58.781871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:58.777926 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-875e8a" Nov 03 20:07:59.095471 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:59.095436 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-875e8a"] Nov 03 20:07:59.095986 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:07:59.095959 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18671ee8_6e47_40a6_9cb5_2dd963a45f60.slice/crio-6f2de8b27f06ef8201663492c3d49ef8e4d86ea5f8210dd554389bc68fc7894c WatchSource:0}: Error finding container 6f2de8b27f06ef8201663492c3d49ef8e4d86ea5f8210dd554389bc68fc7894c: Status 404 returned error can't find the container with id 6f2de8b27f06ef8201663492c3d49ef8e4d86ea5f8210dd554389bc68fc7894c Nov 03 20:07:59.580318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:59.580284 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-875e8a" event={"ID":"18671ee8-6e47-40a6-9cb5-2dd963a45f60","Type":"ContainerStarted","Data":"61647d722b5f1ee318c71a57ee3bc6626673f54b3a90be8e673f66e715b446fc"} Nov 03 20:07:59.580443 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:59.580335 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-875e8a" event={"ID":"18671ee8-6e47-40a6-9cb5-2dd963a45f60","Type":"ContainerStarted","Data":"6f2de8b27f06ef8201663492c3d49ef8e4d86ea5f8210dd554389bc68fc7894c"} Nov 03 20:07:59.580574 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:59.580533 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7240-160ed4" podUID="de452d33-104d-449f-9fad-31e427a18708" containerName="test-curl-7240-160ed4" containerID="cri-o://393a7840754f6783e548304d707ede9143426a1c8e3de44b907cd2008d62a95e" gracePeriod=2 Nov 03 20:07:59.597824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:59.597782 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7240-875e8a" podStartSLOduration=1.597769026 podStartE2EDuration="1.597769026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:59.595717201 +0000 UTC m=+1873.853514458" watchObservedRunningTime="2026-11-03 20:07:59.597769026 +0000 UTC m=+1873.855566283" Nov 03 20:07:59.598415 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:07:59.598391 5059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-160ed4\" is forbidden: User \"system:node:ip-10-0-132-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-83.us-east-2.compute.internal' and this object" podUID="de452d33-104d-449f-9fad-31e427a18708" pod="opendatahub/test-curl-7240-160ed4" Nov 03 20:07:59.773383 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:59.773353 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-ae2e10" Nov 03 20:07:59.776235 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:07:59.776210 5059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-160ed4\" is forbidden: User \"system:node:ip-10-0-132-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-83.us-east-2.compute.internal' and this object" podUID="de452d33-104d-449f-9fad-31e427a18708" pod="opendatahub/test-curl-7240-160ed4" Nov 03 20:07:59.812156 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:59.812129 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5761498-dbc0-432e-87b8-cfbe123e74b6-kube-api-access-vd46r\" (UniqueName: \"kubernetes.io/projected/c5761498-dbc0-432e-87b8-cfbe123e74b6-kube-api-access-vd46r\") pod \"c5761498-dbc0-432e-87b8-cfbe123e74b6\" (UID: \"c5761498-dbc0-432e-87b8-cfbe123e74b6\") " Nov 03 20:07:59.814252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:59.814222 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5761498-dbc0-432e-87b8-cfbe123e74b6-kube-api-access-vd46r" pod "c5761498-dbc0-432e-87b8-cfbe123e74b6" (UID: "c5761498-dbc0-432e-87b8-cfbe123e74b6"). InnerVolumeSpecName "kube-api-access-vd46r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:07:59.858318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:59.858286 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5761498-dbc0-432e-87b8-cfbe123e74b6" path="/var/lib/kubelet/pods/c5761498-dbc0-432e-87b8-cfbe123e74b6/volumes" Nov 03 20:07:59.915060 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:07:59.914989 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd46r\" (UniqueName: \"kubernetes.io/projected/c5761498-dbc0-432e-87b8-cfbe123e74b6-kube-api-access-vd46r\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:00.333323 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:00.333288 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7240-875e8a"] Nov 03 20:08:00.339242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:00.339208 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7240-875e8a"] Nov 03 20:08:00.596324 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:00.596229 5059 generic.go:356] "Generic (PLEG): container finished" podID="c5761498-dbc0-432e-87b8-cfbe123e74b6" containerID="d73bf95c3bcdb0aad962d4f91420182a80a98ef375386c393ad386886db03169" exitCode=137 Nov 03 20:08:00.596752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:00.596723 5059 scope.go:122] "RemoveContainer" containerID="d73bf95c3bcdb0aad962d4f91420182a80a98ef375386c393ad386886db03169" Nov 03 20:08:00.596752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:00.596745 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-ae2e10" Nov 03 20:08:00.607857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:00.607833 5059 scope.go:122] "RemoveContainer" containerID="d73bf95c3bcdb0aad962d4f91420182a80a98ef375386c393ad386886db03169" Nov 03 20:08:00.608221 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:08:00.608156 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d73bf95c3bcdb0aad962d4f91420182a80a98ef375386c393ad386886db03169\": container with ID starting with d73bf95c3bcdb0aad962d4f91420182a80a98ef375386c393ad386886db03169 not found: ID does not exist" containerID="d73bf95c3bcdb0aad962d4f91420182a80a98ef375386c393ad386886db03169" Nov 03 20:08:00.608221 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:00.608192 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d73bf95c3bcdb0aad962d4f91420182a80a98ef375386c393ad386886db03169"} err="failed to get container status \"d73bf95c3bcdb0aad962d4f91420182a80a98ef375386c393ad386886db03169\": rpc error: code = NotFound desc = could not find container \"d73bf95c3bcdb0aad962d4f91420182a80a98ef375386c393ad386886db03169\": container with ID starting with d73bf95c3bcdb0aad962d4f91420182a80a98ef375386c393ad386886db03169 not found: ID does not exist" Nov 03 20:08:01.609076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:01.609036 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7240-875e8a" podUID="18671ee8-6e47-40a6-9cb5-2dd963a45f60" containerName="test-curl-7240-875e8a" containerID="cri-o://61647d722b5f1ee318c71a57ee3bc6626673f54b3a90be8e673f66e715b446fc" gracePeriod=2 Nov 03 20:08:01.612101 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:08:01.612068 5059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-875e8a\" is forbidden: User \"system:node:ip-10-0-132-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-83.us-east-2.compute.internal' and this object" podUID="18671ee8-6e47-40a6-9cb5-2dd963a45f60" pod="opendatahub/test-curl-7240-875e8a" Nov 03 20:08:01.887626 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:01.887578 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-160ed4" Nov 03 20:08:02.013684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:02.013437 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de452d33-104d-449f-9fad-31e427a18708-kube-api-access-484z2\" (UniqueName: \"kubernetes.io/projected/de452d33-104d-449f-9fad-31e427a18708-kube-api-access-484z2\") pod \"de452d33-104d-449f-9fad-31e427a18708\" (UID: \"de452d33-104d-449f-9fad-31e427a18708\") " Nov 03 20:08:02.027266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:02.027205 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de452d33-104d-449f-9fad-31e427a18708-kube-api-access-484z2" pod "de452d33-104d-449f-9fad-31e427a18708" (UID: "de452d33-104d-449f-9fad-31e427a18708"). InnerVolumeSpecName "kube-api-access-484z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:02.121992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:02.121904 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-484z2\" (UniqueName: \"kubernetes.io/projected/de452d33-104d-449f-9fad-31e427a18708-kube-api-access-484z2\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:02.670685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:02.657874 5059 generic.go:356] "Generic (PLEG): container finished" podID="de452d33-104d-449f-9fad-31e427a18708" containerID="393a7840754f6783e548304d707ede9143426a1c8e3de44b907cd2008d62a95e" exitCode=137 Nov 03 20:08:02.671548 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:02.669868 5059 scope.go:122] "RemoveContainer" containerID="393a7840754f6783e548304d707ede9143426a1c8e3de44b907cd2008d62a95e" Nov 03 20:08:02.671779 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:02.669987 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-160ed4" Nov 03 20:08:02.750696 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:08:02.741241 5059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-160ed4\" is forbidden: User \"system:node:ip-10-0-132-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-83.us-east-2.compute.internal' and this object" podUID="de452d33-104d-449f-9fad-31e427a18708" pod="opendatahub/test-curl-7240-160ed4" Nov 03 20:08:02.906061 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:02.901490 5059 scope.go:122] "RemoveContainer" containerID="393a7840754f6783e548304d707ede9143426a1c8e3de44b907cd2008d62a95e" Nov 03 20:08:02.906061 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:08:02.902438 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"393a7840754f6783e548304d707ede9143426a1c8e3de44b907cd2008d62a95e\": container with ID starting with 393a7840754f6783e548304d707ede9143426a1c8e3de44b907cd2008d62a95e not found: ID does not exist" containerID="393a7840754f6783e548304d707ede9143426a1c8e3de44b907cd2008d62a95e" Nov 03 20:08:02.906061 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:02.902479 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"393a7840754f6783e548304d707ede9143426a1c8e3de44b907cd2008d62a95e"} err="failed to get container status \"393a7840754f6783e548304d707ede9143426a1c8e3de44b907cd2008d62a95e\": rpc error: code = NotFound desc = could not find container \"393a7840754f6783e548304d707ede9143426a1c8e3de44b907cd2008d62a95e\": container with ID starting with 393a7840754f6783e548304d707ede9143426a1c8e3de44b907cd2008d62a95e not found: ID does not exist" Nov 03 20:08:03.822175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:03.821797 5059 generic.go:356] "Generic (PLEG): container finished" podID="18671ee8-6e47-40a6-9cb5-2dd963a45f60" containerID="61647d722b5f1ee318c71a57ee3bc6626673f54b3a90be8e673f66e715b446fc" exitCode=137 Nov 03 20:08:03.862018 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:03.861982 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de452d33-104d-449f-9fad-31e427a18708" path="/var/lib/kubelet/pods/de452d33-104d-449f-9fad-31e427a18708/volumes" Nov 03 20:08:04.023560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:04.023535 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-875e8a" Nov 03 20:08:04.028078 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:04.028052 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3abefc2-9ff2n"] Nov 03 20:08:04.031515 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:04.031488 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18671ee8-6e47-40a6-9cb5-2dd963a45f60-kube-api-access-tnswm\" (UniqueName: \"kubernetes.io/projected/18671ee8-6e47-40a6-9cb5-2dd963a45f60-kube-api-access-tnswm\") pod \"18671ee8-6e47-40a6-9cb5-2dd963a45f60\" (UID: \"18671ee8-6e47-40a6-9cb5-2dd963a45f60\") " Nov 03 20:08:04.033083 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:04.033063 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3abefc2-9ff2n"] Nov 03 20:08:04.033772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:04.033752 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18671ee8-6e47-40a6-9cb5-2dd963a45f60-kube-api-access-tnswm" pod "18671ee8-6e47-40a6-9cb5-2dd963a45f60" (UID: "18671ee8-6e47-40a6-9cb5-2dd963a45f60"). InnerVolumeSpecName "kube-api-access-tnswm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:04.134039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:04.134005 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnswm\" (UniqueName: \"kubernetes.io/projected/18671ee8-6e47-40a6-9cb5-2dd963a45f60-kube-api-access-tnswm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:04.832145 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:04.832115 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-875e8a" Nov 03 20:08:04.832567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:04.832174 5059 scope.go:122] "RemoveContainer" containerID="61647d722b5f1ee318c71a57ee3bc6626673f54b3a90be8e673f66e715b446fc" Nov 03 20:08:04.847961 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:08:04.847922 5059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-875e8a\" is forbidden: User \"system:node:ip-10-0-132-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-83.us-east-2.compute.internal' and this object" podUID="18671ee8-6e47-40a6-9cb5-2dd963a45f60" pod="opendatahub/test-curl-7240-875e8a" Nov 03 20:08:05.044545 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.044500 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584f7a67-tqzsq"] Nov 03 20:08:05.044744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.044569 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584f7a67-tqzsq"] Nov 03 20:08:05.288736 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.287724 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5321be46-mtmr8"] Nov 03 20:08:05.288919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.288794 5059 cpu_manager.go:414] "containerMap: removing container" podUID="18671ee8-6e47-40a6-9cb5-2dd963a45f60" containerName="test-curl-7240-875e8a" Nov 03 20:08:05.288919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.288816 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18671ee8-6e47-40a6-9cb5-2dd963a45f60" containerName="test-curl-7240-875e8a" Nov 03 20:08:05.289042 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.289010 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="18671ee8-6e47-40a6-9cb5-2dd963a45f60" containerName="test-curl-7240-875e8a" Nov 03 20:08:05.298230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.294865 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5321be46-mtmr8" Nov 03 20:08:05.298230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.297984 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5321be46-mtmr8"] Nov 03 20:08:05.303638 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.302992 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:08:05.303638 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.303056 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:08:05.319774 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:08:05.319744 5059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-875e8a\" is forbidden: User \"system:node:ip-10-0-132-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-83.us-east-2.compute.internal' and this object" podUID="18671ee8-6e47-40a6-9cb5-2dd963a45f60" pod="opendatahub/test-curl-7240-875e8a" Nov 03 20:08:05.397110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.397070 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfrsk\" (UniqueName: \"kubernetes.io/projected/1274ddc7-26e8-4b34-be7a-bb7ac8bf7404-kube-api-access-gfrsk\") pod \"maas-api-revoke-sub-5321be46-mtmr8\" (UID: \"1274ddc7-26e8-4b34-be7a-bb7ac8bf7404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5321be46-mtmr8" Nov 03 20:08:05.397257 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.397131 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1274ddc7-26e8-4b34-be7a-bb7ac8bf7404-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5321be46-mtmr8\" (UID: \"1274ddc7-26e8-4b34-be7a-bb7ac8bf7404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5321be46-mtmr8" Nov 03 20:08:05.397257 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.397228 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1274ddc7-26e8-4b34-be7a-bb7ac8bf7404-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5321be46-mtmr8\" (UID: \"1274ddc7-26e8-4b34-be7a-bb7ac8bf7404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5321be46-mtmr8" Nov 03 20:08:05.498283 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.498248 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfrsk\" (UniqueName: \"kubernetes.io/projected/1274ddc7-26e8-4b34-be7a-bb7ac8bf7404-kube-api-access-gfrsk\") pod \"maas-api-revoke-sub-5321be46-mtmr8\" (UID: \"1274ddc7-26e8-4b34-be7a-bb7ac8bf7404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5321be46-mtmr8" Nov 03 20:08:05.498446 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.498310 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1274ddc7-26e8-4b34-be7a-bb7ac8bf7404-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5321be46-mtmr8\" (UID: \"1274ddc7-26e8-4b34-be7a-bb7ac8bf7404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5321be46-mtmr8" Nov 03 20:08:05.498446 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.498346 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1274ddc7-26e8-4b34-be7a-bb7ac8bf7404-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5321be46-mtmr8\" (UID: \"1274ddc7-26e8-4b34-be7a-bb7ac8bf7404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5321be46-mtmr8" Nov 03 20:08:05.499061 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.499040 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1274ddc7-26e8-4b34-be7a-bb7ac8bf7404-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5321be46-mtmr8\" (UID: \"1274ddc7-26e8-4b34-be7a-bb7ac8bf7404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5321be46-mtmr8" Nov 03 20:08:05.500623 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.500600 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1274ddc7-26e8-4b34-be7a-bb7ac8bf7404-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5321be46-mtmr8\" (UID: \"1274ddc7-26e8-4b34-be7a-bb7ac8bf7404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5321be46-mtmr8" Nov 03 20:08:05.518964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.518941 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfrsk\" (UniqueName: \"kubernetes.io/projected/1274ddc7-26e8-4b34-be7a-bb7ac8bf7404-kube-api-access-gfrsk\") pod \"maas-api-revoke-sub-5321be46-mtmr8\" (UID: \"1274ddc7-26e8-4b34-be7a-bb7ac8bf7404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5321be46-mtmr8" Nov 03 20:08:05.619787 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.619754 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5321be46-mtmr8" Nov 03 20:08:05.694510 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.693912 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-198a5abc-42nff"] Nov 03 20:08:05.701913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.701889 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-198a5abc-42nff"] Nov 03 20:08:05.702098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.702002 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-198a5abc-42nff" Nov 03 20:08:05.731892 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:08:05.731855 5059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-875e8a\" is forbidden: User \"system:node:ip-10-0-132-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-83.us-east-2.compute.internal' and this object" podUID="18671ee8-6e47-40a6-9cb5-2dd963a45f60" pod="opendatahub/test-curl-7240-875e8a" Nov 03 20:08:05.802585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.802545 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l57dc\" (UniqueName: \"kubernetes.io/projected/c098791a-1c42-4ec9-aeaf-d1f82b2c12da-kube-api-access-l57dc\") pod \"maas-api-revoke-sub-198a5abc-42nff\" (UID: \"c098791a-1c42-4ec9-aeaf-d1f82b2c12da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-198a5abc-42nff" Nov 03 20:08:05.802784 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.802716 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c098791a-1c42-4ec9-aeaf-d1f82b2c12da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-198a5abc-42nff\" (UID: \"c098791a-1c42-4ec9-aeaf-d1f82b2c12da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-198a5abc-42nff" Nov 03 20:08:05.802932 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.802898 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c098791a-1c42-4ec9-aeaf-d1f82b2c12da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-198a5abc-42nff\" (UID: \"c098791a-1c42-4ec9-aeaf-d1f82b2c12da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-198a5abc-42nff" Nov 03 20:08:05.882994 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:08:05.882860 5059 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-875e8a\" is forbidden: User \"system:node:ip-10-0-132-83.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-83.us-east-2.compute.internal' and this object" podUID="18671ee8-6e47-40a6-9cb5-2dd963a45f60" pod="opendatahub/test-curl-7240-875e8a" Nov 03 20:08:05.886916 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.886891 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18671ee8-6e47-40a6-9cb5-2dd963a45f60" path="/var/lib/kubelet/pods/18671ee8-6e47-40a6-9cb5-2dd963a45f60/volumes" Nov 03 20:08:05.887417 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.887400 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24c7c194-490b-4fcf-b043-249e7aa0aa15" path="/var/lib/kubelet/pods/24c7c194-490b-4fcf-b043-249e7aa0aa15/volumes" Nov 03 20:08:05.887904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.887888 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1" path="/var/lib/kubelet/pods/ac60a0ea-6cbc-441e-9d6b-986d5eb0c6a1/volumes" Nov 03 20:08:05.904091 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.904061 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c098791a-1c42-4ec9-aeaf-d1f82b2c12da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-198a5abc-42nff\" (UID: \"c098791a-1c42-4ec9-aeaf-d1f82b2c12da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-198a5abc-42nff" Nov 03 20:08:05.904239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.904225 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c098791a-1c42-4ec9-aeaf-d1f82b2c12da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-198a5abc-42nff\" (UID: \"c098791a-1c42-4ec9-aeaf-d1f82b2c12da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-198a5abc-42nff" Nov 03 20:08:05.904320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.904266 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l57dc\" (UniqueName: \"kubernetes.io/projected/c098791a-1c42-4ec9-aeaf-d1f82b2c12da-kube-api-access-l57dc\") pod \"maas-api-revoke-sub-198a5abc-42nff\" (UID: \"c098791a-1c42-4ec9-aeaf-d1f82b2c12da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-198a5abc-42nff" Nov 03 20:08:05.904717 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.904697 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c098791a-1c42-4ec9-aeaf-d1f82b2c12da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-198a5abc-42nff\" (UID: \"c098791a-1c42-4ec9-aeaf-d1f82b2c12da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-198a5abc-42nff" Nov 03 20:08:05.906620 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.906600 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c098791a-1c42-4ec9-aeaf-d1f82b2c12da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-198a5abc-42nff\" (UID: \"c098791a-1c42-4ec9-aeaf-d1f82b2c12da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-198a5abc-42nff" Nov 03 20:08:05.916388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.916368 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l57dc\" (UniqueName: \"kubernetes.io/projected/c098791a-1c42-4ec9-aeaf-d1f82b2c12da-kube-api-access-l57dc\") pod \"maas-api-revoke-sub-198a5abc-42nff\" (UID: \"c098791a-1c42-4ec9-aeaf-d1f82b2c12da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-198a5abc-42nff" Nov 03 20:08:05.945533 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:05.945509 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5321be46-mtmr8"] Nov 03 20:08:05.945951 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:08:05.945927 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1274ddc7_26e8_4b34_be7a_bb7ac8bf7404.slice/crio-8f9b33cefcb64b935025386579931b9d57dbc60e5e69c011eab940ba3ef47fd8 WatchSource:0}: Error finding container 8f9b33cefcb64b935025386579931b9d57dbc60e5e69c011eab940ba3ef47fd8: Status 404 returned error can't find the container with id 8f9b33cefcb64b935025386579931b9d57dbc60e5e69c011eab940ba3ef47fd8 Nov 03 20:08:06.029736 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:06.029706 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-198a5abc-42nff" Nov 03 20:08:06.426612 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:06.426579 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-198a5abc-42nff"] Nov 03 20:08:06.427239 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:08:06.427200 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc098791a_1c42_4ec9_aeaf_d1f82b2c12da.slice/crio-2669903ccab605dd09d4fe6e58f99720e829976d7ec1448e35aeab5c2b55b57b WatchSource:0}: Error finding container 2669903ccab605dd09d4fe6e58f99720e829976d7ec1448e35aeab5c2b55b57b: Status 404 returned error can't find the container with id 2669903ccab605dd09d4fe6e58f99720e829976d7ec1448e35aeab5c2b55b57b Nov 03 20:08:06.897858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:06.897809 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-198a5abc-42nff" event={"ID":"c098791a-1c42-4ec9-aeaf-d1f82b2c12da","Type":"ContainerStarted","Data":"2669903ccab605dd09d4fe6e58f99720e829976d7ec1448e35aeab5c2b55b57b"} Nov 03 20:08:06.902412 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:06.902382 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5321be46-mtmr8" event={"ID":"1274ddc7-26e8-4b34-be7a-bb7ac8bf7404","Type":"ContainerStarted","Data":"c30f1dd89550433b076a505de418c277da5f2eea4729c0bca5b0deb1e1dddc7b"} Nov 03 20:08:06.902582 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:06.902569 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5321be46-mtmr8" event={"ID":"1274ddc7-26e8-4b34-be7a-bb7ac8bf7404","Type":"ContainerStarted","Data":"8f9b33cefcb64b935025386579931b9d57dbc60e5e69c011eab940ba3ef47fd8"} Nov 03 20:08:06.929722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:06.929656 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5321be46-mtmr8" podStartSLOduration=1.929643276 podStartE2EDuration="1.929643276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:06.927147831 +0000 UTC m=+1881.184945088" watchObservedRunningTime="2026-11-03 20:08:06.929643276 +0000 UTC m=+1881.187440577" Nov 03 20:08:07.916489 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:07.916457 5059 generic.go:356] "Generic (PLEG): container finished" podID="c098791a-1c42-4ec9-aeaf-d1f82b2c12da" containerID="5dd04b77ab04596d2aadc80578199ac5a2c8f34a4fc8b357771411a4c706116c" exitCode=0 Nov 03 20:08:07.916912 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:07.916537 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-198a5abc-42nff" event={"ID":"c098791a-1c42-4ec9-aeaf-d1f82b2c12da","Type":"ContainerDied","Data":"5dd04b77ab04596d2aadc80578199ac5a2c8f34a4fc8b357771411a4c706116c"} Nov 03 20:08:07.918266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:07.918235 5059 generic.go:356] "Generic (PLEG): container finished" podID="1274ddc7-26e8-4b34-be7a-bb7ac8bf7404" containerID="c30f1dd89550433b076a505de418c277da5f2eea4729c0bca5b0deb1e1dddc7b" exitCode=0 Nov 03 20:08:07.918266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:07.918263 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5321be46-mtmr8" event={"ID":"1274ddc7-26e8-4b34-be7a-bb7ac8bf7404","Type":"ContainerDied","Data":"c30f1dd89550433b076a505de418c277da5f2eea4729c0bca5b0deb1e1dddc7b"} Nov 03 20:08:09.236709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.236680 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5321be46-mtmr8" Nov 03 20:08:09.240768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.240745 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-198a5abc-42nff" Nov 03 20:08:09.303991 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.303957 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c098791a-1c42-4ec9-aeaf-d1f82b2c12da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c098791a-1c42-4ec9-aeaf-d1f82b2c12da-maas-api-openshift-ca\") pod \"c098791a-1c42-4ec9-aeaf-d1f82b2c12da\" (UID: \"c098791a-1c42-4ec9-aeaf-d1f82b2c12da\") " Nov 03 20:08:09.304160 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.303999 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1274ddc7-26e8-4b34-be7a-bb7ac8bf7404-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1274ddc7-26e8-4b34-be7a-bb7ac8bf7404-maas-api-openshift-ca\") pod \"1274ddc7-26e8-4b34-be7a-bb7ac8bf7404\" (UID: \"1274ddc7-26e8-4b34-be7a-bb7ac8bf7404\") " Nov 03 20:08:09.304160 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.304085 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1274ddc7-26e8-4b34-be7a-bb7ac8bf7404-kube-api-access-gfrsk\" (UniqueName: \"kubernetes.io/projected/1274ddc7-26e8-4b34-be7a-bb7ac8bf7404-kube-api-access-gfrsk\") pod \"1274ddc7-26e8-4b34-be7a-bb7ac8bf7404\" (UID: \"1274ddc7-26e8-4b34-be7a-bb7ac8bf7404\") " Nov 03 20:08:09.304357 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.304333 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1274ddc7-26e8-4b34-be7a-bb7ac8bf7404-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1274ddc7-26e8-4b34-be7a-bb7ac8bf7404-maas-api-xks-ca\") pod \"1274ddc7-26e8-4b34-be7a-bb7ac8bf7404\" (UID: \"1274ddc7-26e8-4b34-be7a-bb7ac8bf7404\") " Nov 03 20:08:09.304421 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.304384 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c098791a-1c42-4ec9-aeaf-d1f82b2c12da-maas-api-openshift-ca" pod "c098791a-1c42-4ec9-aeaf-d1f82b2c12da" (UID: "c098791a-1c42-4ec9-aeaf-d1f82b2c12da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:09.304482 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.304424 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c098791a-1c42-4ec9-aeaf-d1f82b2c12da-kube-api-access-l57dc\" (UniqueName: \"kubernetes.io/projected/c098791a-1c42-4ec9-aeaf-d1f82b2c12da-kube-api-access-l57dc\") pod \"c098791a-1c42-4ec9-aeaf-d1f82b2c12da\" (UID: \"c098791a-1c42-4ec9-aeaf-d1f82b2c12da\") " Nov 03 20:08:09.304482 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.304463 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1274ddc7-26e8-4b34-be7a-bb7ac8bf7404-maas-api-openshift-ca" pod "1274ddc7-26e8-4b34-be7a-bb7ac8bf7404" (UID: "1274ddc7-26e8-4b34-be7a-bb7ac8bf7404"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:09.304581 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.304487 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c098791a-1c42-4ec9-aeaf-d1f82b2c12da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c098791a-1c42-4ec9-aeaf-d1f82b2c12da-maas-api-xks-ca\") pod \"c098791a-1c42-4ec9-aeaf-d1f82b2c12da\" (UID: \"c098791a-1c42-4ec9-aeaf-d1f82b2c12da\") " Nov 03 20:08:09.305285 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.305258 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c098791a-1c42-4ec9-aeaf-d1f82b2c12da-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:09.305408 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.305288 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1274ddc7-26e8-4b34-be7a-bb7ac8bf7404-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:09.306909 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.306885 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1274ddc7-26e8-4b34-be7a-bb7ac8bf7404-maas-api-xks-ca" pod "1274ddc7-26e8-4b34-be7a-bb7ac8bf7404" (UID: "1274ddc7-26e8-4b34-be7a-bb7ac8bf7404"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:08:09.307349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.307323 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c098791a-1c42-4ec9-aeaf-d1f82b2c12da-kube-api-access-l57dc" pod "c098791a-1c42-4ec9-aeaf-d1f82b2c12da" (UID: "c098791a-1c42-4ec9-aeaf-d1f82b2c12da"). InnerVolumeSpecName "kube-api-access-l57dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:09.307561 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.307535 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1274ddc7-26e8-4b34-be7a-bb7ac8bf7404-kube-api-access-gfrsk" pod "1274ddc7-26e8-4b34-be7a-bb7ac8bf7404" (UID: "1274ddc7-26e8-4b34-be7a-bb7ac8bf7404"). InnerVolumeSpecName "kube-api-access-gfrsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:09.307864 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.307835 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c098791a-1c42-4ec9-aeaf-d1f82b2c12da-maas-api-xks-ca" pod "c098791a-1c42-4ec9-aeaf-d1f82b2c12da" (UID: "c098791a-1c42-4ec9-aeaf-d1f82b2c12da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:08:09.406412 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.406371 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1274ddc7-26e8-4b34-be7a-bb7ac8bf7404-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:09.406412 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.406408 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l57dc\" (UniqueName: \"kubernetes.io/projected/c098791a-1c42-4ec9-aeaf-d1f82b2c12da-kube-api-access-l57dc\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:09.406412 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.406421 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c098791a-1c42-4ec9-aeaf-d1f82b2c12da-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:09.406645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.406431 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfrsk\" (UniqueName: \"kubernetes.io/projected/1274ddc7-26e8-4b34-be7a-bb7ac8bf7404-kube-api-access-gfrsk\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:09.943108 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.943077 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-198a5abc-42nff" Nov 03 20:08:09.943108 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.943090 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-198a5abc-42nff" event={"ID":"c098791a-1c42-4ec9-aeaf-d1f82b2c12da","Type":"ContainerDied","Data":"2669903ccab605dd09d4fe6e58f99720e829976d7ec1448e35aeab5c2b55b57b"} Nov 03 20:08:09.943319 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.943143 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2669903ccab605dd09d4fe6e58f99720e829976d7ec1448e35aeab5c2b55b57b" Nov 03 20:08:09.944859 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.944832 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5321be46-mtmr8" Nov 03 20:08:09.944996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.944865 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5321be46-mtmr8" event={"ID":"1274ddc7-26e8-4b34-be7a-bb7ac8bf7404","Type":"ContainerDied","Data":"8f9b33cefcb64b935025386579931b9d57dbc60e5e69c011eab940ba3ef47fd8"} Nov 03 20:08:09.944996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:09.944905 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f9b33cefcb64b935025386579931b9d57dbc60e5e69c011eab940ba3ef47fd8" Nov 03 20:08:10.610152 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:08:10.610107 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:08:12.370886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.370845 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj"] Nov 03 20:08:12.371940 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.371915 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c098791a-1c42-4ec9-aeaf-d1f82b2c12da" containerName="revoke-keys" Nov 03 20:08:12.371940 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.371940 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c098791a-1c42-4ec9-aeaf-d1f82b2c12da" containerName="revoke-keys" Nov 03 20:08:12.372142 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.371992 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1274ddc7-26e8-4b34-be7a-bb7ac8bf7404" containerName="revoke-keys" Nov 03 20:08:12.372142 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.372000 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1274ddc7-26e8-4b34-be7a-bb7ac8bf7404" containerName="revoke-keys" Nov 03 20:08:12.372261 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.372213 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1274ddc7-26e8-4b34-be7a-bb7ac8bf7404" containerName="revoke-keys" Nov 03 20:08:12.372261 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.372238 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c098791a-1c42-4ec9-aeaf-d1f82b2c12da" containerName="revoke-keys" Nov 03 20:08:12.407814 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.407782 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj"] Nov 03 20:08:12.407991 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.407888 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.412105 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.412085 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-eb69cb-openshift-default-dockercfg-w96q5\"" Nov 03 20:08:12.580698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.576046 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.580698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.576132 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.580698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.576196 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.580698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.576247 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6qbw\" (UniqueName: \"kubernetes.io/projected/c9721811-d27d-4e13-a53d-4ae81e16fb96-kube-api-access-d6qbw\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.580698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.576285 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.580698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.576440 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.580698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.576498 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.580698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.576618 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9721811-d27d-4e13-a53d-4ae81e16fb96-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.580698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.576691 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.696746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.691092 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.696746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.691294 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.696746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.691365 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.696746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.691511 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9721811-d27d-4e13-a53d-4ae81e16fb96-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.696746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.691569 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.696746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.691634 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.696746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.691721 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.696746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.691790 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.696746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.691846 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6qbw\" (UniqueName: \"kubernetes.io/projected/c9721811-d27d-4e13-a53d-4ae81e16fb96-kube-api-access-d6qbw\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.696746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.692656 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.696746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.694991 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.696746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.696106 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9721811-d27d-4e13-a53d-4ae81e16fb96-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.696746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.696349 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.697611 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.697487 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.755768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.754178 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6qbw\" (UniqueName: \"kubernetes.io/projected/c9721811-d27d-4e13-a53d-4ae81e16fb96-kube-api-access-d6qbw\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.755768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.754528 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.762413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.762334 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.763048 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.763004 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:12.798199 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:12.798146 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:13.151734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:13.151706 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj"] Nov 03 20:08:13.153385 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:08:13.153355 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9721811_d27d_4e13_a53d_4ae81e16fb96.slice/crio-d7ff0c8bb110c798a57bd78b4c449469b094a557594b7ef3ed9540f8a707fec2 WatchSource:0}: Error finding container d7ff0c8bb110c798a57bd78b4c449469b094a557594b7ef3ed9540f8a707fec2: Status 404 returned error can't find the container with id d7ff0c8bb110c798a57bd78b4c449469b094a557594b7ef3ed9540f8a707fec2 Nov 03 20:08:13.156410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:13.156373 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:08:13.156516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:13.156458 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:08:13.156516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:13.156503 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:08:14.039303 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:14.039264 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" event={"ID":"c9721811-d27d-4e13-a53d-4ae81e16fb96","Type":"ContainerStarted","Data":"05da4fb16c45de93fc800ccb459f5ce97d6b836ada2d5d7bf411997c2312e7b3"} Nov 03 20:08:14.039872 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:14.039332 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" event={"ID":"c9721811-d27d-4e13-a53d-4ae81e16fb96","Type":"ContainerStarted","Data":"d7ff0c8bb110c798a57bd78b4c449469b094a557594b7ef3ed9540f8a707fec2"} Nov 03 20:08:14.067016 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:14.066959 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" podStartSLOduration=2.066941264 podStartE2EDuration="2.066941264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:14.06434398 +0000 UTC m=+1888.322141237" watchObservedRunningTime="2026-11-03 20:08:14.066941264 +0000 UTC m=+1888.324738521" Nov 03 20:08:14.805754 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:14.805698 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:14.823879 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:14.823621 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:15.047240 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:15.047208 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:15.048300 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:15.048275 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:08:15.534287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:15.534252 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0796d7-hhv2k"] Nov 03 20:08:15.575726 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:15.575689 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0796d7-hhv2k"] Nov 03 20:08:15.575895 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:15.575774 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0796d7-hhv2k" Nov 03 20:08:15.579917 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:15.579878 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:08:15.580081 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:15.580048 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:08:15.777737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:15.763739 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae874737-754b-4414-a571-2333feda6d47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf0796d7-hhv2k\" (UID: \"ae874737-754b-4414-a571-2333feda6d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0796d7-hhv2k" Nov 03 20:08:15.777737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:15.763916 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kspj\" (UniqueName: \"kubernetes.io/projected/ae874737-754b-4414-a571-2333feda6d47-kube-api-access-8kspj\") pod \"maas-api-revoke-sub-bf0796d7-hhv2k\" (UID: \"ae874737-754b-4414-a571-2333feda6d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0796d7-hhv2k" Nov 03 20:08:15.777737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:15.764295 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae874737-754b-4414-a571-2333feda6d47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf0796d7-hhv2k\" (UID: \"ae874737-754b-4414-a571-2333feda6d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0796d7-hhv2k" Nov 03 20:08:15.865981 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:15.865945 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae874737-754b-4414-a571-2333feda6d47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf0796d7-hhv2k\" (UID: \"ae874737-754b-4414-a571-2333feda6d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0796d7-hhv2k" Nov 03 20:08:15.866183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:15.866044 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kspj\" (UniqueName: \"kubernetes.io/projected/ae874737-754b-4414-a571-2333feda6d47-kube-api-access-8kspj\") pod \"maas-api-revoke-sub-bf0796d7-hhv2k\" (UID: \"ae874737-754b-4414-a571-2333feda6d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0796d7-hhv2k" Nov 03 20:08:15.866270 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:15.866182 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae874737-754b-4414-a571-2333feda6d47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf0796d7-hhv2k\" (UID: \"ae874737-754b-4414-a571-2333feda6d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0796d7-hhv2k" Nov 03 20:08:15.866833 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:15.866806 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae874737-754b-4414-a571-2333feda6d47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf0796d7-hhv2k\" (UID: \"ae874737-754b-4414-a571-2333feda6d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0796d7-hhv2k" Nov 03 20:08:15.868435 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:15.868407 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae874737-754b-4414-a571-2333feda6d47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf0796d7-hhv2k\" (UID: \"ae874737-754b-4414-a571-2333feda6d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0796d7-hhv2k" Nov 03 20:08:15.879415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:15.879391 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kspj\" (UniqueName: \"kubernetes.io/projected/ae874737-754b-4414-a571-2333feda6d47-kube-api-access-8kspj\") pod \"maas-api-revoke-sub-bf0796d7-hhv2k\" (UID: \"ae874737-754b-4414-a571-2333feda6d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0796d7-hhv2k" Nov 03 20:08:15.914349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:15.914326 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0796d7-hhv2k" Nov 03 20:08:16.016172 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:16.016139 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59649767-prkjd"] Nov 03 20:08:16.043224 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:16.043166 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59649767-prkjd"] Nov 03 20:08:16.043224 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:16.043196 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59649767-prkjd" Nov 03 20:08:16.069212 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:16.068994 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a4c119-410d-4e61-8db1-d440e0c371c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59649767-prkjd\" (UID: \"06a4c119-410d-4e61-8db1-d440e0c371c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59649767-prkjd" Nov 03 20:08:16.069212 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:16.069069 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a4c119-410d-4e61-8db1-d440e0c371c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59649767-prkjd\" (UID: \"06a4c119-410d-4e61-8db1-d440e0c371c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59649767-prkjd" Nov 03 20:08:16.069212 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:16.069165 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncsgb\" (UniqueName: \"kubernetes.io/projected/06a4c119-410d-4e61-8db1-d440e0c371c9-kube-api-access-ncsgb\") pod \"maas-api-revoke-sub-59649767-prkjd\" (UID: \"06a4c119-410d-4e61-8db1-d440e0c371c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59649767-prkjd" Nov 03 20:08:16.170708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:16.170577 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncsgb\" (UniqueName: \"kubernetes.io/projected/06a4c119-410d-4e61-8db1-d440e0c371c9-kube-api-access-ncsgb\") pod \"maas-api-revoke-sub-59649767-prkjd\" (UID: \"06a4c119-410d-4e61-8db1-d440e0c371c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59649767-prkjd" Nov 03 20:08:16.170932 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:16.170889 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a4c119-410d-4e61-8db1-d440e0c371c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59649767-prkjd\" (UID: \"06a4c119-410d-4e61-8db1-d440e0c371c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59649767-prkjd" Nov 03 20:08:16.171032 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:16.170932 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a4c119-410d-4e61-8db1-d440e0c371c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59649767-prkjd\" (UID: \"06a4c119-410d-4e61-8db1-d440e0c371c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59649767-prkjd" Nov 03 20:08:16.171634 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:16.171613 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a4c119-410d-4e61-8db1-d440e0c371c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59649767-prkjd\" (UID: \"06a4c119-410d-4e61-8db1-d440e0c371c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59649767-prkjd" Nov 03 20:08:16.173334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:16.173312 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a4c119-410d-4e61-8db1-d440e0c371c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59649767-prkjd\" (UID: \"06a4c119-410d-4e61-8db1-d440e0c371c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59649767-prkjd" Nov 03 20:08:16.183087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:16.183063 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncsgb\" (UniqueName: \"kubernetes.io/projected/06a4c119-410d-4e61-8db1-d440e0c371c9-kube-api-access-ncsgb\") pod \"maas-api-revoke-sub-59649767-prkjd\" (UID: \"06a4c119-410d-4e61-8db1-d440e0c371c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59649767-prkjd" Nov 03 20:08:16.235063 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:08:16.235030 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae874737_754b_4414_a571_2333feda6d47.slice/crio-474684a6dfc86168086ee79aa9023a45f7b33ccdcd0a774595dbb6d61b40f1fb WatchSource:0}: Error finding container 474684a6dfc86168086ee79aa9023a45f7b33ccdcd0a774595dbb6d61b40f1fb: Status 404 returned error can't find the container with id 474684a6dfc86168086ee79aa9023a45f7b33ccdcd0a774595dbb6d61b40f1fb Nov 03 20:08:16.235698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:16.235523 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0796d7-hhv2k"] Nov 03 20:08:16.391527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:16.391499 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59649767-prkjd" Nov 03 20:08:16.715338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:16.715312 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59649767-prkjd"] Nov 03 20:08:16.716361 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:08:16.716337 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a4c119_410d_4e61_8db1_d440e0c371c9.slice/crio-0d1c936ff6d2eff7db2952f08292f3edc0e99192e8fc2d3cae43116e8ccc36c8 WatchSource:0}: Error finding container 0d1c936ff6d2eff7db2952f08292f3edc0e99192e8fc2d3cae43116e8ccc36c8: Status 404 returned error can't find the container with id 0d1c936ff6d2eff7db2952f08292f3edc0e99192e8fc2d3cae43116e8ccc36c8 Nov 03 20:08:17.077081 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:17.077050 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59649767-prkjd" event={"ID":"06a4c119-410d-4e61-8db1-d440e0c371c9","Type":"ContainerStarted","Data":"0d1c936ff6d2eff7db2952f08292f3edc0e99192e8fc2d3cae43116e8ccc36c8"} Nov 03 20:08:17.079136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:17.079104 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0796d7-hhv2k" event={"ID":"ae874737-754b-4414-a571-2333feda6d47","Type":"ContainerStarted","Data":"d211a674673719c5b9691712d00ea28d3258a40aa94f820bb32431725834b965"} Nov 03 20:08:17.079266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:17.079158 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0796d7-hhv2k" event={"ID":"ae874737-754b-4414-a571-2333feda6d47","Type":"ContainerStarted","Data":"474684a6dfc86168086ee79aa9023a45f7b33ccdcd0a774595dbb6d61b40f1fb"} Nov 03 20:08:17.102459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:17.102419 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0796d7-hhv2k" podStartSLOduration=2.102407404 podStartE2EDuration="2.102407404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:17.097312989 +0000 UTC m=+1891.355110274" watchObservedRunningTime="2026-11-03 20:08:17.102407404 +0000 UTC m=+1891.360204661" Nov 03 20:08:18.038531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.038498 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g"] Nov 03 20:08:18.071736 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.071711 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g"] Nov 03 20:08:18.071869 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.071804 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.123912 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.123864 5059 generic.go:356] "Generic (PLEG): container finished" podID="ae874737-754b-4414-a571-2333feda6d47" containerID="d211a674673719c5b9691712d00ea28d3258a40aa94f820bb32431725834b965" exitCode=0 Nov 03 20:08:18.124316 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.123945 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0796d7-hhv2k" event={"ID":"ae874737-754b-4414-a571-2333feda6d47","Type":"ContainerDied","Data":"d211a674673719c5b9691712d00ea28d3258a40aa94f820bb32431725834b965"} Nov 03 20:08:18.126941 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.126919 5059 generic.go:356] "Generic (PLEG): container finished" podID="06a4c119-410d-4e61-8db1-d440e0c371c9" containerID="0b3d9bc6f51a12d437b7e1e86a549559399a9de59e0cde12f1a3d52b95ecdf17" exitCode=0 Nov 03 20:08:18.127067 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.127000 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59649767-prkjd" event={"ID":"06a4c119-410d-4e61-8db1-d440e0c371c9","Type":"ContainerDied","Data":"0b3d9bc6f51a12d437b7e1e86a549559399a9de59e0cde12f1a3d52b95ecdf17"} Nov 03 20:08:18.180106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.180072 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-podinfo\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.180276 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.180202 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzmbv\" (UniqueName: \"kubernetes.io/projected/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-kube-api-access-jzmbv\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.180276 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.180249 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-data\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.180360 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.180324 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-envoy\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.180397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.180384 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-credential-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.180440 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.180429 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-workload-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.180586 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.180561 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-workload-certs\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.180731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.180710 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istiod-ca-cert\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.180861 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.180844 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-token\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.282535 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.282453 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-workload-certs\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.282535 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.282501 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istiod-ca-cert\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.282810 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.282563 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-token\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.282810 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.282617 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-podinfo\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.282810 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.282709 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzmbv\" (UniqueName: \"kubernetes.io/projected/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-kube-api-access-jzmbv\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.282810 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.282758 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-data\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.283047 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.282981 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-workload-certs\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.283105 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.283051 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-envoy\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.283148 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.283125 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-credential-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.283210 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.283157 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-data\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.283264 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.283204 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-workload-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.283436 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.283415 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-credential-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.283545 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.283423 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istiod-ca-cert\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.283545 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.283536 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-workload-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.285212 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.285193 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-envoy\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.285694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.285652 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-podinfo\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.295762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.295739 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-token\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.300635 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.300611 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzmbv\" (UniqueName: \"kubernetes.io/projected/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-kube-api-access-jzmbv\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.424989 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.424952 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:18.788836 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.788807 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g"] Nov 03 20:08:18.790764 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:08:18.790736 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea38154_8a1f_4e1f_8fa0_50481ba4fec0.slice/crio-25f8aee821c0a81ef57b7376c1dcec476f8f49ac04b243baa0808bf0af58ce9c WatchSource:0}: Error finding container 25f8aee821c0a81ef57b7376c1dcec476f8f49ac04b243baa0808bf0af58ce9c: Status 404 returned error can't find the container with id 25f8aee821c0a81ef57b7376c1dcec476f8f49ac04b243baa0808bf0af58ce9c Nov 03 20:08:18.794095 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.794060 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:08:18.794188 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.794129 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:08:18.794188 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:18.794183 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:08:19.138352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.138310 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" event={"ID":"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0","Type":"ContainerStarted","Data":"cc7a98566be4a3d14d59e43fa3c73400eb83547051245560b3c66cac856b8485"} Nov 03 20:08:19.138842 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.138383 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" event={"ID":"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0","Type":"ContainerStarted","Data":"25f8aee821c0a81ef57b7376c1dcec476f8f49ac04b243baa0808bf0af58ce9c"} Nov 03 20:08:19.177199 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.177128 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" podStartSLOduration=2.177105537 podStartE2EDuration="2.177105537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:19.162113779 +0000 UTC m=+1893.419911038" watchObservedRunningTime="2026-11-03 20:08:19.177105537 +0000 UTC m=+1893.434902795" Nov 03 20:08:19.440027 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.438186 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:19.441410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.441083 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Nov 03 20:08:19.441410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.441147 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" podUID="5ea38154-8a1f-4e1f-8fa0-50481ba4fec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Nov 03 20:08:19.503957 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.503913 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59649767-prkjd" Nov 03 20:08:19.552689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.550673 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0796d7-hhv2k" Nov 03 20:08:19.652997 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.652908 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06a4c119-410d-4e61-8db1-d440e0c371c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a4c119-410d-4e61-8db1-d440e0c371c9-maas-api-xks-ca\") pod \"06a4c119-410d-4e61-8db1-d440e0c371c9\" (UID: \"06a4c119-410d-4e61-8db1-d440e0c371c9\") " Nov 03 20:08:19.653270 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.653244 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae874737-754b-4414-a571-2333feda6d47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae874737-754b-4414-a571-2333feda6d47-maas-api-openshift-ca\") pod \"ae874737-754b-4414-a571-2333feda6d47\" (UID: \"ae874737-754b-4414-a571-2333feda6d47\") " Nov 03 20:08:19.653433 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.653414 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae874737-754b-4414-a571-2333feda6d47-kube-api-access-8kspj\" (UniqueName: \"kubernetes.io/projected/ae874737-754b-4414-a571-2333feda6d47-kube-api-access-8kspj\") pod \"ae874737-754b-4414-a571-2333feda6d47\" (UID: \"ae874737-754b-4414-a571-2333feda6d47\") " Nov 03 20:08:19.653765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.653742 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06a4c119-410d-4e61-8db1-d440e0c371c9-kube-api-access-ncsgb\" (UniqueName: \"kubernetes.io/projected/06a4c119-410d-4e61-8db1-d440e0c371c9-kube-api-access-ncsgb\") pod \"06a4c119-410d-4e61-8db1-d440e0c371c9\" (UID: \"06a4c119-410d-4e61-8db1-d440e0c371c9\") " Nov 03 20:08:19.654034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.654013 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06a4c119-410d-4e61-8db1-d440e0c371c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a4c119-410d-4e61-8db1-d440e0c371c9-maas-api-openshift-ca\") pod \"06a4c119-410d-4e61-8db1-d440e0c371c9\" (UID: \"06a4c119-410d-4e61-8db1-d440e0c371c9\") " Nov 03 20:08:19.654191 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.654163 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae874737-754b-4414-a571-2333feda6d47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae874737-754b-4414-a571-2333feda6d47-maas-api-xks-ca\") pod \"ae874737-754b-4414-a571-2333feda6d47\" (UID: \"ae874737-754b-4414-a571-2333feda6d47\") " Nov 03 20:08:19.654304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.654275 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae874737-754b-4414-a571-2333feda6d47-maas-api-openshift-ca" pod "ae874737-754b-4414-a571-2333feda6d47" (UID: "ae874737-754b-4414-a571-2333feda6d47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:19.655167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.655144 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a4c119-410d-4e61-8db1-d440e0c371c9-maas-api-openshift-ca" pod "06a4c119-410d-4e61-8db1-d440e0c371c9" (UID: "06a4c119-410d-4e61-8db1-d440e0c371c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:19.655276 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.655235 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a4c119-410d-4e61-8db1-d440e0c371c9-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:19.655276 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.655249 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae874737-754b-4414-a571-2333feda6d47-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:19.669055 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.666884 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae874737-754b-4414-a571-2333feda6d47-kube-api-access-8kspj" pod "ae874737-754b-4414-a571-2333feda6d47" (UID: "ae874737-754b-4414-a571-2333feda6d47"). InnerVolumeSpecName "kube-api-access-8kspj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:19.669055 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.667213 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae874737-754b-4414-a571-2333feda6d47-maas-api-xks-ca" pod "ae874737-754b-4414-a571-2333feda6d47" (UID: "ae874737-754b-4414-a571-2333feda6d47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:08:19.698689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.698232 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06a4c119-410d-4e61-8db1-d440e0c371c9-maas-api-xks-ca" pod "06a4c119-410d-4e61-8db1-d440e0c371c9" (UID: "06a4c119-410d-4e61-8db1-d440e0c371c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:08:19.700175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.700143 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a4c119-410d-4e61-8db1-d440e0c371c9-kube-api-access-ncsgb" pod "06a4c119-410d-4e61-8db1-d440e0c371c9" (UID: "06a4c119-410d-4e61-8db1-d440e0c371c9"). InnerVolumeSpecName "kube-api-access-ncsgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:19.756720 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.756692 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a4c119-410d-4e61-8db1-d440e0c371c9-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:19.756720 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.756719 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kspj\" (UniqueName: \"kubernetes.io/projected/ae874737-754b-4414-a571-2333feda6d47-kube-api-access-8kspj\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:19.756900 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.756729 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncsgb\" (UniqueName: \"kubernetes.io/projected/06a4c119-410d-4e61-8db1-d440e0c371c9-kube-api-access-ncsgb\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:19.756900 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:19.756738 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae874737-754b-4414-a571-2333feda6d47-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:20.151994 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:20.151963 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0796d7-hhv2k" Nov 03 20:08:20.153379 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:20.152032 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0796d7-hhv2k" event={"ID":"ae874737-754b-4414-a571-2333feda6d47","Type":"ContainerDied","Data":"474684a6dfc86168086ee79aa9023a45f7b33ccdcd0a774595dbb6d61b40f1fb"} Nov 03 20:08:20.153566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:20.153552 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="474684a6dfc86168086ee79aa9023a45f7b33ccdcd0a774595dbb6d61b40f1fb" Nov 03 20:08:20.157589 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:20.157563 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59649767-prkjd" Nov 03 20:08:20.162083 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:20.158725 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59649767-prkjd" event={"ID":"06a4c119-410d-4e61-8db1-d440e0c371c9","Type":"ContainerDied","Data":"0d1c936ff6d2eff7db2952f08292f3edc0e99192e8fc2d3cae43116e8ccc36c8"} Nov 03 20:08:20.162316 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:20.162294 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d1c936ff6d2eff7db2952f08292f3edc0e99192e8fc2d3cae43116e8ccc36c8" Nov 03 20:08:20.425822 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:20.425781 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Nov 03 20:08:20.426024 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:20.425859 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" podUID="5ea38154-8a1f-4e1f-8fa0-50481ba4fec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Nov 03 20:08:20.708183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:20.708149 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": read tcp 10.132.0.2:52978->10.132.0.220:15021: read: connection reset by peer" start-of-body= Nov 03 20:08:20.708361 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:20.708227 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" podUID="04b5af1a-f4e4-45a3-a7ee-59037e562b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": read tcp 10.132.0.2:52978->10.132.0.220:15021: read: connection reset by peer" Nov 03 20:08:21.204165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:21.204132 5059 generic.go:356] "Generic (PLEG): container finished" podID="04b5af1a-f4e4-45a3-a7ee-59037e562b8c" containerID="d2ed43f59e564d0a049376f043011d192ceaea1235be3c18c1ae208bbb3f3aa9" exitCode=0 Nov 03 20:08:21.204616 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:21.204204 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" event={"ID":"04b5af1a-f4e4-45a3-a7ee-59037e562b8c","Type":"ContainerDied","Data":"d2ed43f59e564d0a049376f043011d192ceaea1235be3c18c1ae208bbb3f3aa9"} Nov 03 20:08:21.204961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:21.204939 5059 scope.go:122] "RemoveContainer" containerID="d2ed43f59e564d0a049376f043011d192ceaea1235be3c18c1ae208bbb3f3aa9" Nov 03 20:08:21.206737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:21.206702 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:08:21.206859 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:21.206803 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:08:21.206929 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:21.206871 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:08:21.425640 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:21.425608 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Nov 03 20:08:21.425811 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:21.425697 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" podUID="5ea38154-8a1f-4e1f-8fa0-50481ba4fec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Nov 03 20:08:22.221492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:22.219756 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" event={"ID":"04b5af1a-f4e4-45a3-a7ee-59037e562b8c","Type":"ContainerStarted","Data":"14facf3af6388868103be0c86ac9a7621270e968cb1494f92df19d7fbcf7b809"} Nov 03 20:08:22.425500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:22.425457 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Nov 03 20:08:22.425726 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:22.425518 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" podUID="5ea38154-8a1f-4e1f-8fa0-50481ba4fec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Nov 03 20:08:23.208244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:23.208184 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:08:23.208458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:23.208406 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 03 20:08:23.208536 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:23.208472 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" podUID="04b5af1a-f4e4-45a3-a7ee-59037e562b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 03 20:08:23.425859 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:23.425826 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Nov 03 20:08:23.426284 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:23.425881 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" podUID="5ea38154-8a1f-4e1f-8fa0-50481ba4fec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Nov 03 20:08:24.209815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:24.208137 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 03 20:08:24.209815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:24.208195 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" podUID="04b5af1a-f4e4-45a3-a7ee-59037e562b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 03 20:08:24.427121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:24.427077 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Nov 03 20:08:24.427747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:24.427713 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" podUID="5ea38154-8a1f-4e1f-8fa0-50481ba4fec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Nov 03 20:08:25.210747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:25.210090 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 03 20:08:25.210747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:25.210156 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" podUID="04b5af1a-f4e4-45a3-a7ee-59037e562b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 03 20:08:25.444209 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:25.444174 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:25.444863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:25.444694 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:25.445927 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:25.445897 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:08:26.208359 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:26.208323 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 03 20:08:26.208546 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:26.208383 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" podUID="04b5af1a-f4e4-45a3-a7ee-59037e562b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 03 20:08:27.208505 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:27.208469 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 03 20:08:27.208901 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:27.208543 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" podUID="04b5af1a-f4e4-45a3-a7ee-59037e562b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 03 20:08:27.946446 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:27.946413 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv"] Nov 03 20:08:28.223323 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:28.223236 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:08:28.225223 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:28.225197 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:08:28.226837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:28.226817 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:08:28.846042 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:28.846004 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss"] Nov 03 20:08:31.007750 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.007711 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42"] Nov 03 20:08:31.030684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.009396 5059 cpu_manager.go:414] "containerMap: removing container" podUID="06a4c119-410d-4e61-8db1-d440e0c371c9" containerName="revoke-keys" Nov 03 20:08:31.030918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.030895 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a4c119-410d-4e61-8db1-d440e0c371c9" containerName="revoke-keys" Nov 03 20:08:31.031032 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.031021 5059 cpu_manager.go:414] "containerMap: removing container" podUID="ae874737-754b-4414-a571-2333feda6d47" containerName="revoke-keys" Nov 03 20:08:31.031117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.031105 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae874737-754b-4414-a571-2333feda6d47" containerName="revoke-keys" Nov 03 20:08:31.031475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.031454 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a4c119-410d-4e61-8db1-d440e0c371c9" containerName="revoke-keys" Nov 03 20:08:31.031626 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.031612 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae874737-754b-4414-a571-2333feda6d47" containerName="revoke-keys" Nov 03 20:08:31.313087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.313007 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42"] Nov 03 20:08:31.313311 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.313046 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.323651 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.323312 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a0d4ec-openshift-default-dockercfg-gsslf\"" Nov 03 20:08:31.347061 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.347029 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-envoy\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.347213 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.347100 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istiod-ca-cert\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.347213 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.347138 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-token\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.347309 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.347235 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-workload-certs\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.347309 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.347272 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-workload-socket\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.347394 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.347325 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-podinfo\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.347394 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.347382 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-data\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.347545 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.347411 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjdzj\" (UniqueName: \"kubernetes.io/projected/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-kube-api-access-bjdzj\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.347545 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.347484 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-credential-socket\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.448434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.448402 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-envoy\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.448636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.448462 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istiod-ca-cert\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.448636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.448494 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-token\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.448636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.448541 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-workload-certs\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.448636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.448587 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-workload-socket\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.448636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.448615 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-podinfo\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.448978 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.448730 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-data\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.448978 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.448786 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjdzj\" (UniqueName: \"kubernetes.io/projected/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-kube-api-access-bjdzj\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.448978 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.448865 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-credential-socket\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.449151 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.449040 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-workload-certs\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.449151 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.449135 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-data\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.449316 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.449288 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-workload-socket\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.449472 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.449337 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istiod-ca-cert\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.449472 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.449342 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-credential-socket\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.451551 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.451507 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-envoy\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.451879 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.451860 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-podinfo\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.462074 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.462052 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-token\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.462362 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.462341 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjdzj\" (UniqueName: \"kubernetes.io/projected/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-kube-api-access-bjdzj\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.640629 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.640592 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:31.938296 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.938266 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42"] Nov 03 20:08:31.938556 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:08:31.938527 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda594e899_889e_4f7f_a8ba_b7e7a389ba8a.slice/crio-72e807611fccf6e4b9d3a050087a30f7d4a435174a7369909a9e64cfc19552bf WatchSource:0}: Error finding container 72e807611fccf6e4b9d3a050087a30f7d4a435174a7369909a9e64cfc19552bf: Status 404 returned error can't find the container with id 72e807611fccf6e4b9d3a050087a30f7d4a435174a7369909a9e64cfc19552bf Nov 03 20:08:31.941915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.941875 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:08:31.942006 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.941964 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:08:31.942049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:31.942011 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:08:32.346828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:32.346784 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" event={"ID":"a594e899-889e-4f7f-a8ba-b7e7a389ba8a","Type":"ContainerStarted","Data":"6eff9a042806518844e626a38bbf88994cc517a23f4a2adcbbff67af78d07b64"} Nov 03 20:08:32.347241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:32.346850 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" event={"ID":"a594e899-889e-4f7f-a8ba-b7e7a389ba8a","Type":"ContainerStarted","Data":"72e807611fccf6e4b9d3a050087a30f7d4a435174a7369909a9e64cfc19552bf"} Nov 03 20:08:32.372919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:32.372870 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" podStartSLOduration=2.372856621 podStartE2EDuration="2.372856621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:32.370372108 +0000 UTC m=+1906.628169366" watchObservedRunningTime="2026-11-03 20:08:32.372856621 +0000 UTC m=+1906.630653877" Nov 03 20:08:32.641049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:32.640958 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:34.104691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:34.104494 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:34.255414 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:34.249743 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:34.550085 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:34.549996 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:34.571416 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:34.571385 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:08:36.347147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:36.347108 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh"] Nov 03 20:08:36.347591 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:36.347358 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" podUID="7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5" containerName="istio-proxy" containerID="cri-o://b2abe7e147b8cbdfcc2c9422f064572e7cb631adeb4ac0fe474279b47dc73326" gracePeriod=30 Nov 03 20:08:36.352197 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:36.352162 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc"] Nov 03 20:08:36.352438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:36.352410 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" podUID="4e91375b-123b-4c6b-b589-d1a6716be6df" containerName="istio-proxy" containerID="cri-o://883d52d005f1c301ec66a2cc37ef13c20cf63f3a44d00d881c47f3045f54d03c" gracePeriod=30 Nov 03 20:08:36.356931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:36.356903 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg"] Nov 03 20:08:36.357111 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:36.357091 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" podUID="8f5fe53c-7085-4890-99ad-0c4e9381a2a3" containerName="istio-proxy" containerID="cri-o://d1293ad477191c8e2709b91c327804bb9bf71ec3889c9546e956d8ceb782cc26" gracePeriod=30 Nov 03 20:08:36.985519 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:36.985485 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2388a678-8z68z"] Nov 03 20:08:36.989957 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:36.989938 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2388a678-8z68z" Nov 03 20:08:36.995572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:36.995545 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:08:36.995741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:36.995652 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:08:36.999299 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:36.999275 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2388a678-8z68z"] Nov 03 20:08:37.098323 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:37.098285 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fbae90-2272-43f8-a58d-392ccefd6028-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2388a678-8z68z\" (UID: \"93fbae90-2272-43f8-a58d-392ccefd6028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2388a678-8z68z" Nov 03 20:08:37.098585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:37.098561 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6482x\" (UniqueName: \"kubernetes.io/projected/93fbae90-2272-43f8-a58d-392ccefd6028-kube-api-access-6482x\") pod \"maas-api-revoke-sub-2388a678-8z68z\" (UID: \"93fbae90-2272-43f8-a58d-392ccefd6028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2388a678-8z68z" Nov 03 20:08:37.098652 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:37.098621 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fbae90-2272-43f8-a58d-392ccefd6028-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2388a678-8z68z\" (UID: \"93fbae90-2272-43f8-a58d-392ccefd6028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2388a678-8z68z" Nov 03 20:08:37.200433 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:37.200401 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6482x\" (UniqueName: \"kubernetes.io/projected/93fbae90-2272-43f8-a58d-392ccefd6028-kube-api-access-6482x\") pod \"maas-api-revoke-sub-2388a678-8z68z\" (UID: \"93fbae90-2272-43f8-a58d-392ccefd6028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2388a678-8z68z" Nov 03 20:08:37.200587 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:37.200447 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fbae90-2272-43f8-a58d-392ccefd6028-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2388a678-8z68z\" (UID: \"93fbae90-2272-43f8-a58d-392ccefd6028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2388a678-8z68z" Nov 03 20:08:37.200645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:37.200603 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fbae90-2272-43f8-a58d-392ccefd6028-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2388a678-8z68z\" (UID: \"93fbae90-2272-43f8-a58d-392ccefd6028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2388a678-8z68z" Nov 03 20:08:37.201286 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:37.201260 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fbae90-2272-43f8-a58d-392ccefd6028-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2388a678-8z68z\" (UID: \"93fbae90-2272-43f8-a58d-392ccefd6028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2388a678-8z68z" Nov 03 20:08:37.203208 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:37.203182 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fbae90-2272-43f8-a58d-392ccefd6028-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2388a678-8z68z\" (UID: \"93fbae90-2272-43f8-a58d-392ccefd6028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2388a678-8z68z" Nov 03 20:08:37.212646 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:37.212625 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6482x\" (UniqueName: \"kubernetes.io/projected/93fbae90-2272-43f8-a58d-392ccefd6028-kube-api-access-6482x\") pod \"maas-api-revoke-sub-2388a678-8z68z\" (UID: \"93fbae90-2272-43f8-a58d-392ccefd6028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2388a678-8z68z" Nov 03 20:08:37.310423 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:37.310370 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2388a678-8z68z" Nov 03 20:08:37.593126 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:08:37.593097 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93fbae90_2272_43f8_a58d_392ccefd6028.slice/crio-064cabd5a0183e3787822f113a086aa60edc48bfdb97d49a1f13e5ede889543e WatchSource:0}: Error finding container 064cabd5a0183e3787822f113a086aa60edc48bfdb97d49a1f13e5ede889543e: Status 404 returned error can't find the container with id 064cabd5a0183e3787822f113a086aa60edc48bfdb97d49a1f13e5ede889543e Nov 03 20:08:37.593497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:37.593472 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2388a678-8z68z"] Nov 03 20:08:38.587350 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:38.587318 5059 generic.go:356] "Generic (PLEG): container finished" podID="93fbae90-2272-43f8-a58d-392ccefd6028" containerID="323db27c05933987cd87b4fff65b5db0b6716ecde2e1db56e18277ce0a06dc13" exitCode=0 Nov 03 20:08:38.587515 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:38.587401 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2388a678-8z68z" event={"ID":"93fbae90-2272-43f8-a58d-392ccefd6028","Type":"ContainerDied","Data":"323db27c05933987cd87b4fff65b5db0b6716ecde2e1db56e18277ce0a06dc13"} Nov 03 20:08:38.587515 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:38.587456 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2388a678-8z68z" event={"ID":"93fbae90-2272-43f8-a58d-392ccefd6028","Type":"ContainerStarted","Data":"064cabd5a0183e3787822f113a086aa60edc48bfdb97d49a1f13e5ede889543e"} Nov 03 20:08:38.686730 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:38.686630 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b708137-54g7m"] Nov 03 20:08:38.692506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:38.692483 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b708137-54g7m" Nov 03 20:08:38.701779 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:38.701753 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b708137-54g7m"] Nov 03 20:08:38.737309 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:38.737239 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc45bfa-3edf-4f57-9db4-1b8c402e5017-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b708137-54g7m\" (UID: \"6bc45bfa-3edf-4f57-9db4-1b8c402e5017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b708137-54g7m" Nov 03 20:08:38.737476 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:38.737456 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87995\" (UniqueName: \"kubernetes.io/projected/6bc45bfa-3edf-4f57-9db4-1b8c402e5017-kube-api-access-87995\") pod \"maas-api-revoke-sub-9b708137-54g7m\" (UID: \"6bc45bfa-3edf-4f57-9db4-1b8c402e5017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b708137-54g7m" Nov 03 20:08:38.737533 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:38.737501 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc45bfa-3edf-4f57-9db4-1b8c402e5017-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b708137-54g7m\" (UID: \"6bc45bfa-3edf-4f57-9db4-1b8c402e5017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b708137-54g7m" Nov 03 20:08:38.839358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:38.839325 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc45bfa-3edf-4f57-9db4-1b8c402e5017-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b708137-54g7m\" (UID: \"6bc45bfa-3edf-4f57-9db4-1b8c402e5017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b708137-54g7m" Nov 03 20:08:38.839535 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:38.839516 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87995\" (UniqueName: \"kubernetes.io/projected/6bc45bfa-3edf-4f57-9db4-1b8c402e5017-kube-api-access-87995\") pod \"maas-api-revoke-sub-9b708137-54g7m\" (UID: \"6bc45bfa-3edf-4f57-9db4-1b8c402e5017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b708137-54g7m" Nov 03 20:08:38.839606 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:38.839565 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc45bfa-3edf-4f57-9db4-1b8c402e5017-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b708137-54g7m\" (UID: \"6bc45bfa-3edf-4f57-9db4-1b8c402e5017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b708137-54g7m" Nov 03 20:08:38.840078 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:38.840053 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc45bfa-3edf-4f57-9db4-1b8c402e5017-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b708137-54g7m\" (UID: \"6bc45bfa-3edf-4f57-9db4-1b8c402e5017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b708137-54g7m" Nov 03 20:08:38.841852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:38.841823 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc45bfa-3edf-4f57-9db4-1b8c402e5017-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b708137-54g7m\" (UID: \"6bc45bfa-3edf-4f57-9db4-1b8c402e5017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b708137-54g7m" Nov 03 20:08:38.852389 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:38.852362 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87995\" (UniqueName: \"kubernetes.io/projected/6bc45bfa-3edf-4f57-9db4-1b8c402e5017-kube-api-access-87995\") pod \"maas-api-revoke-sub-9b708137-54g7m\" (UID: \"6bc45bfa-3edf-4f57-9db4-1b8c402e5017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b708137-54g7m" Nov 03 20:08:39.013184 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:39.013097 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b708137-54g7m" Nov 03 20:08:39.301571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:39.301530 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b708137-54g7m"] Nov 03 20:08:39.301720 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:08:39.301681 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc45bfa_3edf_4f57_9db4_1b8c402e5017.slice/crio-aef297e053267fa0d26999f44e7640b63b237694fd0733c46f00439eae020243 WatchSource:0}: Error finding container aef297e053267fa0d26999f44e7640b63b237694fd0733c46f00439eae020243: Status 404 returned error can't find the container with id aef297e053267fa0d26999f44e7640b63b237694fd0733c46f00439eae020243 Nov 03 20:08:39.596589 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:39.596557 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b708137-54g7m" event={"ID":"6bc45bfa-3edf-4f57-9db4-1b8c402e5017","Type":"ContainerStarted","Data":"aef297e053267fa0d26999f44e7640b63b237694fd0733c46f00439eae020243"} Nov 03 20:08:39.784393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:39.784371 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2388a678-8z68z" Nov 03 20:08:39.862726 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:39.860730 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93fbae90-2272-43f8-a58d-392ccefd6028-kube-api-access-6482x\" (UniqueName: \"kubernetes.io/projected/93fbae90-2272-43f8-a58d-392ccefd6028-kube-api-access-6482x\") pod \"93fbae90-2272-43f8-a58d-392ccefd6028\" (UID: \"93fbae90-2272-43f8-a58d-392ccefd6028\") " Nov 03 20:08:39.862726 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:39.860800 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93fbae90-2272-43f8-a58d-392ccefd6028-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fbae90-2272-43f8-a58d-392ccefd6028-maas-api-xks-ca\") pod \"93fbae90-2272-43f8-a58d-392ccefd6028\" (UID: \"93fbae90-2272-43f8-a58d-392ccefd6028\") " Nov 03 20:08:39.862726 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:39.860905 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93fbae90-2272-43f8-a58d-392ccefd6028-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fbae90-2272-43f8-a58d-392ccefd6028-maas-api-openshift-ca\") pod \"93fbae90-2272-43f8-a58d-392ccefd6028\" (UID: \"93fbae90-2272-43f8-a58d-392ccefd6028\") " Nov 03 20:08:39.862726 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:39.862111 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93fbae90-2272-43f8-a58d-392ccefd6028-maas-api-openshift-ca" pod "93fbae90-2272-43f8-a58d-392ccefd6028" (UID: "93fbae90-2272-43f8-a58d-392ccefd6028"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:39.876783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:39.874812 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93fbae90-2272-43f8-a58d-392ccefd6028-maas-api-xks-ca" pod "93fbae90-2272-43f8-a58d-392ccefd6028" (UID: "93fbae90-2272-43f8-a58d-392ccefd6028"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:08:39.878154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:39.878117 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93fbae90-2272-43f8-a58d-392ccefd6028-kube-api-access-6482x" pod "93fbae90-2272-43f8-a58d-392ccefd6028" (UID: "93fbae90-2272-43f8-a58d-392ccefd6028"). InnerVolumeSpecName "kube-api-access-6482x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:39.990684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:39.966567 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6482x\" (UniqueName: \"kubernetes.io/projected/93fbae90-2272-43f8-a58d-392ccefd6028-kube-api-access-6482x\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:39.990684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:39.966611 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fbae90-2272-43f8-a58d-392ccefd6028-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:39.990684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:39.966629 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fbae90-2272-43f8-a58d-392ccefd6028-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:40.607733 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:40.607696 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2388a678-8z68z" event={"ID":"93fbae90-2272-43f8-a58d-392ccefd6028","Type":"ContainerDied","Data":"064cabd5a0183e3787822f113a086aa60edc48bfdb97d49a1f13e5ede889543e"} Nov 03 20:08:40.607733 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:40.607717 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2388a678-8z68z" Nov 03 20:08:40.607985 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:40.607756 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="064cabd5a0183e3787822f113a086aa60edc48bfdb97d49a1f13e5ede889543e" Nov 03 20:08:40.609574 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:40.609544 5059 generic.go:356] "Generic (PLEG): container finished" podID="6bc45bfa-3edf-4f57-9db4-1b8c402e5017" containerID="f30d5e1b8f1a57b499fa1e0f545efc84621156d348838104dafc06d3649467f7" exitCode=0 Nov 03 20:08:40.609735 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:40.609628 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b708137-54g7m" event={"ID":"6bc45bfa-3edf-4f57-9db4-1b8c402e5017","Type":"ContainerDied","Data":"f30d5e1b8f1a57b499fa1e0f545efc84621156d348838104dafc06d3649467f7"} Nov 03 20:08:40.617547 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:08:40.617523 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:08:41.627299 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.627230 5059 generic.go:356] "Generic (PLEG): container finished" podID="4e91375b-123b-4c6b-b589-d1a6716be6df" containerID="883d52d005f1c301ec66a2cc37ef13c20cf63f3a44d00d881c47f3045f54d03c" exitCode=0 Nov 03 20:08:41.627717 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.627548 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" event={"ID":"4e91375b-123b-4c6b-b589-d1a6716be6df","Type":"ContainerDied","Data":"883d52d005f1c301ec66a2cc37ef13c20cf63f3a44d00d881c47f3045f54d03c"} Nov 03 20:08:41.629556 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.629531 5059 generic.go:356] "Generic (PLEG): container finished" podID="7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5" containerID="b2abe7e147b8cbdfcc2c9422f064572e7cb631adeb4ac0fe474279b47dc73326" exitCode=0 Nov 03 20:08:41.629820 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.629731 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" event={"ID":"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5","Type":"ContainerDied","Data":"b2abe7e147b8cbdfcc2c9422f064572e7cb631adeb4ac0fe474279b47dc73326"} Nov 03 20:08:41.633147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.633125 5059 generic.go:356] "Generic (PLEG): container finished" podID="8f5fe53c-7085-4890-99ad-0c4e9381a2a3" containerID="d1293ad477191c8e2709b91c327804bb9bf71ec3889c9546e956d8ceb782cc26" exitCode=0 Nov 03 20:08:41.633322 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.633302 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" event={"ID":"8f5fe53c-7085-4890-99ad-0c4e9381a2a3","Type":"ContainerDied","Data":"d1293ad477191c8e2709b91c327804bb9bf71ec3889c9546e956d8ceb782cc26"} Nov 03 20:08:41.682612 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.682267 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 20:08:41.752236 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.751886 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:08:41.758885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.758856 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 20:08:41.818115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.818071 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-podinfo\") pod \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " Nov 03 20:08:41.818290 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.818271 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istiod-ca-cert\") pod \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " Nov 03 20:08:41.818376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.818317 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-envoy\") pod \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " Nov 03 20:08:41.818496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.818398 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-kube-api-access-cmg2l\" (UniqueName: \"kubernetes.io/projected/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-kube-api-access-cmg2l\") pod \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " Nov 03 20:08:41.818569 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.818521 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-workload-certs\") pod \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " Nov 03 20:08:41.818627 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.818567 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-credential-socket\") pod \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " Nov 03 20:08:41.818712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.818634 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-data\") pod \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " Nov 03 20:08:41.818712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.818691 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-workload-socket\") pod \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " Nov 03 20:08:41.818836 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.818746 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-token\" (UniqueName: \"kubernetes.io/projected/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-token\") pod \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\" (UID: \"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5\") " Nov 03 20:08:41.820303 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.820271 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-credential-socket" pod "7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5" (UID: "7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:41.820460 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.820426 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-data" pod "7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5" (UID: "7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:41.820561 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.820514 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-workload-certs" pod "7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5" (UID: "7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:41.820645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.820561 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-workload-socket" pod "7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5" (UID: "7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:41.820767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.820643 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istiod-ca-cert" pod "7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5" (UID: "7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:41.832171 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.832132 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-envoy" pod "7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5" (UID: "7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:41.832299 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.832246 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-token" pod "7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5" (UID: "7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:41.832377 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.832340 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-kube-api-access-cmg2l" pod "7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5" (UID: "7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5"). InnerVolumeSpecName "kube-api-access-cmg2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:41.834638 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.834613 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-podinfo" pod "7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5" (UID: "7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:08:41.889965 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.889942 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b708137-54g7m" Nov 03 20:08:41.921072 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.921038 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istiod-ca-cert\") pod \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " Nov 03 20:08:41.921226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.921087 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-credential-socket\") pod \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " Nov 03 20:08:41.921226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.921172 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-token\" (UniqueName: \"kubernetes.io/projected/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-token\") pod \"4e91375b-123b-4c6b-b589-d1a6716be6df\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " Nov 03 20:08:41.921226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.921202 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e91375b-123b-4c6b-b589-d1a6716be6df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e91375b-123b-4c6b-b589-d1a6716be6df-istiod-ca-cert\") pod \"4e91375b-123b-4c6b-b589-d1a6716be6df\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " Nov 03 20:08:41.921440 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.921264 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-credential-socket\") pod \"4e91375b-123b-4c6b-b589-d1a6716be6df\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " Nov 03 20:08:41.921440 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.921325 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-podinfo\") pod \"4e91375b-123b-4c6b-b589-d1a6716be6df\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " Nov 03 20:08:41.921440 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.921355 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-credential-socket" pod "8f5fe53c-7085-4890-99ad-0c4e9381a2a3" (UID: "8f5fe53c-7085-4890-99ad-0c4e9381a2a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:41.921440 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.921365 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-data\") pod \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " Nov 03 20:08:41.921440 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.921422 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istiod-ca-cert" pod "8f5fe53c-7085-4890-99ad-0c4e9381a2a3" (UID: "8f5fe53c-7085-4890-99ad-0c4e9381a2a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:41.921440 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.921427 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-workload-socket\") pod \"4e91375b-123b-4c6b-b589-d1a6716be6df\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " Nov 03 20:08:41.921826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.921497 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-envoy\") pod \"4e91375b-123b-4c6b-b589-d1a6716be6df\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " Nov 03 20:08:41.921826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.921594 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-data" pod "8f5fe53c-7085-4890-99ad-0c4e9381a2a3" (UID: "8f5fe53c-7085-4890-99ad-0c4e9381a2a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:41.921826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.921641 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-token\" (UniqueName: \"kubernetes.io/projected/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-token\") pod \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " Nov 03 20:08:41.921826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.921720 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e91375b-123b-4c6b-b589-d1a6716be6df-kube-api-access-69k5k\" (UniqueName: \"kubernetes.io/projected/4e91375b-123b-4c6b-b589-d1a6716be6df-kube-api-access-69k5k\") pod \"4e91375b-123b-4c6b-b589-d1a6716be6df\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " Nov 03 20:08:41.921826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.921780 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-kube-api-access-jrp8r\" (UniqueName: \"kubernetes.io/projected/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-kube-api-access-jrp8r\") pod \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " Nov 03 20:08:41.921826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.921801 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e91375b-123b-4c6b-b589-d1a6716be6df-istiod-ca-cert" pod "4e91375b-123b-4c6b-b589-d1a6716be6df" (UID: "4e91375b-123b-4c6b-b589-d1a6716be6df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:41.922102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.921860 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-workload-certs\") pod \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " Nov 03 20:08:41.922102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.921898 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-podinfo\") pod \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " Nov 03 20:08:41.922102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.921940 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-credential-socket" pod "4e91375b-123b-4c6b-b589-d1a6716be6df" (UID: "4e91375b-123b-4c6b-b589-d1a6716be6df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:41.922102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.921958 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-workload-socket" pod "4e91375b-123b-4c6b-b589-d1a6716be6df" (UID: "4e91375b-123b-4c6b-b589-d1a6716be6df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:41.922102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.921974 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-workload-certs\") pod \"4e91375b-123b-4c6b-b589-d1a6716be6df\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " Nov 03 20:08:41.922102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.922067 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-workload-socket\") pod \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " Nov 03 20:08:41.922365 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.922115 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-data\") pod \"4e91375b-123b-4c6b-b589-d1a6716be6df\" (UID: \"4e91375b-123b-4c6b-b589-d1a6716be6df\") " Nov 03 20:08:41.922365 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.922150 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-envoy\") pod \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\" (UID: \"8f5fe53c-7085-4890-99ad-0c4e9381a2a3\") " Nov 03 20:08:41.923908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.923064 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:41.923908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.923093 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:41.923908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.923108 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmg2l\" (UniqueName: \"kubernetes.io/projected/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-kube-api-access-cmg2l\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:41.923908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.923125 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:41.923908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.923138 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:41.923908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.923150 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:41.923908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.923164 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:41.923908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.923181 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:41.923908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.923196 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:41.923908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.923209 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:41.923908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.923223 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e91375b-123b-4c6b-b589-d1a6716be6df-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:41.923908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.923236 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:41.923908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.923248 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:41.923908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.923261 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:41.923908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.923273 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:41.923908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.923310 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-workload-socket" pod "8f5fe53c-7085-4890-99ad-0c4e9381a2a3" (UID: "8f5fe53c-7085-4890-99ad-0c4e9381a2a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:41.923908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.923609 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-workload-certs" pod "4e91375b-123b-4c6b-b589-d1a6716be6df" (UID: "4e91375b-123b-4c6b-b589-d1a6716be6df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:41.924861 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.924833 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-token" pod "4e91375b-123b-4c6b-b589-d1a6716be6df" (UID: "4e91375b-123b-4c6b-b589-d1a6716be6df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:41.925089 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.925057 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-data" pod "4e91375b-123b-4c6b-b589-d1a6716be6df" (UID: "4e91375b-123b-4c6b-b589-d1a6716be6df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:41.925283 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.925261 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-workload-certs" pod "8f5fe53c-7085-4890-99ad-0c4e9381a2a3" (UID: "8f5fe53c-7085-4890-99ad-0c4e9381a2a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:41.925711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.925647 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e91375b-123b-4c6b-b589-d1a6716be6df-kube-api-access-69k5k" pod "4e91375b-123b-4c6b-b589-d1a6716be6df" (UID: "4e91375b-123b-4c6b-b589-d1a6716be6df"). InnerVolumeSpecName "kube-api-access-69k5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:41.925891 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.925768 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-podinfo" pod "4e91375b-123b-4c6b-b589-d1a6716be6df" (UID: "4e91375b-123b-4c6b-b589-d1a6716be6df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:08:41.926480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.926451 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-token" pod "8f5fe53c-7085-4890-99ad-0c4e9381a2a3" (UID: "8f5fe53c-7085-4890-99ad-0c4e9381a2a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:41.926935 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.926910 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-envoy" pod "4e91375b-123b-4c6b-b589-d1a6716be6df" (UID: "4e91375b-123b-4c6b-b589-d1a6716be6df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:41.927296 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.927275 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-envoy" pod "8f5fe53c-7085-4890-99ad-0c4e9381a2a3" (UID: "8f5fe53c-7085-4890-99ad-0c4e9381a2a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:41.927412 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.927390 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-podinfo" pod "8f5fe53c-7085-4890-99ad-0c4e9381a2a3" (UID: "8f5fe53c-7085-4890-99ad-0c4e9381a2a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:08:41.928413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:41.928391 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-kube-api-access-jrp8r" pod "8f5fe53c-7085-4890-99ad-0c4e9381a2a3" (UID: "8f5fe53c-7085-4890-99ad-0c4e9381a2a3"). InnerVolumeSpecName "kube-api-access-jrp8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:42.025732 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.024689 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc45bfa-3edf-4f57-9db4-1b8c402e5017-kube-api-access-87995\" (UniqueName: \"kubernetes.io/projected/6bc45bfa-3edf-4f57-9db4-1b8c402e5017-kube-api-access-87995\") pod \"6bc45bfa-3edf-4f57-9db4-1b8c402e5017\" (UID: \"6bc45bfa-3edf-4f57-9db4-1b8c402e5017\") " Nov 03 20:08:42.025732 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.024838 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bc45bfa-3edf-4f57-9db4-1b8c402e5017-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc45bfa-3edf-4f57-9db4-1b8c402e5017-maas-api-openshift-ca\") pod \"6bc45bfa-3edf-4f57-9db4-1b8c402e5017\" (UID: \"6bc45bfa-3edf-4f57-9db4-1b8c402e5017\") " Nov 03 20:08:42.025732 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.025037 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bc45bfa-3edf-4f57-9db4-1b8c402e5017-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc45bfa-3edf-4f57-9db4-1b8c402e5017-maas-api-xks-ca\") pod \"6bc45bfa-3edf-4f57-9db4-1b8c402e5017\" (UID: \"6bc45bfa-3edf-4f57-9db4-1b8c402e5017\") " Nov 03 20:08:42.027158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.026496 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:42.027158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.026527 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:42.027158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.026541 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:42.027158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.026554 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:42.027158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.026569 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69k5k\" (UniqueName: \"kubernetes.io/projected/4e91375b-123b-4c6b-b589-d1a6716be6df-kube-api-access-69k5k\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:42.027158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.026583 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrp8r\" (UniqueName: \"kubernetes.io/projected/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-kube-api-access-jrp8r\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:42.027158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.026598 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:42.027158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.026612 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:42.027158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.026693 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:42.027158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.026732 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:42.027158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.026746 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e91375b-123b-4c6b-b589-d1a6716be6df-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:42.027158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.026764 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f5fe53c-7085-4890-99ad-0c4e9381a2a3-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:42.030297 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.030266 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc45bfa-3edf-4f57-9db4-1b8c402e5017-maas-api-openshift-ca" pod "6bc45bfa-3edf-4f57-9db4-1b8c402e5017" (UID: "6bc45bfa-3edf-4f57-9db4-1b8c402e5017"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:42.031214 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.031135 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bc45bfa-3edf-4f57-9db4-1b8c402e5017-maas-api-xks-ca" pod "6bc45bfa-3edf-4f57-9db4-1b8c402e5017" (UID: "6bc45bfa-3edf-4f57-9db4-1b8c402e5017"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:08:42.033820 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.033787 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc45bfa-3edf-4f57-9db4-1b8c402e5017-kube-api-access-87995" pod "6bc45bfa-3edf-4f57-9db4-1b8c402e5017" (UID: "6bc45bfa-3edf-4f57-9db4-1b8c402e5017"). InnerVolumeSpecName "kube-api-access-87995". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:42.128031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.128001 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc45bfa-3edf-4f57-9db4-1b8c402e5017-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:42.128031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.128028 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87995\" (UniqueName: \"kubernetes.io/projected/6bc45bfa-3edf-4f57-9db4-1b8c402e5017-kube-api-access-87995\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:42.128267 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.128039 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc45bfa-3edf-4f57-9db4-1b8c402e5017-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:42.644502 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.644420 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b708137-54g7m" Nov 03 20:08:42.645051 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.644455 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b708137-54g7m" event={"ID":"6bc45bfa-3edf-4f57-9db4-1b8c402e5017","Type":"ContainerDied","Data":"aef297e053267fa0d26999f44e7640b63b237694fd0733c46f00439eae020243"} Nov 03 20:08:42.645186 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.645078 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aef297e053267fa0d26999f44e7640b63b237694fd0733c46f00439eae020243" Nov 03 20:08:42.646478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.646405 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" event={"ID":"7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5","Type":"ContainerDied","Data":"fe913e89e3800deb12d201df21ed470aee93e4a175ee4dd15cffe54289881a09"} Nov 03 20:08:42.646653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.646481 5059 scope.go:122] "RemoveContainer" containerID="b2abe7e147b8cbdfcc2c9422f064572e7cb631adeb4ac0fe474279b47dc73326" Nov 03 20:08:42.646762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.646642 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh" Nov 03 20:08:42.651566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.651021 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" Nov 03 20:08:42.651566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.651110 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" event={"ID":"8f5fe53c-7085-4890-99ad-0c4e9381a2a3","Type":"ContainerDied","Data":"3ded4288ac515f1b40cb1a2d33f09c25df46585a69d8eabd0f1d5dee0fb583d7"} Nov 03 20:08:42.653882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.653862 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" event={"ID":"4e91375b-123b-4c6b-b589-d1a6716be6df","Type":"ContainerDied","Data":"ca17ae84cdac2bb8ef9c9278d7a1ba99c554b1bf7d4c7ac5d92af1a7bc5d3ec4"} Nov 03 20:08:42.653989 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.653931 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc" Nov 03 20:08:42.668757 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.668734 5059 scope.go:122] "RemoveContainer" containerID="d1293ad477191c8e2709b91c327804bb9bf71ec3889c9546e956d8ceb782cc26" Nov 03 20:08:42.676261 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.676239 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh"] Nov 03 20:08:42.683235 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.683206 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-6xjzh"] Nov 03 20:08:42.683767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.683742 5059 scope.go:122] "RemoveContainer" containerID="883d52d005f1c301ec66a2cc37ef13c20cf63f3a44d00d881c47f3045f54d03c" Nov 03 20:08:42.703793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.703768 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc"] Nov 03 20:08:42.708038 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.708015 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-rs6nc"] Nov 03 20:08:42.719056 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.719033 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg"] Nov 03 20:08:42.723480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.723458 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg"] Nov 03 20:08:42.738717 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.738676 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:08:42.738980 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.738949 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-7x6cg" podUID="8f5fe53c-7085-4890-99ad-0c4e9381a2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:42.968635 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:42.968541 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" podUID="a66dcf87-0a9a-4bdd-981c-a529e797ba29" containerName="main" containerID="cri-o://7fe9df6e9dc941bdc5c989ff624f231d29d7d890e6e99c5dd70c051a232d47a2" gracePeriod=45 Nov 03 20:08:43.088803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.088764 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" podUID="a66dcf87-0a9a-4bdd-981c-a529e797ba29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.215:8000/ready\": dial tcp 10.132.0.215:8000: connect: connection refused" Nov 03 20:08:43.295244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.295217 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:08:43.472352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.457881 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-home\" (UniqueName: \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-home\") pod \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " Nov 03 20:08:43.472352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.457945 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-model-cache\") pod \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " Nov 03 20:08:43.472352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.458013 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-dshm\") pod \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " Nov 03 20:08:43.472352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.458149 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-tmp-dir\") pod \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " Nov 03 20:08:43.472352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.458191 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a66dcf87-0a9a-4bdd-981c-a529e797ba29-tls-certs\" (UniqueName: \"kubernetes.io/secret/a66dcf87-0a9a-4bdd-981c-a529e797ba29-tls-certs\") pod \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " Nov 03 20:08:43.472352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.458228 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a66dcf87-0a9a-4bdd-981c-a529e797ba29-kube-api-access-lx5th\" (UniqueName: \"kubernetes.io/projected/a66dcf87-0a9a-4bdd-981c-a529e797ba29-kube-api-access-lx5th\") pod \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " Nov 03 20:08:43.472352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.465513 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-home" pod "a66dcf87-0a9a-4bdd-981c-a529e797ba29" (UID: "a66dcf87-0a9a-4bdd-981c-a529e797ba29"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:43.472352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.465904 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-model-cache" pod "a66dcf87-0a9a-4bdd-981c-a529e797ba29" (UID: "a66dcf87-0a9a-4bdd-981c-a529e797ba29"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:43.472352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.466203 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-tmp-dir" pod "a66dcf87-0a9a-4bdd-981c-a529e797ba29" (UID: "a66dcf87-0a9a-4bdd-981c-a529e797ba29"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:43.538766 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.500008 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a66dcf87-0a9a-4bdd-981c-a529e797ba29-tls-certs" pod "a66dcf87-0a9a-4bdd-981c-a529e797ba29" (UID: "a66dcf87-0a9a-4bdd-981c-a529e797ba29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:08:43.565046 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.543921 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a66dcf87-0a9a-4bdd-981c-a529e797ba29-kube-api-access-lx5th" pod "a66dcf87-0a9a-4bdd-981c-a529e797ba29" (UID: "a66dcf87-0a9a-4bdd-981c-a529e797ba29"). InnerVolumeSpecName "kube-api-access-lx5th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:43.577718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.573774 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7243-385b32"] Nov 03 20:08:43.577718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.575022 5059 cpu_manager.go:414] "containerMap: removing container" podUID="4e91375b-123b-4c6b-b589-d1a6716be6df" containerName="istio-proxy" Nov 03 20:08:43.577718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.575046 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e91375b-123b-4c6b-b589-d1a6716be6df" containerName="istio-proxy" Nov 03 20:08:43.577718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.575066 5059 cpu_manager.go:414] "containerMap: removing container" podUID="8f5fe53c-7085-4890-99ad-0c4e9381a2a3" containerName="istio-proxy" Nov 03 20:08:43.577718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.575075 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f5fe53c-7085-4890-99ad-0c4e9381a2a3" containerName="istio-proxy" Nov 03 20:08:43.577718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.575123 5059 cpu_manager.go:414] "containerMap: removing container" podUID="6bc45bfa-3edf-4f57-9db4-1b8c402e5017" containerName="revoke-keys" Nov 03 20:08:43.577718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.575131 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc45bfa-3edf-4f57-9db4-1b8c402e5017" containerName="revoke-keys" Nov 03 20:08:43.577718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.575145 5059 cpu_manager.go:414] "containerMap: removing container" podUID="93fbae90-2272-43f8-a58d-392ccefd6028" containerName="revoke-keys" Nov 03 20:08:43.577718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.575153 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93fbae90-2272-43f8-a58d-392ccefd6028" containerName="revoke-keys" Nov 03 20:08:43.577718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.575174 5059 cpu_manager.go:414] "containerMap: removing container" podUID="a66dcf87-0a9a-4bdd-981c-a529e797ba29" containerName="main" Nov 03 20:08:43.577718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.575182 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a66dcf87-0a9a-4bdd-981c-a529e797ba29" containerName="main" Nov 03 20:08:43.577718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.575214 5059 cpu_manager.go:414] "containerMap: removing container" podUID="7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5" containerName="istio-proxy" Nov 03 20:08:43.577718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.575223 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5" containerName="istio-proxy" Nov 03 20:08:43.577718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.575388 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="93fbae90-2272-43f8-a58d-392ccefd6028" containerName="revoke-keys" Nov 03 20:08:43.577718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.575402 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a66dcf87-0a9a-4bdd-981c-a529e797ba29" containerName="main" Nov 03 20:08:43.577718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.575417 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5" containerName="istio-proxy" Nov 03 20:08:43.577718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.575429 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e91375b-123b-4c6b-b589-d1a6716be6df" containerName="istio-proxy" Nov 03 20:08:43.577718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.575440 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f5fe53c-7085-4890-99ad-0c4e9381a2a3" containerName="istio-proxy" Nov 03 20:08:43.577718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.575455 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc45bfa-3edf-4f57-9db4-1b8c402e5017" containerName="revoke-keys" Nov 03 20:08:43.581715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.579336 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-dshm" pod "a66dcf87-0a9a-4bdd-981c-a529e797ba29" (UID: "a66dcf87-0a9a-4bdd-981c-a529e797ba29"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:43.581785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.581739 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-dshm\") pod \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\" (UID: \"a66dcf87-0a9a-4bdd-981c-a529e797ba29\") " Nov 03 20:08:43.582384 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.582360 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:43.582497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.582389 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:43.582497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.582404 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:43.582497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.582416 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a66dcf87-0a9a-4bdd-981c-a529e797ba29-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:43.582497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.582431 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx5th\" (UniqueName: \"kubernetes.io/projected/a66dcf87-0a9a-4bdd-981c-a529e797ba29-kube-api-access-lx5th\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:43.582767 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:08:43.582498 5059 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/a66dcf87-0a9a-4bdd-981c-a529e797ba29/volumes/kubernetes.io~empty-dir/dshm Nov 03 20:08:43.582767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.582511 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-dshm" pod "a66dcf87-0a9a-4bdd-981c-a529e797ba29" (UID: "a66dcf87-0a9a-4bdd-981c-a529e797ba29"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:43.584561 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.583306 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7243-385b32"] Nov 03 20:08:43.584561 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.583398 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7243-385b32" Nov 03 20:08:43.588432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.588215 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 20:08:43.588432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.588317 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-tz7np\"" Nov 03 20:08:43.588432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.588402 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 20:08:43.675301 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.675262 5059 generic.go:356] "Generic (PLEG): container finished" podID="a66dcf87-0a9a-4bdd-981c-a529e797ba29" containerID="7fe9df6e9dc941bdc5c989ff624f231d29d7d890e6e99c5dd70c051a232d47a2" exitCode=0 Nov 03 20:08:43.675755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.675328 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" Nov 03 20:08:43.675755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.675349 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" event={"ID":"a66dcf87-0a9a-4bdd-981c-a529e797ba29","Type":"ContainerDied","Data":"7fe9df6e9dc941bdc5c989ff624f231d29d7d890e6e99c5dd70c051a232d47a2"} Nov 03 20:08:43.675755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.675394 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv" event={"ID":"a66dcf87-0a9a-4bdd-981c-a529e797ba29","Type":"ContainerDied","Data":"1e6b0e93b0c7ae5a2fc034f91df3c65c499d697210bb6a110223f4890753337b"} Nov 03 20:08:43.675755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.675417 5059 scope.go:122] "RemoveContainer" containerID="7fe9df6e9dc941bdc5c989ff624f231d29d7d890e6e99c5dd70c051a232d47a2" Nov 03 20:08:43.683391 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.683364 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khv2s\" (UniqueName: \"kubernetes.io/projected/3c6d0b26-70f3-435d-a1bb-1fd82ae8c712-kube-api-access-khv2s\") pod \"test-curl-7243-385b32\" (UID: \"3c6d0b26-70f3-435d-a1bb-1fd82ae8c712\") " pod="opendatahub/test-curl-7243-385b32" Nov 03 20:08:43.683636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.683621 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a66dcf87-0a9a-4bdd-981c-a529e797ba29-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:43.692633 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.692557 5059 scope.go:122] "RemoveContainer" containerID="7fe9df6e9dc941bdc5c989ff624f231d29d7d890e6e99c5dd70c051a232d47a2" Nov 03 20:08:43.693123 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:08:43.693100 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe9df6e9dc941bdc5c989ff624f231d29d7d890e6e99c5dd70c051a232d47a2\": container with ID starting with 7fe9df6e9dc941bdc5c989ff624f231d29d7d890e6e99c5dd70c051a232d47a2 not found: ID does not exist" containerID="7fe9df6e9dc941bdc5c989ff624f231d29d7d890e6e99c5dd70c051a232d47a2" Nov 03 20:08:43.693225 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.693129 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe9df6e9dc941bdc5c989ff624f231d29d7d890e6e99c5dd70c051a232d47a2"} err="failed to get container status \"7fe9df6e9dc941bdc5c989ff624f231d29d7d890e6e99c5dd70c051a232d47a2\": rpc error: code = NotFound desc = could not find container \"7fe9df6e9dc941bdc5c989ff624f231d29d7d890e6e99c5dd70c051a232d47a2\": container with ID starting with 7fe9df6e9dc941bdc5c989ff624f231d29d7d890e6e99c5dd70c051a232d47a2 not found: ID does not exist" Nov 03 20:08:43.704704 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.704646 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv"] Nov 03 20:08:43.708835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.708815 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-4fb1d0/auth-test-model-4fb1d0-kserve-57c6789f85-2dbmv"] Nov 03 20:08:43.784957 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.784932 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khv2s\" (UniqueName: \"kubernetes.io/projected/3c6d0b26-70f3-435d-a1bb-1fd82ae8c712-kube-api-access-khv2s\") pod \"test-curl-7243-385b32\" (UID: \"3c6d0b26-70f3-435d-a1bb-1fd82ae8c712\") " pod="opendatahub/test-curl-7243-385b32" Nov 03 20:08:43.797343 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.797282 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khv2s\" (UniqueName: \"kubernetes.io/projected/3c6d0b26-70f3-435d-a1bb-1fd82ae8c712-kube-api-access-khv2s\") pod \"test-curl-7243-385b32\" (UID: \"3c6d0b26-70f3-435d-a1bb-1fd82ae8c712\") " pod="opendatahub/test-curl-7243-385b32" Nov 03 20:08:43.858926 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.858885 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e91375b-123b-4c6b-b589-d1a6716be6df" path="/var/lib/kubelet/pods/4e91375b-123b-4c6b-b589-d1a6716be6df/volumes" Nov 03 20:08:43.859487 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.859469 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5" path="/var/lib/kubelet/pods/7a8b7ef6-cd1f-4b59-82a4-34a1097aebb5/volumes" Nov 03 20:08:43.860016 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.859999 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f5fe53c-7085-4890-99ad-0c4e9381a2a3" path="/var/lib/kubelet/pods/8f5fe53c-7085-4890-99ad-0c4e9381a2a3/volumes" Nov 03 20:08:43.860768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.860751 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a66dcf87-0a9a-4bdd-981c-a529e797ba29" path="/var/lib/kubelet/pods/a66dcf87-0a9a-4bdd-981c-a529e797ba29/volumes" Nov 03 20:08:43.869125 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.869100 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" podUID="14015184-8b33-4358-b3f2-af083975b689" containerName="main" containerID="cri-o://f1c92f88aca7b16e844bcdf4089de485f2e60dfdddc3a048bf22342485dae7bf" gracePeriod=45 Nov 03 20:08:43.905445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:43.905423 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7243-385b32" Nov 03 20:08:44.137797 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.137773 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:08:44.188730 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:08:44.188657 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c6d0b26_70f3_435d_a1bb_1fd82ae8c712.slice/crio-f4650792a865226f259856be9badeeaf9a2393e9cc75e8907a9df628b245d453 WatchSource:0}: Error finding container f4650792a865226f259856be9badeeaf9a2393e9cc75e8907a9df628b245d453: Status 404 returned error can't find the container with id f4650792a865226f259856be9badeeaf9a2393e9cc75e8907a9df628b245d453 Nov 03 20:08:44.189885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.189860 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-tmp-dir\") pod \"14015184-8b33-4358-b3f2-af083975b689\" (UID: \"14015184-8b33-4358-b3f2-af083975b689\") " Nov 03 20:08:44.190030 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.189914 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-home\" (UniqueName: \"kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-home\") pod \"14015184-8b33-4358-b3f2-af083975b689\" (UID: \"14015184-8b33-4358-b3f2-af083975b689\") " Nov 03 20:08:44.190105 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.190027 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-model-cache\") pod \"14015184-8b33-4358-b3f2-af083975b689\" (UID: \"14015184-8b33-4358-b3f2-af083975b689\") " Nov 03 20:08:44.190242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.190221 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14015184-8b33-4358-b3f2-af083975b689-kube-api-access-x26b4\" (UniqueName: \"kubernetes.io/projected/14015184-8b33-4358-b3f2-af083975b689-kube-api-access-x26b4\") pod \"14015184-8b33-4358-b3f2-af083975b689\" (UID: \"14015184-8b33-4358-b3f2-af083975b689\") " Nov 03 20:08:44.190332 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.190245 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-home" pod "14015184-8b33-4358-b3f2-af083975b689" (UID: "14015184-8b33-4358-b3f2-af083975b689"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:44.190332 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.190302 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-tmp-dir" pod "14015184-8b33-4358-b3f2-af083975b689" (UID: "14015184-8b33-4358-b3f2-af083975b689"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:44.190332 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.190315 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-dshm\" (UniqueName: \"kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-dshm\") pod \"14015184-8b33-4358-b3f2-af083975b689\" (UID: \"14015184-8b33-4358-b3f2-af083975b689\") " Nov 03 20:08:44.190560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.190398 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14015184-8b33-4358-b3f2-af083975b689-tls-certs\" (UniqueName: \"kubernetes.io/secret/14015184-8b33-4358-b3f2-af083975b689-tls-certs\") pod \"14015184-8b33-4358-b3f2-af083975b689\" (UID: \"14015184-8b33-4358-b3f2-af083975b689\") " Nov 03 20:08:44.190560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.190390 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-model-cache" pod "14015184-8b33-4358-b3f2-af083975b689" (UID: "14015184-8b33-4358-b3f2-af083975b689"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:44.190701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.190632 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7243-385b32"] Nov 03 20:08:44.191073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.191044 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:44.191198 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.191076 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:44.191198 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.191090 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:44.192981 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.192957 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-dshm" pod "14015184-8b33-4358-b3f2-af083975b689" (UID: "14015184-8b33-4358-b3f2-af083975b689"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:44.193209 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.193183 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14015184-8b33-4358-b3f2-af083975b689-tls-certs" pod "14015184-8b33-4358-b3f2-af083975b689" (UID: "14015184-8b33-4358-b3f2-af083975b689"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:08:44.193476 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.193450 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14015184-8b33-4358-b3f2-af083975b689-kube-api-access-x26b4" pod "14015184-8b33-4358-b3f2-af083975b689" (UID: "14015184-8b33-4358-b3f2-af083975b689"). InnerVolumeSpecName "kube-api-access-x26b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:44.292439 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.292405 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x26b4\" (UniqueName: \"kubernetes.io/projected/14015184-8b33-4358-b3f2-af083975b689-kube-api-access-x26b4\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:44.292439 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.292434 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14015184-8b33-4358-b3f2-af083975b689-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:44.292439 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.292445 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14015184-8b33-4358-b3f2-af083975b689-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:44.690613 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.690571 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7243-385b32" event={"ID":"3c6d0b26-70f3-435d-a1bb-1fd82ae8c712","Type":"ContainerStarted","Data":"5dff47f0da0105899ee506164cb479534b9e2c643e8255d25008c7ecc06e148d"} Nov 03 20:08:44.691098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.690625 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7243-385b32" event={"ID":"3c6d0b26-70f3-435d-a1bb-1fd82ae8c712","Type":"ContainerStarted","Data":"f4650792a865226f259856be9badeeaf9a2393e9cc75e8907a9df628b245d453"} Nov 03 20:08:44.692342 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.692309 5059 generic.go:356] "Generic (PLEG): container finished" podID="14015184-8b33-4358-b3f2-af083975b689" containerID="f1c92f88aca7b16e844bcdf4089de485f2e60dfdddc3a048bf22342485dae7bf" exitCode=0 Nov 03 20:08:44.692499 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.692344 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" Nov 03 20:08:44.692499 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.692350 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" event={"ID":"14015184-8b33-4358-b3f2-af083975b689","Type":"ContainerDied","Data":"f1c92f88aca7b16e844bcdf4089de485f2e60dfdddc3a048bf22342485dae7bf"} Nov 03 20:08:44.692499 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.692390 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss" event={"ID":"14015184-8b33-4358-b3f2-af083975b689","Type":"ContainerDied","Data":"ca1e0afd046cd85e49a5fcc7ab5a3f1eaf5b1b792b41b794258ee05ae9a78842"} Nov 03 20:08:44.692499 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.692408 5059 scope.go:122] "RemoveContainer" containerID="f1c92f88aca7b16e844bcdf4089de485f2e60dfdddc3a048bf22342485dae7bf" Nov 03 20:08:44.705083 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.705060 5059 scope.go:122] "RemoveContainer" containerID="f1c92f88aca7b16e844bcdf4089de485f2e60dfdddc3a048bf22342485dae7bf" Nov 03 20:08:44.705351 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:08:44.705331 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1c92f88aca7b16e844bcdf4089de485f2e60dfdddc3a048bf22342485dae7bf\": container with ID starting with f1c92f88aca7b16e844bcdf4089de485f2e60dfdddc3a048bf22342485dae7bf not found: ID does not exist" containerID="f1c92f88aca7b16e844bcdf4089de485f2e60dfdddc3a048bf22342485dae7bf" Nov 03 20:08:44.705450 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.705361 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1c92f88aca7b16e844bcdf4089de485f2e60dfdddc3a048bf22342485dae7bf"} err="failed to get container status \"f1c92f88aca7b16e844bcdf4089de485f2e60dfdddc3a048bf22342485dae7bf\": rpc error: code = NotFound desc = could not find container \"f1c92f88aca7b16e844bcdf4089de485f2e60dfdddc3a048bf22342485dae7bf\": container with ID starting with f1c92f88aca7b16e844bcdf4089de485f2e60dfdddc3a048bf22342485dae7bf not found: ID does not exist" Nov 03 20:08:44.723567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.723525 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7243-385b32" podStartSLOduration=1.723512742 podStartE2EDuration="1.723512742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:44.714486587 +0000 UTC m=+1918.972283845" watchObservedRunningTime="2026-11-03 20:08:44.723512742 +0000 UTC m=+1918.981310000" Nov 03 20:08:44.740206 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.740181 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss"] Nov 03 20:08:44.744807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:44.744761 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-81a5ae/auth-test-model-81a5ae-kserve-5666559957-l6wss"] Nov 03 20:08:45.546696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:45.537744 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7243-385b32"] Nov 03 20:08:45.577774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:45.577722 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7243-385b32"] Nov 03 20:08:45.877409 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:45.877376 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14015184-8b33-4358-b3f2-af083975b689" path="/var/lib/kubelet/pods/14015184-8b33-4358-b3f2-af083975b689/volumes" Nov 03 20:08:46.510628 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.510594 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj"] Nov 03 20:08:46.511311 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.511296 5059 cpu_manager.go:414] "containerMap: removing container" podUID="3c6d0b26-70f3-435d-a1bb-1fd82ae8c712" containerName="test-curl-7243-385b32" Nov 03 20:08:46.511353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.511313 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c6d0b26-70f3-435d-a1bb-1fd82ae8c712" containerName="test-curl-7243-385b32" Nov 03 20:08:46.511353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.511346 5059 cpu_manager.go:414] "containerMap: removing container" podUID="14015184-8b33-4358-b3f2-af083975b689" containerName="main" Nov 03 20:08:46.511353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.511351 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14015184-8b33-4358-b3f2-af083975b689" containerName="main" Nov 03 20:08:46.511475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.511459 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="14015184-8b33-4358-b3f2-af083975b689" containerName="main" Nov 03 20:08:46.511475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.511472 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c6d0b26-70f3-435d-a1bb-1fd82ae8c712" containerName="test-curl-7243-385b32" Nov 03 20:08:46.516828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.516806 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj" Nov 03 20:08:46.526277 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.526252 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a0d4ec\"" Nov 03 20:08:46.526445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.526332 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a0d4ec-dockercfg-d47xx\"" Nov 03 20:08:46.532947 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.532926 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj"] Nov 03 20:08:46.537585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.537562 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8"] Nov 03 20:08:46.541946 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.541924 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8" Nov 03 20:08:46.552052 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.552011 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8"] Nov 03 20:08:46.587328 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.587287 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmx4m\" (UniqueName: \"kubernetes.io/projected/06b78cd0-94df-40f9-a1ed-87c1751675d3-kube-api-access-rmx4m\") pod \"payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj\" (UID: \"06b78cd0-94df-40f9-a1ed-87c1751675d3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj" Nov 03 20:08:46.587490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.587408 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b78cd0-94df-40f9-a1ed-87c1751675d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj\" (UID: \"06b78cd0-94df-40f9-a1ed-87c1751675d3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj" Nov 03 20:08:46.587490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.587454 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705dea2b-ad85-4b1c-a3c6-a9f705ac9039-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8\" (UID: \"705dea2b-ad85-4b1c-a3c6-a9f705ac9039\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8" Nov 03 20:08:46.587597 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.587536 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p57rb\" (UniqueName: \"kubernetes.io/projected/705dea2b-ad85-4b1c-a3c6-a9f705ac9039-kube-api-access-p57rb\") pod \"payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8\" (UID: \"705dea2b-ad85-4b1c-a3c6-a9f705ac9039\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8" Nov 03 20:08:46.701687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.698632 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705dea2b-ad85-4b1c-a3c6-a9f705ac9039-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8\" (UID: \"705dea2b-ad85-4b1c-a3c6-a9f705ac9039\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8" Nov 03 20:08:46.701687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.698769 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p57rb\" (UniqueName: \"kubernetes.io/projected/705dea2b-ad85-4b1c-a3c6-a9f705ac9039-kube-api-access-p57rb\") pod \"payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8\" (UID: \"705dea2b-ad85-4b1c-a3c6-a9f705ac9039\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8" Nov 03 20:08:46.701687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.699025 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmx4m\" (UniqueName: \"kubernetes.io/projected/06b78cd0-94df-40f9-a1ed-87c1751675d3-kube-api-access-rmx4m\") pod \"payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj\" (UID: \"06b78cd0-94df-40f9-a1ed-87c1751675d3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj" Nov 03 20:08:46.701687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.699129 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b78cd0-94df-40f9-a1ed-87c1751675d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj\" (UID: \"06b78cd0-94df-40f9-a1ed-87c1751675d3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj" Nov 03 20:08:46.701687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.700115 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b78cd0-94df-40f9-a1ed-87c1751675d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj\" (UID: \"06b78cd0-94df-40f9-a1ed-87c1751675d3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj" Nov 03 20:08:46.701687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.700864 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705dea2b-ad85-4b1c-a3c6-a9f705ac9039-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8\" (UID: \"705dea2b-ad85-4b1c-a3c6-a9f705ac9039\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8" Nov 03 20:08:46.815597 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.815516 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p57rb\" (UniqueName: \"kubernetes.io/projected/705dea2b-ad85-4b1c-a3c6-a9f705ac9039-kube-api-access-p57rb\") pod \"payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8\" (UID: \"705dea2b-ad85-4b1c-a3c6-a9f705ac9039\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8" Nov 03 20:08:46.827508 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.827473 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmx4m\" (UniqueName: \"kubernetes.io/projected/06b78cd0-94df-40f9-a1ed-87c1751675d3-kube-api-access-rmx4m\") pod \"payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj\" (UID: \"06b78cd0-94df-40f9-a1ed-87c1751675d3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj" Nov 03 20:08:46.833915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.833873 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj" Nov 03 20:08:46.860766 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.860733 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8" Nov 03 20:08:46.904627 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:46.901813 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7243-385b32" podUID="3c6d0b26-70f3-435d-a1bb-1fd82ae8c712" containerName="test-curl-7243-385b32" containerID="cri-o://5dff47f0da0105899ee506164cb479534b9e2c643e8255d25008c7ecc06e148d" gracePeriod=2 Nov 03 20:08:47.257246 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.257213 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-684e69fc-dpdgd"] Nov 03 20:08:47.263657 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.263634 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e69fc-dpdgd" Nov 03 20:08:47.269026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.267988 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:08:47.269026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.268175 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:08:47.271583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.271557 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-684e69fc-dpdgd"] Nov 03 20:08:47.308732 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.308708 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8"] Nov 03 20:08:47.309478 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:08:47.309451 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705dea2b_ad85_4b1c_a3c6_a9f705ac9039.slice/crio-47d752e074652163b0738c275ed94995338fa980323fecafc676c3c822967ee7 WatchSource:0}: Error finding container 47d752e074652163b0738c275ed94995338fa980323fecafc676c3c822967ee7: Status 404 returned error can't find the container with id 47d752e074652163b0738c275ed94995338fa980323fecafc676c3c822967ee7 Nov 03 20:08:47.314838 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:08:47.314812 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b78cd0_94df_40f9_a1ed_87c1751675d3.slice/crio-79ba33873a545827d72cc34d9ead20855333ee4a7c7053d21cb8dfd4094a631f WatchSource:0}: Error finding container 79ba33873a545827d72cc34d9ead20855333ee4a7c7053d21cb8dfd4094a631f: Status 404 returned error can't find the container with id 79ba33873a545827d72cc34d9ead20855333ee4a7c7053d21cb8dfd4094a631f Nov 03 20:08:47.315098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.315074 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj"] Nov 03 20:08:47.416798 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.416768 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa74ca6a-b680-401c-9e7e-bb3c1bba2458-maas-api-xks-ca\") pod \"maas-api-revoke-sub-684e69fc-dpdgd\" (UID: \"aa74ca6a-b680-401c-9e7e-bb3c1bba2458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e69fc-dpdgd" Nov 03 20:08:47.416910 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.416889 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa74ca6a-b680-401c-9e7e-bb3c1bba2458-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-684e69fc-dpdgd\" (UID: \"aa74ca6a-b680-401c-9e7e-bb3c1bba2458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e69fc-dpdgd" Nov 03 20:08:47.416981 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.416968 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9srkn\" (UniqueName: \"kubernetes.io/projected/aa74ca6a-b680-401c-9e7e-bb3c1bba2458-kube-api-access-9srkn\") pod \"maas-api-revoke-sub-684e69fc-dpdgd\" (UID: \"aa74ca6a-b680-401c-9e7e-bb3c1bba2458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e69fc-dpdgd" Nov 03 20:08:47.525248 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.525214 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw"] Nov 03 20:08:47.525795 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.525758 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa74ca6a-b680-401c-9e7e-bb3c1bba2458-maas-api-xks-ca\") pod \"maas-api-revoke-sub-684e69fc-dpdgd\" (UID: \"aa74ca6a-b680-401c-9e7e-bb3c1bba2458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e69fc-dpdgd" Nov 03 20:08:47.525928 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.525851 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa74ca6a-b680-401c-9e7e-bb3c1bba2458-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-684e69fc-dpdgd\" (UID: \"aa74ca6a-b680-401c-9e7e-bb3c1bba2458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e69fc-dpdgd" Nov 03 20:08:47.526015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.525931 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9srkn\" (UniqueName: \"kubernetes.io/projected/aa74ca6a-b680-401c-9e7e-bb3c1bba2458-kube-api-access-9srkn\") pod \"maas-api-revoke-sub-684e69fc-dpdgd\" (UID: \"aa74ca6a-b680-401c-9e7e-bb3c1bba2458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e69fc-dpdgd" Nov 03 20:08:47.526971 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.526924 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa74ca6a-b680-401c-9e7e-bb3c1bba2458-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-684e69fc-dpdgd\" (UID: \"aa74ca6a-b680-401c-9e7e-bb3c1bba2458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e69fc-dpdgd" Nov 03 20:08:47.528928 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.528906 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa74ca6a-b680-401c-9e7e-bb3c1bba2458-maas-api-xks-ca\") pod \"maas-api-revoke-sub-684e69fc-dpdgd\" (UID: \"aa74ca6a-b680-401c-9e7e-bb3c1bba2458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e69fc-dpdgd" Nov 03 20:08:47.532817 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.532796 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.536626 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.536598 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw"] Nov 03 20:08:47.546017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.545994 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9srkn\" (UniqueName: \"kubernetes.io/projected/aa74ca6a-b680-401c-9e7e-bb3c1bba2458-kube-api-access-9srkn\") pod \"maas-api-revoke-sub-684e69fc-dpdgd\" (UID: \"aa74ca6a-b680-401c-9e7e-bb3c1bba2458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e69fc-dpdgd" Nov 03 20:08:47.590524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.590493 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e69fc-dpdgd" Nov 03 20:08:47.616071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.616036 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-8b7351-openshift-default-dockercfg-lx6hb\"" Nov 03 20:08:47.699684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.699173 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-envoy\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.707692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.706756 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-workload-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.707692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.706854 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-data\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.707692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.706935 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-credential-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.707692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.706975 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6hvf\" (UniqueName: \"kubernetes.io/projected/154d0ad4-2191-4633-af2e-57ca9b458f42-kube-api-access-g6hvf\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.707692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.707045 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-workload-certs\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.707692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.707115 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154d0ad4-2191-4633-af2e-57ca9b458f42-istiod-ca-cert\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.707692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.707194 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-token\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.707692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.707277 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-podinfo\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.841752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.833567 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-workload-certs\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.841752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.833618 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-workload-certs\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.841752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.833729 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154d0ad4-2191-4633-af2e-57ca9b458f42-istiod-ca-cert\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.841752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.833810 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-token\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.841752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.833891 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-podinfo\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.841752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.834063 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-envoy\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.841752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.834132 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-workload-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.841752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.834208 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-data\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.841752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.834286 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-credential-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.841752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.834328 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6hvf\" (UniqueName: \"kubernetes.io/projected/154d0ad4-2191-4633-af2e-57ca9b458f42-kube-api-access-g6hvf\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.841752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.835396 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154d0ad4-2191-4633-af2e-57ca9b458f42-istiod-ca-cert\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.841752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.836467 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-workload-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.847354 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.847044 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-credential-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.876695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.847830 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-data\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.890707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.881958 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-envoy\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:47.924152 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:47.913546 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-podinfo\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:48.050994 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:48.050913 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-token\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:48.054617 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:48.052084 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6hvf\" (UniqueName: \"kubernetes.io/projected/154d0ad4-2191-4633-af2e-57ca9b458f42-kube-api-access-g6hvf\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:48.084932 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:48.084904 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj" event={"ID":"06b78cd0-94df-40f9-a1ed-87c1751675d3","Type":"ContainerStarted","Data":"e188f63192d3371f9c88f1fcfd31fbc6a5bb4e3b477faa591137913f4bacb386"} Nov 03 20:08:48.085071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:48.084954 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj" event={"ID":"06b78cd0-94df-40f9-a1ed-87c1751675d3","Type":"ContainerStarted","Data":"79ba33873a545827d72cc34d9ead20855333ee4a7c7053d21cb8dfd4094a631f"} Nov 03 20:08:48.085820 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:48.085802 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj" Nov 03 20:08:48.088433 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:48.088414 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8" event={"ID":"705dea2b-ad85-4b1c-a3c6-a9f705ac9039","Type":"ContainerStarted","Data":"c1c608afee5d8326b20645c4ef715824387423915b54182a5c217c34ffe83cf2"} Nov 03 20:08:48.088564 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:48.088553 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8" event={"ID":"705dea2b-ad85-4b1c-a3c6-a9f705ac9039","Type":"ContainerStarted","Data":"47d752e074652163b0738c275ed94995338fa980323fecafc676c3c822967ee7"} Nov 03 20:08:48.088940 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:48.088914 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8" Nov 03 20:08:48.175746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:48.166520 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:48.207298 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:48.207242 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8" podStartSLOduration=2.2072226759999998 podStartE2EDuration="2.207222676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:48.204492647 +0000 UTC m=+1922.462289904" watchObservedRunningTime="2026-11-03 20:08:48.207222676 +0000 UTC m=+1922.465019936" Nov 03 20:08:48.207762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:48.207718 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj" podStartSLOduration=2.207703409 podStartE2EDuration="2.207703409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:48.153085759 +0000 UTC m=+1922.410883017" watchObservedRunningTime="2026-11-03 20:08:48.207703409 +0000 UTC m=+1922.465500667" Nov 03 20:08:48.265514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:48.265480 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-684e69fc-dpdgd"] Nov 03 20:08:48.265806 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:08:48.265787 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa74ca6a_b680_401c_9e7e_bb3c1bba2458.slice/crio-4a76ff442674e98797d5158abf2a88bdcde9caf59a9e674813c2ea0fa2610b5e WatchSource:0}: Error finding container 4a76ff442674e98797d5158abf2a88bdcde9caf59a9e674813c2ea0fa2610b5e: Status 404 returned error can't find the container with id 4a76ff442674e98797d5158abf2a88bdcde9caf59a9e674813c2ea0fa2610b5e Nov 03 20:08:48.483635 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:08:48.483599 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154d0ad4_2191_4633_af2e_57ca9b458f42.slice/crio-93f67f4ca8fa5d2065339ba5b1f30d76dd256df96f5bc0b9b581163d2d0afdf0 WatchSource:0}: Error finding container 93f67f4ca8fa5d2065339ba5b1f30d76dd256df96f5bc0b9b581163d2d0afdf0: Status 404 returned error can't find the container with id 93f67f4ca8fa5d2065339ba5b1f30d76dd256df96f5bc0b9b581163d2d0afdf0 Nov 03 20:08:48.484271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:48.484248 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw"] Nov 03 20:08:48.487715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:48.487643 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:08:48.488095 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:48.488049 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:08:48.488195 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:48.488171 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:08:49.025916 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.025885 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31e83134-kg9wj"] Nov 03 20:08:49.030892 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.030871 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e83134-kg9wj" Nov 03 20:08:49.063709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.063618 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31e83134-kg9wj"] Nov 03 20:08:49.092709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.089074 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a948c428-547e-49e1-8d81-a4c3a0732f79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31e83134-kg9wj\" (UID: \"a948c428-547e-49e1-8d81-a4c3a0732f79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e83134-kg9wj" Nov 03 20:08:49.092709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.089170 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a948c428-547e-49e1-8d81-a4c3a0732f79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31e83134-kg9wj\" (UID: \"a948c428-547e-49e1-8d81-a4c3a0732f79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e83134-kg9wj" Nov 03 20:08:49.092709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.089296 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbxzm\" (UniqueName: \"kubernetes.io/projected/a948c428-547e-49e1-8d81-a4c3a0732f79-kube-api-access-pbxzm\") pod \"maas-api-revoke-sub-31e83134-kg9wj\" (UID: \"a948c428-547e-49e1-8d81-a4c3a0732f79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e83134-kg9wj" Nov 03 20:08:49.104057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.103512 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" event={"ID":"154d0ad4-2191-4633-af2e-57ca9b458f42","Type":"ContainerStarted","Data":"efd770fcbd173d9ab92d2bc525a9123ed2006ef5998ec9ce1906a3109519a3de"} Nov 03 20:08:49.104057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.103575 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" event={"ID":"154d0ad4-2191-4633-af2e-57ca9b458f42","Type":"ContainerStarted","Data":"93f67f4ca8fa5d2065339ba5b1f30d76dd256df96f5bc0b9b581163d2d0afdf0"} Nov 03 20:08:49.107280 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.107057 5059 generic.go:356] "Generic (PLEG): container finished" podID="3c6d0b26-70f3-435d-a1bb-1fd82ae8c712" containerID="5dff47f0da0105899ee506164cb479534b9e2c643e8255d25008c7ecc06e148d" exitCode=137 Nov 03 20:08:49.109673 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.109639 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e69fc-dpdgd" event={"ID":"aa74ca6a-b680-401c-9e7e-bb3c1bba2458","Type":"ContainerStarted","Data":"1702ed65b32c81888c4aca4681d437b77a3dbd27b1bcf66b6c100f5886f185ad"} Nov 03 20:08:49.109938 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.109917 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e69fc-dpdgd" event={"ID":"aa74ca6a-b680-401c-9e7e-bb3c1bba2458","Type":"ContainerStarted","Data":"4a76ff442674e98797d5158abf2a88bdcde9caf59a9e674813c2ea0fa2610b5e"} Nov 03 20:08:49.146195 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.146128 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" podStartSLOduration=2.146109479 podStartE2EDuration="2.146109479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:49.131047791 +0000 UTC m=+1923.388845049" watchObservedRunningTime="2026-11-03 20:08:49.146109479 +0000 UTC m=+1923.403906738" Nov 03 20:08:49.164798 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.164654 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e69fc-dpdgd" podStartSLOduration=2.164641016 podStartE2EDuration="2.164641016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:49.164149393 +0000 UTC m=+1923.421946651" watchObservedRunningTime="2026-11-03 20:08:49.164641016 +0000 UTC m=+1923.422438272" Nov 03 20:08:49.167955 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.167928 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:49.178159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.178134 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:49.191482 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.191460 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a948c428-547e-49e1-8d81-a4c3a0732f79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31e83134-kg9wj\" (UID: \"a948c428-547e-49e1-8d81-a4c3a0732f79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e83134-kg9wj" Nov 03 20:08:49.191620 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.191548 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a948c428-547e-49e1-8d81-a4c3a0732f79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31e83134-kg9wj\" (UID: \"a948c428-547e-49e1-8d81-a4c3a0732f79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e83134-kg9wj" Nov 03 20:08:49.191709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.191691 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbxzm\" (UniqueName: \"kubernetes.io/projected/a948c428-547e-49e1-8d81-a4c3a0732f79-kube-api-access-pbxzm\") pod \"maas-api-revoke-sub-31e83134-kg9wj\" (UID: \"a948c428-547e-49e1-8d81-a4c3a0732f79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e83134-kg9wj" Nov 03 20:08:49.192103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.192084 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a948c428-547e-49e1-8d81-a4c3a0732f79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31e83134-kg9wj\" (UID: \"a948c428-547e-49e1-8d81-a4c3a0732f79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e83134-kg9wj" Nov 03 20:08:49.194219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.194194 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a948c428-547e-49e1-8d81-a4c3a0732f79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31e83134-kg9wj\" (UID: \"a948c428-547e-49e1-8d81-a4c3a0732f79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e83134-kg9wj" Nov 03 20:08:49.205090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.205069 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbxzm\" (UniqueName: \"kubernetes.io/projected/a948c428-547e-49e1-8d81-a4c3a0732f79-kube-api-access-pbxzm\") pod \"maas-api-revoke-sub-31e83134-kg9wj\" (UID: \"a948c428-547e-49e1-8d81-a4c3a0732f79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e83134-kg9wj" Nov 03 20:08:49.278757 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.269839 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7243-385b32" Nov 03 20:08:49.317771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.317734 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c6d0b26-70f3-435d-a1bb-1fd82ae8c712-kube-api-access-khv2s\" (UniqueName: \"kubernetes.io/projected/3c6d0b26-70f3-435d-a1bb-1fd82ae8c712-kube-api-access-khv2s\") pod \"3c6d0b26-70f3-435d-a1bb-1fd82ae8c712\" (UID: \"3c6d0b26-70f3-435d-a1bb-1fd82ae8c712\") " Nov 03 20:08:49.321961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.321925 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6d0b26-70f3-435d-a1bb-1fd82ae8c712-kube-api-access-khv2s" pod "3c6d0b26-70f3-435d-a1bb-1fd82ae8c712" (UID: "3c6d0b26-70f3-435d-a1bb-1fd82ae8c712"). InnerVolumeSpecName "kube-api-access-khv2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:49.376119 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.376094 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e83134-kg9wj" Nov 03 20:08:49.421152 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.421089 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khv2s\" (UniqueName: \"kubernetes.io/projected/3c6d0b26-70f3-435d-a1bb-1fd82ae8c712-kube-api-access-khv2s\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:49.664038 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:08:49.664005 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda948c428_547e_49e1_8d81_a4c3a0732f79.slice/crio-53a2a4a0d58310bd0ded737d5b9970d33549bd96b912b8aed9fdd3747d666e9a WatchSource:0}: Error finding container 53a2a4a0d58310bd0ded737d5b9970d33549bd96b912b8aed9fdd3747d666e9a: Status 404 returned error can't find the container with id 53a2a4a0d58310bd0ded737d5b9970d33549bd96b912b8aed9fdd3747d666e9a Nov 03 20:08:49.664155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.664044 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31e83134-kg9wj"] Nov 03 20:08:49.859477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:49.859439 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c6d0b26-70f3-435d-a1bb-1fd82ae8c712" path="/var/lib/kubelet/pods/3c6d0b26-70f3-435d-a1bb-1fd82ae8c712/volumes" Nov 03 20:08:50.120089 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:50.120057 5059 generic.go:356] "Generic (PLEG): container finished" podID="aa74ca6a-b680-401c-9e7e-bb3c1bba2458" containerID="1702ed65b32c81888c4aca4681d437b77a3dbd27b1bcf66b6c100f5886f185ad" exitCode=0 Nov 03 20:08:50.120539 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:50.120138 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e69fc-dpdgd" event={"ID":"aa74ca6a-b680-401c-9e7e-bb3c1bba2458","Type":"ContainerDied","Data":"1702ed65b32c81888c4aca4681d437b77a3dbd27b1bcf66b6c100f5886f185ad"} Nov 03 20:08:50.121993 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:50.121958 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e83134-kg9wj" event={"ID":"a948c428-547e-49e1-8d81-a4c3a0732f79","Type":"ContainerStarted","Data":"60567573d80bb56bd0def80ec667c579e8d02a0503a3699f3818c3e78110cb73"} Nov 03 20:08:50.122076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:50.122010 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e83134-kg9wj" event={"ID":"a948c428-547e-49e1-8d81-a4c3a0732f79","Type":"ContainerStarted","Data":"53a2a4a0d58310bd0ded737d5b9970d33549bd96b912b8aed9fdd3747d666e9a"} Nov 03 20:08:50.123383 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:50.123366 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7243-385b32" Nov 03 20:08:50.123489 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:50.123431 5059 scope.go:122] "RemoveContainer" containerID="5dff47f0da0105899ee506164cb479534b9e2c643e8255d25008c7ecc06e148d" Nov 03 20:08:50.123996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:50.123971 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:50.125617 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:50.125595 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:08:50.160321 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:50.160268 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e83134-kg9wj" podStartSLOduration=1.1602516139999999 podStartE2EDuration="1.160251614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:50.156215749 +0000 UTC m=+1924.414013008" watchObservedRunningTime="2026-11-03 20:08:50.160251614 +0000 UTC m=+1924.418048872" Nov 03 20:08:51.133976 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:51.133943 5059 generic.go:356] "Generic (PLEG): container finished" podID="a948c428-547e-49e1-8d81-a4c3a0732f79" containerID="60567573d80bb56bd0def80ec667c579e8d02a0503a3699f3818c3e78110cb73" exitCode=0 Nov 03 20:08:51.134415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:51.134029 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e83134-kg9wj" event={"ID":"a948c428-547e-49e1-8d81-a4c3a0732f79","Type":"ContainerDied","Data":"60567573d80bb56bd0def80ec667c579e8d02a0503a3699f3818c3e78110cb73"} Nov 03 20:08:51.318593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:51.318566 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e69fc-dpdgd" Nov 03 20:08:51.514580 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:51.508283 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa74ca6a-b680-401c-9e7e-bb3c1bba2458-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa74ca6a-b680-401c-9e7e-bb3c1bba2458-maas-api-xks-ca\") pod \"aa74ca6a-b680-401c-9e7e-bb3c1bba2458\" (UID: \"aa74ca6a-b680-401c-9e7e-bb3c1bba2458\") " Nov 03 20:08:51.514580 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:51.508572 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa74ca6a-b680-401c-9e7e-bb3c1bba2458-kube-api-access-9srkn\" (UniqueName: \"kubernetes.io/projected/aa74ca6a-b680-401c-9e7e-bb3c1bba2458-kube-api-access-9srkn\") pod \"aa74ca6a-b680-401c-9e7e-bb3c1bba2458\" (UID: \"aa74ca6a-b680-401c-9e7e-bb3c1bba2458\") " Nov 03 20:08:51.514580 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:51.508626 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa74ca6a-b680-401c-9e7e-bb3c1bba2458-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa74ca6a-b680-401c-9e7e-bb3c1bba2458-maas-api-openshift-ca\") pod \"aa74ca6a-b680-401c-9e7e-bb3c1bba2458\" (UID: \"aa74ca6a-b680-401c-9e7e-bb3c1bba2458\") " Nov 03 20:08:51.524203 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:51.523976 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa74ca6a-b680-401c-9e7e-bb3c1bba2458-maas-api-openshift-ca" pod "aa74ca6a-b680-401c-9e7e-bb3c1bba2458" (UID: "aa74ca6a-b680-401c-9e7e-bb3c1bba2458"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:51.538689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:51.538439 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa74ca6a-b680-401c-9e7e-bb3c1bba2458-maas-api-xks-ca" pod "aa74ca6a-b680-401c-9e7e-bb3c1bba2458" (UID: "aa74ca6a-b680-401c-9e7e-bb3c1bba2458"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:08:51.545678 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:51.542307 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa74ca6a-b680-401c-9e7e-bb3c1bba2458-kube-api-access-9srkn" pod "aa74ca6a-b680-401c-9e7e-bb3c1bba2458" (UID: "aa74ca6a-b680-401c-9e7e-bb3c1bba2458"). InnerVolumeSpecName "kube-api-access-9srkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:51.611577 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:51.611548 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9srkn\" (UniqueName: \"kubernetes.io/projected/aa74ca6a-b680-401c-9e7e-bb3c1bba2458-kube-api-access-9srkn\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:51.611577 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:51.611579 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa74ca6a-b680-401c-9e7e-bb3c1bba2458-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:51.611744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:51.611589 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa74ca6a-b680-401c-9e7e-bb3c1bba2458-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:52.145957 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:52.145867 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e69fc-dpdgd" Nov 03 20:08:52.145957 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:52.145903 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e69fc-dpdgd" event={"ID":"aa74ca6a-b680-401c-9e7e-bb3c1bba2458","Type":"ContainerDied","Data":"4a76ff442674e98797d5158abf2a88bdcde9caf59a9e674813c2ea0fa2610b5e"} Nov 03 20:08:52.145957 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:52.145957 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a76ff442674e98797d5158abf2a88bdcde9caf59a9e674813c2ea0fa2610b5e" Nov 03 20:08:52.377573 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:52.377541 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e83134-kg9wj" Nov 03 20:08:52.424451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:52.423401 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a948c428-547e-49e1-8d81-a4c3a0732f79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a948c428-547e-49e1-8d81-a4c3a0732f79-maas-api-openshift-ca\") pod \"a948c428-547e-49e1-8d81-a4c3a0732f79\" (UID: \"a948c428-547e-49e1-8d81-a4c3a0732f79\") " Nov 03 20:08:52.424451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:52.423455 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a948c428-547e-49e1-8d81-a4c3a0732f79-kube-api-access-pbxzm\" (UniqueName: \"kubernetes.io/projected/a948c428-547e-49e1-8d81-a4c3a0732f79-kube-api-access-pbxzm\") pod \"a948c428-547e-49e1-8d81-a4c3a0732f79\" (UID: \"a948c428-547e-49e1-8d81-a4c3a0732f79\") " Nov 03 20:08:52.424451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:52.423613 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a948c428-547e-49e1-8d81-a4c3a0732f79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a948c428-547e-49e1-8d81-a4c3a0732f79-maas-api-xks-ca\") pod \"a948c428-547e-49e1-8d81-a4c3a0732f79\" (UID: \"a948c428-547e-49e1-8d81-a4c3a0732f79\") " Nov 03 20:08:52.424451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:52.423849 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a948c428-547e-49e1-8d81-a4c3a0732f79-maas-api-openshift-ca" pod "a948c428-547e-49e1-8d81-a4c3a0732f79" (UID: "a948c428-547e-49e1-8d81-a4c3a0732f79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:52.424451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:52.424372 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a948c428-547e-49e1-8d81-a4c3a0732f79-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:52.425725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:52.425695 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a948c428-547e-49e1-8d81-a4c3a0732f79-kube-api-access-pbxzm" pod "a948c428-547e-49e1-8d81-a4c3a0732f79" (UID: "a948c428-547e-49e1-8d81-a4c3a0732f79"). InnerVolumeSpecName "kube-api-access-pbxzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:52.426203 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:52.426167 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a948c428-547e-49e1-8d81-a4c3a0732f79-maas-api-xks-ca" pod "a948c428-547e-49e1-8d81-a4c3a0732f79" (UID: "a948c428-547e-49e1-8d81-a4c3a0732f79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:08:52.526392 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:52.526356 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbxzm\" (UniqueName: \"kubernetes.io/projected/a948c428-547e-49e1-8d81-a4c3a0732f79-kube-api-access-pbxzm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:52.526392 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:52.526388 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a948c428-547e-49e1-8d81-a4c3a0732f79-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:53.274683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:53.260887 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e83134-kg9wj" event={"ID":"a948c428-547e-49e1-8d81-a4c3a0732f79","Type":"ContainerDied","Data":"53a2a4a0d58310bd0ded737d5b9970d33549bd96b912b8aed9fdd3747d666e9a"} Nov 03 20:08:53.274683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:53.260957 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53a2a4a0d58310bd0ded737d5b9970d33549bd96b912b8aed9fdd3747d666e9a" Nov 03 20:08:53.274683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:53.261028 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e83134-kg9wj" Nov 03 20:08:54.173914 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:54.173883 5059 scope.go:122] "RemoveContainer" containerID="673b24a7ca1f1d3180a9a878a642de7a77773958213ccd7f39ee93062ba8d629" Nov 03 20:08:54.186011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:54.185987 5059 scope.go:122] "RemoveContainer" containerID="1e8b5ec575cf8860e615ae6e140ec0e37a39068e5a9285a1544505d7b1b326e6" Nov 03 20:08:54.196882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:54.196861 5059 scope.go:122] "RemoveContainer" containerID="20731930d7c6361891df9a9b39fdb761ac072612c83e6f102d5a6aea384aae6e" Nov 03 20:08:54.206062 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:54.206043 5059 scope.go:122] "RemoveContainer" containerID="6fb909a8b978acddb86a4265052ea2982fb88641f65a989e41d09c720c2c93fc" Nov 03 20:08:54.215244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:54.215222 5059 scope.go:122] "RemoveContainer" containerID="0fa8daa2bb513452f6e11d8859bcd5159b4218aad834fd86aebad08807b0b170" Nov 03 20:08:54.226408 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:54.226357 5059 scope.go:122] "RemoveContainer" containerID="d564999636dfe02ab63256d324b4ff599b8034e06ec7444d1ea0dab9ae115e83" Nov 03 20:08:54.236941 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:54.236905 5059 scope.go:122] "RemoveContainer" containerID="89ff664338bb927f838ffab3dc041ca737f5a8bdac000bec04e282765d3cd95c" Nov 03 20:08:54.246282 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:54.246263 5059 scope.go:122] "RemoveContainer" containerID="1221bae475bd9e70dc904bfc4a69edde256ca351a86211ae77cfecb90da7c033" Nov 03 20:08:54.257738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:54.257718 5059 scope.go:122] "RemoveContainer" containerID="f45a5ba6d0a6597f36d533287eea3dacb5392b6373db535d57f78e7b1c48721c" Nov 03 20:08:54.267858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:54.267843 5059 scope.go:122] "RemoveContainer" containerID="41824370ba2ad3319dab6d9bbfa7a3f63744067163de4417cf5bbc2373257cdd" Nov 03 20:08:58.378592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.374910 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk"] Nov 03 20:08:58.378592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.376103 5059 cpu_manager.go:414] "containerMap: removing container" podUID="aa74ca6a-b680-401c-9e7e-bb3c1bba2458" containerName="revoke-keys" Nov 03 20:08:58.378592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.376127 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa74ca6a-b680-401c-9e7e-bb3c1bba2458" containerName="revoke-keys" Nov 03 20:08:58.378592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.376149 5059 cpu_manager.go:414] "containerMap: removing container" podUID="a948c428-547e-49e1-8d81-a4c3a0732f79" containerName="revoke-keys" Nov 03 20:08:58.378592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.376157 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a948c428-547e-49e1-8d81-a4c3a0732f79" containerName="revoke-keys" Nov 03 20:08:58.378592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.376315 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa74ca6a-b680-401c-9e7e-bb3c1bba2458" containerName="revoke-keys" Nov 03 20:08:58.378592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.376328 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a948c428-547e-49e1-8d81-a4c3a0732f79" containerName="revoke-keys" Nov 03 20:08:58.390418 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.389075 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.397614 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.397579 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-46a6d7-openshift-default-dockercfg-5jblx\"" Nov 03 20:08:58.401261 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.401239 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk"] Nov 03 20:08:58.580684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.542889 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.580684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.543097 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.580684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.543136 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vrqz\" (UniqueName: \"kubernetes.io/projected/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-kube-api-access-6vrqz\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.580684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.543210 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.580684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.568367 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.580684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.568562 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.580684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.568762 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.580684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.568949 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.580684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.568974 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.684736 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.661095 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:58.812071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.773745 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.842827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.842791 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.843312 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.843285 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.843559 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.843541 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.843705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.843684 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vrqz\" (UniqueName: \"kubernetes.io/projected/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-kube-api-access-6vrqz\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.892683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.891757 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.892683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.892010 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.892683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.892037 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.892683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.892085 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.892683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.816005 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.893113 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.893019 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.893113 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.813432 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" podUID="7331481c-0dd0-4acf-8c73-35a38eece3af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.218:8000/ready\": context deadline exceeded" Nov 03 20:08:58.980774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.859827 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.980774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.903083 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.980774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.849045 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.980774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.905792 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:58.980774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:58.980232 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vrqz\" (UniqueName: \"kubernetes.io/projected/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-kube-api-access-6vrqz\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:59.042686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:59.034403 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:59.106369 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:59.106288 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:59.130864 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:59.116411 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8" Nov 03 20:08:59.317730 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:59.317626 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:08:59.513201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:59.513165 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p"] Nov 03 20:08:59.728829 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:08:59.728770 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b30b4c4_debd_4919_a4e6_67f7ee38eb90.slice/crio-dec408d6192a31fdba83a1d0a1facdcb6c5238a3754650932a40f4abf48c9e5a WatchSource:0}: Error finding container dec408d6192a31fdba83a1d0a1facdcb6c5238a3754650932a40f4abf48c9e5a: Status 404 returned error can't find the container with id dec408d6192a31fdba83a1d0a1facdcb6c5238a3754650932a40f4abf48c9e5a Nov 03 20:08:59.748875 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:59.746716 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk"] Nov 03 20:08:59.760031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:59.759152 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:08:59.760031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:59.759255 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:08:59.760031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:08:59.759308 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:09:00.128295 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:00.128267 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj" Nov 03 20:09:00.412133 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:00.412044 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" event={"ID":"5b30b4c4-debd-4919-a4e6-67f7ee38eb90","Type":"ContainerStarted","Data":"552ec3f06918ad1d09fc456296125df0f1384648fad63a5f292c3a93c5e6ece9"} Nov 03 20:09:00.412133 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:00.412102 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" event={"ID":"5b30b4c4-debd-4919-a4e6-67f7ee38eb90","Type":"ContainerStarted","Data":"dec408d6192a31fdba83a1d0a1facdcb6c5238a3754650932a40f4abf48c9e5a"} Nov 03 20:09:00.455000 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:00.454948 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" podStartSLOduration=2.454933973 podStartE2EDuration="2.454933973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:00.450072946 +0000 UTC m=+1934.707870206" watchObservedRunningTime="2026-11-03 20:09:00.454933973 +0000 UTC m=+1934.712731230" Nov 03 20:09:01.326200 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:01.324989 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:09:01.365179 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:01.365144 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:09:01.420586 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:01.420552 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:09:01.422009 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:01.421887 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:09:05.900909 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:05.900878 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p"] Nov 03 20:09:05.901402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:05.901057 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" podUID="e60b061f-60f0-48bb-8668-fd828a372a8a" containerName="istio-proxy" containerID="cri-o://2ba5e6c6035d1030506e79f3a951df6422700eaabd987c4ba622506c7cd704e7" gracePeriod=30 Nov 03 20:09:05.907316 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:05.907289 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l"] Nov 03 20:09:05.907545 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:05.907504 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" podUID="2cf601d7-8af9-4beb-b64e-f2e9c6b00711" containerName="istio-proxy" containerID="cri-o://84938316ee8fac845a7ea3c47c48e02b03c836b41aeb68977686718055dab32e" gracePeriod=30 Nov 03 20:09:10.272180 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:10.272138 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx"] Nov 03 20:09:10.736373 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:09:10.736327 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:09:11.217904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.217871 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:09:11.244613 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.244581 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e60b061f-60f0-48bb-8668-fd828a372a8a-kube-api-access-jxct6\" (UniqueName: \"kubernetes.io/projected/e60b061f-60f0-48bb-8668-fd828a372a8a-kube-api-access-jxct6\") pod \"e60b061f-60f0-48bb-8668-fd828a372a8a\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " Nov 03 20:09:11.244789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.244638 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-token\" (UniqueName: \"kubernetes.io/projected/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-token\") pod \"e60b061f-60f0-48bb-8668-fd828a372a8a\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " Nov 03 20:09:11.244848 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.244785 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-workload-certs\") pod \"e60b061f-60f0-48bb-8668-fd828a372a8a\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " Nov 03 20:09:11.244848 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.244817 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-credential-socket\") pod \"e60b061f-60f0-48bb-8668-fd828a372a8a\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " Nov 03 20:09:11.244964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.244860 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e60b061f-60f0-48bb-8668-fd828a372a8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60b061f-60f0-48bb-8668-fd828a372a8a-istiod-ca-cert\") pod \"e60b061f-60f0-48bb-8668-fd828a372a8a\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " Nov 03 20:09:11.244964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.244957 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-envoy\") pod \"e60b061f-60f0-48bb-8668-fd828a372a8a\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " Nov 03 20:09:11.245083 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.245055 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-workload-socket\") pod \"e60b061f-60f0-48bb-8668-fd828a372a8a\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " Nov 03 20:09:11.245145 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.245097 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-data\") pod \"e60b061f-60f0-48bb-8668-fd828a372a8a\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " Nov 03 20:09:11.245145 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.245125 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-podinfo\") pod \"e60b061f-60f0-48bb-8668-fd828a372a8a\" (UID: \"e60b061f-60f0-48bb-8668-fd828a372a8a\") " Nov 03 20:09:11.245263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.245050 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-workload-certs" pod "e60b061f-60f0-48bb-8668-fd828a372a8a" (UID: "e60b061f-60f0-48bb-8668-fd828a372a8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:11.245389 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.245367 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-workload-socket" pod "e60b061f-60f0-48bb-8668-fd828a372a8a" (UID: "e60b061f-60f0-48bb-8668-fd828a372a8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:11.245473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.245403 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e60b061f-60f0-48bb-8668-fd828a372a8a-istiod-ca-cert" pod "e60b061f-60f0-48bb-8668-fd828a372a8a" (UID: "e60b061f-60f0-48bb-8668-fd828a372a8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:09:11.245587 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.245566 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-credential-socket" pod "e60b061f-60f0-48bb-8668-fd828a372a8a" (UID: "e60b061f-60f0-48bb-8668-fd828a372a8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:11.246478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.246015 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-data" pod "e60b061f-60f0-48bb-8668-fd828a372a8a" (UID: "e60b061f-60f0-48bb-8668-fd828a372a8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:11.246478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.246367 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:11.246478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.246390 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:11.246478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.246405 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:11.246478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.246420 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:11.246478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.246433 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60b061f-60f0-48bb-8668-fd828a372a8a-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:11.247733 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.247698 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-envoy" pod "e60b061f-60f0-48bb-8668-fd828a372a8a" (UID: "e60b061f-60f0-48bb-8668-fd828a372a8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:11.250033 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.250010 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:09:11.261323 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.261059 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-token" pod "e60b061f-60f0-48bb-8668-fd828a372a8a" (UID: "e60b061f-60f0-48bb-8668-fd828a372a8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:09:11.261323 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.261184 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60b061f-60f0-48bb-8668-fd828a372a8a-kube-api-access-jxct6" pod "e60b061f-60f0-48bb-8668-fd828a372a8a" (UID: "e60b061f-60f0-48bb-8668-fd828a372a8a"). InnerVolumeSpecName "kube-api-access-jxct6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:09:11.261502 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.261403 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-podinfo" pod "e60b061f-60f0-48bb-8668-fd828a372a8a" (UID: "e60b061f-60f0-48bb-8668-fd828a372a8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:09:11.347900 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.347864 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-data\") pod \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " Nov 03 20:09:11.348348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.347938 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-credential-socket\") pod \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " Nov 03 20:09:11.348348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.348061 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-workload-certs\") pod \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " Nov 03 20:09:11.348348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.348093 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istiod-ca-cert\") pod \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " Nov 03 20:09:11.348348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.348178 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-data" pod "2cf601d7-8af9-4beb-b64e-f2e9c6b00711" (UID: "2cf601d7-8af9-4beb-b64e-f2e9c6b00711"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:11.348348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.348191 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-credential-socket" pod "2cf601d7-8af9-4beb-b64e-f2e9c6b00711" (UID: "2cf601d7-8af9-4beb-b64e-f2e9c6b00711"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:11.348348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.348263 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-podinfo\") pod \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " Nov 03 20:09:11.348348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.348301 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-workload-socket\") pod \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " Nov 03 20:09:11.348348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.348344 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-token\" (UniqueName: \"kubernetes.io/projected/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-token\") pod \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " Nov 03 20:09:11.348871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.348450 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-envoy\") pod \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " Nov 03 20:09:11.348871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.348500 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-workload-certs" pod "2cf601d7-8af9-4beb-b64e-f2e9c6b00711" (UID: "2cf601d7-8af9-4beb-b64e-f2e9c6b00711"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:11.348871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.348555 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istiod-ca-cert" pod "2cf601d7-8af9-4beb-b64e-f2e9c6b00711" (UID: "2cf601d7-8af9-4beb-b64e-f2e9c6b00711"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:09:11.348871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.348558 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-kube-api-access-mp4jg\" (UniqueName: \"kubernetes.io/projected/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-kube-api-access-mp4jg\") pod \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\" (UID: \"2cf601d7-8af9-4beb-b64e-f2e9c6b00711\") " Nov 03 20:09:11.348871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.348832 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-workload-socket" pod "2cf601d7-8af9-4beb-b64e-f2e9c6b00711" (UID: "2cf601d7-8af9-4beb-b64e-f2e9c6b00711"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:11.349524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.349499 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:11.349612 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.349531 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:11.349612 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.349547 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:11.349612 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.349563 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:11.349612 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.349577 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:11.349612 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.349591 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:11.349612 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.349605 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:11.349857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.349619 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxct6\" (UniqueName: \"kubernetes.io/projected/e60b061f-60f0-48bb-8668-fd828a372a8a-kube-api-access-jxct6\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:11.349857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.349635 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60b061f-60f0-48bb-8668-fd828a372a8a-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:11.351281 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.351116 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-podinfo" pod "2cf601d7-8af9-4beb-b64e-f2e9c6b00711" (UID: "2cf601d7-8af9-4beb-b64e-f2e9c6b00711"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:09:11.351399 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.351262 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-envoy" pod "2cf601d7-8af9-4beb-b64e-f2e9c6b00711" (UID: "2cf601d7-8af9-4beb-b64e-f2e9c6b00711"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:11.351479 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.351391 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-token" pod "2cf601d7-8af9-4beb-b64e-f2e9c6b00711" (UID: "2cf601d7-8af9-4beb-b64e-f2e9c6b00711"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:09:11.351553 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.351531 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-kube-api-access-mp4jg" pod "2cf601d7-8af9-4beb-b64e-f2e9c6b00711" (UID: "2cf601d7-8af9-4beb-b64e-f2e9c6b00711"). InnerVolumeSpecName "kube-api-access-mp4jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:09:11.451002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.450969 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:11.451002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.450998 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:11.451002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.451009 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:11.451228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.451018 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp4jg\" (UniqueName: \"kubernetes.io/projected/2cf601d7-8af9-4beb-b64e-f2e9c6b00711-kube-api-access-mp4jg\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:11.670455 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.670357 5059 generic.go:356] "Generic (PLEG): container finished" podID="e60b061f-60f0-48bb-8668-fd828a372a8a" containerID="2ba5e6c6035d1030506e79f3a951df6422700eaabd987c4ba622506c7cd704e7" exitCode=0 Nov 03 20:09:11.670721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.670473 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" Nov 03 20:09:11.670819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.670760 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" event={"ID":"e60b061f-60f0-48bb-8668-fd828a372a8a","Type":"ContainerDied","Data":"2ba5e6c6035d1030506e79f3a951df6422700eaabd987c4ba622506c7cd704e7"} Nov 03 20:09:11.670819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.670814 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p" event={"ID":"e60b061f-60f0-48bb-8668-fd828a372a8a","Type":"ContainerDied","Data":"567231dcb6e8df2363f355c4ac1cc809d01976747c63de08d98a3ae8b6e76df8"} Nov 03 20:09:11.670918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.670846 5059 scope.go:122] "RemoveContainer" containerID="2ba5e6c6035d1030506e79f3a951df6422700eaabd987c4ba622506c7cd704e7" Nov 03 20:09:11.675585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.673254 5059 generic.go:356] "Generic (PLEG): container finished" podID="2cf601d7-8af9-4beb-b64e-f2e9c6b00711" containerID="84938316ee8fac845a7ea3c47c48e02b03c836b41aeb68977686718055dab32e" exitCode=0 Nov 03 20:09:11.675585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.673299 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" event={"ID":"2cf601d7-8af9-4beb-b64e-f2e9c6b00711","Type":"ContainerDied","Data":"84938316ee8fac845a7ea3c47c48e02b03c836b41aeb68977686718055dab32e"} Nov 03 20:09:11.676076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.676046 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" event={"ID":"2cf601d7-8af9-4beb-b64e-f2e9c6b00711","Type":"ContainerDied","Data":"f2543d77f2decc8ee1bea967fea0369c0aeed38031f2a8770cb5da2b14b75e92"} Nov 03 20:09:11.676194 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.676135 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l" Nov 03 20:09:11.686245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.686224 5059 scope.go:122] "RemoveContainer" containerID="2ba5e6c6035d1030506e79f3a951df6422700eaabd987c4ba622506c7cd704e7" Nov 03 20:09:11.686579 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:09:11.686557 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ba5e6c6035d1030506e79f3a951df6422700eaabd987c4ba622506c7cd704e7\": container with ID starting with 2ba5e6c6035d1030506e79f3a951df6422700eaabd987c4ba622506c7cd704e7 not found: ID does not exist" containerID="2ba5e6c6035d1030506e79f3a951df6422700eaabd987c4ba622506c7cd704e7" Nov 03 20:09:11.686689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.686587 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ba5e6c6035d1030506e79f3a951df6422700eaabd987c4ba622506c7cd704e7"} err="failed to get container status \"2ba5e6c6035d1030506e79f3a951df6422700eaabd987c4ba622506c7cd704e7\": rpc error: code = NotFound desc = could not find container \"2ba5e6c6035d1030506e79f3a951df6422700eaabd987c4ba622506c7cd704e7\": container with ID starting with 2ba5e6c6035d1030506e79f3a951df6422700eaabd987c4ba622506c7cd704e7 not found: ID does not exist" Nov 03 20:09:11.686689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.686609 5059 scope.go:122] "RemoveContainer" containerID="84938316ee8fac845a7ea3c47c48e02b03c836b41aeb68977686718055dab32e" Nov 03 20:09:11.697259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.697228 5059 scope.go:122] "RemoveContainer" containerID="84938316ee8fac845a7ea3c47c48e02b03c836b41aeb68977686718055dab32e" Nov 03 20:09:11.697603 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:09:11.697575 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84938316ee8fac845a7ea3c47c48e02b03c836b41aeb68977686718055dab32e\": container with ID starting with 84938316ee8fac845a7ea3c47c48e02b03c836b41aeb68977686718055dab32e not found: ID does not exist" containerID="84938316ee8fac845a7ea3c47c48e02b03c836b41aeb68977686718055dab32e" Nov 03 20:09:11.697758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.697612 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84938316ee8fac845a7ea3c47c48e02b03c836b41aeb68977686718055dab32e"} err="failed to get container status \"84938316ee8fac845a7ea3c47c48e02b03c836b41aeb68977686718055dab32e\": rpc error: code = NotFound desc = could not find container \"84938316ee8fac845a7ea3c47c48e02b03c836b41aeb68977686718055dab32e\": container with ID starting with 84938316ee8fac845a7ea3c47c48e02b03c836b41aeb68977686718055dab32e not found: ID does not exist" Nov 03 20:09:11.703232 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.703210 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p"] Nov 03 20:09:11.715571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.715536 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ndt4p"] Nov 03 20:09:11.735166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.734556 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l"] Nov 03 20:09:11.739605 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.739582 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-vfd7l"] Nov 03 20:09:11.859641 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.859609 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cf601d7-8af9-4beb-b64e-f2e9c6b00711" path="/var/lib/kubelet/pods/2cf601d7-8af9-4beb-b64e-f2e9c6b00711/volumes" Nov 03 20:09:11.860213 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:11.860196 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e60b061f-60f0-48bb-8668-fd828a372a8a" path="/var/lib/kubelet/pods/e60b061f-60f0-48bb-8668-fd828a372a8a/volumes" Nov 03 20:09:12.595874 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.595825 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd"] Nov 03 20:09:12.598153 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.598071 5059 cpu_manager.go:414] "containerMap: removing container" podUID="2cf601d7-8af9-4beb-b64e-f2e9c6b00711" containerName="istio-proxy" Nov 03 20:09:12.598432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.598411 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf601d7-8af9-4beb-b64e-f2e9c6b00711" containerName="istio-proxy" Nov 03 20:09:12.598782 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.598752 5059 cpu_manager.go:414] "containerMap: removing container" podUID="e60b061f-60f0-48bb-8668-fd828a372a8a" containerName="istio-proxy" Nov 03 20:09:12.601698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.601677 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e60b061f-60f0-48bb-8668-fd828a372a8a" containerName="istio-proxy" Nov 03 20:09:12.602205 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.602182 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e60b061f-60f0-48bb-8668-fd828a372a8a" containerName="istio-proxy" Nov 03 20:09:12.602346 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.602332 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf601d7-8af9-4beb-b64e-f2e9c6b00711" containerName="istio-proxy" Nov 03 20:09:12.618750 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.618722 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd"] Nov 03 20:09:12.618888 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.618839 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.621995 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.621969 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd"] Nov 03 20:09:12.624239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.624208 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-98529d-openshift-default-dockercfg-l472v\"" Nov 03 20:09:12.627223 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.627062 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.628263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.628244 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd"] Nov 03 20:09:12.658719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.658685 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-36556d-openshift-default-dockercfg-hnpg9\"" Nov 03 20:09:12.707398 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.707361 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-token\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.707544 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.707424 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-workload-certs\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.707614 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.707533 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-workload-socket\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.707614 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.707581 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-podinfo\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.707726 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.707682 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-credential-socket\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.707778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.707737 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-workload-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.707824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.707775 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d60eb56-9852-4c25-ae50-8a932e752793-istio-podinfo\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.707858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.707837 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-data\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.707894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.707881 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-istio-data\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.707932 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.707910 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flp9z\" (UniqueName: \"kubernetes.io/projected/4be209ff-6d62-42d5-a0e3-22373143ecf5-kube-api-access-flp9z\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.708053 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.708033 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-workload-certs\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.708106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.708079 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d60eb56-9852-4c25-ae50-8a932e752793-istiod-ca-cert\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.708106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.708099 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d60eb56-9852-4c25-ae50-8a932e752793-istio-token\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.708265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.708247 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be209ff-6d62-42d5-a0e3-22373143ecf5-istiod-ca-cert\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.708330 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.708291 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q79lh\" (UniqueName: \"kubernetes.io/projected/9d60eb56-9852-4c25-ae50-8a932e752793-kube-api-access-q79lh\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.708330 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.708324 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-istio-envoy\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.708416 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.708358 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-envoy\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.708453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.708420 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-credential-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.810073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.810038 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-workload-certs\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.810254 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.810093 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d60eb56-9852-4c25-ae50-8a932e752793-istiod-ca-cert\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.810254 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.810215 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d60eb56-9852-4c25-ae50-8a932e752793-istio-token\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.810402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.810348 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be209ff-6d62-42d5-a0e3-22373143ecf5-istiod-ca-cert\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.810402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.810386 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q79lh\" (UniqueName: \"kubernetes.io/projected/9d60eb56-9852-4c25-ae50-8a932e752793-kube-api-access-q79lh\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.810531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.810420 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-istio-envoy\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.810531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.810452 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-envoy\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.810531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.810490 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-credential-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.810531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.810514 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-workload-certs\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.810531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.810527 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-token\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.810814 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.810687 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-workload-certs\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.810814 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.810746 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-workload-socket\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.810814 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.810785 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-podinfo\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.811004 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.810822 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-credential-socket\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.811004 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.810880 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-workload-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.811004 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.810922 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d60eb56-9852-4c25-ae50-8a932e752793-istio-podinfo\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.811004 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.810925 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d60eb56-9852-4c25-ae50-8a932e752793-istiod-ca-cert\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.811004 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.810953 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-data\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.811004 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.811003 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-istio-data\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.811298 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.811030 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flp9z\" (UniqueName: \"kubernetes.io/projected/4be209ff-6d62-42d5-a0e3-22373143ecf5-kube-api-access-flp9z\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.811298 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.811051 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-workload-certs\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.811298 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.811122 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be209ff-6d62-42d5-a0e3-22373143ecf5-istiod-ca-cert\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.811298 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.811235 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-credential-socket\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.811521 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.811431 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-credential-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.811582 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.811519 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-istio-data\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.811582 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.811522 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-data\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.811750 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.811718 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-workload-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.811824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.811767 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-workload-socket\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.813336 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.813311 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-envoy\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.813720 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.813696 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-istio-envoy\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.814026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.814008 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d60eb56-9852-4c25-ae50-8a932e752793-istio-podinfo\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.814149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.814126 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-podinfo\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.823481 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.823451 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d60eb56-9852-4c25-ae50-8a932e752793-istio-token\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.823648 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.823624 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-token\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.824303 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.824280 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flp9z\" (UniqueName: \"kubernetes.io/projected/4be209ff-6d62-42d5-a0e3-22373143ecf5-kube-api-access-flp9z\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nlwxd\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:12.824513 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.824497 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q79lh\" (UniqueName: \"kubernetes.io/projected/9d60eb56-9852-4c25-ae50-8a932e752793-kube-api-access-q79lh\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.947086 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.936900 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:12.947086 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:12.945914 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:13.471986 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:09:13.471957 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d60eb56_9852_4c25_ae50_8a932e752793.slice/crio-b8f814c27581a58bb22d904094ae0a72ad87eb9aa97e7d3369103603db7a97ce WatchSource:0}: Error finding container b8f814c27581a58bb22d904094ae0a72ad87eb9aa97e7d3369103603db7a97ce: Status 404 returned error can't find the container with id b8f814c27581a58bb22d904094ae0a72ad87eb9aa97e7d3369103603db7a97ce Nov 03 20:09:13.472129 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:13.472053 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd"] Nov 03 20:09:13.475342 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:09:13.475311 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be209ff_6d62_42d5_a0e3_22373143ecf5.slice/crio-98bdcf25f4397353fe737fdc4f6d13cb878c020361c324fc2bac181aafe9f510 WatchSource:0}: Error finding container 98bdcf25f4397353fe737fdc4f6d13cb878c020361c324fc2bac181aafe9f510: Status 404 returned error can't find the container with id 98bdcf25f4397353fe737fdc4f6d13cb878c020361c324fc2bac181aafe9f510 Nov 03 20:09:13.475725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:13.475683 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:09:13.475789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:13.475771 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:09:13.475829 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:13.475816 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:09:13.478932 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:13.478835 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:09:13.479107 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:13.478950 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:09:13.479107 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:13.478998 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:09:13.479251 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:13.478997 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd"] Nov 03 20:09:13.706836 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:13.706788 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" event={"ID":"9d60eb56-9852-4c25-ae50-8a932e752793","Type":"ContainerStarted","Data":"6dc7ad3073c40aab49d043739b212390e9623e46a6f995b1c1ca2fa55cf79644"} Nov 03 20:09:13.707307 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:13.706861 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" event={"ID":"9d60eb56-9852-4c25-ae50-8a932e752793","Type":"ContainerStarted","Data":"b8f814c27581a58bb22d904094ae0a72ad87eb9aa97e7d3369103603db7a97ce"} Nov 03 20:09:13.710483 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:13.709681 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" event={"ID":"4be209ff-6d62-42d5-a0e3-22373143ecf5","Type":"ContainerStarted","Data":"8ceb4111ba8cc2ad05515d7fe5bc2fe674b5f039e258707d2fdfd55fd1df3f2e"} Nov 03 20:09:13.710483 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:13.709735 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" event={"ID":"4be209ff-6d62-42d5-a0e3-22373143ecf5","Type":"ContainerStarted","Data":"98bdcf25f4397353fe737fdc4f6d13cb878c020361c324fc2bac181aafe9f510"} Nov 03 20:09:13.768237 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:13.767910 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" podStartSLOduration=1.767890532 podStartE2EDuration="1.767890532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:13.743921546 +0000 UTC m=+1948.001718802" watchObservedRunningTime="2026-11-03 20:09:13.767890532 +0000 UTC m=+1948.025687790" Nov 03 20:09:13.771980 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:13.771938 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" podStartSLOduration=1.771925368 podStartE2EDuration="1.771925368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:13.767262721 +0000 UTC m=+1948.025059999" watchObservedRunningTime="2026-11-03 20:09:13.771925368 +0000 UTC m=+1948.029722625" Nov 03 20:09:13.946699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:13.940946 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:13.955754 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:13.950758 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:14.534708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:14.534654 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" podUID="abb3d35a-13d2-4fa9-a21c-0830efce2e7c" containerName="main" containerID="cri-o://fb8da14e89cf10f509bfa14c33a07660777ad613c5be9ff76210f29b1180fdf0" gracePeriod=45 Nov 03 20:09:14.720931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:14.720898 5059 generic.go:356] "Generic (PLEG): container finished" podID="abb3d35a-13d2-4fa9-a21c-0830efce2e7c" containerID="fb8da14e89cf10f509bfa14c33a07660777ad613c5be9ff76210f29b1180fdf0" exitCode=0 Nov 03 20:09:14.721412 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:14.720972 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" event={"ID":"abb3d35a-13d2-4fa9-a21c-0830efce2e7c","Type":"ContainerDied","Data":"fb8da14e89cf10f509bfa14c33a07660777ad613c5be9ff76210f29b1180fdf0"} Nov 03 20:09:14.721506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:14.721482 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" podUID="abb3d35a-13d2-4fa9-a21c-0830efce2e7c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.217:8000/ready\": dial tcp 10.132.0.217:8000: connect: connection refused" Nov 03 20:09:14.844137 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:14.844111 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:09:14.944163 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:14.944132 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:14.954260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:14.954232 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:14.977685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:14.975185 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-kube-api-access-v4bxz\" (UniqueName: \"kubernetes.io/projected/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-kube-api-access-v4bxz\") pod \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\" (UID: \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\") " Nov 03 20:09:14.977685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:14.975254 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-dshm\") pod \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\" (UID: \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\") " Nov 03 20:09:14.977685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:14.975331 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-model-cache\") pod \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\" (UID: \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\") " Nov 03 20:09:14.977685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:14.975363 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-tmp-dir\") pod \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\" (UID: \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\") " Nov 03 20:09:14.977685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:14.975414 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-tls-certs\" (UniqueName: \"kubernetes.io/secret/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-tls-certs\") pod \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\" (UID: \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\") " Nov 03 20:09:14.977685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:14.975511 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-home\" (UniqueName: \"kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-home\") pod \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\" (UID: \"abb3d35a-13d2-4fa9-a21c-0830efce2e7c\") " Nov 03 20:09:14.977685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:14.977415 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-model-cache" pod "abb3d35a-13d2-4fa9-a21c-0830efce2e7c" (UID: "abb3d35a-13d2-4fa9-a21c-0830efce2e7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:14.981678 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:14.978243 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-home" pod "abb3d35a-13d2-4fa9-a21c-0830efce2e7c" (UID: "abb3d35a-13d2-4fa9-a21c-0830efce2e7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:14.993259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:14.993226 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-tmp-dir" pod "abb3d35a-13d2-4fa9-a21c-0830efce2e7c" (UID: "abb3d35a-13d2-4fa9-a21c-0830efce2e7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:14.996354 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:14.996327 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-tls-certs" pod "abb3d35a-13d2-4fa9-a21c-0830efce2e7c" (UID: "abb3d35a-13d2-4fa9-a21c-0830efce2e7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:09:14.996464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:14.996451 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-kube-api-access-v4bxz" pod "abb3d35a-13d2-4fa9-a21c-0830efce2e7c" (UID: "abb3d35a-13d2-4fa9-a21c-0830efce2e7c"). InnerVolumeSpecName "kube-api-access-v4bxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:09:14.997251 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:14.997228 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-dshm" pod "abb3d35a-13d2-4fa9-a21c-0830efce2e7c" (UID: "abb3d35a-13d2-4fa9-a21c-0830efce2e7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:15.077832 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:15.077801 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4bxz\" (UniqueName: \"kubernetes.io/projected/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-kube-api-access-v4bxz\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:15.077979 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:15.077835 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:15.077979 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:15.077860 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:15.077979 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:15.077875 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:15.077979 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:15.077890 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:15.077979 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:15.077902 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abb3d35a-13d2-4fa9-a21c-0830efce2e7c-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:15.729986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:15.729957 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" Nov 03 20:09:15.730440 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:15.729996 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p" event={"ID":"abb3d35a-13d2-4fa9-a21c-0830efce2e7c","Type":"ContainerDied","Data":"a5f300b3cc6f5f95ad9995a3b30bfd6b00184d7e4001d0eeb3da4e915b5094ca"} Nov 03 20:09:15.730440 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:15.730049 5059 scope.go:122] "RemoveContainer" containerID="fb8da14e89cf10f509bfa14c33a07660777ad613c5be9ff76210f29b1180fdf0" Nov 03 20:09:15.730873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:15.730851 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:15.731070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:15.731047 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:15.731917 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:15.731487 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:09:15.731917 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:15.731552 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:09:15.830130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:15.829994 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p"] Nov 03 20:09:15.834068 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:15.834032 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-942930/auth-test-model-942930-kserve-5859b4bbb9-cjt8p"] Nov 03 20:09:15.861677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:15.861640 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abb3d35a-13d2-4fa9-a21c-0830efce2e7c" path="/var/lib/kubelet/pods/abb3d35a-13d2-4fa9-a21c-0830efce2e7c/volumes" Nov 03 20:09:16.544701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:16.543439 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv"] Nov 03 20:09:16.544701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:16.543703 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" podUID="fd6f9e73-1c5a-4151-a720-96f136b7b3a4" containerName="istio-proxy" containerID="cri-o://3879c9e81754313a4cb696508516f2230f2f9c2c215c51e2eb7ec23ec4b34aa3" gracePeriod=30 Nov 03 20:09:17.567157 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:17.567125 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps"] Nov 03 20:09:17.567898 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:17.567882 5059 cpu_manager.go:414] "containerMap: removing container" podUID="abb3d35a-13d2-4fa9-a21c-0830efce2e7c" containerName="main" Nov 03 20:09:17.567948 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:17.567900 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abb3d35a-13d2-4fa9-a21c-0830efce2e7c" containerName="main" Nov 03 20:09:17.568011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:17.568002 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="abb3d35a-13d2-4fa9-a21c-0830efce2e7c" containerName="main" Nov 03 20:09:17.572585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:17.572565 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps" Nov 03 20:09:17.581010 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:17.580938 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-98529d\"" Nov 03 20:09:17.588503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:17.588476 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps"] Nov 03 20:09:17.714688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:17.708184 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8n84\" (UniqueName: \"kubernetes.io/projected/2faa14fa-ec86-4633-a968-c3f761d13443-kube-api-access-g8n84\") pod \"payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps\" (UID: \"2faa14fa-ec86-4633-a968-c3f761d13443\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps" Nov 03 20:09:17.715074 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:17.715039 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2faa14fa-ec86-4633-a968-c3f761d13443-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps\" (UID: \"2faa14fa-ec86-4633-a968-c3f761d13443\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps" Nov 03 20:09:17.883686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:17.864294 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8n84\" (UniqueName: \"kubernetes.io/projected/2faa14fa-ec86-4633-a968-c3f761d13443-kube-api-access-g8n84\") pod \"payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps\" (UID: \"2faa14fa-ec86-4633-a968-c3f761d13443\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps" Nov 03 20:09:17.883686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:17.864432 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2faa14fa-ec86-4633-a968-c3f761d13443-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps\" (UID: \"2faa14fa-ec86-4633-a968-c3f761d13443\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps" Nov 03 20:09:17.896684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:17.885732 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2faa14fa-ec86-4633-a968-c3f761d13443-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps\" (UID: \"2faa14fa-ec86-4633-a968-c3f761d13443\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps" Nov 03 20:09:18.021922 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:18.021886 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8n84\" (UniqueName: \"kubernetes.io/projected/2faa14fa-ec86-4633-a968-c3f761d13443-kube-api-access-g8n84\") pod \"payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps\" (UID: \"2faa14fa-ec86-4633-a968-c3f761d13443\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps" Nov 03 20:09:18.190649 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:18.190579 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps" Nov 03 20:09:18.492481 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:09:18.492445 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2faa14fa_ec86_4633_a968_c3f761d13443.slice/crio-c44513f85db26c84eec581e4101bcb4caf3fe658b6bb76d6c6fe7f404623dd5d WatchSource:0}: Error finding container c44513f85db26c84eec581e4101bcb4caf3fe658b6bb76d6c6fe7f404623dd5d: Status 404 returned error can't find the container with id c44513f85db26c84eec581e4101bcb4caf3fe658b6bb76d6c6fe7f404623dd5d Nov 03 20:09:18.501382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:18.501346 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps"] Nov 03 20:09:18.835690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:18.830645 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255"] Nov 03 20:09:18.835690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:18.830883 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" podUID="34484729-bde1-4b0f-9758-e934b14f33d0" containerName="istio-proxy" containerID="cri-o://c05cf95cff32ad56454940f6ca93b383dfeefc5aaef39852a6353bdb13080488" gracePeriod=30 Nov 03 20:09:19.082692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:19.082299 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps" event={"ID":"2faa14fa-ec86-4633-a968-c3f761d13443","Type":"ContainerStarted","Data":"d610230fd1e961d1b794119c73b53766e612967c8196f325df7359012d12af34"} Nov 03 20:09:19.082692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:19.082369 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps" event={"ID":"2faa14fa-ec86-4633-a968-c3f761d13443","Type":"ContainerStarted","Data":"c44513f85db26c84eec581e4101bcb4caf3fe658b6bb76d6c6fe7f404623dd5d"} Nov 03 20:09:19.084126 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:19.083644 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps" Nov 03 20:09:19.109685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:19.109578 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps" podStartSLOduration=2.109564261 podStartE2EDuration="2.109564261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:19.10685509 +0000 UTC m=+1953.364652372" watchObservedRunningTime="2026-11-03 20:09:19.109564261 +0000 UTC m=+1953.367361517" Nov 03 20:09:22.107874 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.107843 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:09:22.242741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.242708 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-envoy\") pod \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " Nov 03 20:09:22.242961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.242756 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-token\" (UniqueName: \"kubernetes.io/projected/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-token\") pod \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " Nov 03 20:09:22.242961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.242779 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-workload-socket\") pod \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " Nov 03 20:09:22.242961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.242830 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-workload-certs\") pod \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " Nov 03 20:09:22.242961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.242866 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-kube-api-access-zp7gv\" (UniqueName: \"kubernetes.io/projected/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-kube-api-access-zp7gv\") pod \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " Nov 03 20:09:22.243213 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.243004 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-data\") pod \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " Nov 03 20:09:22.243213 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.243038 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istiod-ca-cert\") pod \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " Nov 03 20:09:22.243213 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.243126 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-podinfo\") pod \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " Nov 03 20:09:22.243213 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.243148 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-workload-socket" pod "fd6f9e73-1c5a-4151-a720-96f136b7b3a4" (UID: "fd6f9e73-1c5a-4151-a720-96f136b7b3a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:22.243213 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.243191 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-credential-socket\") pod \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\" (UID: \"fd6f9e73-1c5a-4151-a720-96f136b7b3a4\") " Nov 03 20:09:22.243496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.243245 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-workload-certs" pod "fd6f9e73-1c5a-4151-a720-96f136b7b3a4" (UID: "fd6f9e73-1c5a-4151-a720-96f136b7b3a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:22.243496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.243463 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istiod-ca-cert" pod "fd6f9e73-1c5a-4151-a720-96f136b7b3a4" (UID: "fd6f9e73-1c5a-4151-a720-96f136b7b3a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:09:22.243619 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.243588 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-credential-socket" pod "fd6f9e73-1c5a-4151-a720-96f136b7b3a4" (UID: "fd6f9e73-1c5a-4151-a720-96f136b7b3a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:22.243725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.243655 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-data" pod "fd6f9e73-1c5a-4151-a720-96f136b7b3a4" (UID: "fd6f9e73-1c5a-4151-a720-96f136b7b3a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:22.244370 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.244181 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:22.244508 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.244379 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:22.244508 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.244394 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:22.244508 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.244407 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:22.244508 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.244422 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:22.245813 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.245785 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-kube-api-access-zp7gv" pod "fd6f9e73-1c5a-4151-a720-96f136b7b3a4" (UID: "fd6f9e73-1c5a-4151-a720-96f136b7b3a4"). InnerVolumeSpecName "kube-api-access-zp7gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:09:22.245935 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.245823 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-token" pod "fd6f9e73-1c5a-4151-a720-96f136b7b3a4" (UID: "fd6f9e73-1c5a-4151-a720-96f136b7b3a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:09:22.246478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.246439 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-podinfo" pod "fd6f9e73-1c5a-4151-a720-96f136b7b3a4" (UID: "fd6f9e73-1c5a-4151-a720-96f136b7b3a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:09:22.246600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.246515 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-envoy" pod "fd6f9e73-1c5a-4151-a720-96f136b7b3a4" (UID: "fd6f9e73-1c5a-4151-a720-96f136b7b3a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:22.345990 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.345902 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:22.345990 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.345934 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:22.345990 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.345943 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:22.345990 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.345952 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp7gv\" (UniqueName: \"kubernetes.io/projected/fd6f9e73-1c5a-4151-a720-96f136b7b3a4-kube-api-access-zp7gv\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:22.370077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.370045 5059 generic.go:356] "Generic (PLEG): container finished" podID="fd6f9e73-1c5a-4151-a720-96f136b7b3a4" containerID="3879c9e81754313a4cb696508516f2230f2f9c2c215c51e2eb7ec23ec4b34aa3" exitCode=0 Nov 03 20:09:22.370260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.370157 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" event={"ID":"fd6f9e73-1c5a-4151-a720-96f136b7b3a4","Type":"ContainerDied","Data":"3879c9e81754313a4cb696508516f2230f2f9c2c215c51e2eb7ec23ec4b34aa3"} Nov 03 20:09:22.370260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.370179 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" Nov 03 20:09:22.370260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.370212 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv" event={"ID":"fd6f9e73-1c5a-4151-a720-96f136b7b3a4","Type":"ContainerDied","Data":"dbccc6a151c723ededb6e3105f57f7e135e704ff03418168552d033cbd4a28eb"} Nov 03 20:09:22.370260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.370235 5059 scope.go:122] "RemoveContainer" containerID="3879c9e81754313a4cb696508516f2230f2f9c2c215c51e2eb7ec23ec4b34aa3" Nov 03 20:09:22.382874 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.382854 5059 scope.go:122] "RemoveContainer" containerID="3879c9e81754313a4cb696508516f2230f2f9c2c215c51e2eb7ec23ec4b34aa3" Nov 03 20:09:22.383144 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:09:22.383126 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3879c9e81754313a4cb696508516f2230f2f9c2c215c51e2eb7ec23ec4b34aa3\": container with ID starting with 3879c9e81754313a4cb696508516f2230f2f9c2c215c51e2eb7ec23ec4b34aa3 not found: ID does not exist" containerID="3879c9e81754313a4cb696508516f2230f2f9c2c215c51e2eb7ec23ec4b34aa3" Nov 03 20:09:22.383228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.383150 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3879c9e81754313a4cb696508516f2230f2f9c2c215c51e2eb7ec23ec4b34aa3"} err="failed to get container status \"3879c9e81754313a4cb696508516f2230f2f9c2c215c51e2eb7ec23ec4b34aa3\": rpc error: code = NotFound desc = could not find container \"3879c9e81754313a4cb696508516f2230f2f9c2c215c51e2eb7ec23ec4b34aa3\": container with ID starting with 3879c9e81754313a4cb696508516f2230f2f9c2c215c51e2eb7ec23ec4b34aa3 not found: ID does not exist" Nov 03 20:09:22.402101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.400761 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv"] Nov 03 20:09:22.414547 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:22.414512 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-kdqmv"] Nov 03 20:09:23.860094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:23.860057 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd6f9e73-1c5a-4151-a720-96f136b7b3a4" path="/var/lib/kubelet/pods/fd6f9e73-1c5a-4151-a720-96f136b7b3a4/volumes" Nov 03 20:09:24.304041 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.304018 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 20:09:24.405475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.405389 5059 generic.go:356] "Generic (PLEG): container finished" podID="34484729-bde1-4b0f-9758-e934b14f33d0" containerID="c05cf95cff32ad56454940f6ca93b383dfeefc5aaef39852a6353bdb13080488" exitCode=0 Nov 03 20:09:24.405475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.405436 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" Nov 03 20:09:24.405738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.405476 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" event={"ID":"34484729-bde1-4b0f-9758-e934b14f33d0","Type":"ContainerDied","Data":"c05cf95cff32ad56454940f6ca93b383dfeefc5aaef39852a6353bdb13080488"} Nov 03 20:09:24.405738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.405562 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255" event={"ID":"34484729-bde1-4b0f-9758-e934b14f33d0","Type":"ContainerDied","Data":"667ef8ff7cf2400fbf5fb14999b09d30ef4076fa8d683b04e71a8e6a8629d152"} Nov 03 20:09:24.405738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.405595 5059 scope.go:122] "RemoveContainer" containerID="c05cf95cff32ad56454940f6ca93b383dfeefc5aaef39852a6353bdb13080488" Nov 03 20:09:24.410049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.410024 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34484729-bde1-4b0f-9758-e934b14f33d0-istio-token\" (UniqueName: \"kubernetes.io/projected/34484729-bde1-4b0f-9758-e934b14f33d0-istio-token\") pod \"34484729-bde1-4b0f-9758-e934b14f33d0\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " Nov 03 20:09:24.410161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.410148 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34484729-bde1-4b0f-9758-e934b14f33d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34484729-bde1-4b0f-9758-e934b14f33d0-istio-podinfo\") pod \"34484729-bde1-4b0f-9758-e934b14f33d0\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " Nov 03 20:09:24.410296 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.410279 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-workload-certs\") pod \"34484729-bde1-4b0f-9758-e934b14f33d0\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " Nov 03 20:09:24.410367 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.410326 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34484729-bde1-4b0f-9758-e934b14f33d0-kube-api-access-4bp2g\" (UniqueName: \"kubernetes.io/projected/34484729-bde1-4b0f-9758-e934b14f33d0-kube-api-access-4bp2g\") pod \"34484729-bde1-4b0f-9758-e934b14f33d0\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " Nov 03 20:09:24.410441 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.410422 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-istio-data\") pod \"34484729-bde1-4b0f-9758-e934b14f33d0\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " Nov 03 20:09:24.410511 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.410457 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-credential-socket\") pod \"34484729-bde1-4b0f-9758-e934b14f33d0\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " Nov 03 20:09:24.410570 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.410509 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-workload-socket\") pod \"34484729-bde1-4b0f-9758-e934b14f33d0\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " Nov 03 20:09:24.410570 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.410554 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-workload-certs" pod "34484729-bde1-4b0f-9758-e934b14f33d0" (UID: "34484729-bde1-4b0f-9758-e934b14f33d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:24.410701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.410640 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-istio-envoy\") pod \"34484729-bde1-4b0f-9758-e934b14f33d0\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " Nov 03 20:09:24.410782 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.410766 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34484729-bde1-4b0f-9758-e934b14f33d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34484729-bde1-4b0f-9758-e934b14f33d0-istiod-ca-cert\") pod \"34484729-bde1-4b0f-9758-e934b14f33d0\" (UID: \"34484729-bde1-4b0f-9758-e934b14f33d0\") " Nov 03 20:09:24.410926 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.410897 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-istio-data" pod "34484729-bde1-4b0f-9758-e934b14f33d0" (UID: "34484729-bde1-4b0f-9758-e934b14f33d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:24.410926 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.410905 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-workload-socket" pod "34484729-bde1-4b0f-9758-e934b14f33d0" (UID: "34484729-bde1-4b0f-9758-e934b14f33d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:24.411100 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.410987 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-credential-socket" pod "34484729-bde1-4b0f-9758-e934b14f33d0" (UID: "34484729-bde1-4b0f-9758-e934b14f33d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:24.411315 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.411287 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34484729-bde1-4b0f-9758-e934b14f33d0-istiod-ca-cert" pod "34484729-bde1-4b0f-9758-e934b14f33d0" (UID: "34484729-bde1-4b0f-9758-e934b14f33d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:09:24.411514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.411494 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:24.411594 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.411515 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:24.411594 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.411524 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:24.411594 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.411533 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:24.411594 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.411542 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34484729-bde1-4b0f-9758-e934b14f33d0-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:24.412592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.412568 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34484729-bde1-4b0f-9758-e934b14f33d0-istio-podinfo" pod "34484729-bde1-4b0f-9758-e934b14f33d0" (UID: "34484729-bde1-4b0f-9758-e934b14f33d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:09:24.412699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.412611 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34484729-bde1-4b0f-9758-e934b14f33d0-istio-token" pod "34484729-bde1-4b0f-9758-e934b14f33d0" (UID: "34484729-bde1-4b0f-9758-e934b14f33d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:09:24.412990 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.412973 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-istio-envoy" pod "34484729-bde1-4b0f-9758-e934b14f33d0" (UID: "34484729-bde1-4b0f-9758-e934b14f33d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:24.413118 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.413096 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34484729-bde1-4b0f-9758-e934b14f33d0-kube-api-access-4bp2g" pod "34484729-bde1-4b0f-9758-e934b14f33d0" (UID: "34484729-bde1-4b0f-9758-e934b14f33d0"). InnerVolumeSpecName "kube-api-access-4bp2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:09:24.418636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.418617 5059 scope.go:122] "RemoveContainer" containerID="c05cf95cff32ad56454940f6ca93b383dfeefc5aaef39852a6353bdb13080488" Nov 03 20:09:24.418986 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:09:24.418964 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c05cf95cff32ad56454940f6ca93b383dfeefc5aaef39852a6353bdb13080488\": container with ID starting with c05cf95cff32ad56454940f6ca93b383dfeefc5aaef39852a6353bdb13080488 not found: ID does not exist" containerID="c05cf95cff32ad56454940f6ca93b383dfeefc5aaef39852a6353bdb13080488" Nov 03 20:09:24.419081 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.418998 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c05cf95cff32ad56454940f6ca93b383dfeefc5aaef39852a6353bdb13080488"} err="failed to get container status \"c05cf95cff32ad56454940f6ca93b383dfeefc5aaef39852a6353bdb13080488\": rpc error: code = NotFound desc = could not find container \"c05cf95cff32ad56454940f6ca93b383dfeefc5aaef39852a6353bdb13080488\": container with ID starting with c05cf95cff32ad56454940f6ca93b383dfeefc5aaef39852a6353bdb13080488 not found: ID does not exist" Nov 03 20:09:24.513588 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.513545 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34484729-bde1-4b0f-9758-e934b14f33d0-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:24.513588 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.513587 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bp2g\" (UniqueName: \"kubernetes.io/projected/34484729-bde1-4b0f-9758-e934b14f33d0-kube-api-access-4bp2g\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:24.513588 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.513598 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34484729-bde1-4b0f-9758-e934b14f33d0-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:24.513914 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.513607 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34484729-bde1-4b0f-9758-e934b14f33d0-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:24.734352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.734321 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255"] Nov 03 20:09:24.740716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:24.740689 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-s4255"] Nov 03 20:09:25.298632 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:25.298593 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" podUID="7331481c-0dd0-4acf-8c73-35a38eece3af" containerName="main" containerID="cri-o://d081881808411dc88c6aff949d2bb38279ffa9fcdfcaf1e1805e834b89a1081c" gracePeriod=45 Nov 03 20:09:25.605812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:25.605788 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:09:25.738417 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:25.738377 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-tmp-dir\") pod \"7331481c-0dd0-4acf-8c73-35a38eece3af\" (UID: \"7331481c-0dd0-4acf-8c73-35a38eece3af\") " Nov 03 20:09:25.738624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:25.738480 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-home\" (UniqueName: \"kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-home\") pod \"7331481c-0dd0-4acf-8c73-35a38eece3af\" (UID: \"7331481c-0dd0-4acf-8c73-35a38eece3af\") " Nov 03 20:09:25.738624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:25.738520 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-dshm\") pod \"7331481c-0dd0-4acf-8c73-35a38eece3af\" (UID: \"7331481c-0dd0-4acf-8c73-35a38eece3af\") " Nov 03 20:09:25.738624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:25.738559 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7331481c-0dd0-4acf-8c73-35a38eece3af-tls-certs\" (UniqueName: \"kubernetes.io/secret/7331481c-0dd0-4acf-8c73-35a38eece3af-tls-certs\") pod \"7331481c-0dd0-4acf-8c73-35a38eece3af\" (UID: \"7331481c-0dd0-4acf-8c73-35a38eece3af\") " Nov 03 20:09:25.738624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:25.738591 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7331481c-0dd0-4acf-8c73-35a38eece3af-kube-api-access-bm7xb\" (UniqueName: \"kubernetes.io/projected/7331481c-0dd0-4acf-8c73-35a38eece3af-kube-api-access-bm7xb\") pod \"7331481c-0dd0-4acf-8c73-35a38eece3af\" (UID: \"7331481c-0dd0-4acf-8c73-35a38eece3af\") " Nov 03 20:09:25.738908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:25.738716 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-tmp-dir" pod "7331481c-0dd0-4acf-8c73-35a38eece3af" (UID: "7331481c-0dd0-4acf-8c73-35a38eece3af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:25.738908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:25.738775 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-model-cache\") pod \"7331481c-0dd0-4acf-8c73-35a38eece3af\" (UID: \"7331481c-0dd0-4acf-8c73-35a38eece3af\") " Nov 03 20:09:25.738908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:25.738802 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-home" pod "7331481c-0dd0-4acf-8c73-35a38eece3af" (UID: "7331481c-0dd0-4acf-8c73-35a38eece3af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:25.739401 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:25.739374 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-model-cache" pod "7331481c-0dd0-4acf-8c73-35a38eece3af" (UID: "7331481c-0dd0-4acf-8c73-35a38eece3af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:25.739645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:25.739621 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:25.739645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:25.739646 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:25.739860 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:25.739676 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:25.742084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:25.742059 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-dshm" pod "7331481c-0dd0-4acf-8c73-35a38eece3af" (UID: "7331481c-0dd0-4acf-8c73-35a38eece3af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:25.742203 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:25.742066 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7331481c-0dd0-4acf-8c73-35a38eece3af-tls-certs" pod "7331481c-0dd0-4acf-8c73-35a38eece3af" (UID: "7331481c-0dd0-4acf-8c73-35a38eece3af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:09:25.743305 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:25.743281 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7331481c-0dd0-4acf-8c73-35a38eece3af-kube-api-access-bm7xb" pod "7331481c-0dd0-4acf-8c73-35a38eece3af" (UID: "7331481c-0dd0-4acf-8c73-35a38eece3af"). InnerVolumeSpecName "kube-api-access-bm7xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:09:25.841698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:25.841591 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7331481c-0dd0-4acf-8c73-35a38eece3af-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:25.841698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:25.841621 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7331481c-0dd0-4acf-8c73-35a38eece3af-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:25.841698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:25.841641 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm7xb\" (UniqueName: \"kubernetes.io/projected/7331481c-0dd0-4acf-8c73-35a38eece3af-kube-api-access-bm7xb\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:25.861426 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:25.861395 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34484729-bde1-4b0f-9758-e934b14f33d0" path="/var/lib/kubelet/pods/34484729-bde1-4b0f-9758-e934b14f33d0/volumes" Nov 03 20:09:26.434750 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:26.434712 5059 generic.go:356] "Generic (PLEG): container finished" podID="7331481c-0dd0-4acf-8c73-35a38eece3af" containerID="d081881808411dc88c6aff949d2bb38279ffa9fcdfcaf1e1805e834b89a1081c" exitCode=0 Nov 03 20:09:26.434750 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:26.434742 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" event={"ID":"7331481c-0dd0-4acf-8c73-35a38eece3af","Type":"ContainerDied","Data":"d081881808411dc88c6aff949d2bb38279ffa9fcdfcaf1e1805e834b89a1081c"} Nov 03 20:09:26.435427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:26.434780 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" event={"ID":"7331481c-0dd0-4acf-8c73-35a38eece3af","Type":"ContainerDied","Data":"1357d0cb48541bf119201f22ef60acc15a0272819d3ecd7ad11db6c9cb1c0ef2"} Nov 03 20:09:26.435427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:26.434797 5059 scope.go:122] "RemoveContainer" containerID="d081881808411dc88c6aff949d2bb38279ffa9fcdfcaf1e1805e834b89a1081c" Nov 03 20:09:26.435427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:26.434725 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx" Nov 03 20:09:26.445805 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:26.445683 5059 scope.go:122] "RemoveContainer" containerID="d081881808411dc88c6aff949d2bb38279ffa9fcdfcaf1e1805e834b89a1081c" Nov 03 20:09:26.445982 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:09:26.445953 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d081881808411dc88c6aff949d2bb38279ffa9fcdfcaf1e1805e834b89a1081c\": container with ID starting with d081881808411dc88c6aff949d2bb38279ffa9fcdfcaf1e1805e834b89a1081c not found: ID does not exist" containerID="d081881808411dc88c6aff949d2bb38279ffa9fcdfcaf1e1805e834b89a1081c" Nov 03 20:09:26.446108 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:26.445988 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d081881808411dc88c6aff949d2bb38279ffa9fcdfcaf1e1805e834b89a1081c"} err="failed to get container status \"d081881808411dc88c6aff949d2bb38279ffa9fcdfcaf1e1805e834b89a1081c\": rpc error: code = NotFound desc = could not find container \"d081881808411dc88c6aff949d2bb38279ffa9fcdfcaf1e1805e834b89a1081c\": container with ID starting with d081881808411dc88c6aff949d2bb38279ffa9fcdfcaf1e1805e834b89a1081c not found: ID does not exist" Nov 03 20:09:26.459692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:26.459652 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx"] Nov 03 20:09:26.464606 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:26.464585 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-88a18e/auth-test-model-88a18e-kserve-749fd8c9c-86ccx"] Nov 03 20:09:26.964354 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:26.964314 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f"] Nov 03 20:09:26.965435 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:26.965408 5059 cpu_manager.go:414] "containerMap: removing container" podUID="7331481c-0dd0-4acf-8c73-35a38eece3af" containerName="main" Nov 03 20:09:26.965435 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:26.965431 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7331481c-0dd0-4acf-8c73-35a38eece3af" containerName="main" Nov 03 20:09:26.965647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:26.965449 5059 cpu_manager.go:414] "containerMap: removing container" podUID="34484729-bde1-4b0f-9758-e934b14f33d0" containerName="istio-proxy" Nov 03 20:09:26.965647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:26.965457 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34484729-bde1-4b0f-9758-e934b14f33d0" containerName="istio-proxy" Nov 03 20:09:26.965647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:26.965514 5059 cpu_manager.go:414] "containerMap: removing container" podUID="fd6f9e73-1c5a-4151-a720-96f136b7b3a4" containerName="istio-proxy" Nov 03 20:09:26.965647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:26.965522 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd6f9e73-1c5a-4151-a720-96f136b7b3a4" containerName="istio-proxy" Nov 03 20:09:26.965881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:26.965746 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd6f9e73-1c5a-4151-a720-96f136b7b3a4" containerName="istio-proxy" Nov 03 20:09:26.965881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:26.965772 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7331481c-0dd0-4acf-8c73-35a38eece3af" containerName="main" Nov 03 20:09:26.965881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:26.965784 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="34484729-bde1-4b0f-9758-e934b14f33d0" containerName="istio-proxy" Nov 03 20:09:26.971015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:26.970975 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:26.984106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:26.984082 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0e1546-openshift-default-dockercfg-jcbdc\"" Nov 03 20:09:26.989519 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:26.989492 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f"] Nov 03 20:09:27.061556 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.061521 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-workload-certs\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.061716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.061646 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-workload-socket\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.061760 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.061729 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-data\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.061760 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.061749 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-token\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.061910 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.061891 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvlmp\" (UniqueName: \"kubernetes.io/projected/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-kube-api-access-pvlmp\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.061967 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.061955 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-envoy\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.062004 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.061976 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-credential-socket\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.062004 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.061998 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istiod-ca-cert\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.062104 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.062093 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-podinfo\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.167390 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.167346 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-workload-socket\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.168176 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.168152 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-data\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.189202 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.189175 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-token\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.189863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.189834 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvlmp\" (UniqueName: \"kubernetes.io/projected/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-kube-api-access-pvlmp\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.190115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.190094 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-envoy\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.190257 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.190240 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-credential-socket\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.190378 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.190363 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istiod-ca-cert\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.190533 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.190519 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-podinfo\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.205696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.200866 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-workload-certs\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.205696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.201384 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-workload-certs\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.205696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.204224 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-podinfo\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.205696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.177072 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-data\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.205696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.168089 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-workload-socket\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.206435 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.206411 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-credential-socket\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.225721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.207190 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istiod-ca-cert\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.236682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.230961 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-envoy\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.266689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.237189 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-token\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.283432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.283363 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvlmp\" (UniqueName: \"kubernetes.io/projected/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-kube-api-access-pvlmp\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.312096 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.304265 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:27.839765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.836222 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f"] Nov 03 20:09:27.871496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.871420 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7331481c-0dd0-4acf-8c73-35a38eece3af" path="/var/lib/kubelet/pods/7331481c-0dd0-4acf-8c73-35a38eece3af/volumes" Nov 03 20:09:27.875959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.875550 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:09:27.877021 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.875648 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:09:27.877021 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:27.876198 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:09:28.521692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:28.519251 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" event={"ID":"14a06bdf-e15d-406c-bb76-47ae5dd06b2d","Type":"ContainerStarted","Data":"56b80e220cc2df35563d8697c1e14db7387aa8d2faaa80e6b681d5be3a5c172e"} Nov 03 20:09:28.521692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:28.519377 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" event={"ID":"14a06bdf-e15d-406c-bb76-47ae5dd06b2d","Type":"ContainerStarted","Data":"145f74cb397cea989e22d486f3d9a839d7f366e5a050980469a0ae282b447e35"} Nov 03 20:09:29.326440 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:29.317788 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:29.379685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:29.375975 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:29.428235 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:29.428165 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" podStartSLOduration=3.428147444 podStartE2EDuration="3.428147444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:28.562992379 +0000 UTC m=+1962.820789648" watchObservedRunningTime="2026-11-03 20:09:29.428147444 +0000 UTC m=+1963.685944704" Nov 03 20:09:29.566686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:29.564601 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:29.597357 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:29.596973 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:09:31.383193 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:31.383160 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps" Nov 03 20:09:32.298530 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:32.298489 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44"] Nov 03 20:09:32.304500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:32.304473 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44"] Nov 03 20:09:32.304687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:32.304559 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44" Nov 03 20:09:32.310043 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:32.310021 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-0e1546-dockercfg-hwbh2\"" Nov 03 20:09:32.341155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:32.341118 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26375297-81f4-427f-94c3-d56f7a6a11fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44\" (UID: \"26375297-81f4-427f-94c3-d56f7a6a11fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44" Nov 03 20:09:32.341513 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:32.341492 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp2bg\" (UniqueName: \"kubernetes.io/projected/26375297-81f4-427f-94c3-d56f7a6a11fb-kube-api-access-pp2bg\") pod \"payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44\" (UID: \"26375297-81f4-427f-94c3-d56f7a6a11fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44" Nov 03 20:09:32.400130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:32.399942 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0e1546\"" Nov 03 20:09:32.444687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:32.443899 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp2bg\" (UniqueName: \"kubernetes.io/projected/26375297-81f4-427f-94c3-d56f7a6a11fb-kube-api-access-pp2bg\") pod \"payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44\" (UID: \"26375297-81f4-427f-94c3-d56f7a6a11fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44" Nov 03 20:09:32.444687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:32.444050 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26375297-81f4-427f-94c3-d56f7a6a11fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44\" (UID: \"26375297-81f4-427f-94c3-d56f7a6a11fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44" Nov 03 20:09:32.445129 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:32.445099 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26375297-81f4-427f-94c3-d56f7a6a11fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44\" (UID: \"26375297-81f4-427f-94c3-d56f7a6a11fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44" Nov 03 20:09:32.492687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:32.488652 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp2bg\" (UniqueName: \"kubernetes.io/projected/26375297-81f4-427f-94c3-d56f7a6a11fb-kube-api-access-pp2bg\") pod \"payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44\" (UID: \"26375297-81f4-427f-94c3-d56f7a6a11fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44" Nov 03 20:09:32.637025 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:32.636992 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44" Nov 03 20:09:33.091591 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:09:33.091553 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26375297_81f4_427f_94c3_d56f7a6a11fb.slice/crio-e99e44e38a56617ba12741f6caaea0dcc706de1443380b8c96103d96c6112496 WatchSource:0}: Error finding container e99e44e38a56617ba12741f6caaea0dcc706de1443380b8c96103d96c6112496: Status 404 returned error can't find the container with id e99e44e38a56617ba12741f6caaea0dcc706de1443380b8c96103d96c6112496 Nov 03 20:09:33.095249 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:33.095224 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44"] Nov 03 20:09:33.785226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:33.785188 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44" event={"ID":"26375297-81f4-427f-94c3-d56f7a6a11fb","Type":"ContainerStarted","Data":"93172d79a58c2d8458ec95cdf845557fbb10e251f6cf98dda510ae64f827325b"} Nov 03 20:09:33.785695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:33.785254 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44" event={"ID":"26375297-81f4-427f-94c3-d56f7a6a11fb","Type":"ContainerStarted","Data":"e99e44e38a56617ba12741f6caaea0dcc706de1443380b8c96103d96c6112496"} Nov 03 20:09:33.786126 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:33.786105 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44" Nov 03 20:09:33.851683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:33.847329 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44" podStartSLOduration=1.847306927 podStartE2EDuration="1.847306927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:33.803677388 +0000 UTC m=+1968.061474658" watchObservedRunningTime="2026-11-03 20:09:33.847306927 +0000 UTC m=+1968.105104184" Nov 03 20:09:40.609086 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:09:40.608998 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:09:41.937245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:41.937196 5059 generic.go:356] "Generic (PLEG): container finished" podID="a594e899-889e-4f7f-a8ba-b7e7a389ba8a" containerID="6eff9a042806518844e626a38bbf88994cc517a23f4a2adcbbff67af78d07b64" exitCode=0 Nov 03 20:09:41.937730 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:41.937266 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" event={"ID":"a594e899-889e-4f7f-a8ba-b7e7a389ba8a","Type":"ContainerDied","Data":"6eff9a042806518844e626a38bbf88994cc517a23f4a2adcbbff67af78d07b64"} Nov 03 20:09:41.937730 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:41.937685 5059 scope.go:122] "RemoveContainer" containerID="6eff9a042806518844e626a38bbf88994cc517a23f4a2adcbbff67af78d07b64" Nov 03 20:09:41.939626 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:41.939591 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:09:41.939743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:41.939699 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:09:41.939792 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:41.939753 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:09:42.950477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:42.950427 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" event={"ID":"a594e899-889e-4f7f-a8ba-b7e7a389ba8a","Type":"ContainerStarted","Data":"034deb3d094b43323ad58f8415dd48a1c89f624ddffe2676cc27ac841dc9e817"} Nov 03 20:09:43.641824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:43.641783 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:09:43.642049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:43.641995 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Nov 03 20:09:43.642116 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:43.642049 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" podUID="a594e899-889e-4f7f-a8ba-b7e7a389ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Nov 03 20:09:44.641344 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:44.641310 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Nov 03 20:09:44.641822 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:44.641374 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" podUID="a594e899-889e-4f7f-a8ba-b7e7a389ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Nov 03 20:09:45.641808 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:45.641768 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Nov 03 20:09:45.642443 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:45.642410 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" podUID="a594e899-889e-4f7f-a8ba-b7e7a389ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Nov 03 20:09:45.935327 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:45.935257 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44" Nov 03 20:09:46.641750 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:46.641707 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Nov 03 20:09:46.641957 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:46.641775 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" podUID="a594e899-889e-4f7f-a8ba-b7e7a389ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Nov 03 20:09:46.887735 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:46.887651 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676"] Nov 03 20:09:46.893810 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:46.893742 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676"] Nov 03 20:09:46.893950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:46.893826 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676" Nov 03 20:09:46.900796 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:46.900769 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"kube-root-ca.crt\"" Nov 03 20:09:46.900932 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:46.900840 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"payload-processing-external-model-plugins-e2e-extmod-b-46a6d7\"" Nov 03 20:09:46.900932 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:46.900886 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"routing-overlay\"" Nov 03 20:09:46.900932 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:46.900897 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"openshift-service-ca.crt\"" Nov 03 20:09:46.901194 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:46.900948 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"agc-cred-9d0b4d393cba\"" Nov 03 20:09:46.901194 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:46.901048 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"payload-processing-external-model-e2e--a1bcdb2a-dockercfg-5sh7w\"" Nov 03 20:09:47.186738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:47.180881 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7gzs\" (UniqueName: \"kubernetes.io/projected/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-kube-api-access-q7gzs\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676\" (UID: \"aee62de0-0be9-4d5c-9158-c1ca8e86a9be\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676" Nov 03 20:09:47.186738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:47.181076 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676\" (UID: \"aee62de0-0be9-4d5c-9158-c1ca8e86a9be\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676" Nov 03 20:09:47.186738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:47.181143 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676\" (UID: \"aee62de0-0be9-4d5c-9158-c1ca8e86a9be\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676" Nov 03 20:09:47.186738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:47.181209 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676\" (UID: \"aee62de0-0be9-4d5c-9158-c1ca8e86a9be\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676" Nov 03 20:09:47.294684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:47.291871 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7gzs\" (UniqueName: \"kubernetes.io/projected/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-kube-api-access-q7gzs\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676\" (UID: \"aee62de0-0be9-4d5c-9158-c1ca8e86a9be\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676" Nov 03 20:09:47.294684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:47.292076 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676\" (UID: \"aee62de0-0be9-4d5c-9158-c1ca8e86a9be\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676" Nov 03 20:09:47.294684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:47.292155 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676\" (UID: \"aee62de0-0be9-4d5c-9158-c1ca8e86a9be\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676" Nov 03 20:09:47.308617 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:47.308580 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676\" (UID: \"aee62de0-0be9-4d5c-9158-c1ca8e86a9be\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676" Nov 03 20:09:47.310155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:47.310121 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676\" (UID: \"aee62de0-0be9-4d5c-9158-c1ca8e86a9be\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676" Nov 03 20:09:47.311077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:47.311045 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676\" (UID: \"aee62de0-0be9-4d5c-9158-c1ca8e86a9be\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676" Nov 03 20:09:47.334537 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:47.334499 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676\" (UID: \"aee62de0-0be9-4d5c-9158-c1ca8e86a9be\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676" Nov 03 20:09:47.358152 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:47.345095 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7gzs\" (UniqueName: \"kubernetes.io/projected/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-kube-api-access-q7gzs\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676\" (UID: \"aee62de0-0be9-4d5c-9158-c1ca8e86a9be\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676" Nov 03 20:09:47.511935 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:47.511847 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676" Nov 03 20:09:47.641637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:47.641218 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Nov 03 20:09:47.641637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:47.641274 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" podUID="a594e899-889e-4f7f-a8ba-b7e7a389ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Nov 03 20:09:47.852591 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:47.845623 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676"] Nov 03 20:09:47.914204 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:09:47.914151 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee62de0_0be9_4d5c_9158_c1ca8e86a9be.slice/crio-e76fd6409bfd6b6fc2f3437d78006f7a976b292c975333b2831e25f3fc0b2bb0 WatchSource:0}: Error finding container e76fd6409bfd6b6fc2f3437d78006f7a976b292c975333b2831e25f3fc0b2bb0: Status 404 returned error can't find the container with id e76fd6409bfd6b6fc2f3437d78006f7a976b292c975333b2831e25f3fc0b2bb0 Nov 03 20:09:48.236466 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:48.236433 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676" event={"ID":"aee62de0-0be9-4d5c-9158-c1ca8e86a9be","Type":"ContainerStarted","Data":"3530d5e98f6dd413c771b6ab06e6e590f0758c03e86b6ba6bd7ea14915534a19"} Nov 03 20:09:48.236704 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:48.236493 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676" event={"ID":"aee62de0-0be9-4d5c-9158-c1ca8e86a9be","Type":"ContainerStarted","Data":"e76fd6409bfd6b6fc2f3437d78006f7a976b292c975333b2831e25f3fc0b2bb0"} Nov 03 20:09:48.236704 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:48.236521 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676" Nov 03 20:09:48.286877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:48.286821 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676" podStartSLOduration=2.286806698 podStartE2EDuration="2.286806698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:48.274251924 +0000 UTC m=+1982.532049182" watchObservedRunningTime="2026-11-03 20:09:48.286806698 +0000 UTC m=+1982.544603955" Nov 03 20:09:48.657695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:48.654123 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Nov 03 20:09:48.657695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:48.654195 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" podUID="a594e899-889e-4f7f-a8ba-b7e7a389ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Nov 03 20:09:49.550478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:49.550375 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:09:49.652976 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:49.652934 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:09:50.256241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:50.256207 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:09:54.466684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.458104 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x"] Nov 03 20:09:54.570721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.569957 5059 scope.go:122] "RemoveContainer" containerID="ac82175b4c92dd7084229550834cdd419f955a76a25aacbc757f7a5944d129fe" Nov 03 20:09:54.603009 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.598940 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.619681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.613980 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0e4ce7-openshift-default-dockercfg-nppzh\"" Nov 03 20:09:54.619681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.598831 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x"] Nov 03 20:09:54.695688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.675729 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:09:54.708688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.701397 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-workload-socket\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.708688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.705472 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-credential-socket\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.708688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.705552 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zksbz\" (UniqueName: \"kubernetes.io/projected/7909b42c-1572-4fe2-816a-447020594fe4-kube-api-access-zksbz\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.708688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.705596 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-istio-envoy\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.708688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.705626 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7909b42c-1572-4fe2-816a-447020594fe4-istio-token\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.708688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.705718 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-istio-data\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.708688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.705836 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7909b42c-1572-4fe2-816a-447020594fe4-istiod-ca-cert\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.708688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.705895 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7909b42c-1572-4fe2-816a-447020594fe4-istio-podinfo\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.708688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.706017 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-workload-certs\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.826116 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.813555 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-istio-data\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.826116 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.813735 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7909b42c-1572-4fe2-816a-447020594fe4-istiod-ca-cert\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.826116 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.813797 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7909b42c-1572-4fe2-816a-447020594fe4-istio-podinfo\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.826116 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.813897 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-workload-certs\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.826116 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.814002 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-workload-socket\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.826116 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.814072 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-credential-socket\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.826116 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.814138 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zksbz\" (UniqueName: \"kubernetes.io/projected/7909b42c-1572-4fe2-816a-447020594fe4-kube-api-access-zksbz\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.826116 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.814182 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-istio-envoy\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.826116 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.814215 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7909b42c-1572-4fe2-816a-447020594fe4-istio-token\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.849527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.831977 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-istio-data\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.849527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.846809 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-workload-certs\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.849527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.847063 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-workload-socket\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.849527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.847275 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-credential-socket\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.850498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.850471 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7909b42c-1572-4fe2-816a-447020594fe4-istiod-ca-cert\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.922695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.921780 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-istio-envoy\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.930693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.924237 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7909b42c-1572-4fe2-816a-447020594fe4-istio-podinfo\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.930693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.924770 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7909b42c-1572-4fe2-816a-447020594fe4-istio-token\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.930693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.930307 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zksbz\" (UniqueName: \"kubernetes.io/projected/7909b42c-1572-4fe2-816a-447020594fe4-kube-api-access-zksbz\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:54.948694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:54.946112 5059 scope.go:122] "RemoveContainer" containerID="33b2fa04c0c89641a426819ffbf53835935a93b8ed77c9e64765b51cf4eaa1e3" Nov 03 20:09:55.181272 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:55.181016 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:56.446092 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:56.445655 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x"] Nov 03 20:09:56.446473 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:09:56.446142 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7909b42c_1572_4fe2_816a_447020594fe4.slice/crio-0f34b01f77f5c5dfd37d65b1c806d09a59df2776b37e895036af224434dd2faa WatchSource:0}: Error finding container 0f34b01f77f5c5dfd37d65b1c806d09a59df2776b37e895036af224434dd2faa: Status 404 returned error can't find the container with id 0f34b01f77f5c5dfd37d65b1c806d09a59df2776b37e895036af224434dd2faa Nov 03 20:09:56.449442 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:56.449406 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:09:56.449569 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:56.449500 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:09:56.449644 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:56.449564 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:09:56.797959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:56.795487 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" event={"ID":"7909b42c-1572-4fe2-816a-447020594fe4","Type":"ContainerStarted","Data":"0f34b01f77f5c5dfd37d65b1c806d09a59df2776b37e895036af224434dd2faa"} Nov 03 20:09:58.021012 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.009639 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" event={"ID":"7909b42c-1572-4fe2-816a-447020594fe4","Type":"ContainerStarted","Data":"f2e801a6eee34b07dcc874c7303b6ec332d7470d1b006afc0f98c9a4ae0ba486"} Nov 03 20:09:58.058684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.058182 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" podStartSLOduration=5.058155282 podStartE2EDuration="5.058155282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:58.053139136 +0000 UTC m=+1992.310936430" watchObservedRunningTime="2026-11-03 20:09:58.058155282 +0000 UTC m=+1992.315952539" Nov 03 20:09:58.196680 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.196162 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:58.272741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.271731 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:58.648550 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.648513 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p"] Nov 03 20:09:58.674686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.668025 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:09:58.732690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.712158 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-36556d\"/\"default-dockercfg-xp2s4\"" Nov 03 20:09:58.755216 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.755183 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-36556d\"/\"openshift-service-ca.crt\"" Nov 03 20:09:58.755404 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.755238 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-36556d\"/\"kube-root-ca.crt\"" Nov 03 20:09:58.756409 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.756379 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p"] Nov 03 20:09:58.784808 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.781208 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9ed680-cec0-4abe-9d39-40c70da96e55-tls-certs\") pod \"autores-model-36556d-kserve-cb957bc65-bjp8p\" (UID: \"bb9ed680-cec0-4abe-9d39-40c70da96e55\") " pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:09:58.784808 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.781260 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-home\") pod \"autores-model-36556d-kserve-cb957bc65-bjp8p\" (UID: \"bb9ed680-cec0-4abe-9d39-40c70da96e55\") " pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:09:58.784808 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.781295 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-tmp-dir\") pod \"autores-model-36556d-kserve-cb957bc65-bjp8p\" (UID: \"bb9ed680-cec0-4abe-9d39-40c70da96e55\") " pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:09:58.784808 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.781372 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-dshm\") pod \"autores-model-36556d-kserve-cb957bc65-bjp8p\" (UID: \"bb9ed680-cec0-4abe-9d39-40c70da96e55\") " pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:09:58.784808 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.781426 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7glgb\" (UniqueName: \"kubernetes.io/projected/bb9ed680-cec0-4abe-9d39-40c70da96e55-kube-api-access-7glgb\") pod \"autores-model-36556d-kserve-cb957bc65-bjp8p\" (UID: \"bb9ed680-cec0-4abe-9d39-40c70da96e55\") " pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:09:58.784808 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.781609 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-model-cache\") pod \"autores-model-36556d-kserve-cb957bc65-bjp8p\" (UID: \"bb9ed680-cec0-4abe-9d39-40c70da96e55\") " pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:09:58.784808 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.782002 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-36556d\"/\"autores-model-36556d-kserve-self-signed-certs\"" Nov 03 20:09:58.922611 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.922523 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-model-cache\") pod \"autores-model-36556d-kserve-cb957bc65-bjp8p\" (UID: \"bb9ed680-cec0-4abe-9d39-40c70da96e55\") " pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:09:58.923090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.923063 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9ed680-cec0-4abe-9d39-40c70da96e55-tls-certs\") pod \"autores-model-36556d-kserve-cb957bc65-bjp8p\" (UID: \"bb9ed680-cec0-4abe-9d39-40c70da96e55\") " pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:09:58.923235 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.923221 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-home\") pod \"autores-model-36556d-kserve-cb957bc65-bjp8p\" (UID: \"bb9ed680-cec0-4abe-9d39-40c70da96e55\") " pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:09:58.923362 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.923349 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-tmp-dir\") pod \"autores-model-36556d-kserve-cb957bc65-bjp8p\" (UID: \"bb9ed680-cec0-4abe-9d39-40c70da96e55\") " pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:09:58.935838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.935805 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-dshm\") pod \"autores-model-36556d-kserve-cb957bc65-bjp8p\" (UID: \"bb9ed680-cec0-4abe-9d39-40c70da96e55\") " pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:09:58.936066 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.936045 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7glgb\" (UniqueName: \"kubernetes.io/projected/bb9ed680-cec0-4abe-9d39-40c70da96e55-kube-api-access-7glgb\") pod \"autores-model-36556d-kserve-cb957bc65-bjp8p\" (UID: \"bb9ed680-cec0-4abe-9d39-40c70da96e55\") " pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:09:58.961685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.937091 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-model-cache\") pod \"autores-model-36556d-kserve-cb957bc65-bjp8p\" (UID: \"bb9ed680-cec0-4abe-9d39-40c70da96e55\") " pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:09:59.085769 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.944413 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-home\") pod \"autores-model-36556d-kserve-cb957bc65-bjp8p\" (UID: \"bb9ed680-cec0-4abe-9d39-40c70da96e55\") " pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:09:59.086353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:58.944657 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-tmp-dir\") pod \"autores-model-36556d-kserve-cb957bc65-bjp8p\" (UID: \"bb9ed680-cec0-4abe-9d39-40c70da96e55\") " pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:09:59.206730 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:59.198883 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-dshm\") pod \"autores-model-36556d-kserve-cb957bc65-bjp8p\" (UID: \"bb9ed680-cec0-4abe-9d39-40c70da96e55\") " pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:09:59.230685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:59.229407 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9ed680-cec0-4abe-9d39-40c70da96e55-tls-certs\") pod \"autores-model-36556d-kserve-cb957bc65-bjp8p\" (UID: \"bb9ed680-cec0-4abe-9d39-40c70da96e55\") " pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:09:59.239718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:59.236842 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7glgb\" (UniqueName: \"kubernetes.io/projected/bb9ed680-cec0-4abe-9d39-40c70da96e55-kube-api-access-7glgb\") pod \"autores-model-36556d-kserve-cb957bc65-bjp8p\" (UID: \"bb9ed680-cec0-4abe-9d39-40c70da96e55\") " pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:09:59.245290 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:59.245259 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:59.253252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:59.252868 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:09:59.255106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:59.254323 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676" Nov 03 20:09:59.404686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:59.404574 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:09:59.776135 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:09:59.776100 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb9ed680_cec0_4abe_9d39_40c70da96e55.slice/crio-f7f8d07976a02e68f4aef87cd0b3fd5b9c3d273360cd1c6cae622dce93b5c8b6 WatchSource:0}: Error finding container f7f8d07976a02e68f4aef87cd0b3fd5b9c3d273360cd1c6cae622dce93b5c8b6: Status 404 returned error can't find the container with id f7f8d07976a02e68f4aef87cd0b3fd5b9c3d273360cd1c6cae622dce93b5c8b6 Nov 03 20:09:59.777859 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:09:59.777834 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p"] Nov 03 20:10:00.256867 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:00.256828 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" event={"ID":"bb9ed680-cec0-4abe-9d39-40c70da96e55","Type":"ContainerStarted","Data":"dbaaceaee7f969aeea4cdefe42d387cba2eb64d7c5306e79351bfa4c0e23d85f"} Nov 03 20:10:00.257355 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:00.256884 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" event={"ID":"bb9ed680-cec0-4abe-9d39-40c70da96e55","Type":"ContainerStarted","Data":"f7f8d07976a02e68f4aef87cd0b3fd5b9c3d273360cd1c6cae622dce93b5c8b6"} Nov 03 20:10:00.290082 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:00.290021 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" podStartSLOduration=2.290002023 podStartE2EDuration="2.290002023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:00.28261757 +0000 UTC m=+1994.540414829" watchObservedRunningTime="2026-11-03 20:10:00.290002023 +0000 UTC m=+1994.547799282" Nov 03 20:10:00.417361 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:00.406859 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:10:01.803980 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:01.803426 5059 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:10:01.803980 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:01.803522 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" podUID="4be209ff-6d62-42d5-a0e3-22373143ecf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:10:01.803980 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:01.803872 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:10:01.803980 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:01.803911 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" podUID="9d60eb56-9852-4c25-ae50-8a932e752793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:10:02.462835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:02.462785 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:10:02.463005 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:02.462861 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" podUID="5b30b4c4-debd-4919-a4e6-67f7ee38eb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:10:08.389233 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.389189 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn"] Nov 03 20:10:08.731353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.731276 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn"] Nov 03 20:10:08.731745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.731655 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.793002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.783328 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-token\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.793497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.793464 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-credential-socket\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.793726 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.793699 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-workload-certs\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.793965 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.793942 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-data\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.794231 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.794210 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-podinfo\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.794514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.794495 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-workload-socket\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.794749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.794729 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2779\" (UniqueName: \"kubernetes.io/projected/08b3a993-1a64-4fe5-9577-b0f11a36779e-kube-api-access-d2779\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.794991 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.794969 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b3a993-1a64-4fe5-9577-b0f11a36779e-istiod-ca-cert\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.795211 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.795193 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-envoy\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.812103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.812074 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-b503a8-openshift-default-dockercfg-tchtj\"" Nov 03 20:10:08.900653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.897968 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-credential-socket\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.900653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.898039 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-workload-certs\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.900653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.898076 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-data\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.900653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.898258 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-podinfo\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.900653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.898324 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-workload-socket\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.900653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.898359 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2779\" (UniqueName: \"kubernetes.io/projected/08b3a993-1a64-4fe5-9577-b0f11a36779e-kube-api-access-d2779\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.900653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.898426 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b3a993-1a64-4fe5-9577-b0f11a36779e-istiod-ca-cert\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.900653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.898471 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-envoy\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.900653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.898573 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-token\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.900653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.899900 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-credential-socket\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.900653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.900148 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-workload-certs\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.921265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.920447 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-data\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.921730 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.921633 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b3a993-1a64-4fe5-9577-b0f11a36779e-istiod-ca-cert\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.921877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.921828 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-workload-socket\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.933175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.933049 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2779\" (UniqueName: \"kubernetes.io/projected/08b3a993-1a64-4fe5-9577-b0f11a36779e-kube-api-access-d2779\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.933370 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.933344 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-envoy\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.933475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.933433 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-podinfo\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:08.933631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:08.933609 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-token\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:09.229802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:09.228656 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:09.407253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:09.407216 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:10:09.420467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:09.420442 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:10:10.412897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:10.412867 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:10:11.289828 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:10:11.289790 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:10:11.609333 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:11.609275 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn"] Nov 03 20:10:11.612979 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:10:11.612942 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b3a993_1a64_4fe5_9577_b0f11a36779e.slice/crio-945b49aab43d7640ffe6f9eabe8651bd8c94dc095e160eb5fb327ca5d696ada1 WatchSource:0}: Error finding container 945b49aab43d7640ffe6f9eabe8651bd8c94dc095e160eb5fb327ca5d696ada1: Status 404 returned error can't find the container with id 945b49aab43d7640ffe6f9eabe8651bd8c94dc095e160eb5fb327ca5d696ada1 Nov 03 20:10:11.616550 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:11.616512 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:10:11.616646 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:11.616594 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:10:11.616646 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:11.616639 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:10:12.428719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:12.428676 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" event={"ID":"08b3a993-1a64-4fe5-9577-b0f11a36779e","Type":"ContainerStarted","Data":"3f07fce3b6aeb311a72b724e58639936894c5cc3f1b194d161e8c7b428a68157"} Nov 03 20:10:12.429004 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:12.428746 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" event={"ID":"08b3a993-1a64-4fe5-9577-b0f11a36779e","Type":"ContainerStarted","Data":"945b49aab43d7640ffe6f9eabe8651bd8c94dc095e160eb5fb327ca5d696ada1"} Nov 03 20:10:12.484512 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:12.484452 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" podStartSLOduration=4.484431576 podStartE2EDuration="4.484431576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:12.478108597 +0000 UTC m=+2006.735905861" watchObservedRunningTime="2026-11-03 20:10:12.484431576 +0000 UTC m=+2006.742228833" Nov 03 20:10:12.756713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:12.756591 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr"] Nov 03 20:10:12.833808 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:12.833776 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr"] Nov 03 20:10:12.834015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:12.833882 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:12.918490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:12.918454 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-token\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:12.918703 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:12.918618 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-credential-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:12.918791 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:12.918771 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-workload-certs\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:12.918871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:12.918857 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-envoy\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:12.919065 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:12.919044 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-podinfo\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:12.919139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:12.919115 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb966fa-cc63-40e1-9cb2-aabb05c56717-istiod-ca-cert\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:12.919221 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:12.919203 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh2c4\" (UniqueName: \"kubernetes.io/projected/beb966fa-cc63-40e1-9cb2-aabb05c56717-kube-api-access-fh2c4\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:12.919292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:12.919277 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-data\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:12.919343 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:12.919311 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-workload-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:13.020681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.020580 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-token\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:13.020865 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.020683 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-credential-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:13.020865 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.020753 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-workload-certs\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:13.020865 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.020801 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-envoy\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:13.021048 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.020926 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-podinfo\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:13.021048 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.020999 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb966fa-cc63-40e1-9cb2-aabb05c56717-istiod-ca-cert\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:13.021048 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.021034 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh2c4\" (UniqueName: \"kubernetes.io/projected/beb966fa-cc63-40e1-9cb2-aabb05c56717-kube-api-access-fh2c4\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:13.021207 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.021066 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-data\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:13.021207 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.021099 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-workload-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:13.021207 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.021118 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-credential-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:13.021207 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.021184 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-workload-certs\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:13.021434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.021406 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-workload-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:13.021510 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.021487 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-data\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:13.022026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.022003 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb966fa-cc63-40e1-9cb2-aabb05c56717-istiod-ca-cert\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:13.023960 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.023937 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-envoy\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:13.024115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.024090 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-podinfo\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:13.033165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.033140 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh2c4\" (UniqueName: \"kubernetes.io/projected/beb966fa-cc63-40e1-9cb2-aabb05c56717-kube-api-access-fh2c4\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:13.033535 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.033511 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-token\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-bzpxr\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:13.194033 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.193999 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:13.229359 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.229333 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:13.236215 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.236190 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:13.444553 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.442144 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:13.444553 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.443084 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:10:13.556727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.556608 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m"] Nov 03 20:10:13.609632 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.609591 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m"] Nov 03 20:10:13.609909 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.609846 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.700390 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:10:13.700346 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb966fa_cc63_40e1_9cb2_aabb05c56717.slice/crio-621c9695ae2f64df2abb541ccd57ddbed4c41a25d6e12f6ef69d87cb18cb9682 WatchSource:0}: Error finding container 621c9695ae2f64df2abb541ccd57ddbed4c41a25d6e12f6ef69d87cb18cb9682: Status 404 returned error can't find the container with id 621c9695ae2f64df2abb541ccd57ddbed4c41a25d6e12f6ef69d87cb18cb9682 Nov 03 20:10:13.701829 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.701799 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr"] Nov 03 20:10:13.706977 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.706316 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:10:13.706977 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.706406 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:10:13.706977 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.706453 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:10:13.737240 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.737207 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.737359 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.737274 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15cd101-19a6-4a3c-8381-b175e1182bce-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.737429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.737414 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.737491 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.737460 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.737564 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.737492 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.737626 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.737575 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.737732 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.737653 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.737797 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.737783 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.737857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.737803 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9zfr\" (UniqueName: \"kubernetes.io/projected/c15cd101-19a6-4a3c-8381-b175e1182bce-kube-api-access-g9zfr\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.839183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.839145 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.839614 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.839288 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.839614 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.839395 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.839614 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.839428 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9zfr\" (UniqueName: \"kubernetes.io/projected/c15cd101-19a6-4a3c-8381-b175e1182bce-kube-api-access-g9zfr\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.839614 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.839557 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.839878 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.839644 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15cd101-19a6-4a3c-8381-b175e1182bce-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.839878 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.839696 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.839878 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.839757 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.839878 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.839786 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.840257 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.840233 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.840837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.840815 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.841483 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.841461 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15cd101-19a6-4a3c-8381-b175e1182bce-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.842295 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.842136 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.842295 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.842246 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.842518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.842497 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.844458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.844372 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.860346 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.860318 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:13.861355 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:13.861331 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9zfr\" (UniqueName: \"kubernetes.io/projected/c15cd101-19a6-4a3c-8381-b175e1182bce-kube-api-access-g9zfr\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:14.015969 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:14.015927 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:14.304714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:14.304646 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p"] Nov 03 20:10:14.369708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:14.369610 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m"] Nov 03 20:10:14.370810 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:10:14.370739 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15cd101_19a6_4a3c_8381_b175e1182bce.slice/crio-8031ebfdf68626e4eadfd7cfa669c3bccde1ce227c7c6df6a2760fb6ae1b8f49 WatchSource:0}: Error finding container 8031ebfdf68626e4eadfd7cfa669c3bccde1ce227c7c6df6a2760fb6ae1b8f49: Status 404 returned error can't find the container with id 8031ebfdf68626e4eadfd7cfa669c3bccde1ce227c7c6df6a2760fb6ae1b8f49 Nov 03 20:10:14.374019 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:14.373983 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:10:14.374130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:14.374050 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:10:14.374130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:14.374078 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:10:14.550731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:14.530145 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" event={"ID":"c15cd101-19a6-4a3c-8381-b175e1182bce","Type":"ContainerStarted","Data":"8031ebfdf68626e4eadfd7cfa669c3bccde1ce227c7c6df6a2760fb6ae1b8f49"} Nov 03 20:10:14.550731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:14.541494 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" event={"ID":"beb966fa-cc63-40e1-9cb2-aabb05c56717","Type":"ContainerStarted","Data":"7eef1e1a85c69dbc87d3f8efa0178f9ab2722d386dcaef0fd1da52e567ab07c1"} Nov 03 20:10:14.550731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:14.541554 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" event={"ID":"beb966fa-cc63-40e1-9cb2-aabb05c56717","Type":"ContainerStarted","Data":"621c9695ae2f64df2abb541ccd57ddbed4c41a25d6e12f6ef69d87cb18cb9682"} Nov 03 20:10:15.196993 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:15.194849 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:15.198004 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:15.197629 5059 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Nov 03 20:10:15.198004 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:15.197706 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" podUID="beb966fa-cc63-40e1-9cb2-aabb05c56717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Nov 03 20:10:15.577785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:15.574293 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" event={"ID":"c15cd101-19a6-4a3c-8381-b175e1182bce","Type":"ContainerStarted","Data":"80953a1b9a91f162514cd4bebdcb490c46dfb0ce230b8b37a318d9925a83f540"} Nov 03 20:10:15.647687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:15.646041 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" podStartSLOduration=3.646021015 podStartE2EDuration="3.646021015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:14.637161991 +0000 UTC m=+2008.894959249" watchObservedRunningTime="2026-11-03 20:10:15.646021015 +0000 UTC m=+2009.903818272" Nov 03 20:10:15.651223 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:15.651129 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" podStartSLOduration=2.651111286 podStartE2EDuration="2.651111286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:15.645167244 +0000 UTC m=+2009.902964501" watchObservedRunningTime="2026-11-03 20:10:15.651111286 +0000 UTC m=+2009.908908542" Nov 03 20:10:16.020078 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:16.017902 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:16.020649 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:16.020348 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Nov 03 20:10:16.020649 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:16.020404 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" podUID="c15cd101-19a6-4a3c-8381-b175e1182bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Nov 03 20:10:16.226271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:16.225639 5059 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Nov 03 20:10:16.226271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:16.225756 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" podUID="beb966fa-cc63-40e1-9cb2-aabb05c56717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Nov 03 20:10:16.815050 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:16.813787 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:10:16.815050 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:16.813853 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" podUID="9d60eb56-9852-4c25-ae50-8a932e752793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:10:16.815050 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:16.814941 5059 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:10:16.815050 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:16.814988 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" podUID="4be209ff-6d62-42d5-a0e3-22373143ecf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:10:17.017115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:17.017084 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Nov 03 20:10:17.017405 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:17.017385 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" podUID="c15cd101-19a6-4a3c-8381-b175e1182bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Nov 03 20:10:17.209245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:17.208776 5059 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Nov 03 20:10:17.209245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:17.208846 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" podUID="beb966fa-cc63-40e1-9cb2-aabb05c56717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Nov 03 20:10:17.548732 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:17.547829 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:10:17.548732 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:17.547913 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" podUID="5b30b4c4-debd-4919-a4e6-67f7ee38eb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:10:18.016899 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:18.016520 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Nov 03 20:10:18.016899 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:18.016576 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" podUID="c15cd101-19a6-4a3c-8381-b175e1182bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Nov 03 20:10:18.200944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:18.199101 5059 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Nov 03 20:10:18.200944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:18.199162 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" podUID="beb966fa-cc63-40e1-9cb2-aabb05c56717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Nov 03 20:10:19.016694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:19.016615 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Nov 03 20:10:19.016694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:19.016689 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" podUID="c15cd101-19a6-4a3c-8381-b175e1182bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Nov 03 20:10:19.195620 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:19.195100 5059 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Nov 03 20:10:19.195620 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:19.195177 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" podUID="beb966fa-cc63-40e1-9cb2-aabb05c56717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Nov 03 20:10:20.017612 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:20.017193 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Nov 03 20:10:20.017612 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:20.017252 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" podUID="c15cd101-19a6-4a3c-8381-b175e1182bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Nov 03 20:10:20.200685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:20.200046 5059 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Nov 03 20:10:20.200685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:20.200147 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" podUID="beb966fa-cc63-40e1-9cb2-aabb05c56717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Nov 03 20:10:21.029108 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:21.029076 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Nov 03 20:10:21.029620 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:21.029597 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" podUID="c15cd101-19a6-4a3c-8381-b175e1182bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Nov 03 20:10:21.031722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:21.031601 5059 generic.go:356] "Generic (PLEG): container finished" podID="9d60eb56-9852-4c25-ae50-8a932e752793" containerID="6dc7ad3073c40aab49d043739b212390e9623e46a6f995b1c1ca2fa55cf79644" exitCode=0 Nov 03 20:10:21.031722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:21.031700 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" event={"ID":"9d60eb56-9852-4c25-ae50-8a932e752793","Type":"ContainerDied","Data":"6dc7ad3073c40aab49d043739b212390e9623e46a6f995b1c1ca2fa55cf79644"} Nov 03 20:10:21.034685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:21.032159 5059 scope.go:122] "RemoveContainer" containerID="6dc7ad3073c40aab49d043739b212390e9623e46a6f995b1c1ca2fa55cf79644" Nov 03 20:10:21.034685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:21.034309 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:10:21.034685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:21.034376 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:10:21.034685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:21.034408 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:10:21.202902 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:21.197992 5059 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Nov 03 20:10:21.202902 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:21.198054 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" podUID="beb966fa-cc63-40e1-9cb2-aabb05c56717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Nov 03 20:10:22.048160 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:22.048113 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" event={"ID":"9d60eb56-9852-4c25-ae50-8a932e752793","Type":"ContainerStarted","Data":"1800f61ab969d2f54568614237d89ac56416a680367326b1f485f3183aebbfe9"} Nov 03 20:10:22.078768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:22.078697 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:22.080285 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:22.080138 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:22.081165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:22.081137 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:10:22.208406 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:22.208377 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:22.209564 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:22.208999 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:22.214706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:22.210998 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:10:22.938280 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:22.938247 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:10:22.939349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:22.939320 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 20:10:22.939508 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:22.939370 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" podUID="9d60eb56-9852-4c25-ae50-8a932e752793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 20:10:23.938555 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:23.938516 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 20:10:23.938995 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:23.938581 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" podUID="9d60eb56-9852-4c25-ae50-8a932e752793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 20:10:24.088711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:24.088454 5059 generic.go:356] "Generic (PLEG): container finished" podID="5b30b4c4-debd-4919-a4e6-67f7ee38eb90" containerID="552ec3f06918ad1d09fc456296125df0f1384648fad63a5f292c3a93c5e6ece9" exitCode=0 Nov 03 20:10:24.090304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:24.089602 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" event={"ID":"5b30b4c4-debd-4919-a4e6-67f7ee38eb90","Type":"ContainerDied","Data":"552ec3f06918ad1d09fc456296125df0f1384648fad63a5f292c3a93c5e6ece9"} Nov 03 20:10:24.092957 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:24.091003 5059 scope.go:122] "RemoveContainer" containerID="552ec3f06918ad1d09fc456296125df0f1384648fad63a5f292c3a93c5e6ece9" Nov 03 20:10:24.099211 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:24.093654 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:10:24.099211 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:24.093774 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:10:24.099211 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:24.093826 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:10:24.938035 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:24.938000 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 20:10:24.938220 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:24.938066 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" podUID="9d60eb56-9852-4c25-ae50-8a932e752793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 20:10:25.192744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:25.183389 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" event={"ID":"5b30b4c4-debd-4919-a4e6-67f7ee38eb90","Type":"ContainerStarted","Data":"6c77ae8b3fd95f803333003c66104b729e78fdae56d3aa26d441db19a235aa60"} Nov 03 20:10:25.322686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:25.322021 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:10:25.322686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:25.322227 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 20:10:25.322686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:25.322304 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" podUID="5b30b4c4-debd-4919-a4e6-67f7ee38eb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 20:10:25.938617 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:25.938540 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 20:10:25.938807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:25.938738 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" podUID="9d60eb56-9852-4c25-ae50-8a932e752793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 20:10:26.324796 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:26.323841 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 20:10:26.324796 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:26.323893 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" podUID="5b30b4c4-debd-4919-a4e6-67f7ee38eb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 20:10:26.960696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:26.941102 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 20:10:26.960696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:26.941166 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" podUID="9d60eb56-9852-4c25-ae50-8a932e752793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 20:10:27.318271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:27.318191 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 20:10:27.318271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:27.318261 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" podUID="5b30b4c4-debd-4919-a4e6-67f7ee38eb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 20:10:27.547232 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:27.547184 5059 generic.go:356] "Generic (PLEG): container finished" podID="4be209ff-6d62-42d5-a0e3-22373143ecf5" containerID="8ceb4111ba8cc2ad05515d7fe5bc2fe674b5f039e258707d2fdfd55fd1df3f2e" exitCode=0 Nov 03 20:10:27.547711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:27.547308 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" event={"ID":"4be209ff-6d62-42d5-a0e3-22373143ecf5","Type":"ContainerDied","Data":"8ceb4111ba8cc2ad05515d7fe5bc2fe674b5f039e258707d2fdfd55fd1df3f2e"} Nov 03 20:10:27.551690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:27.547889 5059 scope.go:122] "RemoveContainer" containerID="8ceb4111ba8cc2ad05515d7fe5bc2fe674b5f039e258707d2fdfd55fd1df3f2e" Nov 03 20:10:27.573826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:27.570736 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:10:27.573826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:27.570845 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:10:27.573826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:27.570896 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:10:27.939764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:27.939289 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 20:10:27.939764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:27.939344 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" podUID="9d60eb56-9852-4c25-ae50-8a932e752793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 20:10:28.324128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:28.324085 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 20:10:28.324320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:28.324151 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" podUID="5b30b4c4-debd-4919-a4e6-67f7ee38eb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 20:10:28.633734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:28.633130 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" event={"ID":"4be209ff-6d62-42d5-a0e3-22373143ecf5","Type":"ContainerStarted","Data":"886ffe058a2d7e153fb7bef50ff1ae6a6fd3015f69a3035f02a85dceae7822d8"} Nov 03 20:10:28.938741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:28.938195 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 20:10:28.938741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:28.938259 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" podUID="9d60eb56-9852-4c25-ae50-8a932e752793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 20:10:28.962687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:28.960132 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:10:29.323913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:29.323100 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 20:10:29.323913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:29.323161 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" podUID="5b30b4c4-debd-4919-a4e6-67f7ee38eb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 20:10:29.411063 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:29.406883 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" podUID="bb9ed680-cec0-4abe-9d39-40c70da96e55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.247:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:10:29.801753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:29.789196 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" podUID="bb9ed680-cec0-4abe-9d39-40c70da96e55" containerName="main" containerID="cri-o://dbaaceaee7f969aeea4cdefe42d387cba2eb64d7c5306e79351bfa4c0e23d85f" gracePeriod=45 Nov 03 20:10:29.943302 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:29.943264 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 20:10:29.943586 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:29.943551 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" podUID="9d60eb56-9852-4c25-ae50-8a932e752793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 20:10:29.948998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:29.948971 5059 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Nov 03 20:10:29.949224 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:29.949195 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" podUID="4be209ff-6d62-42d5-a0e3-22373143ecf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Nov 03 20:10:30.550613 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:30.445690 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" podUID="bb9ed680-cec0-4abe-9d39-40c70da96e55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.247:8000/ready\": dial tcp 10.132.0.247:8000: connect: connection refused" Nov 03 20:10:30.550613 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:30.448246 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 20:10:30.550613 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:30.448306 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" podUID="5b30b4c4-debd-4919-a4e6-67f7ee38eb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 20:10:30.739439 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:30.739394 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:10:30.739887 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:30.739862 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:10:30.794711 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:10:30.752609 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb9ed680_cec0_4abe_9d39_40c70da96e55.slice/crio-dbaaceaee7f969aeea4cdefe42d387cba2eb64d7c5306e79351bfa4c0e23d85f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb9ed680_cec0_4abe_9d39_40c70da96e55.slice/crio-conmon-dbaaceaee7f969aeea4cdefe42d387cba2eb64d7c5306e79351bfa4c0e23d85f.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:10:30.962373 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:30.962329 5059 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Nov 03 20:10:31.041989 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:30.962989 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" podUID="4be209ff-6d62-42d5-a0e3-22373143ecf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Nov 03 20:10:31.110833 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.074482 5059 generic.go:356] "Generic (PLEG): container finished" podID="bb9ed680-cec0-4abe-9d39-40c70da96e55" containerID="dbaaceaee7f969aeea4cdefe42d387cba2eb64d7c5306e79351bfa4c0e23d85f" exitCode=0 Nov 03 20:10:31.110833 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.074509 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" event={"ID":"bb9ed680-cec0-4abe-9d39-40c70da96e55","Type":"ContainerDied","Data":"dbaaceaee7f969aeea4cdefe42d387cba2eb64d7c5306e79351bfa4c0e23d85f"} Nov 03 20:10:31.110833 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.099324 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:10:31.111929 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.111900 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:10:31.129100 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.124116 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:10:31.633469 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.633438 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:10:31.653249 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.651501 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 20:10:31.653249 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.651562 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" podUID="5b30b4c4-debd-4919-a4e6-67f7ee38eb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 20:10:31.737770 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.737129 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:10:31.800697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.800647 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:10:31.858686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.838196 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:10:31.858686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.857811 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:10:31.863260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.861564 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-tmp-dir\") pod \"bb9ed680-cec0-4abe-9d39-40c70da96e55\" (UID: \"bb9ed680-cec0-4abe-9d39-40c70da96e55\") " Nov 03 20:10:31.863260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.861687 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-home\" (UniqueName: \"kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-home\") pod \"bb9ed680-cec0-4abe-9d39-40c70da96e55\" (UID: \"bb9ed680-cec0-4abe-9d39-40c70da96e55\") " Nov 03 20:10:31.863260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.861821 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-model-cache\") pod \"bb9ed680-cec0-4abe-9d39-40c70da96e55\" (UID: \"bb9ed680-cec0-4abe-9d39-40c70da96e55\") " Nov 03 20:10:31.863260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.861907 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb9ed680-cec0-4abe-9d39-40c70da96e55-tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9ed680-cec0-4abe-9d39-40c70da96e55-tls-certs\") pod \"bb9ed680-cec0-4abe-9d39-40c70da96e55\" (UID: \"bb9ed680-cec0-4abe-9d39-40c70da96e55\") " Nov 03 20:10:31.863260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.862006 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-dshm\") pod \"bb9ed680-cec0-4abe-9d39-40c70da96e55\" (UID: \"bb9ed680-cec0-4abe-9d39-40c70da96e55\") " Nov 03 20:10:31.863260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.862063 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb9ed680-cec0-4abe-9d39-40c70da96e55-kube-api-access-7glgb\" (UniqueName: \"kubernetes.io/projected/bb9ed680-cec0-4abe-9d39-40c70da96e55-kube-api-access-7glgb\") pod \"bb9ed680-cec0-4abe-9d39-40c70da96e55\" (UID: \"bb9ed680-cec0-4abe-9d39-40c70da96e55\") " Nov 03 20:10:31.864162 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.864138 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-tmp-dir" pod "bb9ed680-cec0-4abe-9d39-40c70da96e55" (UID: "bb9ed680-cec0-4abe-9d39-40c70da96e55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:10:31.865594 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.865574 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-home" pod "bb9ed680-cec0-4abe-9d39-40c70da96e55" (UID: "bb9ed680-cec0-4abe-9d39-40c70da96e55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:10:31.866117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.866097 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-model-cache" pod "bb9ed680-cec0-4abe-9d39-40c70da96e55" (UID: "bb9ed680-cec0-4abe-9d39-40c70da96e55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:10:31.963815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.948071 5059 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Nov 03 20:10:31.963815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.948131 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" podUID="4be209ff-6d62-42d5-a0e3-22373143ecf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Nov 03 20:10:31.967857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.967159 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:31.967857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.967197 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:31.967857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.967212 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:31.973699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.973653 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb9ed680-cec0-4abe-9d39-40c70da96e55-tls-certs" pod "bb9ed680-cec0-4abe-9d39-40c70da96e55" (UID: "bb9ed680-cec0-4abe-9d39-40c70da96e55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:10:31.998493 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:31.998393 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-dshm" pod "bb9ed680-cec0-4abe-9d39-40c70da96e55" (UID: "bb9ed680-cec0-4abe-9d39-40c70da96e55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:10:32.143232 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:32.128905 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9ed680-cec0-4abe-9d39-40c70da96e55-kube-api-access-7glgb" pod "bb9ed680-cec0-4abe-9d39-40c70da96e55" (UID: "bb9ed680-cec0-4abe-9d39-40c70da96e55"). InnerVolumeSpecName "kube-api-access-7glgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:10:32.268738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:32.254608 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9ed680-cec0-4abe-9d39-40c70da96e55-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:32.268738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:32.254648 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9ed680-cec0-4abe-9d39-40c70da96e55-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:32.268738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:32.254685 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7glgb\" (UniqueName: \"kubernetes.io/projected/bb9ed680-cec0-4abe-9d39-40c70da96e55-kube-api-access-7glgb\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:32.333684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:32.324599 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 20:10:32.333684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:32.324704 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" podUID="5b30b4c4-debd-4919-a4e6-67f7ee38eb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 20:10:32.348181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:32.348147 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" Nov 03 20:10:32.391687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:32.380197 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" event={"ID":"bb9ed680-cec0-4abe-9d39-40c70da96e55","Type":"ContainerDied","Data":"f7f8d07976a02e68f4aef87cd0b3fd5b9c3d273360cd1c6cae622dce93b5c8b6"} Nov 03 20:10:32.391687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:32.380261 5059 scope.go:122] "RemoveContainer" containerID="dbaaceaee7f969aeea4cdefe42d387cba2eb64d7c5306e79351bfa4c0e23d85f" Nov 03 20:10:32.454651 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:32.449230 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p"] Nov 03 20:10:32.454651 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:32.453678 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p"] Nov 03 20:10:32.706741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:32.705980 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-36556d/autores-model-36556d-kserve-cb957bc65-bjp8p" podUID="bb9ed680-cec0-4abe-9d39-40c70da96e55" containerName="main" probeResult="failure" output="Get \"https://10.132.0.247:8000/ready\": context deadline exceeded" Nov 03 20:10:32.966741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:32.963207 5059 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Nov 03 20:10:32.966741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:32.963272 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" podUID="4be209ff-6d62-42d5-a0e3-22373143ecf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Nov 03 20:10:33.892687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:33.886170 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb9ed680-cec0-4abe-9d39-40c70da96e55" path="/var/lib/kubelet/pods/bb9ed680-cec0-4abe-9d39-40c70da96e55/volumes" Nov 03 20:10:33.918068 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:33.916243 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:10:33.922693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:33.922063 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:10:33.946870 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:33.946833 5059 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Nov 03 20:10:33.947068 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:33.946965 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" podUID="4be209ff-6d62-42d5-a0e3-22373143ecf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Nov 03 20:10:34.947223 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:34.947182 5059 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Nov 03 20:10:34.947735 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:34.947247 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" podUID="4be209ff-6d62-42d5-a0e3-22373143ecf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Nov 03 20:10:35.946497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:35.946458 5059 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Nov 03 20:10:35.946792 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:35.946511 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" podUID="4be209ff-6d62-42d5-a0e3-22373143ecf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Nov 03 20:10:36.947684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:36.947017 5059 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Nov 03 20:10:36.947684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:36.947087 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" podUID="4be209ff-6d62-42d5-a0e3-22373143ecf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Nov 03 20:10:37.951764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:37.951735 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:10:37.952963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:37.952937 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:10:40.609480 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:10:40.609418 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:10:41.487793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:41.487759 5059 generic.go:356] "Generic (PLEG): container finished" podID="14a06bdf-e15d-406c-bb76-47ae5dd06b2d" containerID="56b80e220cc2df35563d8697c1e14db7387aa8d2faaa80e6b681d5be3a5c172e" exitCode=0 Nov 03 20:10:41.488041 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:41.487849 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" event={"ID":"14a06bdf-e15d-406c-bb76-47ae5dd06b2d","Type":"ContainerDied","Data":"56b80e220cc2df35563d8697c1e14db7387aa8d2faaa80e6b681d5be3a5c172e"} Nov 03 20:10:41.488303 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:41.488280 5059 scope.go:122] "RemoveContainer" containerID="56b80e220cc2df35563d8697c1e14db7387aa8d2faaa80e6b681d5be3a5c172e" Nov 03 20:10:41.490127 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:41.490096 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:10:41.490232 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:41.490167 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:10:41.490232 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:41.490228 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:10:42.676685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:42.663482 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" event={"ID":"14a06bdf-e15d-406c-bb76-47ae5dd06b2d","Type":"ContainerStarted","Data":"cdc8ec4afacb060d845fdcbcf9f9a36756ac0c3cf27302051a5a1dae638f96f6"} Nov 03 20:10:43.305345 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:43.305306 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:10:43.306563 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:43.306534 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Nov 03 20:10:43.306702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:43.306593 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" podUID="14a06bdf-e15d-406c-bb76-47ae5dd06b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Nov 03 20:10:44.308687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:44.308091 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Nov 03 20:10:44.308687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:44.308169 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" podUID="14a06bdf-e15d-406c-bb76-47ae5dd06b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Nov 03 20:10:44.580683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:44.577811 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:10:45.305778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:45.305746 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Nov 03 20:10:45.305958 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:45.305800 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" podUID="14a06bdf-e15d-406c-bb76-47ae5dd06b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Nov 03 20:10:46.305231 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:46.305189 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Nov 03 20:10:46.305746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:46.305280 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" podUID="14a06bdf-e15d-406c-bb76-47ae5dd06b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Nov 03 20:10:47.305775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:47.305733 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Nov 03 20:10:47.306268 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:47.305800 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" podUID="14a06bdf-e15d-406c-bb76-47ae5dd06b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Nov 03 20:10:48.309889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:48.309856 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:10:48.311144 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:48.311116 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:10:57.834703 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:57.834654 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns"] Nov 03 20:10:57.836771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:57.836744 5059 cpu_manager.go:414] "containerMap: removing container" podUID="bb9ed680-cec0-4abe-9d39-40c70da96e55" containerName="main" Nov 03 20:10:57.837185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:57.837166 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb9ed680-cec0-4abe-9d39-40c70da96e55" containerName="main" Nov 03 20:10:57.837601 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:57.837582 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb9ed680-cec0-4abe-9d39-40c70da96e55" containerName="main" Nov 03 20:10:57.844842 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:57.844826 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:57.912566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:57.912531 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns"] Nov 03 20:10:57.991397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:57.991348 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f797b\" (UniqueName: \"kubernetes.io/projected/f900a500-5deb-4f3a-a861-adc38c5b5d23-kube-api-access-f797b\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:57.991599 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:57.991465 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-credential-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:57.991599 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:57.991549 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-envoy\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:57.991775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:57.991740 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-podinfo\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:57.991844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:57.991797 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-data\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:57.991951 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:57.991922 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-workload-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:57.992021 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:57.991992 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f900a500-5deb-4f3a-a861-adc38c5b5d23-istiod-ca-cert\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:57.992085 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:57.992048 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-token\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:57.992145 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:57.992105 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-workload-certs\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:58.099376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:58.096838 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f900a500-5deb-4f3a-a861-adc38c5b5d23-istiod-ca-cert\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:58.099376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:58.096941 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-token\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:58.099376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:58.096991 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-workload-certs\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:58.099376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:58.097059 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f797b\" (UniqueName: \"kubernetes.io/projected/f900a500-5deb-4f3a-a861-adc38c5b5d23-kube-api-access-f797b\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:58.099376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:58.097132 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-credential-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:58.099376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:58.097235 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-envoy\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:58.099376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:58.097373 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-podinfo\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:58.099376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:58.097439 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-data\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:58.099376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:58.097503 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-workload-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:58.109440 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:58.102272 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-workload-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:58.109722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:58.109547 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f900a500-5deb-4f3a-a861-adc38c5b5d23-istiod-ca-cert\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:58.120684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:58.110224 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-workload-certs\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:58.129679 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:58.121134 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-credential-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:58.129679 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:58.121546 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-data\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:58.188685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:58.187819 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-envoy\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:58.188685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:58.188314 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f797b\" (UniqueName: \"kubernetes.io/projected/f900a500-5deb-4f3a-a861-adc38c5b5d23-kube-api-access-f797b\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:58.193747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:58.193186 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-token\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:58.203730 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:58.203188 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-podinfo\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-p7qns\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:58.236530 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:58.236253 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:10:59.019703 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:58.901041 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf"] Nov 03 20:10:59.119683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.118626 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf"] Nov 03 20:10:59.119683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.118817 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.249689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.234779 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df279a7-3d4b-4bad-9fd0-675243359616-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.249689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.234891 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.249689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.234951 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df279a7-3d4b-4bad-9fd0-675243359616-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.249689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.235007 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df279a7-3d4b-4bad-9fd0-675243359616-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.249689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.235088 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.249689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.244175 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.264683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.254983 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.280737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.269751 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-757bf\" (UniqueName: \"kubernetes.io/projected/4df279a7-3d4b-4bad-9fd0-675243359616-kube-api-access-757bf\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.280737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.269900 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.387749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.386939 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.387749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.387173 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.387749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.387246 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.387749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.387285 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-757bf\" (UniqueName: \"kubernetes.io/projected/4df279a7-3d4b-4bad-9fd0-675243359616-kube-api-access-757bf\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.387749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.387359 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.387749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.387490 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df279a7-3d4b-4bad-9fd0-675243359616-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.409897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.403881 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.430913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.420394 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df279a7-3d4b-4bad-9fd0-675243359616-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.430913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.420479 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df279a7-3d4b-4bad-9fd0-675243359616-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.455687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.436234 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.455687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.436515 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.455687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.436784 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.455687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.437343 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.497307 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.497263 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df279a7-3d4b-4bad-9fd0-675243359616-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.662824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.660777 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:10:59.793685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.771463 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-757bf\" (UniqueName: \"kubernetes.io/projected/4df279a7-3d4b-4bad-9fd0-675243359616-kube-api-access-757bf\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.793685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.771923 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df279a7-3d4b-4bad-9fd0-675243359616-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.915652 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.913397 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df279a7-3d4b-4bad-9fd0-675243359616-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:10:59.951686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:10:59.951608 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 03 20:11:00.096751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:00.072312 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:11:00.163673 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:00.163603 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:11:00.249497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:00.249455 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:11:00.267545 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:00.267502 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:11:00.267800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:00.267777 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:11:00.267967 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:00.267940 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:11:00.295126 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:00.295089 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns"] Nov 03 20:11:00.418554 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:00.418519 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:11:01.012545 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:01.012511 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf"] Nov 03 20:11:01.020683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:01.020620 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:11:01.021229 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:01.021148 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:11:01.021229 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:01.021213 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:11:01.313748 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:01.313623 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" event={"ID":"f900a500-5deb-4f3a-a861-adc38c5b5d23","Type":"ContainerStarted","Data":"c3cebbc58993effe006c4b73d8f22aacc3f9f7f36fb2d3f23f3c4875b88790b2"} Nov 03 20:11:01.313748 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:01.313700 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" event={"ID":"f900a500-5deb-4f3a-a861-adc38c5b5d23","Type":"ContainerStarted","Data":"de3b01fe6a2bd45adea1036f5c9b9179f8b85f46ce2afce49b089613c2841bfa"} Nov 03 20:11:01.316653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:01.316628 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" event={"ID":"4df279a7-3d4b-4bad-9fd0-675243359616","Type":"ContainerStarted","Data":"906a4c87ed7c84ea2bc89825e56653fdc587d0ae53e1811d7698a2d74f9cd601"} Nov 03 20:11:01.316750 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:01.316711 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" event={"ID":"4df279a7-3d4b-4bad-9fd0-675243359616","Type":"ContainerStarted","Data":"05d728e5f8be7a486e15b72256103d1e7b63d82681fcecd499794af594a6c750"} Nov 03 20:11:01.342728 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:01.342641 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" podStartSLOduration=4.342618785 podStartE2EDuration="4.342618785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:01.342039204 +0000 UTC m=+2055.599836458" watchObservedRunningTime="2026-11-03 20:11:01.342618785 +0000 UTC m=+2055.600416047" Nov 03 20:11:01.427686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:01.420014 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" podStartSLOduration=3.419993105 podStartE2EDuration="3.419993105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:01.411448259 +0000 UTC m=+2055.669245517" watchObservedRunningTime="2026-11-03 20:11:01.419993105 +0000 UTC m=+2055.677790362" Nov 03 20:11:01.427900 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:01.427780 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:11:02.251504 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:02.249339 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:11:02.251504 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:02.249463 5059 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-p7qns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 20:11:02.251504 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:02.249511 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" podUID="f900a500-5deb-4f3a-a861-adc38c5b5d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 20:11:02.432692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:02.431298 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Nov 03 20:11:02.432692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:02.431682 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" podUID="4df279a7-3d4b-4bad-9fd0-675243359616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Nov 03 20:11:03.243178 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:03.243134 5059 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-p7qns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 20:11:03.243540 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:03.243508 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" podUID="f900a500-5deb-4f3a-a861-adc38c5b5d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 20:11:03.432838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:03.432799 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Nov 03 20:11:03.433438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:03.433390 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" podUID="4df279a7-3d4b-4bad-9fd0-675243359616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Nov 03 20:11:04.238709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:04.238364 5059 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-p7qns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 20:11:04.238709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:04.238418 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" podUID="f900a500-5deb-4f3a-a861-adc38c5b5d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 20:11:04.440296 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:04.440254 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Nov 03 20:11:04.440851 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:04.440327 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" podUID="4df279a7-3d4b-4bad-9fd0-675243359616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Nov 03 20:11:05.239703 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:05.238323 5059 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-p7qns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 20:11:05.239703 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:05.238384 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" podUID="f900a500-5deb-4f3a-a861-adc38c5b5d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 20:11:05.423688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:05.422805 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Nov 03 20:11:05.423688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:05.422867 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" podUID="4df279a7-3d4b-4bad-9fd0-675243359616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Nov 03 20:11:06.239292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:06.239247 5059 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-p7qns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 20:11:06.239749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:06.239314 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" podUID="f900a500-5deb-4f3a-a861-adc38c5b5d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 20:11:06.429685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:06.427630 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Nov 03 20:11:06.429685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:06.427715 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" podUID="4df279a7-3d4b-4bad-9fd0-675243359616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Nov 03 20:11:07.242398 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:07.242351 5059 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-p7qns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 20:11:07.243383 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:07.243344 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" podUID="f900a500-5deb-4f3a-a861-adc38c5b5d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 20:11:07.449447 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:07.449234 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Nov 03 20:11:07.449447 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:07.449301 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" podUID="4df279a7-3d4b-4bad-9fd0-675243359616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Nov 03 20:11:08.245649 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:08.245613 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:11:08.246286 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:08.246259 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:11:08.247202 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:08.247175 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:11:08.482545 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:08.482500 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:11:08.484993 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:08.484961 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:11:08.486072 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:08.486050 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:11:10.612837 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:11:10.612800 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:11:15.290756 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:15.289830 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:11:15.290756 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:15.289906 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" podUID="7909b42c-1572-4fe2-816a-447020594fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:11:15.791420 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:15.791380 5059 generic.go:356] "Generic (PLEG): container finished" podID="7909b42c-1572-4fe2-816a-447020594fe4" containerID="f2e801a6eee34b07dcc874c7303b6ec332d7470d1b006afc0f98c9a4ae0ba486" exitCode=0 Nov 03 20:11:15.791645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:15.791458 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" event={"ID":"7909b42c-1572-4fe2-816a-447020594fe4","Type":"ContainerDied","Data":"f2e801a6eee34b07dcc874c7303b6ec332d7470d1b006afc0f98c9a4ae0ba486"} Nov 03 20:11:15.792053 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:15.792031 5059 scope.go:122] "RemoveContainer" containerID="f2e801a6eee34b07dcc874c7303b6ec332d7470d1b006afc0f98c9a4ae0ba486" Nov 03 20:11:15.794108 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:15.794074 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:11:15.794222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:15.794170 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:11:15.794290 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:15.794221 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:11:16.801622 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:16.801582 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" event={"ID":"7909b42c-1572-4fe2-816a-447020594fe4","Type":"ContainerStarted","Data":"4cc323b3fee429ec7bc1ab6e19bfbfe5f3926b292a99afff5f656543c7dd8e48"} Nov 03 20:11:17.191687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:17.183088 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:11:17.191687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:17.183215 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Nov 03 20:11:17.191687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:17.183280 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" podUID="7909b42c-1572-4fe2-816a-447020594fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Nov 03 20:11:18.185686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:18.185044 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Nov 03 20:11:18.185686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:18.185107 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" podUID="7909b42c-1572-4fe2-816a-447020594fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Nov 03 20:11:19.187129 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:19.187088 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Nov 03 20:11:19.187776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:19.187738 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" podUID="7909b42c-1572-4fe2-816a-447020594fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Nov 03 20:11:20.197686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:20.196770 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Nov 03 20:11:20.197686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:20.196833 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" podUID="7909b42c-1572-4fe2-816a-447020594fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Nov 03 20:11:21.189035 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:21.188996 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Nov 03 20:11:21.189702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:21.189300 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" podUID="7909b42c-1572-4fe2-816a-447020594fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Nov 03 20:11:22.240389 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:22.240346 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Nov 03 20:11:22.245813 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:22.245778 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" podUID="7909b42c-1572-4fe2-816a-447020594fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Nov 03 20:11:23.184115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:23.184072 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Nov 03 20:11:23.184309 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:23.184155 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" podUID="7909b42c-1572-4fe2-816a-447020594fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Nov 03 20:11:24.820937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:24.820873 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:11:24.821550 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:24.821527 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:11:24.822415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:24.822389 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:11:28.443442 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:28.443407 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 03 20:11:28.443875 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:28.443483 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" podUID="08b3a993-1a64-4fe5-9577-b0f11a36779e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 03 20:11:29.227825 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:29.227790 5059 generic.go:356] "Generic (PLEG): container finished" podID="08b3a993-1a64-4fe5-9577-b0f11a36779e" containerID="3f07fce3b6aeb311a72b724e58639936894c5cc3f1b194d161e8c7b428a68157" exitCode=0 Nov 03 20:11:29.228007 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:29.227862 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" event={"ID":"08b3a993-1a64-4fe5-9577-b0f11a36779e","Type":"ContainerDied","Data":"3f07fce3b6aeb311a72b724e58639936894c5cc3f1b194d161e8c7b428a68157"} Nov 03 20:11:29.228293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:29.228277 5059 scope.go:122] "RemoveContainer" containerID="3f07fce3b6aeb311a72b724e58639936894c5cc3f1b194d161e8c7b428a68157" Nov 03 20:11:29.230037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:29.230006 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:11:29.230103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:29.230075 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:11:29.230141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:29.230109 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:11:29.901083 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:29.901049 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd"] Nov 03 20:11:29.901821 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:29.901590 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" podUID="4be209ff-6d62-42d5-a0e3-22373143ecf5" containerName="istio-proxy" containerID="cri-o://886ffe058a2d7e153fb7bef50ff1ae6a6fd3015f69a3035f02a85dceae7822d8" gracePeriod=30 Nov 03 20:11:29.911921 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:29.911862 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns"] Nov 03 20:11:29.912116 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:29.912076 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" podUID="f900a500-5deb-4f3a-a861-adc38c5b5d23" containerName="istio-proxy" containerID="cri-o://c3cebbc58993effe006c4b73d8f22aacc3f9f7f36fb2d3f23f3c4875b88790b2" gracePeriod=30 Nov 03 20:11:29.918770 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:29.918745 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr"] Nov 03 20:11:29.918982 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:29.918950 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" podUID="beb966fa-cc63-40e1-9cb2-aabb05c56717" containerName="istio-proxy" containerID="cri-o://7eef1e1a85c69dbc87d3f8efa0178f9ab2722d386dcaef0fd1da52e567ab07c1" gracePeriod=30 Nov 03 20:11:30.320746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:30.319459 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" event={"ID":"08b3a993-1a64-4fe5-9577-b0f11a36779e","Type":"ContainerStarted","Data":"ab75e4ee856c98aa1f2cad9eecfea81d6df844e47094122246ef45af739d37d5"} Nov 03 20:11:31.229774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:31.229740 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:11:31.230243 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:31.230002 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 03 20:11:31.230243 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:31.230058 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" podUID="08b3a993-1a64-4fe5-9577-b0f11a36779e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 03 20:11:32.217906 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.217463 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt"] Nov 03 20:11:32.221570 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.221547 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.229682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.229304 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 03 20:11:32.229682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.229392 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" podUID="08b3a993-1a64-4fe5-9577-b0f11a36779e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 03 20:11:32.230192 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.229807 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-33e267-openshift-default-dockercfg-lkhx5\"" Nov 03 20:11:32.238956 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.238912 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt"] Nov 03 20:11:32.318020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.317982 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-workload-certs\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.318314 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.318098 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-workload-socket\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.318314 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.318217 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-data\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.318314 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.318268 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq82f\" (UniqueName: \"kubernetes.io/projected/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-kube-api-access-sq82f\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.318536 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.318367 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-podinfo\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.318536 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.318436 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-credential-socket\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.318536 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.318497 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istiod-ca-cert\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.318740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.318548 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-token\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.318740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.318641 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-envoy\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.427683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.424006 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-workload-socket\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.427683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.424159 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-data\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.427683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.424244 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq82f\" (UniqueName: \"kubernetes.io/projected/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-kube-api-access-sq82f\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.427683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.424304 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-podinfo\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.427683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.424407 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-credential-socket\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.427683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.424454 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istiod-ca-cert\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.427683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.424504 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-token\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.427683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.424589 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-envoy\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.427683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.424892 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-workload-certs\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.427683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.425422 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-workload-certs\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.427683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.425853 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-credential-socket\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.427683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.425932 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-data\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.427683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.426219 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-workload-socket\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.440688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.440265 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-podinfo\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.464682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.449807 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istiod-ca-cert\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.464682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.464433 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq82f\" (UniqueName: \"kubernetes.io/projected/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-kube-api-access-sq82f\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.487734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.465010 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-token\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.488239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.488178 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-envoy\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:32.557019 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:32.556976 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:33.171982 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:33.171746 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt"] Nov 03 20:11:33.183710 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:11:33.183679 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd912f1fd_b76b_4bb6_a5d9_ccc8b4536840.slice/crio-a2e2318cab978244150236d259a7997e616a413482cbe8845c7e1d51252684fb WatchSource:0}: Error finding container a2e2318cab978244150236d259a7997e616a413482cbe8845c7e1d51252684fb: Status 404 returned error can't find the container with id a2e2318cab978244150236d259a7997e616a413482cbe8845c7e1d51252684fb Nov 03 20:11:33.187220 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:33.187179 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:11:33.187419 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:33.187396 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:11:33.187560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:33.187542 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:11:33.229791 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:33.229765 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 03 20:11:33.229888 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:33.229821 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" podUID="08b3a993-1a64-4fe5-9577-b0f11a36779e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 03 20:11:33.348891 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:33.348851 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" event={"ID":"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840","Type":"ContainerStarted","Data":"2f57092ac51573f4a21c3d1c3f99cddf2880faf5135c68f41abacbcf61bc2cfa"} Nov 03 20:11:33.349077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:33.348907 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" event={"ID":"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840","Type":"ContainerStarted","Data":"a2e2318cab978244150236d259a7997e616a413482cbe8845c7e1d51252684fb"} Nov 03 20:11:33.389719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:33.389614 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" podStartSLOduration=1.389599074 podStartE2EDuration="1.389599074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:33.376148822 +0000 UTC m=+2087.633946093" watchObservedRunningTime="2026-11-03 20:11:33.389599074 +0000 UTC m=+2087.647396325" Nov 03 20:11:33.603683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:33.596360 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:34.229788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:34.229754 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 03 20:11:34.229973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:34.229805 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" podUID="08b3a993-1a64-4fe5-9577-b0f11a36779e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 03 20:11:34.588963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:34.588934 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:35.234212 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.233914 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 03 20:11:35.234212 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.233995 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" podUID="08b3a993-1a64-4fe5-9577-b0f11a36779e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 03 20:11:35.429761 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.429034 5059 generic.go:356] "Generic (PLEG): container finished" podID="f900a500-5deb-4f3a-a861-adc38c5b5d23" containerID="c3cebbc58993effe006c4b73d8f22aacc3f9f7f36fb2d3f23f3c4875b88790b2" exitCode=0 Nov 03 20:11:35.429761 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.429200 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" event={"ID":"f900a500-5deb-4f3a-a861-adc38c5b5d23","Type":"ContainerDied","Data":"c3cebbc58993effe006c4b73d8f22aacc3f9f7f36fb2d3f23f3c4875b88790b2"} Nov 03 20:11:35.435304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.434240 5059 generic.go:356] "Generic (PLEG): container finished" podID="4be209ff-6d62-42d5-a0e3-22373143ecf5" containerID="886ffe058a2d7e153fb7bef50ff1ae6a6fd3015f69a3035f02a85dceae7822d8" exitCode=0 Nov 03 20:11:35.435304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.434296 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" event={"ID":"4be209ff-6d62-42d5-a0e3-22373143ecf5","Type":"ContainerDied","Data":"886ffe058a2d7e153fb7bef50ff1ae6a6fd3015f69a3035f02a85dceae7822d8"} Nov 03 20:11:35.435304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.434350 5059 scope.go:122] "RemoveContainer" containerID="8ceb4111ba8cc2ad05515d7fe5bc2fe674b5f039e258707d2fdfd55fd1df3f2e" Nov 03 20:11:35.442875 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.438783 5059 generic.go:356] "Generic (PLEG): container finished" podID="beb966fa-cc63-40e1-9cb2-aabb05c56717" containerID="7eef1e1a85c69dbc87d3f8efa0178f9ab2722d386dcaef0fd1da52e567ab07c1" exitCode=0 Nov 03 20:11:35.442875 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.440101 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" event={"ID":"beb966fa-cc63-40e1-9cb2-aabb05c56717","Type":"ContainerDied","Data":"7eef1e1a85c69dbc87d3f8efa0178f9ab2722d386dcaef0fd1da52e567ab07c1"} Nov 03 20:11:35.442875 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.440161 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:35.442875 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.442099 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:11:35.501030 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.499854 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:11:35.518960 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.518930 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:11:35.523395 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.523368 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:11:35.533231 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.533197 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-workload-socket\") pod \"f900a500-5deb-4f3a-a861-adc38c5b5d23\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " Nov 03 20:11:35.533367 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.533258 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-data\") pod \"f900a500-5deb-4f3a-a861-adc38c5b5d23\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " Nov 03 20:11:35.533367 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.533291 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-workload-certs\") pod \"beb966fa-cc63-40e1-9cb2-aabb05c56717\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " Nov 03 20:11:35.533367 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.533336 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-workload-socket\") pod \"4be209ff-6d62-42d5-a0e3-22373143ecf5\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " Nov 03 20:11:35.533563 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.533368 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-data\") pod \"4be209ff-6d62-42d5-a0e3-22373143ecf5\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " Nov 03 20:11:35.533563 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.533401 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-envoy\") pod \"f900a500-5deb-4f3a-a861-adc38c5b5d23\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " Nov 03 20:11:35.533563 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.533439 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-token\" (UniqueName: \"kubernetes.io/projected/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-token\") pod \"beb966fa-cc63-40e1-9cb2-aabb05c56717\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " Nov 03 20:11:35.533563 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.533466 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-workload-socket\") pod \"beb966fa-cc63-40e1-9cb2-aabb05c56717\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " Nov 03 20:11:35.533563 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.533533 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-credential-socket\") pod \"4be209ff-6d62-42d5-a0e3-22373143ecf5\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " Nov 03 20:11:35.533886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.533574 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-envoy\") pod \"beb966fa-cc63-40e1-9cb2-aabb05c56717\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " Nov 03 20:11:35.533886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.533636 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f900a500-5deb-4f3a-a861-adc38c5b5d23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f900a500-5deb-4f3a-a861-adc38c5b5d23-istiod-ca-cert\") pod \"f900a500-5deb-4f3a-a861-adc38c5b5d23\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " Nov 03 20:11:35.533886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.533702 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-workload-certs\") pod \"4be209ff-6d62-42d5-a0e3-22373143ecf5\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " Nov 03 20:11:35.533886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.533732 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beb966fa-cc63-40e1-9cb2-aabb05c56717-kube-api-access-fh2c4\" (UniqueName: \"kubernetes.io/projected/beb966fa-cc63-40e1-9cb2-aabb05c56717-kube-api-access-fh2c4\") pod \"beb966fa-cc63-40e1-9cb2-aabb05c56717\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " Nov 03 20:11:35.533886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.533774 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-token\" (UniqueName: \"kubernetes.io/projected/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-token\") pod \"f900a500-5deb-4f3a-a861-adc38c5b5d23\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " Nov 03 20:11:35.533886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.533812 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-workload-certs\") pod \"f900a500-5deb-4f3a-a861-adc38c5b5d23\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " Nov 03 20:11:35.533886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.533858 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be209ff-6d62-42d5-a0e3-22373143ecf5-kube-api-access-flp9z\" (UniqueName: \"kubernetes.io/projected/4be209ff-6d62-42d5-a0e3-22373143ecf5-kube-api-access-flp9z\") pod \"4be209ff-6d62-42d5-a0e3-22373143ecf5\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " Nov 03 20:11:35.534291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.533893 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4be209ff-6d62-42d5-a0e3-22373143ecf5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be209ff-6d62-42d5-a0e3-22373143ecf5-istiod-ca-cert\") pod \"4be209ff-6d62-42d5-a0e3-22373143ecf5\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " Nov 03 20:11:35.534291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.533946 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-podinfo\") pod \"4be209ff-6d62-42d5-a0e3-22373143ecf5\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " Nov 03 20:11:35.534291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.534059 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-credential-socket\") pod \"f900a500-5deb-4f3a-a861-adc38c5b5d23\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " Nov 03 20:11:35.534291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.534136 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beb966fa-cc63-40e1-9cb2-aabb05c56717-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb966fa-cc63-40e1-9cb2-aabb05c56717-istiod-ca-cert\") pod \"beb966fa-cc63-40e1-9cb2-aabb05c56717\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " Nov 03 20:11:35.534291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.534209 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-podinfo\") pod \"f900a500-5deb-4f3a-a861-adc38c5b5d23\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " Nov 03 20:11:35.534291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.534266 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-token\" (UniqueName: \"kubernetes.io/projected/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-token\") pod \"4be209ff-6d62-42d5-a0e3-22373143ecf5\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " Nov 03 20:11:35.534647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.534301 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-envoy\") pod \"4be209ff-6d62-42d5-a0e3-22373143ecf5\" (UID: \"4be209ff-6d62-42d5-a0e3-22373143ecf5\") " Nov 03 20:11:35.534647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.534350 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-podinfo\") pod \"beb966fa-cc63-40e1-9cb2-aabb05c56717\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " Nov 03 20:11:35.534647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.534378 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-data\") pod \"beb966fa-cc63-40e1-9cb2-aabb05c56717\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " Nov 03 20:11:35.534647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.534429 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f900a500-5deb-4f3a-a861-adc38c5b5d23-kube-api-access-f797b\" (UniqueName: \"kubernetes.io/projected/f900a500-5deb-4f3a-a861-adc38c5b5d23-kube-api-access-f797b\") pod \"f900a500-5deb-4f3a-a861-adc38c5b5d23\" (UID: \"f900a500-5deb-4f3a-a861-adc38c5b5d23\") " Nov 03 20:11:35.534647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.534468 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-credential-socket\") pod \"beb966fa-cc63-40e1-9cb2-aabb05c56717\" (UID: \"beb966fa-cc63-40e1-9cb2-aabb05c56717\") " Nov 03 20:11:35.535825 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.535802 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-workload-socket" pod "f900a500-5deb-4f3a-a861-adc38c5b5d23" (UID: "f900a500-5deb-4f3a-a861-adc38c5b5d23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:35.535949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.535934 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-data" pod "f900a500-5deb-4f3a-a861-adc38c5b5d23" (UID: "f900a500-5deb-4f3a-a861-adc38c5b5d23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:35.536081 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.536067 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-workload-certs" pod "beb966fa-cc63-40e1-9cb2-aabb05c56717" (UID: "beb966fa-cc63-40e1-9cb2-aabb05c56717"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:35.536201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.536188 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-workload-socket" pod "4be209ff-6d62-42d5-a0e3-22373143ecf5" (UID: "4be209ff-6d62-42d5-a0e3-22373143ecf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:35.536311 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.536299 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-data" pod "4be209ff-6d62-42d5-a0e3-22373143ecf5" (UID: "4be209ff-6d62-42d5-a0e3-22373143ecf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:35.537129 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.537100 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-credential-socket" pod "beb966fa-cc63-40e1-9cb2-aabb05c56717" (UID: "beb966fa-cc63-40e1-9cb2-aabb05c56717"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:35.537329 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.537289 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-data" pod "beb966fa-cc63-40e1-9cb2-aabb05c56717" (UID: "beb966fa-cc63-40e1-9cb2-aabb05c56717"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:35.537754 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.537490 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-credential-socket" pod "4be209ff-6d62-42d5-a0e3-22373143ecf5" (UID: "4be209ff-6d62-42d5-a0e3-22373143ecf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:35.537862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.537751 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-workload-certs" pod "4be209ff-6d62-42d5-a0e3-22373143ecf5" (UID: "4be209ff-6d62-42d5-a0e3-22373143ecf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:35.539086 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.538879 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-podinfo" pod "4be209ff-6d62-42d5-a0e3-22373143ecf5" (UID: "4be209ff-6d62-42d5-a0e3-22373143ecf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:11:35.543477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.543449 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-token" pod "4be209ff-6d62-42d5-a0e3-22373143ecf5" (UID: "4be209ff-6d62-42d5-a0e3-22373143ecf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:11:35.543894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.543733 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-credential-socket" pod "f900a500-5deb-4f3a-a861-adc38c5b5d23" (UID: "f900a500-5deb-4f3a-a861-adc38c5b5d23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:35.543894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.543818 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-podinfo" pod "beb966fa-cc63-40e1-9cb2-aabb05c56717" (UID: "beb966fa-cc63-40e1-9cb2-aabb05c56717"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:11:35.543894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.543851 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb966fa-cc63-40e1-9cb2-aabb05c56717-kube-api-access-fh2c4" pod "beb966fa-cc63-40e1-9cb2-aabb05c56717" (UID: "beb966fa-cc63-40e1-9cb2-aabb05c56717"). InnerVolumeSpecName "kube-api-access-fh2c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:11:35.544114 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.543937 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-workload-certs" pod "f900a500-5deb-4f3a-a861-adc38c5b5d23" (UID: "f900a500-5deb-4f3a-a861-adc38c5b5d23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:35.545242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.544473 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-envoy" pod "beb966fa-cc63-40e1-9cb2-aabb05c56717" (UID: "beb966fa-cc63-40e1-9cb2-aabb05c56717"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:35.545242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.545083 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb966fa-cc63-40e1-9cb2-aabb05c56717-istiod-ca-cert" pod "beb966fa-cc63-40e1-9cb2-aabb05c56717" (UID: "beb966fa-cc63-40e1-9cb2-aabb05c56717"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:11:35.545242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.545160 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-workload-socket" pod "beb966fa-cc63-40e1-9cb2-aabb05c56717" (UID: "beb966fa-cc63-40e1-9cb2-aabb05c56717"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:35.545527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.545304 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f900a500-5deb-4f3a-a861-adc38c5b5d23-istiod-ca-cert" pod "f900a500-5deb-4f3a-a861-adc38c5b5d23" (UID: "f900a500-5deb-4f3a-a861-adc38c5b5d23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:11:35.545527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.545461 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be209ff-6d62-42d5-a0e3-22373143ecf5-istiod-ca-cert" pod "4be209ff-6d62-42d5-a0e3-22373143ecf5" (UID: "4be209ff-6d62-42d5-a0e3-22373143ecf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:11:35.547738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.547641 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-envoy" pod "f900a500-5deb-4f3a-a861-adc38c5b5d23" (UID: "f900a500-5deb-4f3a-a861-adc38c5b5d23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:35.547827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.547788 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-token" pod "beb966fa-cc63-40e1-9cb2-aabb05c56717" (UID: "beb966fa-cc63-40e1-9cb2-aabb05c56717"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:11:35.547827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.547794 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-podinfo" pod "f900a500-5deb-4f3a-a861-adc38c5b5d23" (UID: "f900a500-5deb-4f3a-a861-adc38c5b5d23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:11:35.548080 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.547993 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be209ff-6d62-42d5-a0e3-22373143ecf5-kube-api-access-flp9z" pod "4be209ff-6d62-42d5-a0e3-22373143ecf5" (UID: "4be209ff-6d62-42d5-a0e3-22373143ecf5"). InnerVolumeSpecName "kube-api-access-flp9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:11:35.548080 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.548052 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f900a500-5deb-4f3a-a861-adc38c5b5d23-kube-api-access-f797b" pod "f900a500-5deb-4f3a-a861-adc38c5b5d23" (UID: "f900a500-5deb-4f3a-a861-adc38c5b5d23"). InnerVolumeSpecName "kube-api-access-f797b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:11:35.549318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.549291 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-envoy" pod "4be209ff-6d62-42d5-a0e3-22373143ecf5" (UID: "4be209ff-6d62-42d5-a0e3-22373143ecf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:11:35.549899 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.549875 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-token" pod "f900a500-5deb-4f3a-a861-adc38c5b5d23" (UID: "f900a500-5deb-4f3a-a861-adc38c5b5d23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637518 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flp9z\" (UniqueName: \"kubernetes.io/projected/4be209ff-6d62-42d5-a0e3-22373143ecf5-kube-api-access-flp9z\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637554 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be209ff-6d62-42d5-a0e3-22373143ecf5-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637569 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637584 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637598 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb966fa-cc63-40e1-9cb2-aabb05c56717-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637614 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637627 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637641 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637654 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637712 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637726 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f797b\" (UniqueName: \"kubernetes.io/projected/f900a500-5deb-4f3a-a861-adc38c5b5d23-kube-api-access-f797b\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637741 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637754 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637767 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637782 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637796 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637813 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637826 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637840 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637853 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637867 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637882 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb966fa-cc63-40e1-9cb2-aabb05c56717-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637895 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f900a500-5deb-4f3a-a861-adc38c5b5d23-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637909 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be209ff-6d62-42d5-a0e3-22373143ecf5-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637923 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh2c4\" (UniqueName: \"kubernetes.io/projected/beb966fa-cc63-40e1-9cb2-aabb05c56717-kube-api-access-fh2c4\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637937 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f900a500-5deb-4f3a-a861-adc38c5b5d23-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:35.637983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:35.637951 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f900a500-5deb-4f3a-a861-adc38c5b5d23-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:36.235165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:36.235055 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 03 20:11:36.236912 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:36.236873 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" podUID="08b3a993-1a64-4fe5-9577-b0f11a36779e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 03 20:11:36.448101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:36.448070 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" Nov 03 20:11:36.448101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:36.448092 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr" event={"ID":"beb966fa-cc63-40e1-9cb2-aabb05c56717","Type":"ContainerDied","Data":"621c9695ae2f64df2abb541ccd57ddbed4c41a25d6e12f6ef69d87cb18cb9682"} Nov 03 20:11:36.448383 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:36.448157 5059 scope.go:122] "RemoveContainer" containerID="7eef1e1a85c69dbc87d3f8efa0178f9ab2722d386dcaef0fd1da52e567ab07c1" Nov 03 20:11:36.449905 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:36.449884 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" Nov 03 20:11:36.450048 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:36.449896 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns" event={"ID":"f900a500-5deb-4f3a-a861-adc38c5b5d23","Type":"ContainerDied","Data":"de3b01fe6a2bd45adea1036f5c9b9179f8b85f46ce2afce49b089613c2841bfa"} Nov 03 20:11:36.451952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:36.451910 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" event={"ID":"4be209ff-6d62-42d5-a0e3-22373143ecf5","Type":"ContainerDied","Data":"98bdcf25f4397353fe737fdc4f6d13cb878c020361c324fc2bac181aafe9f510"} Nov 03 20:11:36.452155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:36.451959 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd" Nov 03 20:11:36.486469 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:36.486441 5059 scope.go:122] "RemoveContainer" containerID="c3cebbc58993effe006c4b73d8f22aacc3f9f7f36fb2d3f23f3c4875b88790b2" Nov 03 20:11:36.497156 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:36.497134 5059 scope.go:122] "RemoveContainer" containerID="886ffe058a2d7e153fb7bef50ff1ae6a6fd3015f69a3035f02a85dceae7822d8" Nov 03 20:11:36.507062 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:36.507039 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns"] Nov 03 20:11:36.511939 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:36.511918 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-p7qns"] Nov 03 20:11:36.520628 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:36.520606 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr"] Nov 03 20:11:36.526265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:36.526244 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-bzpxr"] Nov 03 20:11:36.538884 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:36.538851 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd"] Nov 03 20:11:36.544625 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:36.544603 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nlwxd"] Nov 03 20:11:37.064591 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:37.063494 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676"] Nov 03 20:11:37.235743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:37.235710 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:11:37.237198 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:37.237176 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:11:37.239101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:37.239079 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:11:37.859170 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:37.859139 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4be209ff-6d62-42d5-a0e3-22373143ecf5" path="/var/lib/kubelet/pods/4be209ff-6d62-42d5-a0e3-22373143ecf5/volumes" Nov 03 20:11:37.859724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:37.859707 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beb966fa-cc63-40e1-9cb2-aabb05c56717" path="/var/lib/kubelet/pods/beb966fa-cc63-40e1-9cb2-aabb05c56717/volumes" Nov 03 20:11:37.860221 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:37.860205 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f900a500-5deb-4f3a-a861-adc38c5b5d23" path="/var/lib/kubelet/pods/f900a500-5deb-4f3a-a861-adc38c5b5d23/volumes" Nov 03 20:11:40.606060 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:11:40.606021 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:11:42.067227 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:42.067165 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676" podUID="aee62de0-0be9-4d5c-9158-c1ca8e86a9be" containerName="payload-processing" containerID="cri-o://3530d5e98f6dd413c771b6ab06e6e590f0758c03e86b6ba6bd7ea14915534a19" gracePeriod=25 Nov 03 20:11:42.725191 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:42.713819 5059 generic.go:356] "Generic (PLEG): container finished" podID="aee62de0-0be9-4d5c-9158-c1ca8e86a9be" containerID="3530d5e98f6dd413c771b6ab06e6e590f0758c03e86b6ba6bd7ea14915534a19" exitCode=0 Nov 03 20:11:42.725191 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:42.714074 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676" event={"ID":"aee62de0-0be9-4d5c-9158-c1ca8e86a9be","Type":"ContainerDied","Data":"3530d5e98f6dd413c771b6ab06e6e590f0758c03e86b6ba6bd7ea14915534a19"} Nov 03 20:11:42.977487 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:42.977429 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676" Nov 03 20:11:43.183142 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:43.182544 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-routing-overlay-volume\") pod \"aee62de0-0be9-4d5c-9158-c1ca8e86a9be\" (UID: \"aee62de0-0be9-4d5c-9158-c1ca8e86a9be\") " Nov 03 20:11:43.183142 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:43.182868 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-plugins-config-volume\") pod \"aee62de0-0be9-4d5c-9158-c1ca8e86a9be\" (UID: \"aee62de0-0be9-4d5c-9158-c1ca8e86a9be\") " Nov 03 20:11:43.183142 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:43.182973 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-provider-credentials\" (UniqueName: \"kubernetes.io/projected/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-provider-credentials\") pod \"aee62de0-0be9-4d5c-9158-c1ca8e86a9be\" (UID: \"aee62de0-0be9-4d5c-9158-c1ca8e86a9be\") " Nov 03 20:11:43.183142 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:43.183007 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-routing-overlay-volume" pod "aee62de0-0be9-4d5c-9158-c1ca8e86a9be" (UID: "aee62de0-0be9-4d5c-9158-c1ca8e86a9be"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:11:43.183142 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:43.183022 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-kube-api-access-q7gzs\" (UniqueName: \"kubernetes.io/projected/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-kube-api-access-q7gzs\") pod \"aee62de0-0be9-4d5c-9158-c1ca8e86a9be\" (UID: \"aee62de0-0be9-4d5c-9158-c1ca8e86a9be\") " Nov 03 20:11:43.183930 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:43.183842 5059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-routing-overlay-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:43.189709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:43.185910 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-kube-api-access-q7gzs" pod "aee62de0-0be9-4d5c-9158-c1ca8e86a9be" (UID: "aee62de0-0be9-4d5c-9158-c1ca8e86a9be"). InnerVolumeSpecName "kube-api-access-q7gzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:11:43.189709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:43.185917 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-plugins-config-volume" pod "aee62de0-0be9-4d5c-9158-c1ca8e86a9be" (UID: "aee62de0-0be9-4d5c-9158-c1ca8e86a9be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:11:43.189908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:43.189837 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-provider-credentials" pod "aee62de0-0be9-4d5c-9158-c1ca8e86a9be" (UID: "aee62de0-0be9-4d5c-9158-c1ca8e86a9be"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:11:43.284949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:43.284886 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:43.284949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:43.284911 5059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-provider-credentials\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:43.284949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:43.284921 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7gzs\" (UniqueName: \"kubernetes.io/projected/aee62de0-0be9-4d5c-9158-c1ca8e86a9be-kube-api-access-q7gzs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:11:43.741376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:43.741343 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676" Nov 03 20:11:43.741376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:43.741370 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676" event={"ID":"aee62de0-0be9-4d5c-9158-c1ca8e86a9be","Type":"ContainerDied","Data":"e76fd6409bfd6b6fc2f3437d78006f7a976b292c975333b2831e25f3fc0b2bb0"} Nov 03 20:11:43.741603 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:43.741435 5059 scope.go:122] "RemoveContainer" containerID="3530d5e98f6dd413c771b6ab06e6e590f0758c03e86b6ba6bd7ea14915534a19" Nov 03 20:11:43.771500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:43.771474 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676"] Nov 03 20:11:43.775952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:43.775928 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799vf676"] Nov 03 20:11:43.858871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:43.858845 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aee62de0-0be9-4d5c-9158-c1ca8e86a9be" path="/var/lib/kubelet/pods/aee62de0-0be9-4d5c-9158-c1ca8e86a9be/volumes" Nov 03 20:11:46.824391 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:46.824354 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 20:11:46.826334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:11:46.826310 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 20:12:10.612826 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:12:10.612785 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:12:18.482685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.474302 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx"] Nov 03 20:12:18.482685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.475645 5059 cpu_manager.go:414] "containerMap: removing container" podUID="4be209ff-6d62-42d5-a0e3-22373143ecf5" containerName="istio-proxy" Nov 03 20:12:18.482685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.475688 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4be209ff-6d62-42d5-a0e3-22373143ecf5" containerName="istio-proxy" Nov 03 20:12:18.482685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.475714 5059 cpu_manager.go:414] "containerMap: removing container" podUID="aee62de0-0be9-4d5c-9158-c1ca8e86a9be" containerName="payload-processing" Nov 03 20:12:18.482685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.475723 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aee62de0-0be9-4d5c-9158-c1ca8e86a9be" containerName="payload-processing" Nov 03 20:12:18.482685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.475739 5059 cpu_manager.go:414] "containerMap: removing container" podUID="4be209ff-6d62-42d5-a0e3-22373143ecf5" containerName="istio-proxy" Nov 03 20:12:18.482685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.475747 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4be209ff-6d62-42d5-a0e3-22373143ecf5" containerName="istio-proxy" Nov 03 20:12:18.482685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.475766 5059 cpu_manager.go:414] "containerMap: removing container" podUID="f900a500-5deb-4f3a-a861-adc38c5b5d23" containerName="istio-proxy" Nov 03 20:12:18.482685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.475773 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f900a500-5deb-4f3a-a861-adc38c5b5d23" containerName="istio-proxy" Nov 03 20:12:18.482685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.475796 5059 cpu_manager.go:414] "containerMap: removing container" podUID="beb966fa-cc63-40e1-9cb2-aabb05c56717" containerName="istio-proxy" Nov 03 20:12:18.482685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.475802 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beb966fa-cc63-40e1-9cb2-aabb05c56717" containerName="istio-proxy" Nov 03 20:12:18.482685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.476014 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="beb966fa-cc63-40e1-9cb2-aabb05c56717" containerName="istio-proxy" Nov 03 20:12:18.482685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.476049 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4be209ff-6d62-42d5-a0e3-22373143ecf5" containerName="istio-proxy" Nov 03 20:12:18.482685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.476065 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="aee62de0-0be9-4d5c-9158-c1ca8e86a9be" containerName="payload-processing" Nov 03 20:12:18.482685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.476079 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4be209ff-6d62-42d5-a0e3-22373143ecf5" containerName="istio-proxy" Nov 03 20:12:18.482685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.476093 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f900a500-5deb-4f3a-a861-adc38c5b5d23" containerName="istio-proxy" Nov 03 20:12:18.722688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.702370 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:18.879072 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.747438 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-4c2a28-openshift-default-dockercfg-vhjbn\"" Nov 03 20:12:18.879072 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.808203 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-data\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:18.879072 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.808266 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-envoy\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:18.879072 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.808328 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-token\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:18.879072 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.808384 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-workload-socket\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:18.879072 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.808433 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdbbr\" (UniqueName: \"kubernetes.io/projected/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-kube-api-access-hdbbr\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:18.879072 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.808480 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-workload-certs\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:18.879072 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.808523 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istiod-ca-cert\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:18.879072 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.808698 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-podinfo\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:18.879072 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.808911 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-credential-socket\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:18.955686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.943729 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-podinfo\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:19.002987 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:19.002944 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-credential-socket\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:19.010687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:19.006770 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-data\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:19.033759 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:19.033721 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-envoy\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:19.034501 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:19.034476 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-token\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:19.053565 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:19.053527 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-workload-socket\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:19.054297 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:19.054269 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdbbr\" (UniqueName: \"kubernetes.io/projected/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-kube-api-access-hdbbr\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:19.089815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:19.089777 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-workload-certs\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:19.110856 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:19.110818 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istiod-ca-cert\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:19.155747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:19.054138 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-workload-socket\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:19.155747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:19.004306 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-credential-socket\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:19.155747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:18.987529 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx"] Nov 03 20:12:19.155747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:19.090409 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-workload-certs\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:19.249860 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:19.249817 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:12:19.299592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:19.016604 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-data\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:19.444738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:19.427570 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istiod-ca-cert\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:19.444738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:19.432623 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdbbr\" (UniqueName: \"kubernetes.io/projected/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-kube-api-access-hdbbr\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:19.459432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:19.454983 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-envoy\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:19.484592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:19.480626 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-podinfo\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:19.484592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:19.481127 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-token\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:19.762059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:19.761960 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:20.307746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:20.307469 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx"] Nov 03 20:12:20.314381 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:20.314339 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:12:20.314525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:20.314433 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:12:20.314525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:20.314477 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:12:20.495647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:20.495608 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" event={"ID":"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193","Type":"ContainerStarted","Data":"f7f451b988823a9abe607e4d43baa7cfb31cca43dcac0c1eb59db7b68d0ee813"} Nov 03 20:12:21.519630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:21.519586 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" event={"ID":"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193","Type":"ContainerStarted","Data":"f0c0cf57d91e43bac1c932a3dc7cf04c1c2e7f51acbb2554396c4f939b1e52cd"} Nov 03 20:12:21.545648 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:21.545567 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" podStartSLOduration=3.545553802 podStartE2EDuration="3.545553802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:21.544060946 +0000 UTC m=+2135.801858203" watchObservedRunningTime="2026-11-03 20:12:21.545553802 +0000 UTC m=+2135.803351059" Nov 03 20:12:21.765706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:21.765653 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:21.787738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:21.787016 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:22.578685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:22.578189 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:22.583455 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:22.583431 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:12:25.956865 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:25.956825 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7"] Nov 03 20:12:26.194929 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:26.194886 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7"] Nov 03 20:12:26.201937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:26.201836 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7" Nov 03 20:12:26.312012 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:26.311908 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qltdw\" (UniqueName: \"kubernetes.io/projected/2c363281-0226-420f-a963-5922d5ca609e-kube-api-access-qltdw\") pod \"payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7\" (UID: \"2c363281-0226-420f-a963-5922d5ca609e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7" Nov 03 20:12:26.325971 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:26.318172 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-4c2a28-dockercfg-psjnh\"" Nov 03 20:12:26.325971 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:26.318303 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-4c2a28\"" Nov 03 20:12:26.333934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:26.333074 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c363281-0226-420f-a963-5922d5ca609e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7\" (UID: \"2c363281-0226-420f-a963-5922d5ca609e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7" Nov 03 20:12:26.436240 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:26.436191 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qltdw\" (UniqueName: \"kubernetes.io/projected/2c363281-0226-420f-a963-5922d5ca609e-kube-api-access-qltdw\") pod \"payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7\" (UID: \"2c363281-0226-420f-a963-5922d5ca609e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7" Nov 03 20:12:26.436402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:26.436279 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c363281-0226-420f-a963-5922d5ca609e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7\" (UID: \"2c363281-0226-420f-a963-5922d5ca609e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7" Nov 03 20:12:26.437940 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:26.437910 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c363281-0226-420f-a963-5922d5ca609e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7\" (UID: \"2c363281-0226-420f-a963-5922d5ca609e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7" Nov 03 20:12:26.451259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:26.451229 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qltdw\" (UniqueName: \"kubernetes.io/projected/2c363281-0226-420f-a963-5922d5ca609e-kube-api-access-qltdw\") pod \"payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7\" (UID: \"2c363281-0226-420f-a963-5922d5ca609e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7" Nov 03 20:12:26.466982 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:26.466954 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7" Nov 03 20:12:26.831528 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:26.831467 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7"] Nov 03 20:12:27.701298 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:27.701242 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7" event={"ID":"2c363281-0226-420f-a963-5922d5ca609e","Type":"ContainerStarted","Data":"a1545bd75166fa5a492a8e6723253de1761cd417b14f627fbdab40e21ab83652"} Nov 03 20:12:27.701976 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:27.701316 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7" event={"ID":"2c363281-0226-420f-a963-5922d5ca609e","Type":"ContainerStarted","Data":"0790100bd49a069e5ae6fe894ddc55096ee6648b0ba9da367a971377af75ae84"} Nov 03 20:12:27.702551 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:27.702522 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7" Nov 03 20:12:27.724832 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:27.724776 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7" podStartSLOduration=2.724759292 podStartE2EDuration="2.724759292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:27.721949105 +0000 UTC m=+2141.979746363" watchObservedRunningTime="2026-11-03 20:12:27.724759292 +0000 UTC m=+2141.982556550" Nov 03 20:12:29.720795 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:29.720698 5059 generic.go:356] "Generic (PLEG): container finished" podID="d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" containerID="2f57092ac51573f4a21c3d1c3f99cddf2880faf5135c68f41abacbcf61bc2cfa" exitCode=0 Nov 03 20:12:29.720795 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:29.720771 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" event={"ID":"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840","Type":"ContainerDied","Data":"2f57092ac51573f4a21c3d1c3f99cddf2880faf5135c68f41abacbcf61bc2cfa"} Nov 03 20:12:29.721296 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:29.721277 5059 scope.go:122] "RemoveContainer" containerID="2f57092ac51573f4a21c3d1c3f99cddf2880faf5135c68f41abacbcf61bc2cfa" Nov 03 20:12:29.723493 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:29.723453 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:12:29.723597 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:29.723530 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:12:29.723597 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:29.723578 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:12:30.732256 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:30.732211 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" event={"ID":"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840","Type":"ContainerStarted","Data":"e7118a165e1c692020e00ab701f84d22189feb4501a06e12ef7daecc27e93292"} Nov 03 20:12:31.568686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:31.564603 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:12:31.568686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:31.564802 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Nov 03 20:12:31.568686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:31.564851 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" podUID="d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Nov 03 20:12:32.565685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:32.562036 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Nov 03 20:12:32.565685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:32.562133 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" podUID="d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Nov 03 20:12:33.046117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.044530 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5"] Nov 03 20:12:33.093600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.093559 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5"] Nov 03 20:12:33.093895 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.093770 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.227967 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.227883 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-data\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.228140 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.228036 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-workload-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.228140 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.228089 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-credential-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.228319 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.228291 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-token\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.228358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.228345 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-envoy\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.228448 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.228434 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istiod-ca-cert\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.228530 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.228502 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbmnk\" (UniqueName: \"kubernetes.io/projected/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-kube-api-access-fbmnk\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.228684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.228653 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-workload-certs\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.228760 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.228715 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-podinfo\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.330636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.330608 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-token\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.330767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.330653 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-envoy\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.330830 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.330814 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istiod-ca-cert\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.330867 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.330847 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbmnk\" (UniqueName: \"kubernetes.io/projected/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-kube-api-access-fbmnk\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.331015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.330998 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-workload-certs\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.331088 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.331036 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-podinfo\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.331150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.331087 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-data\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.331230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.331207 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-workload-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.331290 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.331246 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-workload-certs\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.331290 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.331265 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-credential-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.331505 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.331442 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-data\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.331644 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.331531 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-credential-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.331644 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.331557 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-workload-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.331644 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.331570 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istiod-ca-cert\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.332949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.332929 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-envoy\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.333422 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.333403 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-podinfo\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.342644 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.342612 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-token\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.342878 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.342859 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbmnk\" (UniqueName: \"kubernetes.io/projected/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-kube-api-access-fbmnk\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.424534 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.424502 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:33.572736 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.562197 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Nov 03 20:12:33.572736 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.562280 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" podUID="d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Nov 03 20:12:33.790842 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.790802 5059 generic.go:356] "Generic (PLEG): container finished" podID="154d0ad4-2191-4633-af2e-57ca9b458f42" containerID="efd770fcbd173d9ab92d2bc525a9123ed2006ef5998ec9ce1906a3109519a3de" exitCode=0 Nov 03 20:12:33.791052 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.790886 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" event={"ID":"154d0ad4-2191-4633-af2e-57ca9b458f42","Type":"ContainerDied","Data":"efd770fcbd173d9ab92d2bc525a9123ed2006ef5998ec9ce1906a3109519a3de"} Nov 03 20:12:33.791439 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.791416 5059 scope.go:122] "RemoveContainer" containerID="efd770fcbd173d9ab92d2bc525a9123ed2006ef5998ec9ce1906a3109519a3de" Nov 03 20:12:33.796590 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.796536 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:12:33.796764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.796735 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:12:33.798480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.796849 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:12:33.838419 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:12:33.838381 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab8320f7_2354_433a_a8a7_4d91cc5d37d2.slice/crio-5c734102d818e68d05b3ceb8796ce96b37ab13c4b474fd2808bd7c4bb32e2de8 WatchSource:0}: Error finding container 5c734102d818e68d05b3ceb8796ce96b37ab13c4b474fd2808bd7c4bb32e2de8: Status 404 returned error can't find the container with id 5c734102d818e68d05b3ceb8796ce96b37ab13c4b474fd2808bd7c4bb32e2de8 Nov 03 20:12:33.840971 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.840784 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5"] Nov 03 20:12:33.844706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.841835 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:12:33.844706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.841898 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:12:33.844706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:33.841943 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:12:34.564101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:34.564061 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Nov 03 20:12:34.564388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:34.564355 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" podUID="d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Nov 03 20:12:34.802024 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:34.801985 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" event={"ID":"ab8320f7-2354-433a-a8a7-4d91cc5d37d2","Type":"ContainerStarted","Data":"91f3fb2bb610fbeea4466047557a60f91aee6d1477483a4dc37e84c19840f464"} Nov 03 20:12:34.802448 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:34.802054 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" event={"ID":"ab8320f7-2354-433a-a8a7-4d91cc5d37d2","Type":"ContainerStarted","Data":"5c734102d818e68d05b3ceb8796ce96b37ab13c4b474fd2808bd7c4bb32e2de8"} Nov 03 20:12:34.804264 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:34.804235 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" event={"ID":"154d0ad4-2191-4633-af2e-57ca9b458f42","Type":"ContainerStarted","Data":"049c99efb365c2ff91b8218d5f60551a4bad355fbcc0a7ec9968e46acb798d68"} Nov 03 20:12:34.831287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:34.831234 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" podStartSLOduration=2.8312182359999998 podStartE2EDuration="2.831218236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:34.826407635 +0000 UTC m=+2149.084204893" watchObservedRunningTime="2026-11-03 20:12:34.831218236 +0000 UTC m=+2149.089015492" Nov 03 20:12:35.133744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:35.130321 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:12:35.176685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:35.176254 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:12:35.197686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:35.181808 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 20:12:35.197686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:35.181871 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" podUID="154d0ad4-2191-4633-af2e-57ca9b458f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 20:12:35.436782 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:35.436019 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:35.437819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:35.437789 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Nov 03 20:12:35.437924 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:35.437851 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" podUID="ab8320f7-2354-433a-a8a7-4d91cc5d37d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Nov 03 20:12:35.439935 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:35.439911 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:12:35.578685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:35.569777 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Nov 03 20:12:35.578685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:35.569840 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" podUID="d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Nov 03 20:12:35.933685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:35.932985 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c"] Nov 03 20:12:36.058689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.056790 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c"] Nov 03 20:12:36.058689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.056947 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:36.144525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.094058 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/64153a99-d5f7-4f92-a0c9-f79b39eba97c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5g9c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:36.144525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.094125 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/64153a99-d5f7-4f92-a0c9-f79b39eba97c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5g9c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:36.144525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.094257 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5g9c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:36.144525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.094303 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4rpm\" (UniqueName: \"kubernetes.io/projected/64153a99-d5f7-4f92-a0c9-f79b39eba97c-kube-api-access-p4rpm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5g9c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:36.144525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.094494 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5g9c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:36.144525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.094716 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5g9c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:36.144525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.122853 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5g9c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:36.234507 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.233303 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5g9c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:36.234507 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.233442 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5g9c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:36.234507 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.233514 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/64153a99-d5f7-4f92-a0c9-f79b39eba97c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5g9c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:36.234507 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.233552 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/64153a99-d5f7-4f92-a0c9-f79b39eba97c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5g9c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:36.234507 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.233631 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5g9c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:36.234507 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.233689 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4rpm\" (UniqueName: \"kubernetes.io/projected/64153a99-d5f7-4f92-a0c9-f79b39eba97c-kube-api-access-p4rpm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5g9c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:36.234507 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.233820 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5g9c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:36.237576 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.237459 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5g9c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:36.238079 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.238056 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 20:12:36.238312 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.238270 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" podUID="154d0ad4-2191-4633-af2e-57ca9b458f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 20:12:36.239138 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.239091 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5g9c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:36.253390 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.242548 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5g9c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:36.253390 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.243146 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/64153a99-d5f7-4f92-a0c9-f79b39eba97c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5g9c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:36.253390 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.246348 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/64153a99-d5f7-4f92-a0c9-f79b39eba97c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5g9c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:36.256854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.256825 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5g9c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:36.257481 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.257459 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4rpm\" (UniqueName: \"kubernetes.io/projected/64153a99-d5f7-4f92-a0c9-f79b39eba97c-kube-api-access-p4rpm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l5g9c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:36.428023 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.427558 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Nov 03 20:12:36.428023 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.427621 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" podUID="ab8320f7-2354-433a-a8a7-4d91cc5d37d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Nov 03 20:12:36.464684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.461882 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:36.583943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.583906 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Nov 03 20:12:36.584203 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:36.584174 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" podUID="d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Nov 03 20:12:37.503731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:37.501288 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Nov 03 20:12:37.503731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:37.501370 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" podUID="ab8320f7-2354-433a-a8a7-4d91cc5d37d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Nov 03 20:12:37.865955 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:37.865919 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:12:38.380841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:38.380783 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 20:12:38.466805 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:37.903374 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 20:12:38.467229 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:38.467185 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" podUID="154d0ad4-2191-4633-af2e-57ca9b458f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 20:12:38.553726 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:38.406847 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:12:38.553726 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:38.510700 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" podUID="08b3a993-1a64-4fe5-9577-b0f11a36779e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:12:38.596207 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:38.596060 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:12:38.726808 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:38.642287 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 20:12:38.727115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:38.727071 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" podUID="154d0ad4-2191-4633-af2e-57ca9b458f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 20:12:38.753294 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:38.753241 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:12:38.841708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:38.834603 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Nov 03 20:12:38.891886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:38.891841 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" podUID="ab8320f7-2354-433a-a8a7-4d91cc5d37d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Nov 03 20:12:38.935035 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:38.934990 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:12:38.947740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:38.941095 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:12:38.977058 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:38.976969 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" podUID="5ac27a9c-459f-4310-aef7-8dc643b46c6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:12:39.101687 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:12:39.079499 5059 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.082s" Nov 03 20:12:39.101687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:39.079638 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:12:39.101687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:39.079687 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c"] Nov 03 20:12:39.101687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:39.096592 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:12:39.101687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:39.096716 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:12:39.180706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:39.179493 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 20:12:39.180706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:39.179554 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" podUID="154d0ad4-2191-4633-af2e-57ca9b458f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 20:12:39.315316 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:39.315279 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" event={"ID":"64153a99-d5f7-4f92-a0c9-f79b39eba97c","Type":"ContainerStarted","Data":"8ba1f76672e6305c0b0bdd96ef45512c44b85b7829f85b511af9bea1f49228c9"} Nov 03 20:12:39.428805 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:39.428769 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Nov 03 20:12:39.429105 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:39.429060 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" podUID="ab8320f7-2354-433a-a8a7-4d91cc5d37d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Nov 03 20:12:39.781685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:39.775465 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7" Nov 03 20:12:40.176683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:40.176611 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 20:12:40.177006 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:40.176959 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" podUID="154d0ad4-2191-4633-af2e-57ca9b458f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 20:12:40.355471 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:40.354631 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" event={"ID":"64153a99-d5f7-4f92-a0c9-f79b39eba97c","Type":"ContainerStarted","Data":"1929782c8e0a4859c99bc362063000c768666ab59352d7c235edbdf79f308a9e"} Nov 03 20:12:40.366684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:40.356756 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:40.390685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:40.389822 5059 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-l5g9c container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:12:40.390685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:40.389878 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" podUID="64153a99-d5f7-4f92-a0c9-f79b39eba97c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:12:40.433758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:40.432174 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Nov 03 20:12:40.433758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:40.432236 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" podUID="ab8320f7-2354-433a-a8a7-4d91cc5d37d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Nov 03 20:12:40.447605 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:40.447529 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" podStartSLOduration=5.447510041 podStartE2EDuration="5.447510041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:40.404271383 +0000 UTC m=+2154.662068651" watchObservedRunningTime="2026-11-03 20:12:40.447510041 +0000 UTC m=+2154.705307300" Nov 03 20:12:40.775737 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:12:40.774934 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:12:41.172697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:41.172018 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 20:12:41.172697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:41.172083 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" podUID="154d0ad4-2191-4633-af2e-57ca9b458f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 20:12:41.450004 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:41.449841 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Nov 03 20:12:41.450004 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:41.449913 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" podUID="ab8320f7-2354-433a-a8a7-4d91cc5d37d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Nov 03 20:12:41.512443 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:41.512408 5059 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-l5g9c container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:12:41.512785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:41.512760 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" podUID="64153a99-d5f7-4f92-a0c9-f79b39eba97c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:12:42.433879 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:42.433835 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Nov 03 20:12:42.434402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:42.433903 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" podUID="ab8320f7-2354-433a-a8a7-4d91cc5d37d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Nov 03 20:12:42.697064 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:42.696979 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:12:42.698592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:42.698558 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:12:42.822411 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:42.822366 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cebde73-pqrbc"] Nov 03 20:12:42.827373 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:42.827352 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cebde73-pqrbc" Nov 03 20:12:42.838973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:42.838938 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:12:42.839159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:42.839140 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:12:42.839603 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:42.839575 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cebde73-pqrbc"] Nov 03 20:12:42.984936 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:42.984854 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4dd321e-0d3f-4295-bd95-c2ef50971afc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cebde73-pqrbc\" (UID: \"a4dd321e-0d3f-4295-bd95-c2ef50971afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cebde73-pqrbc" Nov 03 20:12:42.985096 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:42.984985 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4dd321e-0d3f-4295-bd95-c2ef50971afc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cebde73-pqrbc\" (UID: \"a4dd321e-0d3f-4295-bd95-c2ef50971afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cebde73-pqrbc" Nov 03 20:12:42.985213 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:42.985192 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9bwp\" (UniqueName: \"kubernetes.io/projected/a4dd321e-0d3f-4295-bd95-c2ef50971afc-kube-api-access-c9bwp\") pod \"maas-api-revoke-sub-3cebde73-pqrbc\" (UID: \"a4dd321e-0d3f-4295-bd95-c2ef50971afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cebde73-pqrbc" Nov 03 20:12:43.086419 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:43.086384 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9bwp\" (UniqueName: \"kubernetes.io/projected/a4dd321e-0d3f-4295-bd95-c2ef50971afc-kube-api-access-c9bwp\") pod \"maas-api-revoke-sub-3cebde73-pqrbc\" (UID: \"a4dd321e-0d3f-4295-bd95-c2ef50971afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cebde73-pqrbc" Nov 03 20:12:43.086612 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:43.086541 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4dd321e-0d3f-4295-bd95-c2ef50971afc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cebde73-pqrbc\" (UID: \"a4dd321e-0d3f-4295-bd95-c2ef50971afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cebde73-pqrbc" Nov 03 20:12:43.086761 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:43.086625 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4dd321e-0d3f-4295-bd95-c2ef50971afc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cebde73-pqrbc\" (UID: \"a4dd321e-0d3f-4295-bd95-c2ef50971afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cebde73-pqrbc" Nov 03 20:12:43.087355 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:43.087334 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4dd321e-0d3f-4295-bd95-c2ef50971afc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cebde73-pqrbc\" (UID: \"a4dd321e-0d3f-4295-bd95-c2ef50971afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cebde73-pqrbc" Nov 03 20:12:43.088972 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:43.088947 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4dd321e-0d3f-4295-bd95-c2ef50971afc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cebde73-pqrbc\" (UID: \"a4dd321e-0d3f-4295-bd95-c2ef50971afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cebde73-pqrbc" Nov 03 20:12:43.099112 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:43.099088 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9bwp\" (UniqueName: \"kubernetes.io/projected/a4dd321e-0d3f-4295-bd95-c2ef50971afc-kube-api-access-c9bwp\") pod \"maas-api-revoke-sub-3cebde73-pqrbc\" (UID: \"a4dd321e-0d3f-4295-bd95-c2ef50971afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cebde73-pqrbc" Nov 03 20:12:43.182837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:43.182814 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cebde73-pqrbc" Nov 03 20:12:43.440075 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:43.440036 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:43.441628 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:43.441599 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:43.442278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:43.442250 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:12:43.479544 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:12:43.479518 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4dd321e_0d3f_4295_bd95_c2ef50971afc.slice/crio-cfc5a57d06aac5d8f713e87fe5f6f22fb6c79e7eea1528c1d7b5755ffca3934a WatchSource:0}: Error finding container cfc5a57d06aac5d8f713e87fe5f6f22fb6c79e7eea1528c1d7b5755ffca3934a: Status 404 returned error can't find the container with id cfc5a57d06aac5d8f713e87fe5f6f22fb6c79e7eea1528c1d7b5755ffca3934a Nov 03 20:12:43.483176 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:43.483134 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cebde73-pqrbc"] Nov 03 20:12:43.537866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:43.537832 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cebde73-pqrbc" event={"ID":"a4dd321e-0d3f-4295-bd95-c2ef50971afc","Type":"ContainerStarted","Data":"cfc5a57d06aac5d8f713e87fe5f6f22fb6c79e7eea1528c1d7b5755ffca3934a"} Nov 03 20:12:43.845310 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:43.845279 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd"] Nov 03 20:12:43.852457 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:43.852429 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:43.869671 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:43.869629 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd"] Nov 03 20:12:43.939572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:43.939527 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:43.939572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:43.939575 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:43.939824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:43.939716 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:43.939824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:43.939762 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:43.939824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:43.939782 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:43.939946 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:43.939919 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:43.940045 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:43.940019 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f88ae6f3-f1d6-4aa3-ad33-589490696167-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:43.940590 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:43.940327 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkjmc\" (UniqueName: \"kubernetes.io/projected/f88ae6f3-f1d6-4aa3-ad33-589490696167-kube-api-access-vkjmc\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:43.940840 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:43.940820 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:44.042629 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.042545 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:44.042629 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.042589 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:44.042629 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.042632 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:44.042932 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.042778 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:44.042932 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.042814 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:44.042932 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.042898 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:44.042932 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.042928 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f88ae6f3-f1d6-4aa3-ad33-589490696167-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:44.043172 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.043098 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkjmc\" (UniqueName: \"kubernetes.io/projected/f88ae6f3-f1d6-4aa3-ad33-589490696167-kube-api-access-vkjmc\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:44.043237 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.043165 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:44.043237 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.043176 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:44.043375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.043101 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:44.043462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.043444 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:44.043531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.043460 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:44.043830 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.043810 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f88ae6f3-f1d6-4aa3-ad33-589490696167-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:44.045567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.045544 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:44.045782 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.045766 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:44.055211 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.055183 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:44.055407 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.055387 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkjmc\" (UniqueName: \"kubernetes.io/projected/f88ae6f3-f1d6-4aa3-ad33-589490696167-kube-api-access-vkjmc\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:44.187183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.187092 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:44.511812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.511781 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:12:44.532590 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:12:44.532550 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88ae6f3_f1d6_4aa3_ad33_589490696167.slice/crio-307295e491d089d6a319631a56952ba3f148983cbd35b208c9b1fa12d1f37be9 WatchSource:0}: Error finding container 307295e491d089d6a319631a56952ba3f148983cbd35b208c9b1fa12d1f37be9: Status 404 returned error can't find the container with id 307295e491d089d6a319631a56952ba3f148983cbd35b208c9b1fa12d1f37be9 Nov 03 20:12:44.533200 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.533179 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd"] Nov 03 20:12:44.535931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.535895 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:12:44.536033 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.535979 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:12:44.536033 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.536023 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:12:44.551339 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.551270 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" event={"ID":"f88ae6f3-f1d6-4aa3-ad33-589490696167","Type":"ContainerStarted","Data":"307295e491d089d6a319631a56952ba3f148983cbd35b208c9b1fa12d1f37be9"} Nov 03 20:12:44.553542 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.553516 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cebde73-pqrbc" event={"ID":"a4dd321e-0d3f-4295-bd95-c2ef50971afc","Type":"ContainerStarted","Data":"42779f04df1e4721774b3b54da0dbcfc59a67d87a90b4bd237140e7d01ff843b"} Nov 03 20:12:44.577960 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:44.577910 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cebde73-pqrbc" podStartSLOduration=2.577891855 podStartE2EDuration="2.577891855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:44.571533974 +0000 UTC m=+2158.829331232" watchObservedRunningTime="2026-11-03 20:12:44.577891855 +0000 UTC m=+2158.835689113" Nov 03 20:12:45.572082 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:45.572048 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" event={"ID":"f88ae6f3-f1d6-4aa3-ad33-589490696167","Type":"ContainerStarted","Data":"f35a41707de2c0253145d89c0f1f6796087f8f0d6d98dc9ca16c8dcbf8e24a1e"} Nov 03 20:12:45.578638 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:45.574892 5059 generic.go:356] "Generic (PLEG): container finished" podID="a4dd321e-0d3f-4295-bd95-c2ef50971afc" containerID="42779f04df1e4721774b3b54da0dbcfc59a67d87a90b4bd237140e7d01ff843b" exitCode=0 Nov 03 20:12:45.578638 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:45.575147 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cebde73-pqrbc" event={"ID":"a4dd321e-0d3f-4295-bd95-c2ef50971afc","Type":"ContainerDied","Data":"42779f04df1e4721774b3b54da0dbcfc59a67d87a90b4bd237140e7d01ff843b"} Nov 03 20:12:45.640721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:45.634287 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" podStartSLOduration=2.634267226 podStartE2EDuration="2.634267226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:45.633135036 +0000 UTC m=+2159.890932294" watchObservedRunningTime="2026-11-03 20:12:45.634267226 +0000 UTC m=+2159.892064485" Nov 03 20:12:46.010740 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:12:46.010579 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4dd321e_0d3f_4295_bd95_c2ef50971afc.slice/crio-cfc5a57d06aac5d8f713e87fe5f6f22fb6c79e7eea1528c1d7b5755ffca3934a\": RecentStats: unable to find data in memory cache]" Nov 03 20:12:46.188204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:46.188160 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:46.189857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:46.189812 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Nov 03 20:12:46.189980 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:46.189875 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" podUID="f88ae6f3-f1d6-4aa3-ad33-589490696167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Nov 03 20:12:47.188240 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:47.188204 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Nov 03 20:12:47.188678 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:47.188255 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" podUID="f88ae6f3-f1d6-4aa3-ad33-589490696167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Nov 03 20:12:47.216492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:47.216465 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cebde73-pqrbc" Nov 03 20:12:47.296849 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:47.296760 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4dd321e-0d3f-4295-bd95-c2ef50971afc-kube-api-access-c9bwp\" (UniqueName: \"kubernetes.io/projected/a4dd321e-0d3f-4295-bd95-c2ef50971afc-kube-api-access-c9bwp\") pod \"a4dd321e-0d3f-4295-bd95-c2ef50971afc\" (UID: \"a4dd321e-0d3f-4295-bd95-c2ef50971afc\") " Nov 03 20:12:47.297010 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:47.296910 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4dd321e-0d3f-4295-bd95-c2ef50971afc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4dd321e-0d3f-4295-bd95-c2ef50971afc-maas-api-xks-ca\") pod \"a4dd321e-0d3f-4295-bd95-c2ef50971afc\" (UID: \"a4dd321e-0d3f-4295-bd95-c2ef50971afc\") " Nov 03 20:12:47.297147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:47.297124 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4dd321e-0d3f-4295-bd95-c2ef50971afc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4dd321e-0d3f-4295-bd95-c2ef50971afc-maas-api-openshift-ca\") pod \"a4dd321e-0d3f-4295-bd95-c2ef50971afc\" (UID: \"a4dd321e-0d3f-4295-bd95-c2ef50971afc\") " Nov 03 20:12:47.297469 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:47.297432 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4dd321e-0d3f-4295-bd95-c2ef50971afc-maas-api-openshift-ca" pod "a4dd321e-0d3f-4295-bd95-c2ef50971afc" (UID: "a4dd321e-0d3f-4295-bd95-c2ef50971afc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:47.297948 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:47.297927 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4dd321e-0d3f-4295-bd95-c2ef50971afc-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:47.305432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:47.302794 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4dd321e-0d3f-4295-bd95-c2ef50971afc-maas-api-xks-ca" pod "a4dd321e-0d3f-4295-bd95-c2ef50971afc" (UID: "a4dd321e-0d3f-4295-bd95-c2ef50971afc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:47.305432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:47.302851 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4dd321e-0d3f-4295-bd95-c2ef50971afc-kube-api-access-c9bwp" pod "a4dd321e-0d3f-4295-bd95-c2ef50971afc" (UID: "a4dd321e-0d3f-4295-bd95-c2ef50971afc"). InnerVolumeSpecName "kube-api-access-c9bwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:47.399493 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:47.399457 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9bwp\" (UniqueName: \"kubernetes.io/projected/a4dd321e-0d3f-4295-bd95-c2ef50971afc-kube-api-access-c9bwp\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:47.399493 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:47.399490 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4dd321e-0d3f-4295-bd95-c2ef50971afc-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:47.739839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:47.739804 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cebde73-pqrbc" Nov 03 20:12:47.740039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:47.739834 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cebde73-pqrbc" event={"ID":"a4dd321e-0d3f-4295-bd95-c2ef50971afc","Type":"ContainerDied","Data":"cfc5a57d06aac5d8f713e87fe5f6f22fb6c79e7eea1528c1d7b5755ffca3934a"} Nov 03 20:12:47.740039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:47.739899 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfc5a57d06aac5d8f713e87fe5f6f22fb6c79e7eea1528c1d7b5755ffca3934a" Nov 03 20:12:48.188206 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:48.188172 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Nov 03 20:12:48.188386 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:48.188236 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" podUID="f88ae6f3-f1d6-4aa3-ad33-589490696167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Nov 03 20:12:49.188199 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:49.188162 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Nov 03 20:12:49.188392 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:49.188233 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" podUID="f88ae6f3-f1d6-4aa3-ad33-589490696167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Nov 03 20:12:50.188559 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:50.188520 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Nov 03 20:12:50.188984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:50.188578 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" podUID="f88ae6f3-f1d6-4aa3-ad33-589490696167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Nov 03 20:12:51.187556 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:51.187522 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body= Nov 03 20:12:51.187762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:51.187574 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" podUID="f88ae6f3-f1d6-4aa3-ad33-589490696167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" Nov 03 20:12:52.265552 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:52.265522 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:52.266176 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:52.266017 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:12:52.266765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:12:52.266747 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:13:08.812692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:08.810198 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a55f0974-wxtpc"] Nov 03 20:13:08.812692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:08.811563 5059 cpu_manager.go:414] "containerMap: removing container" podUID="a4dd321e-0d3f-4295-bd95-c2ef50971afc" containerName="revoke-keys" Nov 03 20:13:08.812692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:08.811588 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4dd321e-0d3f-4295-bd95-c2ef50971afc" containerName="revoke-keys" Nov 03 20:13:08.812692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:08.811816 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4dd321e-0d3f-4295-bd95-c2ef50971afc" containerName="revoke-keys" Nov 03 20:13:08.845222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:08.828726 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55f0974-wxtpc" Nov 03 20:13:08.948740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:08.938115 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a6ca31-3be0-4803-bcdb-998ffa9636e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a55f0974-wxtpc\" (UID: \"c0a6ca31-3be0-4803-bcdb-998ffa9636e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55f0974-wxtpc" Nov 03 20:13:08.948740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:08.938392 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddxgq\" (UniqueName: \"kubernetes.io/projected/c0a6ca31-3be0-4803-bcdb-998ffa9636e2-kube-api-access-ddxgq\") pod \"maas-api-revoke-sub-a55f0974-wxtpc\" (UID: \"c0a6ca31-3be0-4803-bcdb-998ffa9636e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55f0974-wxtpc" Nov 03 20:13:08.948740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:08.938437 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a6ca31-3be0-4803-bcdb-998ffa9636e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a55f0974-wxtpc\" (UID: \"c0a6ca31-3be0-4803-bcdb-998ffa9636e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55f0974-wxtpc" Nov 03 20:13:09.068876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:09.048925 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a6ca31-3be0-4803-bcdb-998ffa9636e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a55f0974-wxtpc\" (UID: \"c0a6ca31-3be0-4803-bcdb-998ffa9636e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55f0974-wxtpc" Nov 03 20:13:09.114699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:09.107075 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:13:09.114699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:09.107358 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:13:09.225762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:09.166636 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddxgq\" (UniqueName: \"kubernetes.io/projected/c0a6ca31-3be0-4803-bcdb-998ffa9636e2-kube-api-access-ddxgq\") pod \"maas-api-revoke-sub-a55f0974-wxtpc\" (UID: \"c0a6ca31-3be0-4803-bcdb-998ffa9636e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55f0974-wxtpc" Nov 03 20:13:09.358701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:09.358644 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a6ca31-3be0-4803-bcdb-998ffa9636e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a55f0974-wxtpc\" (UID: \"c0a6ca31-3be0-4803-bcdb-998ffa9636e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55f0974-wxtpc" Nov 03 20:13:09.360057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:09.346751 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a6ca31-3be0-4803-bcdb-998ffa9636e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a55f0974-wxtpc\" (UID: \"c0a6ca31-3be0-4803-bcdb-998ffa9636e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55f0974-wxtpc" Nov 03 20:13:09.360201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:09.358027 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a55f0974-wxtpc"] Nov 03 20:13:09.360201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:09.338497 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddxgq\" (UniqueName: \"kubernetes.io/projected/c0a6ca31-3be0-4803-bcdb-998ffa9636e2-kube-api-access-ddxgq\") pod \"maas-api-revoke-sub-a55f0974-wxtpc\" (UID: \"c0a6ca31-3be0-4803-bcdb-998ffa9636e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55f0974-wxtpc" Nov 03 20:13:09.360201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:09.359698 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a6ca31-3be0-4803-bcdb-998ffa9636e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a55f0974-wxtpc\" (UID: \"c0a6ca31-3be0-4803-bcdb-998ffa9636e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55f0974-wxtpc" Nov 03 20:13:09.368721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:09.368695 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5321be46-mtmr8"] Nov 03 20:13:09.375254 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:09.375233 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5321be46-mtmr8"] Nov 03 20:13:09.376264 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:09.376244 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55f0974-wxtpc" Nov 03 20:13:09.673328 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:13:09.673296 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a6ca31_3be0_4803_bcdb_998ffa9636e2.slice/crio-64caceb86f8176a9ab1a54d34181b80827b900cf14194e7ce9931763da62bc01 WatchSource:0}: Error finding container 64caceb86f8176a9ab1a54d34181b80827b900cf14194e7ce9931763da62bc01: Status 404 returned error can't find the container with id 64caceb86f8176a9ab1a54d34181b80827b900cf14194e7ce9931763da62bc01 Nov 03 20:13:09.673904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:09.673780 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a55f0974-wxtpc"] Nov 03 20:13:09.872858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:09.872820 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1274ddc7-26e8-4b34-be7a-bb7ac8bf7404" path="/var/lib/kubelet/pods/1274ddc7-26e8-4b34-be7a-bb7ac8bf7404/volumes" Nov 03 20:13:10.034090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:10.034058 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-198a5abc-42nff"] Nov 03 20:13:10.038656 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:10.038631 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-198a5abc-42nff"] Nov 03 20:13:10.379307 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:10.379273 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55f0974-wxtpc" event={"ID":"c0a6ca31-3be0-4803-bcdb-998ffa9636e2","Type":"ContainerStarted","Data":"5ed25179a720c02f6cd6430787db423d5e4ca1a560471a4494160bbc12737947"} Nov 03 20:13:10.379480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:10.379328 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55f0974-wxtpc" event={"ID":"c0a6ca31-3be0-4803-bcdb-998ffa9636e2","Type":"ContainerStarted","Data":"64caceb86f8176a9ab1a54d34181b80827b900cf14194e7ce9931763da62bc01"} Nov 03 20:13:10.400807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:10.400752 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55f0974-wxtpc" podStartSLOduration=2.400738379 podStartE2EDuration="2.400738379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:10.398218656 +0000 UTC m=+2184.656015915" watchObservedRunningTime="2026-11-03 20:13:10.400738379 +0000 UTC m=+2184.658535654" Nov 03 20:13:10.606763 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:13:10.606726 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:13:11.389742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:11.389707 5059 generic.go:356] "Generic (PLEG): container finished" podID="c0a6ca31-3be0-4803-bcdb-998ffa9636e2" containerID="5ed25179a720c02f6cd6430787db423d5e4ca1a560471a4494160bbc12737947" exitCode=0 Nov 03 20:13:11.390144 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:11.389780 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55f0974-wxtpc" event={"ID":"c0a6ca31-3be0-4803-bcdb-998ffa9636e2","Type":"ContainerDied","Data":"5ed25179a720c02f6cd6430787db423d5e4ca1a560471a4494160bbc12737947"} Nov 03 20:13:12.024680 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:12.022356 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c098791a-1c42-4ec9-aeaf-d1f82b2c12da" path="/var/lib/kubelet/pods/c098791a-1c42-4ec9-aeaf-d1f82b2c12da/volumes" Nov 03 20:13:12.594161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:12.594135 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55f0974-wxtpc" Nov 03 20:13:12.661083 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:12.660953 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0a6ca31-3be0-4803-bcdb-998ffa9636e2-kube-api-access-ddxgq\" (UniqueName: \"kubernetes.io/projected/c0a6ca31-3be0-4803-bcdb-998ffa9636e2-kube-api-access-ddxgq\") pod \"c0a6ca31-3be0-4803-bcdb-998ffa9636e2\" (UID: \"c0a6ca31-3be0-4803-bcdb-998ffa9636e2\") " Nov 03 20:13:12.661266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:12.661166 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0a6ca31-3be0-4803-bcdb-998ffa9636e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a6ca31-3be0-4803-bcdb-998ffa9636e2-maas-api-xks-ca\") pod \"c0a6ca31-3be0-4803-bcdb-998ffa9636e2\" (UID: \"c0a6ca31-3be0-4803-bcdb-998ffa9636e2\") " Nov 03 20:13:12.661345 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:12.661286 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0a6ca31-3be0-4803-bcdb-998ffa9636e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a6ca31-3be0-4803-bcdb-998ffa9636e2-maas-api-openshift-ca\") pod \"c0a6ca31-3be0-4803-bcdb-998ffa9636e2\" (UID: \"c0a6ca31-3be0-4803-bcdb-998ffa9636e2\") " Nov 03 20:13:12.663007 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:12.662964 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0a6ca31-3be0-4803-bcdb-998ffa9636e2-maas-api-openshift-ca" pod "c0a6ca31-3be0-4803-bcdb-998ffa9636e2" (UID: "c0a6ca31-3be0-4803-bcdb-998ffa9636e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:13:12.666081 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:12.666046 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a6ca31-3be0-4803-bcdb-998ffa9636e2-kube-api-access-ddxgq" pod "c0a6ca31-3be0-4803-bcdb-998ffa9636e2" (UID: "c0a6ca31-3be0-4803-bcdb-998ffa9636e2"). InnerVolumeSpecName "kube-api-access-ddxgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:13:12.666246 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:12.666152 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a6ca31-3be0-4803-bcdb-998ffa9636e2-maas-api-xks-ca" pod "c0a6ca31-3be0-4803-bcdb-998ffa9636e2" (UID: "c0a6ca31-3be0-4803-bcdb-998ffa9636e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:13:12.764648 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:12.764608 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddxgq\" (UniqueName: \"kubernetes.io/projected/c0a6ca31-3be0-4803-bcdb-998ffa9636e2-kube-api-access-ddxgq\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:12.764648 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:12.764651 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a6ca31-3be0-4803-bcdb-998ffa9636e2-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:12.764897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:12.764682 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a6ca31-3be0-4803-bcdb-998ffa9636e2-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:13.412734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:13.412700 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55f0974-wxtpc" event={"ID":"c0a6ca31-3be0-4803-bcdb-998ffa9636e2","Type":"ContainerDied","Data":"64caceb86f8176a9ab1a54d34181b80827b900cf14194e7ce9931763da62bc01"} Nov 03 20:13:13.412943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:13.412762 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64caceb86f8176a9ab1a54d34181b80827b900cf14194e7ce9931763da62bc01" Nov 03 20:13:13.412943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:13.412713 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a55f0974-wxtpc" Nov 03 20:13:19.475722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:19.475690 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-605cf807-5dt4l"] Nov 03 20:13:19.477455 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:19.477428 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c0a6ca31-3be0-4803-bcdb-998ffa9636e2" containerName="revoke-keys" Nov 03 20:13:19.735998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:19.735905 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a6ca31-3be0-4803-bcdb-998ffa9636e2" containerName="revoke-keys" Nov 03 20:13:19.736331 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:19.736304 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a6ca31-3be0-4803-bcdb-998ffa9636e2" containerName="revoke-keys" Nov 03 20:13:19.752686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:19.743094 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-605cf807-5dt4l" Nov 03 20:13:19.824036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:19.821821 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:13:19.824036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:19.821918 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:13:19.927710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:19.915059 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2942b7cd-a9df-4503-90c2-4299e5a5ed4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-605cf807-5dt4l\" (UID: \"2942b7cd-a9df-4503-90c2-4299e5a5ed4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605cf807-5dt4l" Nov 03 20:13:19.927710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:19.915179 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5ql4\" (UniqueName: \"kubernetes.io/projected/2942b7cd-a9df-4503-90c2-4299e5a5ed4f-kube-api-access-q5ql4\") pod \"maas-api-revoke-sub-605cf807-5dt4l\" (UID: \"2942b7cd-a9df-4503-90c2-4299e5a5ed4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605cf807-5dt4l" Nov 03 20:13:19.927710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:19.915399 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2942b7cd-a9df-4503-90c2-4299e5a5ed4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-605cf807-5dt4l\" (UID: \"2942b7cd-a9df-4503-90c2-4299e5a5ed4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605cf807-5dt4l" Nov 03 20:13:20.057743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:20.053480 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5ql4\" (UniqueName: \"kubernetes.io/projected/2942b7cd-a9df-4503-90c2-4299e5a5ed4f-kube-api-access-q5ql4\") pod \"maas-api-revoke-sub-605cf807-5dt4l\" (UID: \"2942b7cd-a9df-4503-90c2-4299e5a5ed4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605cf807-5dt4l" Nov 03 20:13:20.057743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:20.053845 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2942b7cd-a9df-4503-90c2-4299e5a5ed4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-605cf807-5dt4l\" (UID: \"2942b7cd-a9df-4503-90c2-4299e5a5ed4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605cf807-5dt4l" Nov 03 20:13:20.057743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:20.054031 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2942b7cd-a9df-4503-90c2-4299e5a5ed4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-605cf807-5dt4l\" (UID: \"2942b7cd-a9df-4503-90c2-4299e5a5ed4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605cf807-5dt4l" Nov 03 20:13:20.057743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:20.055972 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2942b7cd-a9df-4503-90c2-4299e5a5ed4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-605cf807-5dt4l\" (UID: \"2942b7cd-a9df-4503-90c2-4299e5a5ed4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605cf807-5dt4l" Nov 03 20:13:20.068145 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:20.068111 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2942b7cd-a9df-4503-90c2-4299e5a5ed4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-605cf807-5dt4l\" (UID: \"2942b7cd-a9df-4503-90c2-4299e5a5ed4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605cf807-5dt4l" Nov 03 20:13:20.093778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:20.086411 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5ql4\" (UniqueName: \"kubernetes.io/projected/2942b7cd-a9df-4503-90c2-4299e5a5ed4f-kube-api-access-q5ql4\") pod \"maas-api-revoke-sub-605cf807-5dt4l\" (UID: \"2942b7cd-a9df-4503-90c2-4299e5a5ed4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605cf807-5dt4l" Nov 03 20:13:20.094356 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:20.092126 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-605cf807-5dt4l"] Nov 03 20:13:20.120684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:20.120612 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:20.257528 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:20.256582 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-605cf807-5dt4l" Nov 03 20:13:20.632827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:20.632783 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0796d7-hhv2k"] Nov 03 20:13:20.846772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:20.846736 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0796d7-hhv2k"] Nov 03 20:13:20.856477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:20.856208 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59649767-prkjd"] Nov 03 20:13:20.903714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:20.902124 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59649767-prkjd"] Nov 03 20:13:21.107219 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:13:21.107188 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2942b7cd_a9df_4503_90c2_4299e5a5ed4f.slice/crio-b75accc29c5d1e62c9bd71894442bf3592da2aa860dd1449023432346a0dbe7f WatchSource:0}: Error finding container b75accc29c5d1e62c9bd71894442bf3592da2aa860dd1449023432346a0dbe7f: Status 404 returned error can't find the container with id b75accc29c5d1e62c9bd71894442bf3592da2aa860dd1449023432346a0dbe7f Nov 03 20:13:21.109598 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:21.109571 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-605cf807-5dt4l"] Nov 03 20:13:22.094686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:22.067204 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06a4c119-410d-4e61-8db1-d440e0c371c9" path="/var/lib/kubelet/pods/06a4c119-410d-4e61-8db1-d440e0c371c9/volumes" Nov 03 20:13:22.096264 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:22.096225 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae874737-754b-4414-a571-2333feda6d47" path="/var/lib/kubelet/pods/ae874737-754b-4414-a571-2333feda6d47/volumes" Nov 03 20:13:22.118686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:22.105120 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-605cf807-5dt4l" event={"ID":"2942b7cd-a9df-4503-90c2-4299e5a5ed4f","Type":"ContainerStarted","Data":"b75accc29c5d1e62c9bd71894442bf3592da2aa860dd1449023432346a0dbe7f"} Nov 03 20:13:22.256695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:22.255866 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:23.388127 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:23.387114 5059 generic.go:356] "Generic (PLEG): container finished" podID="2942b7cd-a9df-4503-90c2-4299e5a5ed4f" containerID="f699606f369980d8290e3ce1e2cd9be2286f1319b3ed9a3df6365bb1d6c00987" exitCode=0 Nov 03 20:13:23.388127 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:23.387335 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-605cf807-5dt4l" event={"ID":"2942b7cd-a9df-4503-90c2-4299e5a5ed4f","Type":"ContainerDied","Data":"f699606f369980d8290e3ce1e2cd9be2286f1319b3ed9a3df6365bb1d6c00987"} Nov 03 20:13:24.883952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:24.883920 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-605cf807-5dt4l" Nov 03 20:13:24.947957 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:24.947931 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2942b7cd-a9df-4503-90c2-4299e5a5ed4f-kube-api-access-q5ql4\" (UniqueName: \"kubernetes.io/projected/2942b7cd-a9df-4503-90c2-4299e5a5ed4f-kube-api-access-q5ql4\") pod \"2942b7cd-a9df-4503-90c2-4299e5a5ed4f\" (UID: \"2942b7cd-a9df-4503-90c2-4299e5a5ed4f\") " Nov 03 20:13:24.948118 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:24.947991 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2942b7cd-a9df-4503-90c2-4299e5a5ed4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2942b7cd-a9df-4503-90c2-4299e5a5ed4f-maas-api-openshift-ca\") pod \"2942b7cd-a9df-4503-90c2-4299e5a5ed4f\" (UID: \"2942b7cd-a9df-4503-90c2-4299e5a5ed4f\") " Nov 03 20:13:24.948163 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:24.948131 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2942b7cd-a9df-4503-90c2-4299e5a5ed4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2942b7cd-a9df-4503-90c2-4299e5a5ed4f-maas-api-xks-ca\") pod \"2942b7cd-a9df-4503-90c2-4299e5a5ed4f\" (UID: \"2942b7cd-a9df-4503-90c2-4299e5a5ed4f\") " Nov 03 20:13:24.948413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:24.948383 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2942b7cd-a9df-4503-90c2-4299e5a5ed4f-maas-api-openshift-ca" pod "2942b7cd-a9df-4503-90c2-4299e5a5ed4f" (UID: "2942b7cd-a9df-4503-90c2-4299e5a5ed4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:13:24.949265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:24.949155 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2942b7cd-a9df-4503-90c2-4299e5a5ed4f-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:24.950370 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:24.950347 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2942b7cd-a9df-4503-90c2-4299e5a5ed4f-maas-api-xks-ca" pod "2942b7cd-a9df-4503-90c2-4299e5a5ed4f" (UID: "2942b7cd-a9df-4503-90c2-4299e5a5ed4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:13:24.950506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:24.950490 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2942b7cd-a9df-4503-90c2-4299e5a5ed4f-kube-api-access-q5ql4" pod "2942b7cd-a9df-4503-90c2-4299e5a5ed4f" (UID: "2942b7cd-a9df-4503-90c2-4299e5a5ed4f"). InnerVolumeSpecName "kube-api-access-q5ql4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:13:25.051035 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:25.050998 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5ql4\" (UniqueName: \"kubernetes.io/projected/2942b7cd-a9df-4503-90c2-4299e5a5ed4f-kube-api-access-q5ql4\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:25.051035 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:25.051036 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2942b7cd-a9df-4503-90c2-4299e5a5ed4f-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:25.353735 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:25.353652 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d28fd70-dm69h"] Nov 03 20:13:25.354542 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:25.354522 5059 cpu_manager.go:414] "containerMap: removing container" podUID="2942b7cd-a9df-4503-90c2-4299e5a5ed4f" containerName="revoke-keys" Nov 03 20:13:25.354609 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:25.354542 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2942b7cd-a9df-4503-90c2-4299e5a5ed4f" containerName="revoke-keys" Nov 03 20:13:25.354752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:25.354740 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2942b7cd-a9df-4503-90c2-4299e5a5ed4f" containerName="revoke-keys" Nov 03 20:13:25.361685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:25.359840 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d28fd70-dm69h" Nov 03 20:13:25.395855 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:25.395824 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d28fd70-dm69h"] Nov 03 20:13:25.483700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:25.483626 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs5ch\" (UniqueName: \"kubernetes.io/projected/7e962f76-40b4-459c-b600-c1cd72ddd43a-kube-api-access-gs5ch\") pod \"maas-api-revoke-sub-7d28fd70-dm69h\" (UID: \"7e962f76-40b4-459c-b600-c1cd72ddd43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d28fd70-dm69h" Nov 03 20:13:25.484006 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:25.483983 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e962f76-40b4-459c-b600-c1cd72ddd43a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d28fd70-dm69h\" (UID: \"7e962f76-40b4-459c-b600-c1cd72ddd43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d28fd70-dm69h" Nov 03 20:13:25.484078 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:25.484047 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e962f76-40b4-459c-b600-c1cd72ddd43a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d28fd70-dm69h\" (UID: \"7e962f76-40b4-459c-b600-c1cd72ddd43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d28fd70-dm69h" Nov 03 20:13:25.599684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:25.598821 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs5ch\" (UniqueName: \"kubernetes.io/projected/7e962f76-40b4-459c-b600-c1cd72ddd43a-kube-api-access-gs5ch\") pod \"maas-api-revoke-sub-7d28fd70-dm69h\" (UID: \"7e962f76-40b4-459c-b600-c1cd72ddd43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d28fd70-dm69h" Nov 03 20:13:25.599684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:25.599161 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e962f76-40b4-459c-b600-c1cd72ddd43a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d28fd70-dm69h\" (UID: \"7e962f76-40b4-459c-b600-c1cd72ddd43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d28fd70-dm69h" Nov 03 20:13:25.599684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:25.599251 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e962f76-40b4-459c-b600-c1cd72ddd43a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d28fd70-dm69h\" (UID: \"7e962f76-40b4-459c-b600-c1cd72ddd43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d28fd70-dm69h" Nov 03 20:13:25.603677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:25.601176 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e962f76-40b4-459c-b600-c1cd72ddd43a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d28fd70-dm69h\" (UID: \"7e962f76-40b4-459c-b600-c1cd72ddd43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d28fd70-dm69h" Nov 03 20:13:25.607723 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:25.607192 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-605cf807-5dt4l" event={"ID":"2942b7cd-a9df-4503-90c2-4299e5a5ed4f","Type":"ContainerDied","Data":"b75accc29c5d1e62c9bd71894442bf3592da2aa860dd1449023432346a0dbe7f"} Nov 03 20:13:25.607723 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:25.607247 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b75accc29c5d1e62c9bd71894442bf3592da2aa860dd1449023432346a0dbe7f" Nov 03 20:13:25.607723 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:25.607323 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-605cf807-5dt4l" Nov 03 20:13:25.614718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:25.612007 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e962f76-40b4-459c-b600-c1cd72ddd43a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d28fd70-dm69h\" (UID: \"7e962f76-40b4-459c-b600-c1cd72ddd43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d28fd70-dm69h" Nov 03 20:13:25.614718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:25.613919 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs5ch\" (UniqueName: \"kubernetes.io/projected/7e962f76-40b4-459c-b600-c1cd72ddd43a-kube-api-access-gs5ch\") pod \"maas-api-revoke-sub-7d28fd70-dm69h\" (UID: \"7e962f76-40b4-459c-b600-c1cd72ddd43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d28fd70-dm69h" Nov 03 20:13:25.699123 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:25.699086 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d28fd70-dm69h" Nov 03 20:13:26.936691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:26.925309 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d28fd70-dm69h"] Nov 03 20:13:27.736034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:27.735999 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d28fd70-dm69h" event={"ID":"7e962f76-40b4-459c-b600-c1cd72ddd43a","Type":"ContainerStarted","Data":"ee192b7e2870a7575f388924d9b1bf31233af9ee5f207d6799a544ebe36fc804"} Nov 03 20:13:27.736369 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:27.736347 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d28fd70-dm69h" event={"ID":"7e962f76-40b4-459c-b600-c1cd72ddd43a","Type":"ContainerStarted","Data":"4f64eb350100fa426b12551b39fd54d65c32f5da6f2f55376f568eea899e00ce"} Nov 03 20:13:27.789522 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:27.789416 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d28fd70-dm69h" podStartSLOduration=2.789396267 podStartE2EDuration="2.789396267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:27.756486754 +0000 UTC m=+2202.014284010" watchObservedRunningTime="2026-11-03 20:13:27.789396267 +0000 UTC m=+2202.047193525" Nov 03 20:13:28.540587 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:28.540518 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca7e8b9b-jtg7k"] Nov 03 20:13:28.545711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:28.545686 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7e8b9b-jtg7k" Nov 03 20:13:28.555205 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:28.555177 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca7e8b9b-jtg7k"] Nov 03 20:13:28.637633 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:28.637601 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c79062a1-763f-43de-9c97-6bf3b9d27fb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca7e8b9b-jtg7k\" (UID: \"c79062a1-763f-43de-9c97-6bf3b9d27fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7e8b9b-jtg7k" Nov 03 20:13:28.637833 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:28.637782 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c79062a1-763f-43de-9c97-6bf3b9d27fb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca7e8b9b-jtg7k\" (UID: \"c79062a1-763f-43de-9c97-6bf3b9d27fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7e8b9b-jtg7k" Nov 03 20:13:28.637916 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:28.637856 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rc7w\" (UniqueName: \"kubernetes.io/projected/c79062a1-763f-43de-9c97-6bf3b9d27fb9-kube-api-access-5rc7w\") pod \"maas-api-revoke-sub-ca7e8b9b-jtg7k\" (UID: \"c79062a1-763f-43de-9c97-6bf3b9d27fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7e8b9b-jtg7k" Nov 03 20:13:28.739103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:28.739062 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c79062a1-763f-43de-9c97-6bf3b9d27fb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca7e8b9b-jtg7k\" (UID: \"c79062a1-763f-43de-9c97-6bf3b9d27fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7e8b9b-jtg7k" Nov 03 20:13:28.739285 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:28.739123 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rc7w\" (UniqueName: \"kubernetes.io/projected/c79062a1-763f-43de-9c97-6bf3b9d27fb9-kube-api-access-5rc7w\") pod \"maas-api-revoke-sub-ca7e8b9b-jtg7k\" (UID: \"c79062a1-763f-43de-9c97-6bf3b9d27fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7e8b9b-jtg7k" Nov 03 20:13:28.739353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:28.739343 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c79062a1-763f-43de-9c97-6bf3b9d27fb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca7e8b9b-jtg7k\" (UID: \"c79062a1-763f-43de-9c97-6bf3b9d27fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7e8b9b-jtg7k" Nov 03 20:13:28.739996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:28.739967 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c79062a1-763f-43de-9c97-6bf3b9d27fb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca7e8b9b-jtg7k\" (UID: \"c79062a1-763f-43de-9c97-6bf3b9d27fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7e8b9b-jtg7k" Nov 03 20:13:28.741582 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:28.741553 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c79062a1-763f-43de-9c97-6bf3b9d27fb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca7e8b9b-jtg7k\" (UID: \"c79062a1-763f-43de-9c97-6bf3b9d27fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7e8b9b-jtg7k" Nov 03 20:13:28.746409 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:28.746378 5059 generic.go:356] "Generic (PLEG): container finished" podID="7e962f76-40b4-459c-b600-c1cd72ddd43a" containerID="ee192b7e2870a7575f388924d9b1bf31233af9ee5f207d6799a544ebe36fc804" exitCode=0 Nov 03 20:13:28.746516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:28.746417 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d28fd70-dm69h" event={"ID":"7e962f76-40b4-459c-b600-c1cd72ddd43a","Type":"ContainerDied","Data":"ee192b7e2870a7575f388924d9b1bf31233af9ee5f207d6799a544ebe36fc804"} Nov 03 20:13:28.752026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:28.751999 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rc7w\" (UniqueName: \"kubernetes.io/projected/c79062a1-763f-43de-9c97-6bf3b9d27fb9-kube-api-access-5rc7w\") pod \"maas-api-revoke-sub-ca7e8b9b-jtg7k\" (UID: \"c79062a1-763f-43de-9c97-6bf3b9d27fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7e8b9b-jtg7k" Nov 03 20:13:28.878229 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:28.878196 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7e8b9b-jtg7k" Nov 03 20:13:29.272536 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:29.272500 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca7e8b9b-jtg7k"] Nov 03 20:13:29.759271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:29.759235 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7e8b9b-jtg7k" event={"ID":"c79062a1-763f-43de-9c97-6bf3b9d27fb9","Type":"ContainerStarted","Data":"4e655c42295929ba687cf5b5615e912fa9a10072e9f0e29b4771d3d696cb0dcb"} Nov 03 20:13:29.759769 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:29.759301 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7e8b9b-jtg7k" event={"ID":"c79062a1-763f-43de-9c97-6bf3b9d27fb9","Type":"ContainerStarted","Data":"8f4e5a39bdb4e5770a98700911644a8453ef1f8973e7885b789a52c72ae5a23d"} Nov 03 20:13:29.781532 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:29.781482 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7e8b9b-jtg7k" podStartSLOduration=1.7814645690000002 podStartE2EDuration="1.781464569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:29.776960214 +0000 UTC m=+2204.034757474" watchObservedRunningTime="2026-11-03 20:13:29.781464569 +0000 UTC m=+2204.039261839" Nov 03 20:13:30.032593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:30.032558 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d28fd70-dm69h" Nov 03 20:13:30.143755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:30.143711 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e962f76-40b4-459c-b600-c1cd72ddd43a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e962f76-40b4-459c-b600-c1cd72ddd43a-maas-api-xks-ca\") pod \"7e962f76-40b4-459c-b600-c1cd72ddd43a\" (UID: \"7e962f76-40b4-459c-b600-c1cd72ddd43a\") " Nov 03 20:13:30.143935 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:30.143805 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e962f76-40b4-459c-b600-c1cd72ddd43a-kube-api-access-gs5ch\" (UniqueName: \"kubernetes.io/projected/7e962f76-40b4-459c-b600-c1cd72ddd43a-kube-api-access-gs5ch\") pod \"7e962f76-40b4-459c-b600-c1cd72ddd43a\" (UID: \"7e962f76-40b4-459c-b600-c1cd72ddd43a\") " Nov 03 20:13:30.144132 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:30.144111 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e962f76-40b4-459c-b600-c1cd72ddd43a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e962f76-40b4-459c-b600-c1cd72ddd43a-maas-api-openshift-ca\") pod \"7e962f76-40b4-459c-b600-c1cd72ddd43a\" (UID: \"7e962f76-40b4-459c-b600-c1cd72ddd43a\") " Nov 03 20:13:30.144460 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:30.144430 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e962f76-40b4-459c-b600-c1cd72ddd43a-maas-api-openshift-ca" pod "7e962f76-40b4-459c-b600-c1cd72ddd43a" (UID: "7e962f76-40b4-459c-b600-c1cd72ddd43a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:13:30.145102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:30.145073 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e962f76-40b4-459c-b600-c1cd72ddd43a-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:30.146042 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:30.146009 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e962f76-40b4-459c-b600-c1cd72ddd43a-maas-api-xks-ca" pod "7e962f76-40b4-459c-b600-c1cd72ddd43a" (UID: "7e962f76-40b4-459c-b600-c1cd72ddd43a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:13:30.146231 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:30.146205 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e962f76-40b4-459c-b600-c1cd72ddd43a-kube-api-access-gs5ch" pod "7e962f76-40b4-459c-b600-c1cd72ddd43a" (UID: "7e962f76-40b4-459c-b600-c1cd72ddd43a"). InnerVolumeSpecName "kube-api-access-gs5ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:13:30.250204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:30.250107 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e962f76-40b4-459c-b600-c1cd72ddd43a-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:30.250204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:30.250152 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs5ch\" (UniqueName: \"kubernetes.io/projected/7e962f76-40b4-459c-b600-c1cd72ddd43a-kube-api-access-gs5ch\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:30.773307 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:30.773266 5059 generic.go:356] "Generic (PLEG): container finished" podID="c79062a1-763f-43de-9c97-6bf3b9d27fb9" containerID="4e655c42295929ba687cf5b5615e912fa9a10072e9f0e29b4771d3d696cb0dcb" exitCode=0 Nov 03 20:13:30.773854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:30.773357 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7e8b9b-jtg7k" event={"ID":"c79062a1-763f-43de-9c97-6bf3b9d27fb9","Type":"ContainerDied","Data":"4e655c42295929ba687cf5b5615e912fa9a10072e9f0e29b4771d3d696cb0dcb"} Nov 03 20:13:30.775197 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:30.775167 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d28fd70-dm69h" event={"ID":"7e962f76-40b4-459c-b600-c1cd72ddd43a","Type":"ContainerDied","Data":"4f64eb350100fa426b12551b39fd54d65c32f5da6f2f55376f568eea899e00ce"} Nov 03 20:13:30.775342 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:30.775217 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f64eb350100fa426b12551b39fd54d65c32f5da6f2f55376f568eea899e00ce" Nov 03 20:13:30.775342 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:30.775242 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d28fd70-dm69h" Nov 03 20:13:32.036411 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.036331 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7e8b9b-jtg7k" Nov 03 20:13:32.099454 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.099408 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c79062a1-763f-43de-9c97-6bf3b9d27fb9-kube-api-access-5rc7w\" (UniqueName: \"kubernetes.io/projected/c79062a1-763f-43de-9c97-6bf3b9d27fb9-kube-api-access-5rc7w\") pod \"c79062a1-763f-43de-9c97-6bf3b9d27fb9\" (UID: \"c79062a1-763f-43de-9c97-6bf3b9d27fb9\") " Nov 03 20:13:32.099657 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.099602 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c79062a1-763f-43de-9c97-6bf3b9d27fb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c79062a1-763f-43de-9c97-6bf3b9d27fb9-maas-api-xks-ca\") pod \"c79062a1-763f-43de-9c97-6bf3b9d27fb9\" (UID: \"c79062a1-763f-43de-9c97-6bf3b9d27fb9\") " Nov 03 20:13:32.099769 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.099719 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c79062a1-763f-43de-9c97-6bf3b9d27fb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c79062a1-763f-43de-9c97-6bf3b9d27fb9-maas-api-openshift-ca\") pod \"c79062a1-763f-43de-9c97-6bf3b9d27fb9\" (UID: \"c79062a1-763f-43de-9c97-6bf3b9d27fb9\") " Nov 03 20:13:32.100093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.100067 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c79062a1-763f-43de-9c97-6bf3b9d27fb9-maas-api-openshift-ca" pod "c79062a1-763f-43de-9c97-6bf3b9d27fb9" (UID: "c79062a1-763f-43de-9c97-6bf3b9d27fb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:13:32.100370 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.100353 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c79062a1-763f-43de-9c97-6bf3b9d27fb9-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:32.102171 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.102147 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c79062a1-763f-43de-9c97-6bf3b9d27fb9-maas-api-xks-ca" pod "c79062a1-763f-43de-9c97-6bf3b9d27fb9" (UID: "c79062a1-763f-43de-9c97-6bf3b9d27fb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:13:32.102286 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.102191 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c79062a1-763f-43de-9c97-6bf3b9d27fb9-kube-api-access-5rc7w" pod "c79062a1-763f-43de-9c97-6bf3b9d27fb9" (UID: "c79062a1-763f-43de-9c97-6bf3b9d27fb9"). InnerVolumeSpecName "kube-api-access-5rc7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:13:32.205292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.204256 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c79062a1-763f-43de-9c97-6bf3b9d27fb9-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:32.205292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.204305 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rc7w\" (UniqueName: \"kubernetes.io/projected/c79062a1-763f-43de-9c97-6bf3b9d27fb9-kube-api-access-5rc7w\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:32.673858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.673825 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aced2fa-t7d6c"] Nov 03 20:13:32.674657 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.674639 5059 cpu_manager.go:414] "containerMap: removing container" podUID="7e962f76-40b4-459c-b600-c1cd72ddd43a" containerName="revoke-keys" Nov 03 20:13:32.674741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.674671 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e962f76-40b4-459c-b600-c1cd72ddd43a" containerName="revoke-keys" Nov 03 20:13:32.674741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.674690 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c79062a1-763f-43de-9c97-6bf3b9d27fb9" containerName="revoke-keys" Nov 03 20:13:32.674741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.674712 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c79062a1-763f-43de-9c97-6bf3b9d27fb9" containerName="revoke-keys" Nov 03 20:13:32.674868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.674858 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e962f76-40b4-459c-b600-c1cd72ddd43a" containerName="revoke-keys" Nov 03 20:13:32.674908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.674874 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c79062a1-763f-43de-9c97-6bf3b9d27fb9" containerName="revoke-keys" Nov 03 20:13:32.794255 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.794221 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7e8b9b-jtg7k" Nov 03 20:13:32.806060 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.806034 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aced2fa-t7d6c"] Nov 03 20:13:32.806175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.806071 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca7e8b9b-jtg7k" event={"ID":"c79062a1-763f-43de-9c97-6bf3b9d27fb9","Type":"ContainerDied","Data":"8f4e5a39bdb4e5770a98700911644a8453ef1f8973e7885b789a52c72ae5a23d"} Nov 03 20:13:32.806175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.806089 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f4e5a39bdb4e5770a98700911644a8453ef1f8973e7885b789a52c72ae5a23d" Nov 03 20:13:32.806175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.806134 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aced2fa-t7d6c" Nov 03 20:13:32.810098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.810074 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:13:32.810296 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.810276 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:13:32.919267 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.919234 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a00812c-0595-4045-8d46-735a667dc32e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5aced2fa-t7d6c\" (UID: \"1a00812c-0595-4045-8d46-735a667dc32e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aced2fa-t7d6c" Nov 03 20:13:32.919498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.919299 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a00812c-0595-4045-8d46-735a667dc32e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5aced2fa-t7d6c\" (UID: \"1a00812c-0595-4045-8d46-735a667dc32e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aced2fa-t7d6c" Nov 03 20:13:32.919498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:32.919409 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl5w4\" (UniqueName: \"kubernetes.io/projected/1a00812c-0595-4045-8d46-735a667dc32e-kube-api-access-nl5w4\") pod \"maas-api-revoke-sub-5aced2fa-t7d6c\" (UID: \"1a00812c-0595-4045-8d46-735a667dc32e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aced2fa-t7d6c" Nov 03 20:13:33.020946 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:33.020866 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a00812c-0595-4045-8d46-735a667dc32e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5aced2fa-t7d6c\" (UID: \"1a00812c-0595-4045-8d46-735a667dc32e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aced2fa-t7d6c" Nov 03 20:13:33.020946 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:33.020931 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a00812c-0595-4045-8d46-735a667dc32e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5aced2fa-t7d6c\" (UID: \"1a00812c-0595-4045-8d46-735a667dc32e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aced2fa-t7d6c" Nov 03 20:13:33.021145 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:33.020961 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl5w4\" (UniqueName: \"kubernetes.io/projected/1a00812c-0595-4045-8d46-735a667dc32e-kube-api-access-nl5w4\") pod \"maas-api-revoke-sub-5aced2fa-t7d6c\" (UID: \"1a00812c-0595-4045-8d46-735a667dc32e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aced2fa-t7d6c" Nov 03 20:13:33.021552 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:33.021528 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a00812c-0595-4045-8d46-735a667dc32e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5aced2fa-t7d6c\" (UID: \"1a00812c-0595-4045-8d46-735a667dc32e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aced2fa-t7d6c" Nov 03 20:13:33.023393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:33.023371 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a00812c-0595-4045-8d46-735a667dc32e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5aced2fa-t7d6c\" (UID: \"1a00812c-0595-4045-8d46-735a667dc32e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aced2fa-t7d6c" Nov 03 20:13:33.033564 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:33.033539 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl5w4\" (UniqueName: \"kubernetes.io/projected/1a00812c-0595-4045-8d46-735a667dc32e-kube-api-access-nl5w4\") pod \"maas-api-revoke-sub-5aced2fa-t7d6c\" (UID: \"1a00812c-0595-4045-8d46-735a667dc32e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aced2fa-t7d6c" Nov 03 20:13:33.268574 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:33.268539 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aced2fa-t7d6c" Nov 03 20:13:33.718334 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:13:33.718297 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a00812c_0595_4045_8d46_735a667dc32e.slice/crio-4b762d65cd3198d1cf2be689a5621e1d6d1edd49d2334e804536259891640ee9 WatchSource:0}: Error finding container 4b762d65cd3198d1cf2be689a5621e1d6d1edd49d2334e804536259891640ee9: Status 404 returned error can't find the container with id 4b762d65cd3198d1cf2be689a5621e1d6d1edd49d2334e804536259891640ee9 Nov 03 20:13:33.720551 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:33.720516 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aced2fa-t7d6c"] Nov 03 20:13:33.803703 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:33.803632 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aced2fa-t7d6c" event={"ID":"1a00812c-0595-4045-8d46-735a667dc32e","Type":"ContainerStarted","Data":"4b762d65cd3198d1cf2be689a5621e1d6d1edd49d2334e804536259891640ee9"} Nov 03 20:13:34.814013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:34.813930 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aced2fa-t7d6c" event={"ID":"1a00812c-0595-4045-8d46-735a667dc32e","Type":"ContainerStarted","Data":"1c8649494a394f35f22a91b1ef156c134f8777bae822ea21a75e4a1bc8656536"} Nov 03 20:13:34.835405 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:34.835356 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aced2fa-t7d6c" podStartSLOduration=2.835344051 podStartE2EDuration="2.835344051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:34.833285087 +0000 UTC m=+2209.091082340" watchObservedRunningTime="2026-11-03 20:13:34.835344051 +0000 UTC m=+2209.093141308" Nov 03 20:13:35.829170 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:35.829133 5059 generic.go:356] "Generic (PLEG): container finished" podID="1a00812c-0595-4045-8d46-735a667dc32e" containerID="1c8649494a394f35f22a91b1ef156c134f8777bae822ea21a75e4a1bc8656536" exitCode=0 Nov 03 20:13:35.829542 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:35.829214 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aced2fa-t7d6c" event={"ID":"1a00812c-0595-4045-8d46-735a667dc32e","Type":"ContainerDied","Data":"1c8649494a394f35f22a91b1ef156c134f8777bae822ea21a75e4a1bc8656536"} Nov 03 20:13:36.740143 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:36.740105 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdaa2d49-5b9ms"] Nov 03 20:13:36.827626 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:36.827588 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdaa2d49-5b9ms"] Nov 03 20:13:36.827843 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:36.827706 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdaa2d49-5b9ms" Nov 03 20:13:36.929754 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:36.928908 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd71f6f2-b3f5-438c-917a-6645c43c6fd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdaa2d49-5b9ms\" (UID: \"cd71f6f2-b3f5-438c-917a-6645c43c6fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdaa2d49-5b9ms" Nov 03 20:13:36.929754 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:36.929000 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd71f6f2-b3f5-438c-917a-6645c43c6fd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdaa2d49-5b9ms\" (UID: \"cd71f6f2-b3f5-438c-917a-6645c43c6fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdaa2d49-5b9ms" Nov 03 20:13:36.929754 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:36.929075 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j88fw\" (UniqueName: \"kubernetes.io/projected/cd71f6f2-b3f5-438c-917a-6645c43c6fd2-kube-api-access-j88fw\") pod \"maas-api-revoke-sub-bdaa2d49-5b9ms\" (UID: \"cd71f6f2-b3f5-438c-917a-6645c43c6fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdaa2d49-5b9ms" Nov 03 20:13:37.030871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:37.030832 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd71f6f2-b3f5-438c-917a-6645c43c6fd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdaa2d49-5b9ms\" (UID: \"cd71f6f2-b3f5-438c-917a-6645c43c6fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdaa2d49-5b9ms" Nov 03 20:13:37.031054 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:37.030928 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd71f6f2-b3f5-438c-917a-6645c43c6fd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdaa2d49-5b9ms\" (UID: \"cd71f6f2-b3f5-438c-917a-6645c43c6fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdaa2d49-5b9ms" Nov 03 20:13:37.031054 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:37.031001 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j88fw\" (UniqueName: \"kubernetes.io/projected/cd71f6f2-b3f5-438c-917a-6645c43c6fd2-kube-api-access-j88fw\") pod \"maas-api-revoke-sub-bdaa2d49-5b9ms\" (UID: \"cd71f6f2-b3f5-438c-917a-6645c43c6fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdaa2d49-5b9ms" Nov 03 20:13:37.035333 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:37.035300 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd71f6f2-b3f5-438c-917a-6645c43c6fd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdaa2d49-5b9ms\" (UID: \"cd71f6f2-b3f5-438c-917a-6645c43c6fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdaa2d49-5b9ms" Nov 03 20:13:37.040277 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:37.040248 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd71f6f2-b3f5-438c-917a-6645c43c6fd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdaa2d49-5b9ms\" (UID: \"cd71f6f2-b3f5-438c-917a-6645c43c6fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdaa2d49-5b9ms" Nov 03 20:13:37.045767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:37.045721 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j88fw\" (UniqueName: \"kubernetes.io/projected/cd71f6f2-b3f5-438c-917a-6645c43c6fd2-kube-api-access-j88fw\") pod \"maas-api-revoke-sub-bdaa2d49-5b9ms\" (UID: \"cd71f6f2-b3f5-438c-917a-6645c43c6fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdaa2d49-5b9ms" Nov 03 20:13:37.132361 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:37.132337 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aced2fa-t7d6c" Nov 03 20:13:37.200218 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:37.200188 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdaa2d49-5b9ms" Nov 03 20:13:37.233287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:37.233256 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a00812c-0595-4045-8d46-735a667dc32e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a00812c-0595-4045-8d46-735a667dc32e-maas-api-openshift-ca\") pod \"1a00812c-0595-4045-8d46-735a667dc32e\" (UID: \"1a00812c-0595-4045-8d46-735a667dc32e\") " Nov 03 20:13:37.233528 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:37.233504 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a00812c-0595-4045-8d46-735a667dc32e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a00812c-0595-4045-8d46-735a667dc32e-maas-api-xks-ca\") pod \"1a00812c-0595-4045-8d46-735a667dc32e\" (UID: \"1a00812c-0595-4045-8d46-735a667dc32e\") " Nov 03 20:13:37.233613 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:37.233595 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a00812c-0595-4045-8d46-735a667dc32e-kube-api-access-nl5w4\" (UniqueName: \"kubernetes.io/projected/1a00812c-0595-4045-8d46-735a667dc32e-kube-api-access-nl5w4\") pod \"1a00812c-0595-4045-8d46-735a667dc32e\" (UID: \"1a00812c-0595-4045-8d46-735a667dc32e\") " Nov 03 20:13:37.233726 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:37.233699 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a00812c-0595-4045-8d46-735a667dc32e-maas-api-openshift-ca" pod "1a00812c-0595-4045-8d46-735a667dc32e" (UID: "1a00812c-0595-4045-8d46-735a667dc32e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:13:37.234544 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:37.234504 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a00812c-0595-4045-8d46-735a667dc32e-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:37.237118 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:37.237086 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a00812c-0595-4045-8d46-735a667dc32e-maas-api-xks-ca" pod "1a00812c-0595-4045-8d46-735a667dc32e" (UID: "1a00812c-0595-4045-8d46-735a667dc32e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:13:37.237815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:37.237623 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a00812c-0595-4045-8d46-735a667dc32e-kube-api-access-nl5w4" pod "1a00812c-0595-4045-8d46-735a667dc32e" (UID: "1a00812c-0595-4045-8d46-735a667dc32e"). InnerVolumeSpecName "kube-api-access-nl5w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:13:37.337295 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:37.336928 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a00812c-0595-4045-8d46-735a667dc32e-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:37.337295 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:37.336966 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl5w4\" (UniqueName: \"kubernetes.io/projected/1a00812c-0595-4045-8d46-735a667dc32e-kube-api-access-nl5w4\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:37.511873 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:13:37.511834 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd71f6f2_b3f5_438c_917a_6645c43c6fd2.slice/crio-e02e247b38015fbe88bcb28b3fcce2f52ce4054fdd165414345937e92e16da09 WatchSource:0}: Error finding container e02e247b38015fbe88bcb28b3fcce2f52ce4054fdd165414345937e92e16da09: Status 404 returned error can't find the container with id e02e247b38015fbe88bcb28b3fcce2f52ce4054fdd165414345937e92e16da09 Nov 03 20:13:37.512039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:37.511902 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdaa2d49-5b9ms"] Nov 03 20:13:37.851783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:37.851746 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aced2fa-t7d6c" Nov 03 20:13:37.862437 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:37.862408 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdaa2d49-5b9ms" event={"ID":"cd71f6f2-b3f5-438c-917a-6645c43c6fd2","Type":"ContainerStarted","Data":"e02e247b38015fbe88bcb28b3fcce2f52ce4054fdd165414345937e92e16da09"} Nov 03 20:13:37.862581 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:37.862451 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aced2fa-t7d6c" event={"ID":"1a00812c-0595-4045-8d46-735a667dc32e","Type":"ContainerDied","Data":"4b762d65cd3198d1cf2be689a5621e1d6d1edd49d2334e804536259891640ee9"} Nov 03 20:13:37.862581 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:37.862463 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b762d65cd3198d1cf2be689a5621e1d6d1edd49d2334e804536259891640ee9" Nov 03 20:13:38.619816 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:38.619776 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:13:38.620199 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:38.619837 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" podUID="2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:38.862745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:38.862708 5059 generic.go:356] "Generic (PLEG): container finished" podID="cd71f6f2-b3f5-438c-917a-6645c43c6fd2" containerID="877ba5b6bb8922da79517f070ebf5ed7acbd22f66190671c82224d47eb7dc8ac" exitCode=0 Nov 03 20:13:38.862938 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:38.862794 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdaa2d49-5b9ms" event={"ID":"cd71f6f2-b3f5-438c-917a-6645c43c6fd2","Type":"ContainerDied","Data":"877ba5b6bb8922da79517f070ebf5ed7acbd22f66190671c82224d47eb7dc8ac"} Nov 03 20:13:40.210845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.210755 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2388a678-8z68z"] Nov 03 20:13:40.410005 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.409972 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2388a678-8z68z"] Nov 03 20:13:40.423871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.423768 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87ed69a2-2w485"] Nov 03 20:13:40.425457 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.425254 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1a00812c-0595-4045-8d46-735a667dc32e" containerName="revoke-keys" Nov 03 20:13:40.425652 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.425636 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a00812c-0595-4045-8d46-735a667dc32e" containerName="revoke-keys" Nov 03 20:13:40.426040 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.426007 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a00812c-0595-4045-8d46-735a667dc32e" containerName="revoke-keys" Nov 03 20:13:40.475948 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.475878 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87ed69a2-2w485"] Nov 03 20:13:40.476110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.475982 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ed69a2-2w485" Nov 03 20:13:40.491922 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.491899 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdaa2d49-5b9ms" Nov 03 20:13:40.560764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.558718 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd71f6f2-b3f5-438c-917a-6645c43c6fd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd71f6f2-b3f5-438c-917a-6645c43c6fd2-maas-api-xks-ca\") pod \"cd71f6f2-b3f5-438c-917a-6645c43c6fd2\" (UID: \"cd71f6f2-b3f5-438c-917a-6645c43c6fd2\") " Nov 03 20:13:40.560764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.558957 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd71f6f2-b3f5-438c-917a-6645c43c6fd2-kube-api-access-j88fw\" (UniqueName: \"kubernetes.io/projected/cd71f6f2-b3f5-438c-917a-6645c43c6fd2-kube-api-access-j88fw\") pod \"cd71f6f2-b3f5-438c-917a-6645c43c6fd2\" (UID: \"cd71f6f2-b3f5-438c-917a-6645c43c6fd2\") " Nov 03 20:13:40.560764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.559166 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd71f6f2-b3f5-438c-917a-6645c43c6fd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd71f6f2-b3f5-438c-917a-6645c43c6fd2-maas-api-openshift-ca\") pod \"cd71f6f2-b3f5-438c-917a-6645c43c6fd2\" (UID: \"cd71f6f2-b3f5-438c-917a-6645c43c6fd2\") " Nov 03 20:13:40.560764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.559703 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e98c550f-1c06-4553-b266-979be0615bd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87ed69a2-2w485\" (UID: \"e98c550f-1c06-4553-b266-979be0615bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ed69a2-2w485" Nov 03 20:13:40.560764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.559744 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e98c550f-1c06-4553-b266-979be0615bd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87ed69a2-2w485\" (UID: \"e98c550f-1c06-4553-b266-979be0615bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ed69a2-2w485" Nov 03 20:13:40.560764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.559803 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4d7c\" (UniqueName: \"kubernetes.io/projected/e98c550f-1c06-4553-b266-979be0615bd3-kube-api-access-b4d7c\") pod \"maas-api-revoke-sub-87ed69a2-2w485\" (UID: \"e98c550f-1c06-4553-b266-979be0615bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ed69a2-2w485" Nov 03 20:13:40.586990 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.586878 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd71f6f2-b3f5-438c-917a-6645c43c6fd2-maas-api-openshift-ca" pod "cd71f6f2-b3f5-438c-917a-6645c43c6fd2" (UID: "cd71f6f2-b3f5-438c-917a-6645c43c6fd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:13:40.590834 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.589827 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd71f6f2-b3f5-438c-917a-6645c43c6fd2-maas-api-xks-ca" pod "cd71f6f2-b3f5-438c-917a-6645c43c6fd2" (UID: "cd71f6f2-b3f5-438c-917a-6645c43c6fd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:13:40.592574 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.592544 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd71f6f2-b3f5-438c-917a-6645c43c6fd2-kube-api-access-j88fw" pod "cd71f6f2-b3f5-438c-917a-6645c43c6fd2" (UID: "cd71f6f2-b3f5-438c-917a-6645c43c6fd2"). InnerVolumeSpecName "kube-api-access-j88fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:13:40.637778 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:13:40.637682 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:13:40.662053 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.662023 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e98c550f-1c06-4553-b266-979be0615bd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87ed69a2-2w485\" (UID: \"e98c550f-1c06-4553-b266-979be0615bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ed69a2-2w485" Nov 03 20:13:40.662213 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.662059 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e98c550f-1c06-4553-b266-979be0615bd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87ed69a2-2w485\" (UID: \"e98c550f-1c06-4553-b266-979be0615bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ed69a2-2w485" Nov 03 20:13:40.662213 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.662111 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4d7c\" (UniqueName: \"kubernetes.io/projected/e98c550f-1c06-4553-b266-979be0615bd3-kube-api-access-b4d7c\") pod \"maas-api-revoke-sub-87ed69a2-2w485\" (UID: \"e98c550f-1c06-4553-b266-979be0615bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ed69a2-2w485" Nov 03 20:13:40.662386 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.662345 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j88fw\" (UniqueName: \"kubernetes.io/projected/cd71f6f2-b3f5-438c-917a-6645c43c6fd2-kube-api-access-j88fw\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:40.662386 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.662371 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd71f6f2-b3f5-438c-917a-6645c43c6fd2-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:40.662518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.662387 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd71f6f2-b3f5-438c-917a-6645c43c6fd2-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:40.663002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.662984 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e98c550f-1c06-4553-b266-979be0615bd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87ed69a2-2w485\" (UID: \"e98c550f-1c06-4553-b266-979be0615bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ed69a2-2w485" Nov 03 20:13:40.664961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.664941 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e98c550f-1c06-4553-b266-979be0615bd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87ed69a2-2w485\" (UID: \"e98c550f-1c06-4553-b266-979be0615bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ed69a2-2w485" Nov 03 20:13:40.675138 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.675112 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4d7c\" (UniqueName: \"kubernetes.io/projected/e98c550f-1c06-4553-b266-979be0615bd3-kube-api-access-b4d7c\") pod \"maas-api-revoke-sub-87ed69a2-2w485\" (UID: \"e98c550f-1c06-4553-b266-979be0615bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ed69a2-2w485" Nov 03 20:13:40.846010 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.845971 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ed69a2-2w485" Nov 03 20:13:40.883950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.883924 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdaa2d49-5b9ms" event={"ID":"cd71f6f2-b3f5-438c-917a-6645c43c6fd2","Type":"ContainerDied","Data":"e02e247b38015fbe88bcb28b3fcce2f52ce4054fdd165414345937e92e16da09"} Nov 03 20:13:40.884098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.883968 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e02e247b38015fbe88bcb28b3fcce2f52ce4054fdd165414345937e92e16da09" Nov 03 20:13:40.884098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:40.883979 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdaa2d49-5b9ms" Nov 03 20:13:41.177221 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:13:41.177189 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode98c550f_1c06_4553_b266_979be0615bd3.slice/crio-087c2257d2f12f216d44ed881f1293a676fd62d402adac179874c6adf426ca12 WatchSource:0}: Error finding container 087c2257d2f12f216d44ed881f1293a676fd62d402adac179874c6adf426ca12: Status 404 returned error can't find the container with id 087c2257d2f12f216d44ed881f1293a676fd62d402adac179874c6adf426ca12 Nov 03 20:13:41.178264 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:41.178232 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87ed69a2-2w485"] Nov 03 20:13:41.859865 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:41.859789 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93fbae90-2272-43f8-a58d-392ccefd6028" path="/var/lib/kubelet/pods/93fbae90-2272-43f8-a58d-392ccefd6028/volumes" Nov 03 20:13:41.897624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:41.897592 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ed69a2-2w485" event={"ID":"e98c550f-1c06-4553-b266-979be0615bd3","Type":"ContainerStarted","Data":"b0fb18b05faf1c551b3d319859a3a3fbe61243db3aff2e8f5bf61992514fcd4e"} Nov 03 20:13:41.897839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:41.897641 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ed69a2-2w485" event={"ID":"e98c550f-1c06-4553-b266-979be0615bd3","Type":"ContainerStarted","Data":"087c2257d2f12f216d44ed881f1293a676fd62d402adac179874c6adf426ca12"} Nov 03 20:13:41.920105 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:41.920061 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ed69a2-2w485" podStartSLOduration=1.920046412 podStartE2EDuration="1.920046412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:41.917873509 +0000 UTC m=+2216.175670777" watchObservedRunningTime="2026-11-03 20:13:41.920046412 +0000 UTC m=+2216.177843674" Nov 03 20:13:42.025582 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:42.025542 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b708137-54g7m"] Nov 03 20:13:42.030702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:42.030651 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b708137-54g7m"] Nov 03 20:13:42.908265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:42.908234 5059 generic.go:356] "Generic (PLEG): container finished" podID="2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" containerID="f0c0cf57d91e43bac1c932a3dc7cf04c1c2e7f51acbb2554396c4f939b1e52cd" exitCode=0 Nov 03 20:13:42.908816 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:42.908266 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" event={"ID":"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193","Type":"ContainerDied","Data":"f0c0cf57d91e43bac1c932a3dc7cf04c1c2e7f51acbb2554396c4f939b1e52cd"} Nov 03 20:13:42.908816 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:42.908696 5059 scope.go:122] "RemoveContainer" containerID="f0c0cf57d91e43bac1c932a3dc7cf04c1c2e7f51acbb2554396c4f939b1e52cd" Nov 03 20:13:42.910305 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:42.910280 5059 generic.go:356] "Generic (PLEG): container finished" podID="e98c550f-1c06-4553-b266-979be0615bd3" containerID="b0fb18b05faf1c551b3d319859a3a3fbe61243db3aff2e8f5bf61992514fcd4e" exitCode=0 Nov 03 20:13:42.910431 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:42.910360 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ed69a2-2w485" event={"ID":"e98c550f-1c06-4553-b266-979be0615bd3","Type":"ContainerDied","Data":"b0fb18b05faf1c551b3d319859a3a3fbe61243db3aff2e8f5bf61992514fcd4e"} Nov 03 20:13:42.910628 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:42.910599 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:13:42.910719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:42.910689 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:13:42.910768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:42.910739 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:13:43.887335 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:43.887292 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bc45bfa-3edf-4f57-9db4-1b8c402e5017" path="/var/lib/kubelet/pods/6bc45bfa-3edf-4f57-9db4-1b8c402e5017/volumes" Nov 03 20:13:43.958710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:43.957765 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" event={"ID":"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193","Type":"ContainerStarted","Data":"6c9d6b78a3dbde46dbddcc1222495b4fce407ba8a3afc87ef4704aadd4524702"} Nov 03 20:13:44.750229 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:44.750199 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ed69a2-2w485" Nov 03 20:13:44.787657 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:44.787620 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:13:44.787912 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:44.787771 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Nov 03 20:13:44.787912 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:44.787823 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" podUID="2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Nov 03 20:13:44.981957 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:44.981911 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e98c550f-1c06-4553-b266-979be0615bd3-kube-api-access-b4d7c\" (UniqueName: \"kubernetes.io/projected/e98c550f-1c06-4553-b266-979be0615bd3-kube-api-access-b4d7c\") pod \"e98c550f-1c06-4553-b266-979be0615bd3\" (UID: \"e98c550f-1c06-4553-b266-979be0615bd3\") " Nov 03 20:13:45.005827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:45.005572 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e98c550f-1c06-4553-b266-979be0615bd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e98c550f-1c06-4553-b266-979be0615bd3-maas-api-xks-ca\") pod \"e98c550f-1c06-4553-b266-979be0615bd3\" (UID: \"e98c550f-1c06-4553-b266-979be0615bd3\") " Nov 03 20:13:45.006854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:45.006529 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e98c550f-1c06-4553-b266-979be0615bd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e98c550f-1c06-4553-b266-979be0615bd3-maas-api-openshift-ca\") pod \"e98c550f-1c06-4553-b266-979be0615bd3\" (UID: \"e98c550f-1c06-4553-b266-979be0615bd3\") " Nov 03 20:13:45.008832 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:45.008803 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e98c550f-1c06-4553-b266-979be0615bd3-maas-api-openshift-ca" pod "e98c550f-1c06-4553-b266-979be0615bd3" (UID: "e98c550f-1c06-4553-b266-979be0615bd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:13:45.009289 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:45.009271 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e98c550f-1c06-4553-b266-979be0615bd3-kube-api-access-b4d7c" pod "e98c550f-1c06-4553-b266-979be0615bd3" (UID: "e98c550f-1c06-4553-b266-979be0615bd3"). InnerVolumeSpecName "kube-api-access-b4d7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:13:45.010975 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:45.010956 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e98c550f-1c06-4553-b266-979be0615bd3-maas-api-xks-ca" pod "e98c550f-1c06-4553-b266-979be0615bd3" (UID: "e98c550f-1c06-4553-b266-979be0615bd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:13:45.013458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:45.013428 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ed69a2-2w485" event={"ID":"e98c550f-1c06-4553-b266-979be0615bd3","Type":"ContainerDied","Data":"087c2257d2f12f216d44ed881f1293a676fd62d402adac179874c6adf426ca12"} Nov 03 20:13:45.013578 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:45.013486 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="087c2257d2f12f216d44ed881f1293a676fd62d402adac179874c6adf426ca12" Nov 03 20:13:45.013578 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:45.013439 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ed69a2-2w485" Nov 03 20:13:45.110039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:45.109993 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e98c550f-1c06-4553-b266-979be0615bd3-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:45.110039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:45.110034 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4d7c\" (UniqueName: \"kubernetes.io/projected/e98c550f-1c06-4553-b266-979be0615bd3-kube-api-access-b4d7c\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:45.110253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:45.110054 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e98c550f-1c06-4553-b266-979be0615bd3-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:45.781685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:45.780770 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Nov 03 20:13:45.781685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:45.780840 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" podUID="2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Nov 03 20:13:46.763013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:46.762973 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Nov 03 20:13:46.763609 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:46.763573 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" podUID="2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Nov 03 20:13:47.770561 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:47.769389 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Nov 03 20:13:47.770561 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:47.769453 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" podUID="2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Nov 03 20:13:48.778090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:48.778048 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Nov 03 20:13:48.782802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:48.782759 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" podUID="2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Nov 03 20:13:49.877058 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:49.877026 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:13:49.877588 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:49.877403 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:13:49.878559 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:49.878534 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:13:52.026905 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:52.026869 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-684e69fc-dpdgd"] Nov 03 20:13:52.031559 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:52.031534 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-684e69fc-dpdgd"] Nov 03 20:13:53.029065 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:53.029031 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31e83134-kg9wj"] Nov 03 20:13:53.035051 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:53.035020 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31e83134-kg9wj"] Nov 03 20:13:53.860163 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:53.860124 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a948c428-547e-49e1-8d81-a4c3a0732f79" path="/var/lib/kubelet/pods/a948c428-547e-49e1-8d81-a4c3a0732f79/volumes" Nov 03 20:13:53.860743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:13:53.860719 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa74ca6a-b680-401c-9e7e-bb3c1bba2458" path="/var/lib/kubelet/pods/aa74ca6a-b680-401c-9e7e-bb3c1bba2458/volumes" Nov 03 20:14:10.614755 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:14:10.614706 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:14:28.143931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.143896 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf"] Nov 03 20:14:28.145090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.145060 5059 cpu_manager.go:414] "containerMap: removing container" podUID="e98c550f-1c06-4553-b266-979be0615bd3" containerName="revoke-keys" Nov 03 20:14:28.145090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.145088 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e98c550f-1c06-4553-b266-979be0615bd3" containerName="revoke-keys" Nov 03 20:14:28.145368 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.145154 5059 cpu_manager.go:414] "containerMap: removing container" podUID="cd71f6f2-b3f5-438c-917a-6645c43c6fd2" containerName="revoke-keys" Nov 03 20:14:28.145368 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.145163 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd71f6f2-b3f5-438c-917a-6645c43c6fd2" containerName="revoke-keys" Nov 03 20:14:28.145368 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.145326 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e98c550f-1c06-4553-b266-979be0615bd3" containerName="revoke-keys" Nov 03 20:14:28.145368 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.145354 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd71f6f2-b3f5-438c-917a-6645c43c6fd2" containerName="revoke-keys" Nov 03 20:14:28.148939 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.148916 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.172438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.172408 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf"] Nov 03 20:14:28.237121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.237081 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8adff4a5-5376-43ec-9b53-888705ae8236-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.237317 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.237145 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.237317 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.237232 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.237317 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.237264 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.237525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.237346 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8adff4a5-5376-43ec-9b53-888705ae8236-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.237525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.237398 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8adff4a5-5376-43ec-9b53-888705ae8236-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.237525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.237480 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.237761 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.237540 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.237761 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.237617 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw5v4\" (UniqueName: \"kubernetes.io/projected/8adff4a5-5376-43ec-9b53-888705ae8236-kube-api-access-dw5v4\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.339551 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.339514 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8adff4a5-5376-43ec-9b53-888705ae8236-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.339753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.339569 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8adff4a5-5376-43ec-9b53-888705ae8236-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.339753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.339643 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.339753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.339718 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.339892 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.339805 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw5v4\" (UniqueName: \"kubernetes.io/projected/8adff4a5-5376-43ec-9b53-888705ae8236-kube-api-access-dw5v4\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.340015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.339998 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8adff4a5-5376-43ec-9b53-888705ae8236-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.340058 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.340048 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.340117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.340086 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.340117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.340105 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.340229 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.340166 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.340229 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.340181 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.340477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.340451 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.340574 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.340512 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.340801 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.340778 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8adff4a5-5376-43ec-9b53-888705ae8236-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.342328 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.342305 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.342397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.342376 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8adff4a5-5376-43ec-9b53-888705ae8236-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.352191 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.352170 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw5v4\" (UniqueName: \"kubernetes.io/projected/8adff4a5-5376-43ec-9b53-888705ae8236-kube-api-access-dw5v4\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.352332 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.352312 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8adff4a5-5376-43ec-9b53-888705ae8236-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.483406 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.483312 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:28.642215 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.642170 5059 generic.go:356] "Generic (PLEG): container finished" podID="c9721811-d27d-4e13-a53d-4ae81e16fb96" containerID="05da4fb16c45de93fc800ccb459f5ce97d6b836ada2d5d7bf411997c2312e7b3" exitCode=0 Nov 03 20:14:28.642409 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.642246 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" event={"ID":"c9721811-d27d-4e13-a53d-4ae81e16fb96","Type":"ContainerDied","Data":"05da4fb16c45de93fc800ccb459f5ce97d6b836ada2d5d7bf411997c2312e7b3"} Nov 03 20:14:28.642868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.642842 5059 scope.go:122] "RemoveContainer" containerID="05da4fb16c45de93fc800ccb459f5ce97d6b836ada2d5d7bf411997c2312e7b3" Nov 03 20:14:28.645494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.645461 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:14:28.645638 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.645530 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:14:28.645638 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.645568 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:14:28.775694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.775642 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf"] Nov 03 20:14:28.776473 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:14:28.776449 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8adff4a5_5376_43ec_9b53_888705ae8236.slice/crio-ca0dc432182cad3fc0a9b45ad885449be4d6c0dc42dd143b8c84906cb422e240 WatchSource:0}: Error finding container ca0dc432182cad3fc0a9b45ad885449be4d6c0dc42dd143b8c84906cb422e240: Status 404 returned error can't find the container with id ca0dc432182cad3fc0a9b45ad885449be4d6c0dc42dd143b8c84906cb422e240 Nov 03 20:14:28.780111 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.780067 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:14:28.780227 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.780159 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:14:28.780227 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:28.780201 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:14:29.677043 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:29.662634 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" event={"ID":"8adff4a5-5376-43ec-9b53-888705ae8236","Type":"ContainerStarted","Data":"3904c49bb7ab63c515d01a6052936b8d990a8f9bb352ab2810819de54dc7b74c"} Nov 03 20:14:29.677043 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:29.662720 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" event={"ID":"8adff4a5-5376-43ec-9b53-888705ae8236","Type":"ContainerStarted","Data":"ca0dc432182cad3fc0a9b45ad885449be4d6c0dc42dd143b8c84906cb422e240"} Nov 03 20:14:29.714782 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:29.690291 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" event={"ID":"c9721811-d27d-4e13-a53d-4ae81e16fb96","Type":"ContainerStarted","Data":"b4dc2ca466f30e8024a88160d858421a5e6e6e115a118109fb3f8569f83e3ae2"} Nov 03 20:14:29.719728 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:29.715178 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" podStartSLOduration=1.715158731 podStartE2EDuration="1.715158731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:29.714083959 +0000 UTC m=+2263.971881217" watchObservedRunningTime="2026-11-03 20:14:29.715158731 +0000 UTC m=+2263.972955989" Nov 03 20:14:29.812687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:29.809230 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:14:29.822689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:29.816595 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 03 20:14:29.822689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:29.816656 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" podUID="c9721811-d27d-4e13-a53d-4ae81e16fb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 03 20:14:30.047285 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:30.047203 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:14:30.488686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:30.488373 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:30.511893 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:30.511256 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Nov 03 20:14:30.511893 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:30.511314 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" podUID="8adff4a5-5376-43ec-9b53-888705ae8236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Nov 03 20:14:30.799742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:30.799324 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 03 20:14:30.799742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:30.799418 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" podUID="c9721811-d27d-4e13-a53d-4ae81e16fb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 03 20:14:31.489777 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:31.489294 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Nov 03 20:14:31.489777 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:31.489357 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" podUID="8adff4a5-5376-43ec-9b53-888705ae8236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Nov 03 20:14:31.909684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:31.892927 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:31.909684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:31.895479 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 03 20:14:31.909684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:31.895539 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" podUID="c9721811-d27d-4e13-a53d-4ae81e16fb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 03 20:14:31.964688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:31.947428 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 03 20:14:32.150844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:32.150774 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:32.361712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:32.361652 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 20:14:32.505582 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:32.505543 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Nov 03 20:14:32.505920 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:32.505890 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" podUID="8adff4a5-5376-43ec-9b53-888705ae8236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Nov 03 20:14:32.702898 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:32.702790 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:32.803691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:32.801780 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:32.803691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:32.802680 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 03 20:14:32.803691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:32.802728 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" podUID="c9721811-d27d-4e13-a53d-4ae81e16fb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 03 20:14:32.954777 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:32.953026 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:32.954777 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:32.953560 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:14:33.207783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:33.177761 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": context deadline exceeded" Nov 03 20:14:33.458739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:33.457740 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:14:33.489688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:33.488992 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Nov 03 20:14:33.489688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:33.489052 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" podUID="8adff4a5-5376-43ec-9b53-888705ae8236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Nov 03 20:14:33.813849 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:33.813763 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 03 20:14:33.813849 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:33.813823 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" podUID="c9721811-d27d-4e13-a53d-4ae81e16fb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 03 20:14:34.485687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:34.485582 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Nov 03 20:14:34.485687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:34.485647 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" podUID="8adff4a5-5376-43ec-9b53-888705ae8236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Nov 03 20:14:34.809680 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:34.809576 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 03 20:14:34.810438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:34.809923 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" podUID="c9721811-d27d-4e13-a53d-4ae81e16fb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 03 20:14:35.499162 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:35.499123 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Nov 03 20:14:35.499970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:35.499934 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" podUID="8adff4a5-5376-43ec-9b53-888705ae8236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Nov 03 20:14:35.803583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:35.803068 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 03 20:14:35.803583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:35.803134 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" podUID="c9721811-d27d-4e13-a53d-4ae81e16fb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 03 20:14:36.490052 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:36.490010 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Nov 03 20:14:36.490377 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:36.490345 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" podUID="8adff4a5-5376-43ec-9b53-888705ae8236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Nov 03 20:14:36.800250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:36.800158 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 03 20:14:36.805054 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:36.805009 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" podUID="c9721811-d27d-4e13-a53d-4ae81e16fb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 03 20:14:37.550864 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:37.550832 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:37.551553 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:37.551522 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:37.552777 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:37.552756 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:14:37.803895 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:37.803819 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:14:37.804928 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:37.804904 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:14:40.607552 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:14:40.607514 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:14:55.660517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:55.660482 5059 scope.go:122] "RemoveContainer" containerID="f30d5e1b8f1a57b499fa1e0f545efc84621156d348838104dafc06d3649467f7" Nov 03 20:14:55.672809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:55.672783 5059 scope.go:122] "RemoveContainer" containerID="1702ed65b32c81888c4aca4681d437b77a3dbd27b1bcf66b6c100f5886f185ad" Nov 03 20:14:55.684353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:55.684302 5059 scope.go:122] "RemoveContainer" containerID="60567573d80bb56bd0def80ec667c579e8d02a0503a3699f3818c3e78110cb73" Nov 03 20:14:55.697009 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:55.696985 5059 scope.go:122] "RemoveContainer" containerID="c30f1dd89550433b076a505de418c277da5f2eea4729c0bca5b0deb1e1dddc7b" Nov 03 20:14:55.708527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:55.708508 5059 scope.go:122] "RemoveContainer" containerID="5dd04b77ab04596d2aadc80578199ac5a2c8f34a4fc8b357771411a4c706116c" Nov 03 20:14:55.724808 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:55.724780 5059 scope.go:122] "RemoveContainer" containerID="323db27c05933987cd87b4fff65b5db0b6716ecde2e1db56e18277ce0a06dc13" Nov 03 20:14:55.830954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:55.825950 5059 scope.go:122] "RemoveContainer" containerID="0b3d9bc6f51a12d437b7e1e86a549559399a9de59e0cde12f1a3d52b95ecdf17" Nov 03 20:14:55.844291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:55.844269 5059 scope.go:122] "RemoveContainer" containerID="d211a674673719c5b9691712d00ea28d3258a40aa94f820bb32431725834b965" Nov 03 20:14:57.194691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:57.187196 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x"] Nov 03 20:14:57.194691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:14:57.187433 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" podUID="f043a660-eddb-4de5-88a7-cf0b1fda6512" containerName="istio-proxy" containerID="cri-o://f9f4f7f919667909db546636d5524b9efb53ecdd7bf93966afd963aa9742b298" gracePeriod=30 Nov 03 20:15:00.141326 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.141286 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc"] Nov 03 20:15:00.148392 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.148364 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt"] Nov 03 20:15:00.148538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.148465 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc" Nov 03 20:15:00.153093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.153070 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm"] Nov 03 20:15:00.153265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.153243 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt" Nov 03 20:15:00.157745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.157725 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59"] Nov 03 20:15:00.157890 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.157849 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm" Nov 03 20:15:00.158862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.158843 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:15:00.164888 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.164853 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc"] Nov 03 20:15:00.165022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.164976 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59" Nov 03 20:15:00.176517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.172596 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc"] Nov 03 20:15:00.176517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.172724 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc" Nov 03 20:15:00.195380 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.179425 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt"] Nov 03 20:15:00.195380 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.187174 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59"] Nov 03 20:15:00.195380 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.192602 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm"] Nov 03 20:15:00.208122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.200671 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh"] Nov 03 20:15:00.211867 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.211406 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh" Nov 03 20:15:00.230382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.227524 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp"] Nov 03 20:15:00.233734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.233705 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp" Nov 03 20:15:00.242279 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.242249 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p"] Nov 03 20:15:00.247773 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.247748 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p" Nov 03 20:15:00.259586 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.258290 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp"] Nov 03 20:15:00.267063 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.264986 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc"] Nov 03 20:15:00.273684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.273591 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh"] Nov 03 20:15:00.279753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.279728 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p"] Nov 03 20:15:00.392575 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.391835 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlmgg\" (UniqueName: \"kubernetes.io/projected/0b18a5b0-0fc4-4dc8-88b7-affec66830d8-kube-api-access-tlmgg\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59\" (UID: \"0b18a5b0-0fc4-4dc8-88b7-affec66830d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59" Nov 03 20:15:00.392575 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.391989 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1e6b3a-7887-42c5-b23d-b226cc6d0641-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc\" (UID: \"3a1e6b3a-7887-42c5-b23d-b226cc6d0641\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc" Nov 03 20:15:00.392575 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.392094 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv7cg\" (UniqueName: \"kubernetes.io/projected/1be9592b-18d0-4ce7-9322-92898161c6d8-kube-api-access-kv7cg\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp\" (UID: \"1be9592b-18d0-4ce7-9322-92898161c6d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp" Nov 03 20:15:00.392575 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.392236 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmf2s\" (UniqueName: \"kubernetes.io/projected/1774de9e-305c-4b0f-b194-a09995fb25bf-kube-api-access-jmf2s\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc\" (UID: \"1774de9e-305c-4b0f-b194-a09995fb25bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc" Nov 03 20:15:00.392575 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.392283 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql7sk\" (UniqueName: \"kubernetes.io/projected/f1477a68-b92a-4138-b649-fdcd3b17f693-kube-api-access-ql7sk\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt\" (UID: \"f1477a68-b92a-4138-b649-fdcd3b17f693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt" Nov 03 20:15:00.392575 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.392418 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffb6bb03-2f3c-43f0-871a-1140b53f0fb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895615-ghpwh\" (UID: \"ffb6bb03-2f3c-43f0-871a-1140b53f0fb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh" Nov 03 20:15:00.395524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.392455 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1774de9e-305c-4b0f-b194-a09995fb25bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc\" (UID: \"1774de9e-305c-4b0f-b194-a09995fb25bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc" Nov 03 20:15:00.395524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.395214 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkd6f\" (UniqueName: \"kubernetes.io/projected/3a1e6b3a-7887-42c5-b23d-b226cc6d0641-kube-api-access-xkd6f\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc\" (UID: \"3a1e6b3a-7887-42c5-b23d-b226cc6d0641\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc" Nov 03 20:15:00.395524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.395276 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1477a68-b92a-4138-b649-fdcd3b17f693-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt\" (UID: \"f1477a68-b92a-4138-b649-fdcd3b17f693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt" Nov 03 20:15:00.395524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.395310 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8588fce3-f04c-472e-a796-ed5314845eb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm\" (UID: \"8588fce3-f04c-472e-a796-ed5314845eb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm" Nov 03 20:15:00.395524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.395393 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1be9592b-18d0-4ce7-9322-92898161c6d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp\" (UID: \"1be9592b-18d0-4ce7-9322-92898161c6d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp" Nov 03 20:15:00.395524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.395444 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h9c2\" (UniqueName: \"kubernetes.io/projected/ffb6bb03-2f3c-43f0-871a-1140b53f0fb9-kube-api-access-5h9c2\") pod \"maas-api-key-cleanup-29895615-ghpwh\" (UID: \"ffb6bb03-2f3c-43f0-871a-1140b53f0fb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh" Nov 03 20:15:00.396330 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.396086 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7cvn\" (UniqueName: \"kubernetes.io/projected/8588fce3-f04c-472e-a796-ed5314845eb5-kube-api-access-c7cvn\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm\" (UID: \"8588fce3-f04c-472e-a796-ed5314845eb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm" Nov 03 20:15:00.396330 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.396128 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b18a5b0-0fc4-4dc8-88b7-affec66830d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59\" (UID: \"0b18a5b0-0fc4-4dc8-88b7-affec66830d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59" Nov 03 20:15:00.497967 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.497933 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7cvn\" (UniqueName: \"kubernetes.io/projected/8588fce3-f04c-472e-a796-ed5314845eb5-kube-api-access-c7cvn\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm\" (UID: \"8588fce3-f04c-472e-a796-ed5314845eb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm" Nov 03 20:15:00.498132 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.497972 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b18a5b0-0fc4-4dc8-88b7-affec66830d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59\" (UID: \"0b18a5b0-0fc4-4dc8-88b7-affec66830d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59" Nov 03 20:15:00.498132 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.498002 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb7298dc-a500-4715-a360-bcbdf0c7a298-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p\" (UID: \"cb7298dc-a500-4715-a360-bcbdf0c7a298\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p" Nov 03 20:15:00.498132 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.498055 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlmgg\" (UniqueName: \"kubernetes.io/projected/0b18a5b0-0fc4-4dc8-88b7-affec66830d8-kube-api-access-tlmgg\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59\" (UID: \"0b18a5b0-0fc4-4dc8-88b7-affec66830d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59" Nov 03 20:15:00.498266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.498145 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1e6b3a-7887-42c5-b23d-b226cc6d0641-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc\" (UID: \"3a1e6b3a-7887-42c5-b23d-b226cc6d0641\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc" Nov 03 20:15:00.498266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.498194 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzjjb\" (UniqueName: \"kubernetes.io/projected/cb7298dc-a500-4715-a360-bcbdf0c7a298-kube-api-access-bzjjb\") pod \"maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p\" (UID: \"cb7298dc-a500-4715-a360-bcbdf0c7a298\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p" Nov 03 20:15:00.498266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.498254 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv7cg\" (UniqueName: \"kubernetes.io/projected/1be9592b-18d0-4ce7-9322-92898161c6d8-kube-api-access-kv7cg\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp\" (UID: \"1be9592b-18d0-4ce7-9322-92898161c6d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp" Nov 03 20:15:00.498438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.498410 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmf2s\" (UniqueName: \"kubernetes.io/projected/1774de9e-305c-4b0f-b194-a09995fb25bf-kube-api-access-jmf2s\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc\" (UID: \"1774de9e-305c-4b0f-b194-a09995fb25bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc" Nov 03 20:15:00.498488 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.498468 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql7sk\" (UniqueName: \"kubernetes.io/projected/f1477a68-b92a-4138-b649-fdcd3b17f693-kube-api-access-ql7sk\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt\" (UID: \"f1477a68-b92a-4138-b649-fdcd3b17f693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt" Nov 03 20:15:00.498600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.498583 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffb6bb03-2f3c-43f0-871a-1140b53f0fb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895615-ghpwh\" (UID: \"ffb6bb03-2f3c-43f0-871a-1140b53f0fb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh" Nov 03 20:15:00.498686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.498620 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1774de9e-305c-4b0f-b194-a09995fb25bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc\" (UID: \"1774de9e-305c-4b0f-b194-a09995fb25bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc" Nov 03 20:15:00.498803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.498772 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkd6f\" (UniqueName: \"kubernetes.io/projected/3a1e6b3a-7887-42c5-b23d-b226cc6d0641-kube-api-access-xkd6f\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc\" (UID: \"3a1e6b3a-7887-42c5-b23d-b226cc6d0641\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc" Nov 03 20:15:00.498866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.498829 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1477a68-b92a-4138-b649-fdcd3b17f693-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt\" (UID: \"f1477a68-b92a-4138-b649-fdcd3b17f693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt" Nov 03 20:15:00.498924 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.498862 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8588fce3-f04c-472e-a796-ed5314845eb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm\" (UID: \"8588fce3-f04c-472e-a796-ed5314845eb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm" Nov 03 20:15:00.498983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.498936 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1be9592b-18d0-4ce7-9322-92898161c6d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp\" (UID: \"1be9592b-18d0-4ce7-9322-92898161c6d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp" Nov 03 20:15:00.499042 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.498980 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b18a5b0-0fc4-4dc8-88b7-affec66830d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59\" (UID: \"0b18a5b0-0fc4-4dc8-88b7-affec66830d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59" Nov 03 20:15:00.499113 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.498981 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1e6b3a-7887-42c5-b23d-b226cc6d0641-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc\" (UID: \"3a1e6b3a-7887-42c5-b23d-b226cc6d0641\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc" Nov 03 20:15:00.499113 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.498987 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h9c2\" (UniqueName: \"kubernetes.io/projected/ffb6bb03-2f3c-43f0-871a-1140b53f0fb9-kube-api-access-5h9c2\") pod \"maas-api-key-cleanup-29895615-ghpwh\" (UID: \"ffb6bb03-2f3c-43f0-871a-1140b53f0fb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh" Nov 03 20:15:00.499410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.499385 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffb6bb03-2f3c-43f0-871a-1140b53f0fb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895615-ghpwh\" (UID: \"ffb6bb03-2f3c-43f0-871a-1140b53f0fb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh" Nov 03 20:15:00.499487 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.499450 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1774de9e-305c-4b0f-b194-a09995fb25bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc\" (UID: \"1774de9e-305c-4b0f-b194-a09995fb25bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc" Nov 03 20:15:00.499655 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.499638 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1be9592b-18d0-4ce7-9322-92898161c6d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp\" (UID: \"1be9592b-18d0-4ce7-9322-92898161c6d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp" Nov 03 20:15:00.499749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.499728 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1477a68-b92a-4138-b649-fdcd3b17f693-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt\" (UID: \"f1477a68-b92a-4138-b649-fdcd3b17f693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt" Nov 03 20:15:00.499796 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.499781 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8588fce3-f04c-472e-a796-ed5314845eb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm\" (UID: \"8588fce3-f04c-472e-a796-ed5314845eb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm" Nov 03 20:15:00.513499 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.513466 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkd6f\" (UniqueName: \"kubernetes.io/projected/3a1e6b3a-7887-42c5-b23d-b226cc6d0641-kube-api-access-xkd6f\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc\" (UID: \"3a1e6b3a-7887-42c5-b23d-b226cc6d0641\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc" Nov 03 20:15:00.514195 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.514160 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlmgg\" (UniqueName: \"kubernetes.io/projected/0b18a5b0-0fc4-4dc8-88b7-affec66830d8-kube-api-access-tlmgg\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59\" (UID: \"0b18a5b0-0fc4-4dc8-88b7-affec66830d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59" Nov 03 20:15:00.514305 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.514193 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7cvn\" (UniqueName: \"kubernetes.io/projected/8588fce3-f04c-472e-a796-ed5314845eb5-kube-api-access-c7cvn\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm\" (UID: \"8588fce3-f04c-472e-a796-ed5314845eb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm" Nov 03 20:15:00.514305 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.514226 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql7sk\" (UniqueName: \"kubernetes.io/projected/f1477a68-b92a-4138-b649-fdcd3b17f693-kube-api-access-ql7sk\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt\" (UID: \"f1477a68-b92a-4138-b649-fdcd3b17f693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt" Nov 03 20:15:00.514305 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.514271 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h9c2\" (UniqueName: \"kubernetes.io/projected/ffb6bb03-2f3c-43f0-871a-1140b53f0fb9-kube-api-access-5h9c2\") pod \"maas-api-key-cleanup-29895615-ghpwh\" (UID: \"ffb6bb03-2f3c-43f0-871a-1140b53f0fb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh" Nov 03 20:15:00.514305 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.514289 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmf2s\" (UniqueName: \"kubernetes.io/projected/1774de9e-305c-4b0f-b194-a09995fb25bf-kube-api-access-jmf2s\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc\" (UID: \"1774de9e-305c-4b0f-b194-a09995fb25bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc" Nov 03 20:15:00.514451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.514287 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv7cg\" (UniqueName: \"kubernetes.io/projected/1be9592b-18d0-4ce7-9322-92898161c6d8-kube-api-access-kv7cg\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp\" (UID: \"1be9592b-18d0-4ce7-9322-92898161c6d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp" Nov 03 20:15:00.550306 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.550283 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt" Nov 03 20:15:00.567127 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.567104 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm" Nov 03 20:15:00.579934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.579757 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59" Nov 03 20:15:00.590706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.590680 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc" Nov 03 20:15:00.602324 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.601622 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb7298dc-a500-4715-a360-bcbdf0c7a298-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p\" (UID: \"cb7298dc-a500-4715-a360-bcbdf0c7a298\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p" Nov 03 20:15:00.602324 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.601759 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzjjb\" (UniqueName: \"kubernetes.io/projected/cb7298dc-a500-4715-a360-bcbdf0c7a298-kube-api-access-bzjjb\") pod \"maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p\" (UID: \"cb7298dc-a500-4715-a360-bcbdf0c7a298\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p" Nov 03 20:15:00.602324 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.601778 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh" Nov 03 20:15:00.602700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.602642 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb7298dc-a500-4715-a360-bcbdf0c7a298-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p\" (UID: \"cb7298dc-a500-4715-a360-bcbdf0c7a298\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p" Nov 03 20:15:00.629872 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.629349 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzjjb\" (UniqueName: \"kubernetes.io/projected/cb7298dc-a500-4715-a360-bcbdf0c7a298-kube-api-access-bzjjb\") pod \"maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p\" (UID: \"cb7298dc-a500-4715-a360-bcbdf0c7a298\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p" Nov 03 20:15:00.659498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.653181 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp" Nov 03 20:15:00.699144 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.698154 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p" Nov 03 20:15:00.773114 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.773079 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc" Nov 03 20:15:00.846723 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:00.846395 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt"] Nov 03 20:15:00.850132 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:15:00.850100 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1477a68_b92a_4138_b649_fdcd3b17f693.slice/crio-f8b9a57d221b8140062f07fbdd5376564c8e3ed3a7029541d50b221d9d74e72e WatchSource:0}: Error finding container f8b9a57d221b8140062f07fbdd5376564c8e3ed3a7029541d50b221d9d74e72e: Status 404 returned error can't find the container with id f8b9a57d221b8140062f07fbdd5376564c8e3ed3a7029541d50b221d9d74e72e Nov 03 20:15:01.026141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:01.026108 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt" event={"ID":"f1477a68-b92a-4138-b649-fdcd3b17f693","Type":"ContainerStarted","Data":"f8b9a57d221b8140062f07fbdd5376564c8e3ed3a7029541d50b221d9d74e72e"} Nov 03 20:15:01.097822 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:15:01.097789 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1774de9e_305c_4b0f_b194_a09995fb25bf.slice/crio-0151b6dcaef645f4d2056752410a26e86127b7d0ac96168f0dbe37b3c4cf0c54 WatchSource:0}: Error finding container 0151b6dcaef645f4d2056752410a26e86127b7d0ac96168f0dbe37b3c4cf0c54: Status 404 returned error can't find the container with id 0151b6dcaef645f4d2056752410a26e86127b7d0ac96168f0dbe37b3c4cf0c54 Nov 03 20:15:01.099128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:01.098905 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc"] Nov 03 20:15:01.103156 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:15:01.103125 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7298dc_a500_4715_a360_bcbdf0c7a298.slice/crio-d87d71a5a5c9be19d71b4e2b3ad4ab28202b3b84f458132cae3b6777927887db WatchSource:0}: Error finding container d87d71a5a5c9be19d71b4e2b3ad4ab28202b3b84f458132cae3b6777927887db: Status 404 returned error can't find the container with id d87d71a5a5c9be19d71b4e2b3ad4ab28202b3b84f458132cae3b6777927887db Nov 03 20:15:01.105222 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:15:01.105184 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8588fce3_f04c_472e_a796_ed5314845eb5.slice/crio-83337441d5fc8aa3f9f21c6f388bd3717e0a48766933d358c20c3b06f2c975f0 WatchSource:0}: Error finding container 83337441d5fc8aa3f9f21c6f388bd3717e0a48766933d358c20c3b06f2c975f0: Status 404 returned error can't find the container with id 83337441d5fc8aa3f9f21c6f388bd3717e0a48766933d358c20c3b06f2c975f0 Nov 03 20:15:01.106098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:01.106075 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p"] Nov 03 20:15:01.111899 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:01.111879 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm"] Nov 03 20:15:01.116998 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:15:01.116973 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b18a5b0_0fc4_4dc8_88b7_affec66830d8.slice/crio-ddafb52d39e712193525e5596a1ee622603944f333153a926b3dae4e067b4fad WatchSource:0}: Error finding container ddafb52d39e712193525e5596a1ee622603944f333153a926b3dae4e067b4fad: Status 404 returned error can't find the container with id ddafb52d39e712193525e5596a1ee622603944f333153a926b3dae4e067b4fad Nov 03 20:15:01.117765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:01.117569 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59"] Nov 03 20:15:01.147608 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:01.147578 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc"] Nov 03 20:15:01.155161 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:15:01.155139 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1e6b3a_7887_42c5_b23d_b226cc6d0641.slice/crio-e7047c3a50e0e5af63c3aad56c3df9eed9fdbb01524dbc1e1aee5d0a30c1949b WatchSource:0}: Error finding container e7047c3a50e0e5af63c3aad56c3df9eed9fdbb01524dbc1e1aee5d0a30c1949b: Status 404 returned error can't find the container with id e7047c3a50e0e5af63c3aad56c3df9eed9fdbb01524dbc1e1aee5d0a30c1949b Nov 03 20:15:01.193766 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:01.193734 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh"] Nov 03 20:15:01.201006 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:01.200983 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp"] Nov 03 20:15:01.263064 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:15:01.263029 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb6bb03_2f3c_43f0_871a_1140b53f0fb9.slice/crio-ad52f985b75b77f67ea55389f8cb90df283d76b36b98979e8212d7270e641313 WatchSource:0}: Error finding container ad52f985b75b77f67ea55389f8cb90df283d76b36b98979e8212d7270e641313: Status 404 returned error can't find the container with id ad52f985b75b77f67ea55389f8cb90df283d76b36b98979e8212d7270e641313 Nov 03 20:15:01.264549 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:15:01.264457 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be9592b_18d0_4ce7_9322_92898161c6d8.slice/crio-b4f18413a134ddf59191a98736526c70cb01aedb4cb0a51e4737d8566d235d53 WatchSource:0}: Error finding container b4f18413a134ddf59191a98736526c70cb01aedb4cb0a51e4737d8566d235d53: Status 404 returned error can't find the container with id b4f18413a134ddf59191a98736526c70cb01aedb4cb0a51e4737d8566d235d53 Nov 03 20:15:02.036371 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.036272 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc" event={"ID":"1774de9e-305c-4b0f-b194-a09995fb25bf","Type":"ContainerStarted","Data":"552422a49af1b65c96ffae9e92a7ac62288e14a69e60ca302b5b055257e19066"} Nov 03 20:15:02.036371 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.036344 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc" event={"ID":"1774de9e-305c-4b0f-b194-a09995fb25bf","Type":"ContainerStarted","Data":"0151b6dcaef645f4d2056752410a26e86127b7d0ac96168f0dbe37b3c4cf0c54"} Nov 03 20:15:02.038335 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.038297 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59" event={"ID":"0b18a5b0-0fc4-4dc8-88b7-affec66830d8","Type":"ContainerStarted","Data":"07e2e3c445eccf38243f79efeaea824250cb8407bd3cdcbf92f0c8318cb3d346"} Nov 03 20:15:02.038488 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.038352 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59" event={"ID":"0b18a5b0-0fc4-4dc8-88b7-affec66830d8","Type":"ContainerStarted","Data":"ddafb52d39e712193525e5596a1ee622603944f333153a926b3dae4e067b4fad"} Nov 03 20:15:02.040248 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.040213 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt" event={"ID":"f1477a68-b92a-4138-b649-fdcd3b17f693","Type":"ContainerStarted","Data":"7ecb9dfb8b44ab18b2e162028f6ed7ba702990a2f1b6616f870571db2cae037d"} Nov 03 20:15:02.042030 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.041998 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p" event={"ID":"cb7298dc-a500-4715-a360-bcbdf0c7a298","Type":"ContainerStarted","Data":"739c84fbdbd4472de209b03f5016c97d18fa9210f304c910c8b8d69956d55c4e"} Nov 03 20:15:02.042154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.042053 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p" event={"ID":"cb7298dc-a500-4715-a360-bcbdf0c7a298","Type":"ContainerStarted","Data":"d87d71a5a5c9be19d71b4e2b3ad4ab28202b3b84f458132cae3b6777927887db"} Nov 03 20:15:02.043901 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.043874 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc" event={"ID":"3a1e6b3a-7887-42c5-b23d-b226cc6d0641","Type":"ContainerStarted","Data":"8e6471a9040955b10b1d38438cf69bdf99f79bc78a228b96d0fe698ae0462287"} Nov 03 20:15:02.044061 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.043922 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc" event={"ID":"3a1e6b3a-7887-42c5-b23d-b226cc6d0641","Type":"ContainerStarted","Data":"e7047c3a50e0e5af63c3aad56c3df9eed9fdbb01524dbc1e1aee5d0a30c1949b"} Nov 03 20:15:02.045529 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.045500 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm" event={"ID":"8588fce3-f04c-472e-a796-ed5314845eb5","Type":"ContainerStarted","Data":"b32c4d293b8995f06f1fb5ddb3a001137e90eae1a4f7c0fb9e03d191a3a37478"} Nov 03 20:15:02.045639 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.045547 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm" event={"ID":"8588fce3-f04c-472e-a796-ed5314845eb5","Type":"ContainerStarted","Data":"83337441d5fc8aa3f9f21c6f388bd3717e0a48766933d358c20c3b06f2c975f0"} Nov 03 20:15:02.047007 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.046986 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp" event={"ID":"1be9592b-18d0-4ce7-9322-92898161c6d8","Type":"ContainerStarted","Data":"b4f18413a134ddf59191a98736526c70cb01aedb4cb0a51e4737d8566d235d53"} Nov 03 20:15:02.048550 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.048529 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh" event={"ID":"ffb6bb03-2f3c-43f0-871a-1140b53f0fb9","Type":"ContainerStarted","Data":"3e20e22599c182419971be9a99e81987add8911d9eb7888383738e6fa6713192"} Nov 03 20:15:02.048637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.048566 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh" event={"ID":"ffb6bb03-2f3c-43f0-871a-1140b53f0fb9","Type":"ContainerStarted","Data":"ad52f985b75b77f67ea55389f8cb90df283d76b36b98979e8212d7270e641313"} Nov 03 20:15:02.058197 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.058143 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc" podStartSLOduration=2.058125826 podStartE2EDuration="2.058125826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:02.054346844 +0000 UTC m=+2296.312144104" watchObservedRunningTime="2026-11-03 20:15:02.058125826 +0000 UTC m=+2296.315923085" Nov 03 20:15:02.076826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.076776 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59" podStartSLOduration=2.076761507 podStartE2EDuration="2.076761507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:02.073094357 +0000 UTC m=+2296.330891614" watchObservedRunningTime="2026-11-03 20:15:02.076761507 +0000 UTC m=+2296.334558765" Nov 03 20:15:02.096530 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.096480 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt" podStartSLOduration=2.0964668890000002 podStartE2EDuration="2.096466889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:02.093274728 +0000 UTC m=+2296.351071986" watchObservedRunningTime="2026-11-03 20:15:02.096466889 +0000 UTC m=+2296.354264146" Nov 03 20:15:02.111937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.111893 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p" podStartSLOduration=2.111878507 podStartE2EDuration="2.111878507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:02.111087683 +0000 UTC m=+2296.368884941" watchObservedRunningTime="2026-11-03 20:15:02.111878507 +0000 UTC m=+2296.369675766" Nov 03 20:15:02.130833 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.130783 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm" podStartSLOduration=2.130770573 podStartE2EDuration="2.130770573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:02.128850945 +0000 UTC m=+2296.386648202" watchObservedRunningTime="2026-11-03 20:15:02.130770573 +0000 UTC m=+2296.388567879" Nov 03 20:15:02.149910 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.149863 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc" podStartSLOduration=2.149850196 podStartE2EDuration="2.149850196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:02.146580973 +0000 UTC m=+2296.404378236" watchObservedRunningTime="2026-11-03 20:15:02.149850196 +0000 UTC m=+2296.407647452" Nov 03 20:15:02.167297 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.167244 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh" podStartSLOduration=2.167226935 podStartE2EDuration="2.167226935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:02.166398009 +0000 UTC m=+2296.424195267" watchObservedRunningTime="2026-11-03 20:15:02.167226935 +0000 UTC m=+2296.425024192" Nov 03 20:15:02.893339 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.893267 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:15:02.935930 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.935897 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq"] Nov 03 20:15:02.936878 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.936858 5059 cpu_manager.go:414] "containerMap: removing container" podUID="f043a660-eddb-4de5-88a7-cf0b1fda6512" containerName="istio-proxy" Nov 03 20:15:02.936961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.936880 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f043a660-eddb-4de5-88a7-cf0b1fda6512" containerName="istio-proxy" Nov 03 20:15:02.936961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.936924 5059 cpu_manager.go:414] "containerMap: removing container" podUID="f043a660-eddb-4de5-88a7-cf0b1fda6512" containerName="istio-proxy" Nov 03 20:15:02.936961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.936930 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f043a660-eddb-4de5-88a7-cf0b1fda6512" containerName="istio-proxy" Nov 03 20:15:02.937083 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.937060 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f043a660-eddb-4de5-88a7-cf0b1fda6512" containerName="istio-proxy" Nov 03 20:15:02.941721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.941522 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq" Nov 03 20:15:02.944308 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.944073 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg"] Nov 03 20:15:02.946384 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.946316 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f043a660-eddb-4de5-88a7-cf0b1fda6512" containerName="istio-proxy" Nov 03 20:15:02.959138 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.959112 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r"] Nov 03 20:15:02.959729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.959698 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-envoy\") pod \"f043a660-eddb-4de5-88a7-cf0b1fda6512\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " Nov 03 20:15:02.959848 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.959752 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-data\") pod \"f043a660-eddb-4de5-88a7-cf0b1fda6512\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " Nov 03 20:15:02.959848 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.959819 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f043a660-eddb-4de5-88a7-cf0b1fda6512-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f043a660-eddb-4de5-88a7-cf0b1fda6512-istiod-ca-cert\") pod \"f043a660-eddb-4de5-88a7-cf0b1fda6512\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " Nov 03 20:15:02.959966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.959851 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-token\" (UniqueName: \"kubernetes.io/projected/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-token\") pod \"f043a660-eddb-4de5-88a7-cf0b1fda6512\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " Nov 03 20:15:02.959966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.959894 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-credential-socket\") pod \"f043a660-eddb-4de5-88a7-cf0b1fda6512\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " Nov 03 20:15:02.962629 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.960191 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-data" pod "f043a660-eddb-4de5-88a7-cf0b1fda6512" (UID: "f043a660-eddb-4de5-88a7-cf0b1fda6512"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:15:02.962629 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.960646 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-podinfo\") pod \"f043a660-eddb-4de5-88a7-cf0b1fda6512\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " Nov 03 20:15:02.962629 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.960769 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f043a660-eddb-4de5-88a7-cf0b1fda6512-kube-api-access-vh69z\" (UniqueName: \"kubernetes.io/projected/f043a660-eddb-4de5-88a7-cf0b1fda6512-kube-api-access-vh69z\") pod \"f043a660-eddb-4de5-88a7-cf0b1fda6512\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " Nov 03 20:15:02.962629 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.960854 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-workload-socket\") pod \"f043a660-eddb-4de5-88a7-cf0b1fda6512\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " Nov 03 20:15:02.962629 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.961089 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-workload-certs\") pod \"f043a660-eddb-4de5-88a7-cf0b1fda6512\" (UID: \"f043a660-eddb-4de5-88a7-cf0b1fda6512\") " Nov 03 20:15:02.962629 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.961998 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:02.964219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.964189 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f043a660-eddb-4de5-88a7-cf0b1fda6512-istiod-ca-cert" pod "f043a660-eddb-4de5-88a7-cf0b1fda6512" (UID: "f043a660-eddb-4de5-88a7-cf0b1fda6512"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:02.971570 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.965087 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-credential-socket" pod "f043a660-eddb-4de5-88a7-cf0b1fda6512" (UID: "f043a660-eddb-4de5-88a7-cf0b1fda6512"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:15:02.971570 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.965611 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-token" pod "f043a660-eddb-4de5-88a7-cf0b1fda6512" (UID: "f043a660-eddb-4de5-88a7-cf0b1fda6512"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:02.971570 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.965827 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-envoy" pod "f043a660-eddb-4de5-88a7-cf0b1fda6512" (UID: "f043a660-eddb-4de5-88a7-cf0b1fda6512"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:15:02.971570 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.965861 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-workload-socket" pod "f043a660-eddb-4de5-88a7-cf0b1fda6512" (UID: "f043a660-eddb-4de5-88a7-cf0b1fda6512"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:15:02.971570 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.966022 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-workload-certs" pod "f043a660-eddb-4de5-88a7-cf0b1fda6512" (UID: "f043a660-eddb-4de5-88a7-cf0b1fda6512"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:15:02.971570 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.966334 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg" Nov 03 20:15:02.971570 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.967425 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f043a660-eddb-4de5-88a7-cf0b1fda6512-kube-api-access-vh69z" pod "f043a660-eddb-4de5-88a7-cf0b1fda6512" (UID: "f043a660-eddb-4de5-88a7-cf0b1fda6512"). InnerVolumeSpecName "kube-api-access-vh69z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:02.971570 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.969182 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-podinfo" pod "f043a660-eddb-4de5-88a7-cf0b1fda6512" (UID: "f043a660-eddb-4de5-88a7-cf0b1fda6512"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:15:02.973026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.973004 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl"] Nov 03 20:15:02.973276 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.973255 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r" Nov 03 20:15:02.980535 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.980502 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp"] Nov 03 20:15:02.981326 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.981306 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl" Nov 03 20:15:02.992317 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.992290 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s"] Nov 03 20:15:02.992690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:02.992401 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp" Nov 03 20:15:03.007343 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.007316 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9"] Nov 03 20:15:03.008774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.008748 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s" Nov 03 20:15:03.065269 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.065231 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3873f10-84bb-489c-9de5-b907f375ff3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp\" (UID: \"e3873f10-84bb-489c-9de5-b907f375ff3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp" Nov 03 20:15:03.065490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.065466 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9" Nov 03 20:15:03.065644 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.065622 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpvtn\" (UniqueName: \"kubernetes.io/projected/8f7ed20a-4c3c-4023-b1de-479a8ea346a8-kube-api-access-kpvtn\") pod \"maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq\" (UID: \"8f7ed20a-4c3c-4023-b1de-479a8ea346a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq" Nov 03 20:15:03.065873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.065852 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932670b6-eb80-4970-b092-8682e1daa6f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s\" (UID: \"932670b6-eb80-4970-b092-8682e1daa6f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s" Nov 03 20:15:03.066029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.066012 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30d22aa8-3a01-44f1-a59d-b92a70934a7f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl\" (UID: \"30d22aa8-3a01-44f1-a59d-b92a70934a7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl" Nov 03 20:15:03.066160 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.066143 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpn6l\" (UniqueName: \"kubernetes.io/projected/e3873f10-84bb-489c-9de5-b907f375ff3b-kube-api-access-lpn6l\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp\" (UID: \"e3873f10-84bb-489c-9de5-b907f375ff3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp" Nov 03 20:15:03.066590 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.066320 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32905cf1-9e12-4fc4-8da4-dda3d142a0a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r\" (UID: \"32905cf1-9e12-4fc4-8da4-dda3d142a0a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r" Nov 03 20:15:03.070327 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.070303 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9tmz\" (UniqueName: \"kubernetes.io/projected/30d22aa8-3a01-44f1-a59d-b92a70934a7f-kube-api-access-c9tmz\") pod \"maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl\" (UID: \"30d22aa8-3a01-44f1-a59d-b92a70934a7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl" Nov 03 20:15:03.070881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.065382 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft"] Nov 03 20:15:03.073928 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.070578 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcqkz\" (UniqueName: \"kubernetes.io/projected/932670b6-eb80-4970-b092-8682e1daa6f5-kube-api-access-gcqkz\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s\" (UID: \"932670b6-eb80-4970-b092-8682e1daa6f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s" Nov 03 20:15:03.074042 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.074001 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh8fh\" (UniqueName: \"kubernetes.io/projected/32905cf1-9e12-4fc4-8da4-dda3d142a0a0-kube-api-access-hh8fh\") pod \"maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r\" (UID: \"32905cf1-9e12-4fc4-8da4-dda3d142a0a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r" Nov 03 20:15:03.074042 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.074035 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edb7775d-1e94-40ed-9da3-738ad142ef20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg\" (UID: \"edb7775d-1e94-40ed-9da3-738ad142ef20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg" Nov 03 20:15:03.074176 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.074160 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgszl\" (UniqueName: \"kubernetes.io/projected/edb7775d-1e94-40ed-9da3-738ad142ef20-kube-api-access-qgszl\") pod \"maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg\" (UID: \"edb7775d-1e94-40ed-9da3-738ad142ef20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg" Nov 03 20:15:03.074277 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.074262 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7ed20a-4c3c-4023-b1de-479a8ea346a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq\" (UID: \"8f7ed20a-4c3c-4023-b1de-479a8ea346a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq" Nov 03 20:15:03.074544 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.074524 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:03.074624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.074550 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f043a660-eddb-4de5-88a7-cf0b1fda6512-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:03.074624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.074564 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:03.074624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.074579 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:03.074624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.074591 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f043a660-eddb-4de5-88a7-cf0b1fda6512-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:03.074624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.074605 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh69z\" (UniqueName: \"kubernetes.io/projected/f043a660-eddb-4de5-88a7-cf0b1fda6512-kube-api-access-vh69z\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:03.074971 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.074895 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:03.074971 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.074920 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f043a660-eddb-4de5-88a7-cf0b1fda6512-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:03.077574 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.077554 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8"] Nov 03 20:15:03.077829 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.077807 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft" Nov 03 20:15:03.082352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.082327 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m"] Nov 03 20:15:03.082543 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.082523 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8" Nov 03 20:15:03.085927 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.085893 5059 generic.go:356] "Generic (PLEG): container finished" podID="1be9592b-18d0-4ce7-9322-92898161c6d8" containerID="1d214d09944a545dab7667764f096ad403d270e2e5d37c37dfbdf2bebfe94c28" exitCode=60 Nov 03 20:15:03.086862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.086804 5059 scope.go:122] "RemoveContainer" containerID="1d214d09944a545dab7667764f096ad403d270e2e5d37c37dfbdf2bebfe94c28" Nov 03 20:15:03.098316 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.095645 5059 generic.go:356] "Generic (PLEG): container finished" podID="f043a660-eddb-4de5-88a7-cf0b1fda6512" containerID="f9f4f7f919667909db546636d5524b9efb53ecdd7bf93966afd963aa9742b298" exitCode=0 Nov 03 20:15:03.098316 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.095838 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" Nov 03 20:15:03.103059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.103025 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp" event={"ID":"1be9592b-18d0-4ce7-9322-92898161c6d8","Type":"ContainerDied","Data":"1d214d09944a545dab7667764f096ad403d270e2e5d37c37dfbdf2bebfe94c28"} Nov 03 20:15:03.103182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.103077 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" event={"ID":"f043a660-eddb-4de5-88a7-cf0b1fda6512","Type":"ContainerDied","Data":"f9f4f7f919667909db546636d5524b9efb53ecdd7bf93966afd963aa9742b298"} Nov 03 20:15:03.103182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.103105 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x" event={"ID":"f043a660-eddb-4de5-88a7-cf0b1fda6512","Type":"ContainerDied","Data":"4f71494403b009093856f2e7fa63f4a666e4cba8027d0b0e586e9f6a5daa59fc"} Nov 03 20:15:03.103182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.103132 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq"] Nov 03 20:15:03.103182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.103166 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg"] Nov 03 20:15:03.103419 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.103192 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s"] Nov 03 20:15:03.103419 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.103219 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9"] Nov 03 20:15:03.103419 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.103243 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp"] Nov 03 20:15:03.103419 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.103266 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft"] Nov 03 20:15:03.103419 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.103290 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8"] Nov 03 20:15:03.103419 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.103319 5059 scope.go:122] "RemoveContainer" containerID="f9f4f7f919667909db546636d5524b9efb53ecdd7bf93966afd963aa9742b298" Nov 03 20:15:03.103777 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.103487 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m" Nov 03 20:15:03.106847 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.106810 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl"] Nov 03 20:15:03.117913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.117882 5059 generic.go:356] "Generic (PLEG): container finished" podID="ffb6bb03-2f3c-43f0-871a-1140b53f0fb9" containerID="3e20e22599c182419971be9a99e81987add8911d9eb7888383738e6fa6713192" exitCode=60 Nov 03 20:15:03.118230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.118204 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh" event={"ID":"ffb6bb03-2f3c-43f0-871a-1140b53f0fb9","Type":"ContainerDied","Data":"3e20e22599c182419971be9a99e81987add8911d9eb7888383738e6fa6713192"} Nov 03 20:15:03.118385 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.118361 5059 scope.go:122] "RemoveContainer" containerID="3e20e22599c182419971be9a99e81987add8911d9eb7888383738e6fa6713192" Nov 03 20:15:03.123578 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.123329 5059 generic.go:356] "Generic (PLEG): container finished" podID="1774de9e-305c-4b0f-b194-a09995fb25bf" containerID="552422a49af1b65c96ffae9e92a7ac62288e14a69e60ca302b5b055257e19066" exitCode=60 Nov 03 20:15:03.124365 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.123434 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc" event={"ID":"1774de9e-305c-4b0f-b194-a09995fb25bf","Type":"ContainerDied","Data":"552422a49af1b65c96ffae9e92a7ac62288e14a69e60ca302b5b055257e19066"} Nov 03 20:15:03.125787 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.124044 5059 scope.go:122] "RemoveContainer" containerID="552422a49af1b65c96ffae9e92a7ac62288e14a69e60ca302b5b055257e19066" Nov 03 20:15:03.130431 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.130405 5059 generic.go:356] "Generic (PLEG): container finished" podID="0b18a5b0-0fc4-4dc8-88b7-affec66830d8" containerID="07e2e3c445eccf38243f79efeaea824250cb8407bd3cdcbf92f0c8318cb3d346" exitCode=60 Nov 03 20:15:03.130602 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.130576 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59" event={"ID":"0b18a5b0-0fc4-4dc8-88b7-affec66830d8","Type":"ContainerDied","Data":"07e2e3c445eccf38243f79efeaea824250cb8407bd3cdcbf92f0c8318cb3d346"} Nov 03 20:15:03.131116 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.131084 5059 scope.go:122] "RemoveContainer" containerID="07e2e3c445eccf38243f79efeaea824250cb8407bd3cdcbf92f0c8318cb3d346" Nov 03 20:15:03.131873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.131853 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r"] Nov 03 20:15:03.132885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.132861 5059 generic.go:356] "Generic (PLEG): container finished" podID="f1477a68-b92a-4138-b649-fdcd3b17f693" containerID="7ecb9dfb8b44ab18b2e162028f6ed7ba702990a2f1b6616f870571db2cae037d" exitCode=60 Nov 03 20:15:03.133039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.133014 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt" event={"ID":"f1477a68-b92a-4138-b649-fdcd3b17f693","Type":"ContainerDied","Data":"7ecb9dfb8b44ab18b2e162028f6ed7ba702990a2f1b6616f870571db2cae037d"} Nov 03 20:15:03.133161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.133143 5059 scope.go:122] "RemoveContainer" containerID="7ecb9dfb8b44ab18b2e162028f6ed7ba702990a2f1b6616f870571db2cae037d" Nov 03 20:15:03.139593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.139566 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m"] Nov 03 20:15:03.141441 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.141415 5059 generic.go:356] "Generic (PLEG): container finished" podID="cb7298dc-a500-4715-a360-bcbdf0c7a298" containerID="739c84fbdbd4472de209b03f5016c97d18fa9210f304c910c8b8d69956d55c4e" exitCode=60 Nov 03 20:15:03.142581 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.142555 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p" event={"ID":"cb7298dc-a500-4715-a360-bcbdf0c7a298","Type":"ContainerDied","Data":"739c84fbdbd4472de209b03f5016c97d18fa9210f304c910c8b8d69956d55c4e"} Nov 03 20:15:03.143023 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.142999 5059 scope.go:122] "RemoveContainer" containerID="739c84fbdbd4472de209b03f5016c97d18fa9210f304c910c8b8d69956d55c4e" Nov 03 20:15:03.144002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.143979 5059 scope.go:122] "RemoveContainer" containerID="873a0ffe9f526eaa4ece120b96a6e7e29de1a52daefeff130e5e76d7773021ad" Nov 03 20:15:03.147943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.146863 5059 generic.go:356] "Generic (PLEG): container finished" podID="3a1e6b3a-7887-42c5-b23d-b226cc6d0641" containerID="8e6471a9040955b10b1d38438cf69bdf99f79bc78a228b96d0fe698ae0462287" exitCode=60 Nov 03 20:15:03.147943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.146999 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc" event={"ID":"3a1e6b3a-7887-42c5-b23d-b226cc6d0641","Type":"ContainerDied","Data":"8e6471a9040955b10b1d38438cf69bdf99f79bc78a228b96d0fe698ae0462287"} Nov 03 20:15:03.147943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.147177 5059 scope.go:122] "RemoveContainer" containerID="8e6471a9040955b10b1d38438cf69bdf99f79bc78a228b96d0fe698ae0462287" Nov 03 20:15:03.163296 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.163256 5059 generic.go:356] "Generic (PLEG): container finished" podID="8588fce3-f04c-472e-a796-ed5314845eb5" containerID="b32c4d293b8995f06f1fb5ddb3a001137e90eae1a4f7c0fb9e03d191a3a37478" exitCode=60 Nov 03 20:15:03.163628 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.163364 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm" event={"ID":"8588fce3-f04c-472e-a796-ed5314845eb5","Type":"ContainerDied","Data":"b32c4d293b8995f06f1fb5ddb3a001137e90eae1a4f7c0fb9e03d191a3a37478"} Nov 03 20:15:03.164606 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.164581 5059 scope.go:122] "RemoveContainer" containerID="b32c4d293b8995f06f1fb5ddb3a001137e90eae1a4f7c0fb9e03d191a3a37478" Nov 03 20:15:03.176644 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.176606 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpn6l\" (UniqueName: \"kubernetes.io/projected/e3873f10-84bb-489c-9de5-b907f375ff3b-kube-api-access-lpn6l\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp\" (UID: \"e3873f10-84bb-489c-9de5-b907f375ff3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp" Nov 03 20:15:03.176830 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.176791 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32905cf1-9e12-4fc4-8da4-dda3d142a0a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r\" (UID: \"32905cf1-9e12-4fc4-8da4-dda3d142a0a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r" Nov 03 20:15:03.176906 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.176835 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9tmz\" (UniqueName: \"kubernetes.io/projected/30d22aa8-3a01-44f1-a59d-b92a70934a7f-kube-api-access-c9tmz\") pod \"maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl\" (UID: \"30d22aa8-3a01-44f1-a59d-b92a70934a7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl" Nov 03 20:15:03.176906 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.176867 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcqkz\" (UniqueName: \"kubernetes.io/projected/932670b6-eb80-4970-b092-8682e1daa6f5-kube-api-access-gcqkz\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s\" (UID: \"932670b6-eb80-4970-b092-8682e1daa6f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s" Nov 03 20:15:03.177503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.176911 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv5xn\" (UniqueName: \"kubernetes.io/projected/ea54ebf8-4aab-4442-adf0-6930bcb6229b-kube-api-access-tv5xn\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft\" (UID: \"ea54ebf8-4aab-4442-adf0-6930bcb6229b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft" Nov 03 20:15:03.177503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.176949 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh8fh\" (UniqueName: \"kubernetes.io/projected/32905cf1-9e12-4fc4-8da4-dda3d142a0a0-kube-api-access-hh8fh\") pod \"maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r\" (UID: \"32905cf1-9e12-4fc4-8da4-dda3d142a0a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r" Nov 03 20:15:03.177503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.176977 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edb7775d-1e94-40ed-9da3-738ad142ef20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg\" (UID: \"edb7775d-1e94-40ed-9da3-738ad142ef20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg" Nov 03 20:15:03.177503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.177067 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9\" (UID: \"afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9" Nov 03 20:15:03.177503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.177142 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e0bbcca-04df-4474-91fc-c5c5e6521732-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8\" (UID: \"0e0bbcca-04df-4474-91fc-c5c5e6521732\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8" Nov 03 20:15:03.177503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.177193 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd6zd\" (UniqueName: \"kubernetes.io/projected/0e0bbcca-04df-4474-91fc-c5c5e6521732-kube-api-access-jd6zd\") pod \"maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8\" (UID: \"0e0bbcca-04df-4474-91fc-c5c5e6521732\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8" Nov 03 20:15:03.177503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.177217 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgszl\" (UniqueName: \"kubernetes.io/projected/edb7775d-1e94-40ed-9da3-738ad142ef20-kube-api-access-qgszl\") pod \"maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg\" (UID: \"edb7775d-1e94-40ed-9da3-738ad142ef20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg" Nov 03 20:15:03.177503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.177276 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r4s2\" (UniqueName: \"kubernetes.io/projected/545cc204-5445-437d-b58e-9b6494f1a3b7-kube-api-access-6r4s2\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m\" (UID: \"545cc204-5445-437d-b58e-9b6494f1a3b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m" Nov 03 20:15:03.177503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.177309 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea54ebf8-4aab-4442-adf0-6930bcb6229b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft\" (UID: \"ea54ebf8-4aab-4442-adf0-6930bcb6229b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft" Nov 03 20:15:03.177503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.177426 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7ed20a-4c3c-4023-b1de-479a8ea346a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq\" (UID: \"8f7ed20a-4c3c-4023-b1de-479a8ea346a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq" Nov 03 20:15:03.177958 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.177593 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmq9v\" (UniqueName: \"kubernetes.io/projected/afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9-kube-api-access-fmq9v\") pod \"maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9\" (UID: \"afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9" Nov 03 20:15:03.177958 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.177620 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3873f10-84bb-489c-9de5-b907f375ff3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp\" (UID: \"e3873f10-84bb-489c-9de5-b907f375ff3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp" Nov 03 20:15:03.177958 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.177706 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpvtn\" (UniqueName: \"kubernetes.io/projected/8f7ed20a-4c3c-4023-b1de-479a8ea346a8-kube-api-access-kpvtn\") pod \"maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq\" (UID: \"8f7ed20a-4c3c-4023-b1de-479a8ea346a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq" Nov 03 20:15:03.177958 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.177777 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/545cc204-5445-437d-b58e-9b6494f1a3b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m\" (UID: \"545cc204-5445-437d-b58e-9b6494f1a3b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m" Nov 03 20:15:03.177958 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.177868 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932670b6-eb80-4970-b092-8682e1daa6f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s\" (UID: \"932670b6-eb80-4970-b092-8682e1daa6f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s" Nov 03 20:15:03.177958 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.177948 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30d22aa8-3a01-44f1-a59d-b92a70934a7f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl\" (UID: \"30d22aa8-3a01-44f1-a59d-b92a70934a7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl" Nov 03 20:15:03.178153 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.178068 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edb7775d-1e94-40ed-9da3-738ad142ef20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg\" (UID: \"edb7775d-1e94-40ed-9da3-738ad142ef20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg" Nov 03 20:15:03.178572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.178545 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32905cf1-9e12-4fc4-8da4-dda3d142a0a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r\" (UID: \"32905cf1-9e12-4fc4-8da4-dda3d142a0a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r" Nov 03 20:15:03.178779 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.178759 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30d22aa8-3a01-44f1-a59d-b92a70934a7f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl\" (UID: \"30d22aa8-3a01-44f1-a59d-b92a70934a7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl" Nov 03 20:15:03.178982 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.178957 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932670b6-eb80-4970-b092-8682e1daa6f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s\" (UID: \"932670b6-eb80-4970-b092-8682e1daa6f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s" Nov 03 20:15:03.179369 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.179330 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7ed20a-4c3c-4023-b1de-479a8ea346a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq\" (UID: \"8f7ed20a-4c3c-4023-b1de-479a8ea346a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq" Nov 03 20:15:03.179857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.179819 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3873f10-84bb-489c-9de5-b907f375ff3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp\" (UID: \"e3873f10-84bb-489c-9de5-b907f375ff3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp" Nov 03 20:15:03.191323 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.191289 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9tmz\" (UniqueName: \"kubernetes.io/projected/30d22aa8-3a01-44f1-a59d-b92a70934a7f-kube-api-access-c9tmz\") pod \"maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl\" (UID: \"30d22aa8-3a01-44f1-a59d-b92a70934a7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl" Nov 03 20:15:03.191904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.191879 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcqkz\" (UniqueName: \"kubernetes.io/projected/932670b6-eb80-4970-b092-8682e1daa6f5-kube-api-access-gcqkz\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s\" (UID: \"932670b6-eb80-4970-b092-8682e1daa6f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s" Nov 03 20:15:03.195255 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.195232 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpn6l\" (UniqueName: \"kubernetes.io/projected/e3873f10-84bb-489c-9de5-b907f375ff3b-kube-api-access-lpn6l\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp\" (UID: \"e3873f10-84bb-489c-9de5-b907f375ff3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp" Nov 03 20:15:03.198281 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.198255 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpvtn\" (UniqueName: \"kubernetes.io/projected/8f7ed20a-4c3c-4023-b1de-479a8ea346a8-kube-api-access-kpvtn\") pod \"maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq\" (UID: \"8f7ed20a-4c3c-4023-b1de-479a8ea346a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq" Nov 03 20:15:03.198562 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.198542 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgszl\" (UniqueName: \"kubernetes.io/projected/edb7775d-1e94-40ed-9da3-738ad142ef20-kube-api-access-qgszl\") pod \"maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg\" (UID: \"edb7775d-1e94-40ed-9da3-738ad142ef20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg" Nov 03 20:15:03.198650 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.198631 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh8fh\" (UniqueName: \"kubernetes.io/projected/32905cf1-9e12-4fc4-8da4-dda3d142a0a0-kube-api-access-hh8fh\") pod \"maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r\" (UID: \"32905cf1-9e12-4fc4-8da4-dda3d142a0a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r" Nov 03 20:15:03.227717 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.227626 5059 scope.go:122] "RemoveContainer" containerID="f9f4f7f919667909db546636d5524b9efb53ecdd7bf93966afd963aa9742b298" Nov 03 20:15:03.228029 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:03.228004 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f4f7f919667909db546636d5524b9efb53ecdd7bf93966afd963aa9742b298\": container with ID starting with f9f4f7f919667909db546636d5524b9efb53ecdd7bf93966afd963aa9742b298 not found: ID does not exist" containerID="f9f4f7f919667909db546636d5524b9efb53ecdd7bf93966afd963aa9742b298" Nov 03 20:15:03.228120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.228041 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f4f7f919667909db546636d5524b9efb53ecdd7bf93966afd963aa9742b298"} err="failed to get container status \"f9f4f7f919667909db546636d5524b9efb53ecdd7bf93966afd963aa9742b298\": rpc error: code = NotFound desc = could not find container \"f9f4f7f919667909db546636d5524b9efb53ecdd7bf93966afd963aa9742b298\": container with ID starting with f9f4f7f919667909db546636d5524b9efb53ecdd7bf93966afd963aa9742b298 not found: ID does not exist" Nov 03 20:15:03.228120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.228060 5059 scope.go:122] "RemoveContainer" containerID="873a0ffe9f526eaa4ece120b96a6e7e29de1a52daefeff130e5e76d7773021ad" Nov 03 20:15:03.228544 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:03.228429 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"873a0ffe9f526eaa4ece120b96a6e7e29de1a52daefeff130e5e76d7773021ad\": container with ID starting with 873a0ffe9f526eaa4ece120b96a6e7e29de1a52daefeff130e5e76d7773021ad not found: ID does not exist" containerID="873a0ffe9f526eaa4ece120b96a6e7e29de1a52daefeff130e5e76d7773021ad" Nov 03 20:15:03.228544 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.228464 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"873a0ffe9f526eaa4ece120b96a6e7e29de1a52daefeff130e5e76d7773021ad"} err="failed to get container status \"873a0ffe9f526eaa4ece120b96a6e7e29de1a52daefeff130e5e76d7773021ad\": rpc error: code = NotFound desc = could not find container \"873a0ffe9f526eaa4ece120b96a6e7e29de1a52daefeff130e5e76d7773021ad\": container with ID starting with 873a0ffe9f526eaa4ece120b96a6e7e29de1a52daefeff130e5e76d7773021ad not found: ID does not exist" Nov 03 20:15:03.279924 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.279881 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/545cc204-5445-437d-b58e-9b6494f1a3b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m\" (UID: \"545cc204-5445-437d-b58e-9b6494f1a3b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m" Nov 03 20:15:03.280867 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.280325 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq" Nov 03 20:15:03.281415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.281369 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/545cc204-5445-437d-b58e-9b6494f1a3b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m\" (UID: \"545cc204-5445-437d-b58e-9b6494f1a3b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m" Nov 03 20:15:03.282031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.282003 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv5xn\" (UniqueName: \"kubernetes.io/projected/ea54ebf8-4aab-4442-adf0-6930bcb6229b-kube-api-access-tv5xn\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft\" (UID: \"ea54ebf8-4aab-4442-adf0-6930bcb6229b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft" Nov 03 20:15:03.282268 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.282255 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9\" (UID: \"afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9" Nov 03 20:15:03.282445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.282432 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e0bbcca-04df-4474-91fc-c5c5e6521732-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8\" (UID: \"0e0bbcca-04df-4474-91fc-c5c5e6521732\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8" Nov 03 20:15:03.282551 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.282539 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd6zd\" (UniqueName: \"kubernetes.io/projected/0e0bbcca-04df-4474-91fc-c5c5e6521732-kube-api-access-jd6zd\") pod \"maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8\" (UID: \"0e0bbcca-04df-4474-91fc-c5c5e6521732\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8" Nov 03 20:15:03.283099 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.283068 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r4s2\" (UniqueName: \"kubernetes.io/projected/545cc204-5445-437d-b58e-9b6494f1a3b7-kube-api-access-6r4s2\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m\" (UID: \"545cc204-5445-437d-b58e-9b6494f1a3b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m" Nov 03 20:15:03.283344 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.283327 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea54ebf8-4aab-4442-adf0-6930bcb6229b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft\" (UID: \"ea54ebf8-4aab-4442-adf0-6930bcb6229b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft" Nov 03 20:15:03.283780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.283751 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmq9v\" (UniqueName: \"kubernetes.io/projected/afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9-kube-api-access-fmq9v\") pod \"maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9\" (UID: \"afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9" Nov 03 20:15:03.284727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.284598 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e0bbcca-04df-4474-91fc-c5c5e6521732-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8\" (UID: \"0e0bbcca-04df-4474-91fc-c5c5e6521732\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8" Nov 03 20:15:03.285110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.285060 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9\" (UID: \"afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9" Nov 03 20:15:03.286188 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.285762 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea54ebf8-4aab-4442-adf0-6930bcb6229b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft\" (UID: \"ea54ebf8-4aab-4442-adf0-6930bcb6229b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft" Nov 03 20:15:03.294520 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.294481 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg" Nov 03 20:15:03.319675 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.319640 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r" Nov 03 20:15:03.348016 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.347978 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x"] Nov 03 20:15:03.391174 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.381540 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl" Nov 03 20:15:03.413502 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.396804 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp" Nov 03 20:15:03.418368 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.418282 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r4s2\" (UniqueName: \"kubernetes.io/projected/545cc204-5445-437d-b58e-9b6494f1a3b7-kube-api-access-6r4s2\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m\" (UID: \"545cc204-5445-437d-b58e-9b6494f1a3b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m" Nov 03 20:15:03.419354 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.419320 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd6zd\" (UniqueName: \"kubernetes.io/projected/0e0bbcca-04df-4474-91fc-c5c5e6521732-kube-api-access-jd6zd\") pod \"maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8\" (UID: \"0e0bbcca-04df-4474-91fc-c5c5e6521732\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8" Nov 03 20:15:03.438968 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.426808 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s" Nov 03 20:15:03.477989 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.477344 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmq9v\" (UniqueName: \"kubernetes.io/projected/afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9-kube-api-access-fmq9v\") pod \"maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9\" (UID: \"afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9" Nov 03 20:15:03.477989 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.477630 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8" Nov 03 20:15:03.487180 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.486809 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m" Nov 03 20:15:03.487180 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.486929 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv5xn\" (UniqueName: \"kubernetes.io/projected/ea54ebf8-4aab-4442-adf0-6930bcb6229b-kube-api-access-tv5xn\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft\" (UID: \"ea54ebf8-4aab-4442-adf0-6930bcb6229b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft" Nov 03 20:15:03.502609 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.502498 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-pv92x"] Nov 03 20:15:03.727904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.724979 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9" Nov 03 20:15:03.745062 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.745024 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft" Nov 03 20:15:03.870205 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.870161 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f043a660-eddb-4de5-88a7-cf0b1fda6512" path="/var/lib/kubelet/pods/f043a660-eddb-4de5-88a7-cf0b1fda6512/volumes" Nov 03 20:15:03.872094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.872065 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r"] Nov 03 20:15:03.894990 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:03.894706 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq"] Nov 03 20:15:04.143745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:04.143711 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s"] Nov 03 20:15:04.187554 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:04.187517 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh" event={"ID":"ffb6bb03-2f3c-43f0-871a-1140b53f0fb9","Type":"ContainerStarted","Data":"d0ca647ad2a0bb692e9f226eacc2341615eee4d1712d1925878723228313dd6f"} Nov 03 20:15:04.199286 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:04.199192 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8"] Nov 03 20:15:04.200526 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:04.200499 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc" event={"ID":"1774de9e-305c-4b0f-b194-a09995fb25bf","Type":"ContainerStarted","Data":"27c5d4aab2ceeb8f6dcc2fa3afdeac510698928a98e9dfce1c397485f66dd355"} Nov 03 20:15:04.202532 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:04.202507 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r" event={"ID":"32905cf1-9e12-4fc4-8da4-dda3d142a0a0","Type":"ContainerStarted","Data":"7c8856108be7e89fce01d0e65487bc2f6750e128e01dbeedcd21f05f4e887a76"} Nov 03 20:15:04.204817 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:04.204777 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt" event={"ID":"f1477a68-b92a-4138-b649-fdcd3b17f693","Type":"ContainerStarted","Data":"2a791fe1424f5e3f7f01347833a3d0c452d0d885d27375eac99aaed96a99aba1"} Nov 03 20:15:04.208321 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:04.208282 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp" event={"ID":"1be9592b-18d0-4ce7-9322-92898161c6d8","Type":"ContainerStarted","Data":"6d124e9a3e506b7312a6889ad995a1dba8e0e21475f34dc893d8ef6818c32f43"} Nov 03 20:15:04.216870 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:04.216750 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq" event={"ID":"8f7ed20a-4c3c-4023-b1de-479a8ea346a8","Type":"ContainerStarted","Data":"4d9a6aa9c74c714b28ceb76d60fbf0e49b07d9badbbaf661f59b14302eab46cf"} Nov 03 20:15:04.231610 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:04.231583 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m"] Nov 03 20:15:04.238315 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:04.238289 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl"] Nov 03 20:15:04.246747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:04.246713 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp"] Nov 03 20:15:04.270065 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:15:04.269491 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932670b6_eb80_4970_b092_8682e1daa6f5.slice/crio-e3d2f41c255b5346cae2519e6623612f5606b6131525fc2593e624ffc59f03cd WatchSource:0}: Error finding container e3d2f41c255b5346cae2519e6623612f5606b6131525fc2593e624ffc59f03cd: Status 404 returned error can't find the container with id e3d2f41c255b5346cae2519e6623612f5606b6131525fc2593e624ffc59f03cd Nov 03 20:15:04.297155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:04.297119 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg"] Nov 03 20:15:04.306345 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:04.306296 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp" podStartSLOduration=4.306280779 podStartE2EDuration="4.306280779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:04.295043344 +0000 UTC m=+2298.552840615" watchObservedRunningTime="2026-11-03 20:15:04.306280779 +0000 UTC m=+2298.564078052" Nov 03 20:15:04.459051 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:04.458991 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9"] Nov 03 20:15:04.476059 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:15:04.476001 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545cc204_5445_437d_b58e_9b6494f1a3b7.slice/crio-fc9d1eb7fe91d2a484c35c68b0cf16395c040ceb653bfb10a71dd3d1d5fd6787 WatchSource:0}: Error finding container fc9d1eb7fe91d2a484c35c68b0cf16395c040ceb653bfb10a71dd3d1d5fd6787: Status 404 returned error can't find the container with id fc9d1eb7fe91d2a484c35c68b0cf16395c040ceb653bfb10a71dd3d1d5fd6787 Nov 03 20:15:04.479944 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:15:04.479871 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d22aa8_3a01_44f1_a59d_b92a70934a7f.slice/crio-f5ab6fc16c6c116db960954179c62e7333d0f8b7cfb5208e68e0f033ca807e33 WatchSource:0}: Error finding container f5ab6fc16c6c116db960954179c62e7333d0f8b7cfb5208e68e0f033ca807e33: Status 404 returned error can't find the container with id f5ab6fc16c6c116db960954179c62e7333d0f8b7cfb5208e68e0f033ca807e33 Nov 03 20:15:04.769891 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:04.769857 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft"] Nov 03 20:15:04.792897 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:15:04.792862 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea54ebf8_4aab_4442_adf0_6930bcb6229b.slice/crio-763b4652cf02c03cf287b4d600ed5ff7430bcf7dbc6610ac1134c602aff9b73f WatchSource:0}: Error finding container 763b4652cf02c03cf287b4d600ed5ff7430bcf7dbc6610ac1134c602aff9b73f: Status 404 returned error can't find the container with id 763b4652cf02c03cf287b4d600ed5ff7430bcf7dbc6610ac1134c602aff9b73f Nov 03 20:15:05.240407 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.240020 5059 generic.go:356] "Generic (PLEG): container finished" podID="1774de9e-305c-4b0f-b194-a09995fb25bf" containerID="27c5d4aab2ceeb8f6dcc2fa3afdeac510698928a98e9dfce1c397485f66dd355" exitCode=60 Nov 03 20:15:05.241477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.241438 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc" event={"ID":"1774de9e-305c-4b0f-b194-a09995fb25bf","Type":"ContainerDied","Data":"27c5d4aab2ceeb8f6dcc2fa3afdeac510698928a98e9dfce1c397485f66dd355"} Nov 03 20:15:05.241593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.241522 5059 scope.go:122] "RemoveContainer" containerID="552422a49af1b65c96ffae9e92a7ac62288e14a69e60ca302b5b055257e19066" Nov 03 20:15:05.242775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.241996 5059 scope.go:122] "RemoveContainer" containerID="27c5d4aab2ceeb8f6dcc2fa3afdeac510698928a98e9dfce1c397485f66dd355" Nov 03 20:15:05.242775 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:05.242507 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc_odh-ai-gateway-infra(1774de9e-305c-4b0f-b194-a09995fb25bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc" podUID="1774de9e-305c-4b0f-b194-a09995fb25bf" Nov 03 20:15:05.251558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.251523 5059 generic.go:356] "Generic (PLEG): container finished" podID="0b18a5b0-0fc4-4dc8-88b7-affec66830d8" containerID="344db2809ca423c43d7ba4570f981c986e2be112e1c054ff4d7c69f78a2e4da5" exitCode=60 Nov 03 20:15:05.251739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.251643 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59" event={"ID":"0b18a5b0-0fc4-4dc8-88b7-affec66830d8","Type":"ContainerDied","Data":"344db2809ca423c43d7ba4570f981c986e2be112e1c054ff4d7c69f78a2e4da5"} Nov 03 20:15:05.252374 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.252348 5059 scope.go:122] "RemoveContainer" containerID="344db2809ca423c43d7ba4570f981c986e2be112e1c054ff4d7c69f78a2e4da5" Nov 03 20:15:05.252879 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:05.252851 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59_odh-ai-gateway-infra(0b18a5b0-0fc4-4dc8-88b7-affec66830d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59" podUID="0b18a5b0-0fc4-4dc8-88b7-affec66830d8" Nov 03 20:15:05.259990 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.257088 5059 generic.go:356] "Generic (PLEG): container finished" podID="32905cf1-9e12-4fc4-8da4-dda3d142a0a0" containerID="c84575b5df8beccb7c1ee5e4fea2e730526cbda28bc23acb585bc1a231d9a391" exitCode=60 Nov 03 20:15:05.259990 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.257200 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r" event={"ID":"32905cf1-9e12-4fc4-8da4-dda3d142a0a0","Type":"ContainerDied","Data":"c84575b5df8beccb7c1ee5e4fea2e730526cbda28bc23acb585bc1a231d9a391"} Nov 03 20:15:05.259990 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.257649 5059 scope.go:122] "RemoveContainer" containerID="c84575b5df8beccb7c1ee5e4fea2e730526cbda28bc23acb585bc1a231d9a391" Nov 03 20:15:05.264201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.264174 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft" event={"ID":"ea54ebf8-4aab-4442-adf0-6930bcb6229b","Type":"ContainerStarted","Data":"763b4652cf02c03cf287b4d600ed5ff7430bcf7dbc6610ac1134c602aff9b73f"} Nov 03 20:15:05.267286 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.267258 5059 generic.go:356] "Generic (PLEG): container finished" podID="f1477a68-b92a-4138-b649-fdcd3b17f693" containerID="2a791fe1424f5e3f7f01347833a3d0c452d0d885d27375eac99aaed96a99aba1" exitCode=60 Nov 03 20:15:05.267400 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.267333 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt" event={"ID":"f1477a68-b92a-4138-b649-fdcd3b17f693","Type":"ContainerDied","Data":"2a791fe1424f5e3f7f01347833a3d0c452d0d885d27375eac99aaed96a99aba1"} Nov 03 20:15:05.267784 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.267756 5059 scope.go:122] "RemoveContainer" containerID="2a791fe1424f5e3f7f01347833a3d0c452d0d885d27375eac99aaed96a99aba1" Nov 03 20:15:05.268174 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:05.268146 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt_odh-ai-gateway-infra(f1477a68-b92a-4138-b649-fdcd3b17f693)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt" podUID="f1477a68-b92a-4138-b649-fdcd3b17f693" Nov 03 20:15:05.272058 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.271936 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p" event={"ID":"cb7298dc-a500-4715-a360-bcbdf0c7a298","Type":"ContainerStarted","Data":"7d11fe0783239bbf8305bb91e55e123815ecb6c3374828d9bfb606cc84fa4df8"} Nov 03 20:15:05.274429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.274304 5059 generic.go:356] "Generic (PLEG): container finished" podID="3a1e6b3a-7887-42c5-b23d-b226cc6d0641" containerID="d3c79ea4543dd1915df8c46633fc8e2d807a8d10570b2cb61f85bc02cd46eee0" exitCode=60 Nov 03 20:15:05.274429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.274377 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc" event={"ID":"3a1e6b3a-7887-42c5-b23d-b226cc6d0641","Type":"ContainerDied","Data":"d3c79ea4543dd1915df8c46633fc8e2d807a8d10570b2cb61f85bc02cd46eee0"} Nov 03 20:15:05.274865 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.274843 5059 scope.go:122] "RemoveContainer" containerID="d3c79ea4543dd1915df8c46633fc8e2d807a8d10570b2cb61f85bc02cd46eee0" Nov 03 20:15:05.275191 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:05.275165 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc_odh-ai-gateway-infra(3a1e6b3a-7887-42c5-b23d-b226cc6d0641)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc" podUID="3a1e6b3a-7887-42c5-b23d-b226cc6d0641" Nov 03 20:15:05.278446 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.278420 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp" event={"ID":"e3873f10-84bb-489c-9de5-b907f375ff3b","Type":"ContainerStarted","Data":"65538fb8d9a44370fd7c667232b6fa9352b996a2b8afb7322c8ac62772b76dcc"} Nov 03 20:15:05.286595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.286559 5059 generic.go:356] "Generic (PLEG): container finished" podID="8588fce3-f04c-472e-a796-ed5314845eb5" containerID="b31c48815a514c04876a3d2b2b4c979280141874041103d2903c3f2f0c08868b" exitCode=60 Nov 03 20:15:05.287022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.286734 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm" event={"ID":"8588fce3-f04c-472e-a796-ed5314845eb5","Type":"ContainerDied","Data":"b31c48815a514c04876a3d2b2b4c979280141874041103d2903c3f2f0c08868b"} Nov 03 20:15:05.287283 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.287056 5059 scope.go:122] "RemoveContainer" containerID="b31c48815a514c04876a3d2b2b4c979280141874041103d2903c3f2f0c08868b" Nov 03 20:15:05.288006 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:05.287981 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm_odh-ai-gateway-infra(8588fce3-f04c-472e-a796-ed5314845eb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm" podUID="8588fce3-f04c-472e-a796-ed5314845eb5" Nov 03 20:15:05.291601 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.290803 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl" event={"ID":"30d22aa8-3a01-44f1-a59d-b92a70934a7f","Type":"ContainerStarted","Data":"f5ab6fc16c6c116db960954179c62e7333d0f8b7cfb5208e68e0f033ca807e33"} Nov 03 20:15:05.293830 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.293801 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg" event={"ID":"edb7775d-1e94-40ed-9da3-738ad142ef20","Type":"ContainerStarted","Data":"8624b23a907e6fec32e99f3f1e511f5333507b9d493e122bd70694eced8bdaf2"} Nov 03 20:15:05.309943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.308790 5059 generic.go:356] "Generic (PLEG): container finished" podID="1be9592b-18d0-4ce7-9322-92898161c6d8" containerID="6d124e9a3e506b7312a6889ad995a1dba8e0e21475f34dc893d8ef6818c32f43" exitCode=60 Nov 03 20:15:05.309943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.308886 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp" event={"ID":"1be9592b-18d0-4ce7-9322-92898161c6d8","Type":"ContainerDied","Data":"6d124e9a3e506b7312a6889ad995a1dba8e0e21475f34dc893d8ef6818c32f43"} Nov 03 20:15:05.309943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.309300 5059 scope.go:122] "RemoveContainer" containerID="6d124e9a3e506b7312a6889ad995a1dba8e0e21475f34dc893d8ef6818c32f43" Nov 03 20:15:05.309943 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:05.309687 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp_odh-ai-gateway-infra(1be9592b-18d0-4ce7-9322-92898161c6d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp" podUID="1be9592b-18d0-4ce7-9322-92898161c6d8" Nov 03 20:15:05.312390 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.312309 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8" event={"ID":"0e0bbcca-04df-4474-91fc-c5c5e6521732","Type":"ContainerStarted","Data":"9dbc2829e04680ce84fa93cadf2abbe8c17c35e9687b862bc3da1d0ad2b37254"} Nov 03 20:15:05.316109 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.315296 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq" event={"ID":"8f7ed20a-4c3c-4023-b1de-479a8ea346a8","Type":"ContainerStarted","Data":"4f60f7480a57f3caa3f1d5c2e2d92abbf8eb21a07100fc9813002036efb75287"} Nov 03 20:15:05.318795 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.318772 5059 scope.go:122] "RemoveContainer" containerID="07e2e3c445eccf38243f79efeaea824250cb8407bd3cdcbf92f0c8318cb3d346" Nov 03 20:15:05.318892 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.318817 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9" event={"ID":"afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9","Type":"ContainerStarted","Data":"3fe1a2cef26d035e5c43f6701ff9c787904ab6d185c34c939c7a3346236b8068"} Nov 03 20:15:05.322457 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.322408 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s" event={"ID":"932670b6-eb80-4970-b092-8682e1daa6f5","Type":"ContainerStarted","Data":"e3d2f41c255b5346cae2519e6623612f5606b6131525fc2593e624ffc59f03cd"} Nov 03 20:15:05.324283 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.324257 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m" event={"ID":"545cc204-5445-437d-b58e-9b6494f1a3b7","Type":"ContainerStarted","Data":"fc9d1eb7fe91d2a484c35c68b0cf16395c040ceb653bfb10a71dd3d1d5fd6787"} Nov 03 20:15:05.326512 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.326489 5059 generic.go:356] "Generic (PLEG): container finished" podID="ffb6bb03-2f3c-43f0-871a-1140b53f0fb9" containerID="d0ca647ad2a0bb692e9f226eacc2341615eee4d1712d1925878723228313dd6f" exitCode=60 Nov 03 20:15:05.326605 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.326573 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh" event={"ID":"ffb6bb03-2f3c-43f0-871a-1140b53f0fb9","Type":"ContainerDied","Data":"d0ca647ad2a0bb692e9f226eacc2341615eee4d1712d1925878723228313dd6f"} Nov 03 20:15:05.327013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.326995 5059 scope.go:122] "RemoveContainer" containerID="d0ca647ad2a0bb692e9f226eacc2341615eee4d1712d1925878723228313dd6f" Nov 03 20:15:05.327372 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:05.327348 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895615-ghpwh_odh-ai-gateway-infra(ffb6bb03-2f3c-43f0-871a-1140b53f0fb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh" podUID="ffb6bb03-2f3c-43f0-871a-1140b53f0fb9" Nov 03 20:15:05.362898 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.362875 5059 scope.go:122] "RemoveContainer" containerID="7ecb9dfb8b44ab18b2e162028f6ed7ba702990a2f1b6616f870571db2cae037d" Nov 03 20:15:05.379153 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.379129 5059 scope.go:122] "RemoveContainer" containerID="8e6471a9040955b10b1d38438cf69bdf99f79bc78a228b96d0fe698ae0462287" Nov 03 20:15:05.401360 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.401283 5059 scope.go:122] "RemoveContainer" containerID="b32c4d293b8995f06f1fb5ddb3a001137e90eae1a4f7c0fb9e03d191a3a37478" Nov 03 20:15:05.422568 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.422538 5059 scope.go:122] "RemoveContainer" containerID="1d214d09944a545dab7667764f096ad403d270e2e5d37c37dfbdf2bebfe94c28" Nov 03 20:15:05.443404 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.443370 5059 scope.go:122] "RemoveContainer" containerID="3e20e22599c182419971be9a99e81987add8911d9eb7888383738e6fa6713192" Nov 03 20:15:05.461728 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:05.460422 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq" podStartSLOduration=5.460402622 podStartE2EDuration="5.460402622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:05.458628681 +0000 UTC m=+2299.716425938" watchObservedRunningTime="2026-11-03 20:15:05.460402622 +0000 UTC m=+2299.718199881" Nov 03 20:15:06.338837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.338805 5059 scope.go:122] "RemoveContainer" containerID="b31c48815a514c04876a3d2b2b4c979280141874041103d2903c3f2f0c08868b" Nov 03 20:15:06.339234 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:06.339174 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm_odh-ai-gateway-infra(8588fce3-f04c-472e-a796-ed5314845eb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm" podUID="8588fce3-f04c-472e-a796-ed5314845eb5" Nov 03 20:15:06.340284 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.340261 5059 generic.go:356] "Generic (PLEG): container finished" podID="30d22aa8-3a01-44f1-a59d-b92a70934a7f" containerID="8d7d2b586c54f646194ae892c4cc4ecb93d0c609c3d9cb9d84b684b95151a961" exitCode=60 Nov 03 20:15:06.340376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.340325 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl" event={"ID":"30d22aa8-3a01-44f1-a59d-b92a70934a7f","Type":"ContainerDied","Data":"8d7d2b586c54f646194ae892c4cc4ecb93d0c609c3d9cb9d84b684b95151a961"} Nov 03 20:15:06.340558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.340542 5059 scope.go:122] "RemoveContainer" containerID="8d7d2b586c54f646194ae892c4cc4ecb93d0c609c3d9cb9d84b684b95151a961" Nov 03 20:15:06.342266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.342237 5059 generic.go:356] "Generic (PLEG): container finished" podID="edb7775d-1e94-40ed-9da3-738ad142ef20" containerID="97fba9d6d0460a49eada10cf35f50ef277d07fca427ae47751192defed3250e4" exitCode=60 Nov 03 20:15:06.342384 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.342316 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg" event={"ID":"edb7775d-1e94-40ed-9da3-738ad142ef20","Type":"ContainerDied","Data":"97fba9d6d0460a49eada10cf35f50ef277d07fca427ae47751192defed3250e4"} Nov 03 20:15:06.342549 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.342529 5059 scope.go:122] "RemoveContainer" containerID="97fba9d6d0460a49eada10cf35f50ef277d07fca427ae47751192defed3250e4" Nov 03 20:15:06.344908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.344692 5059 scope.go:122] "RemoveContainer" containerID="6d124e9a3e506b7312a6889ad995a1dba8e0e21475f34dc893d8ef6818c32f43" Nov 03 20:15:06.345016 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:06.344990 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp_odh-ai-gateway-infra(1be9592b-18d0-4ce7-9322-92898161c6d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp" podUID="1be9592b-18d0-4ce7-9322-92898161c6d8" Nov 03 20:15:06.346235 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.346201 5059 generic.go:356] "Generic (PLEG): container finished" podID="0e0bbcca-04df-4474-91fc-c5c5e6521732" containerID="dfdb566211057e29c8badd5b8f5e6430f80f97d478afd97cdaf962ed153dad94" exitCode=60 Nov 03 20:15:06.346362 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.346277 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8" event={"ID":"0e0bbcca-04df-4474-91fc-c5c5e6521732","Type":"ContainerDied","Data":"dfdb566211057e29c8badd5b8f5e6430f80f97d478afd97cdaf962ed153dad94"} Nov 03 20:15:06.346571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.346484 5059 scope.go:122] "RemoveContainer" containerID="dfdb566211057e29c8badd5b8f5e6430f80f97d478afd97cdaf962ed153dad94" Nov 03 20:15:06.348633 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.348609 5059 generic.go:356] "Generic (PLEG): container finished" podID="8f7ed20a-4c3c-4023-b1de-479a8ea346a8" containerID="4f60f7480a57f3caa3f1d5c2e2d92abbf8eb21a07100fc9813002036efb75287" exitCode=60 Nov 03 20:15:06.348805 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.348743 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq" event={"ID":"8f7ed20a-4c3c-4023-b1de-479a8ea346a8","Type":"ContainerDied","Data":"4f60f7480a57f3caa3f1d5c2e2d92abbf8eb21a07100fc9813002036efb75287"} Nov 03 20:15:06.348997 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.348977 5059 scope.go:122] "RemoveContainer" containerID="4f60f7480a57f3caa3f1d5c2e2d92abbf8eb21a07100fc9813002036efb75287" Nov 03 20:15:06.350821 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.350633 5059 generic.go:356] "Generic (PLEG): container finished" podID="afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9" containerID="d7f628c75a9cc807cec172fa5fc6287a7dad7d176c66589c409c76f394f7e034" exitCode=60 Nov 03 20:15:06.350821 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.350731 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9" event={"ID":"afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9","Type":"ContainerDied","Data":"d7f628c75a9cc807cec172fa5fc6287a7dad7d176c66589c409c76f394f7e034"} Nov 03 20:15:06.351600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.351138 5059 scope.go:122] "RemoveContainer" containerID="d7f628c75a9cc807cec172fa5fc6287a7dad7d176c66589c409c76f394f7e034" Nov 03 20:15:06.358837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.355098 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s" event={"ID":"932670b6-eb80-4970-b092-8682e1daa6f5","Type":"ContainerStarted","Data":"bd7bcb8e314b81ca67b052cfe0fca6f72a4eeb556cf793496a60745dd66aeacb"} Nov 03 20:15:06.402654 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.378365 5059 generic.go:356] "Generic (PLEG): container finished" podID="545cc204-5445-437d-b58e-9b6494f1a3b7" containerID="acf38b9c201cfc558dda46642bc6e28ea89cb07642ca5a974d0778e0b1c85880" exitCode=60 Nov 03 20:15:06.402654 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.378452 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m" event={"ID":"545cc204-5445-437d-b58e-9b6494f1a3b7","Type":"ContainerDied","Data":"acf38b9c201cfc558dda46642bc6e28ea89cb07642ca5a974d0778e0b1c85880"} Nov 03 20:15:06.402654 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.378960 5059 scope.go:122] "RemoveContainer" containerID="acf38b9c201cfc558dda46642bc6e28ea89cb07642ca5a974d0778e0b1c85880" Nov 03 20:15:06.452408 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.452373 5059 scope.go:122] "RemoveContainer" containerID="d0ca647ad2a0bb692e9f226eacc2341615eee4d1712d1925878723228313dd6f" Nov 03 20:15:06.453369 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:06.452924 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895615-ghpwh_odh-ai-gateway-infra(ffb6bb03-2f3c-43f0-871a-1140b53f0fb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh" podUID="ffb6bb03-2f3c-43f0-871a-1140b53f0fb9" Nov 03 20:15:06.479268 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.479227 5059 scope.go:122] "RemoveContainer" containerID="27c5d4aab2ceeb8f6dcc2fa3afdeac510698928a98e9dfce1c397485f66dd355" Nov 03 20:15:06.479716 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:06.479637 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc_odh-ai-gateway-infra(1774de9e-305c-4b0f-b194-a09995fb25bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc" podUID="1774de9e-305c-4b0f-b194-a09995fb25bf" Nov 03 20:15:06.485105 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.485080 5059 scope.go:122] "RemoveContainer" containerID="344db2809ca423c43d7ba4570f981c986e2be112e1c054ff4d7c69f78a2e4da5" Nov 03 20:15:06.485412 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:06.485389 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59_odh-ai-gateway-infra(0b18a5b0-0fc4-4dc8-88b7-affec66830d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59" podUID="0b18a5b0-0fc4-4dc8-88b7-affec66830d8" Nov 03 20:15:06.488166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.488074 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r" event={"ID":"32905cf1-9e12-4fc4-8da4-dda3d142a0a0","Type":"ContainerStarted","Data":"a107bc168a164406d6ea61ce0a988930867021ec1584134e9bba99cca682cb88"} Nov 03 20:15:06.492151 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.492096 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft" event={"ID":"ea54ebf8-4aab-4442-adf0-6930bcb6229b","Type":"ContainerStarted","Data":"0b0635a36e8a232f1f814fabcc60a86cd5df249945d9cf90694feca3ab729d62"} Nov 03 20:15:06.498136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.498116 5059 scope.go:122] "RemoveContainer" containerID="2a791fe1424f5e3f7f01347833a3d0c452d0d885d27375eac99aaed96a99aba1" Nov 03 20:15:06.499549 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:06.498518 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt_odh-ai-gateway-infra(f1477a68-b92a-4138-b649-fdcd3b17f693)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt" podUID="f1477a68-b92a-4138-b649-fdcd3b17f693" Nov 03 20:15:06.501848 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.501241 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s" podStartSLOduration=6.501229389 podStartE2EDuration="6.501229389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:06.461891877 +0000 UTC m=+2300.719689134" watchObservedRunningTime="2026-11-03 20:15:06.501229389 +0000 UTC m=+2300.759026645" Nov 03 20:15:06.502560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.502535 5059 generic.go:356] "Generic (PLEG): container finished" podID="cb7298dc-a500-4715-a360-bcbdf0c7a298" containerID="7d11fe0783239bbf8305bb91e55e123815ecb6c3374828d9bfb606cc84fa4df8" exitCode=60 Nov 03 20:15:06.502688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.502613 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p" event={"ID":"cb7298dc-a500-4715-a360-bcbdf0c7a298","Type":"ContainerDied","Data":"7d11fe0783239bbf8305bb91e55e123815ecb6c3374828d9bfb606cc84fa4df8"} Nov 03 20:15:06.502817 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.502800 5059 scope.go:122] "RemoveContainer" containerID="739c84fbdbd4472de209b03f5016c97d18fa9210f304c910c8b8d69956d55c4e" Nov 03 20:15:06.502999 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.502976 5059 scope.go:122] "RemoveContainer" containerID="7d11fe0783239bbf8305bb91e55e123815ecb6c3374828d9bfb606cc84fa4df8" Nov 03 20:15:06.503414 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:06.503385 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p_odh-ai-gateway-infra(cb7298dc-a500-4715-a360-bcbdf0c7a298)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p" podUID="cb7298dc-a500-4715-a360-bcbdf0c7a298" Nov 03 20:15:06.504895 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.504871 5059 scope.go:122] "RemoveContainer" containerID="d3c79ea4543dd1915df8c46633fc8e2d807a8d10570b2cb61f85bc02cd46eee0" Nov 03 20:15:06.505234 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:06.505211 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc_odh-ai-gateway-infra(3a1e6b3a-7887-42c5-b23d-b226cc6d0641)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc" podUID="3a1e6b3a-7887-42c5-b23d-b226cc6d0641" Nov 03 20:15:06.507251 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.507209 5059 generic.go:356] "Generic (PLEG): container finished" podID="e3873f10-84bb-489c-9de5-b907f375ff3b" containerID="aff99e33d5b636eb6e448ee05d5fe6d2d297cd59156090e486bb86186fcda8d7" exitCode=60 Nov 03 20:15:06.507373 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.507271 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp" event={"ID":"e3873f10-84bb-489c-9de5-b907f375ff3b","Type":"ContainerDied","Data":"aff99e33d5b636eb6e448ee05d5fe6d2d297cd59156090e486bb86186fcda8d7"} Nov 03 20:15:06.507708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.507644 5059 scope.go:122] "RemoveContainer" containerID="aff99e33d5b636eb6e448ee05d5fe6d2d297cd59156090e486bb86186fcda8d7" Nov 03 20:15:06.699127 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.699017 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r" podStartSLOduration=6.699000164 podStartE2EDuration="6.699000164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:06.692006686 +0000 UTC m=+2300.949803942" watchObservedRunningTime="2026-11-03 20:15:06.699000164 +0000 UTC m=+2300.956797422" Nov 03 20:15:06.715441 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:06.715392 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft" podStartSLOduration=6.715377321 podStartE2EDuration="6.715377321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:06.71204803 +0000 UTC m=+2300.969845289" watchObservedRunningTime="2026-11-03 20:15:06.715377321 +0000 UTC m=+2300.973174578" Nov 03 20:15:07.519644 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:07.519605 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq" event={"ID":"8f7ed20a-4c3c-4023-b1de-479a8ea346a8","Type":"ContainerStarted","Data":"62db5cf0bb31f7b7d7282599d004697222a80261497bbef3283f170cd0cd87a3"} Nov 03 20:15:07.521906 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:07.521829 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9" event={"ID":"afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9","Type":"ContainerStarted","Data":"3cdda42662f948d6383cab1dc729ed28c7c179dd9a55b7685cde56e67593dfc7"} Nov 03 20:15:07.523571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:07.523544 5059 generic.go:356] "Generic (PLEG): container finished" podID="932670b6-eb80-4970-b092-8682e1daa6f5" containerID="bd7bcb8e314b81ca67b052cfe0fca6f72a4eeb556cf793496a60745dd66aeacb" exitCode=60 Nov 03 20:15:07.523725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:07.523619 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s" event={"ID":"932670b6-eb80-4970-b092-8682e1daa6f5","Type":"ContainerDied","Data":"bd7bcb8e314b81ca67b052cfe0fca6f72a4eeb556cf793496a60745dd66aeacb"} Nov 03 20:15:07.523930 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:07.523908 5059 scope.go:122] "RemoveContainer" containerID="bd7bcb8e314b81ca67b052cfe0fca6f72a4eeb556cf793496a60745dd66aeacb" Nov 03 20:15:07.525821 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:07.525780 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m" event={"ID":"545cc204-5445-437d-b58e-9b6494f1a3b7","Type":"ContainerStarted","Data":"a2d590c159c646b7bc90c66c7292004a5c54a7b78cad668d2cd0be92bd7864fd"} Nov 03 20:15:07.528098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:07.527937 5059 generic.go:356] "Generic (PLEG): container finished" podID="32905cf1-9e12-4fc4-8da4-dda3d142a0a0" containerID="a107bc168a164406d6ea61ce0a988930867021ec1584134e9bba99cca682cb88" exitCode=60 Nov 03 20:15:07.528098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:07.528016 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r" event={"ID":"32905cf1-9e12-4fc4-8da4-dda3d142a0a0","Type":"ContainerDied","Data":"a107bc168a164406d6ea61ce0a988930867021ec1584134e9bba99cca682cb88"} Nov 03 20:15:07.528098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:07.528076 5059 scope.go:122] "RemoveContainer" containerID="c84575b5df8beccb7c1ee5e4fea2e730526cbda28bc23acb585bc1a231d9a391" Nov 03 20:15:07.528305 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:07.528249 5059 scope.go:122] "RemoveContainer" containerID="a107bc168a164406d6ea61ce0a988930867021ec1584134e9bba99cca682cb88" Nov 03 20:15:07.528575 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:07.528544 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r_odh-ai-gateway-infra(32905cf1-9e12-4fc4-8da4-dda3d142a0a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r" podUID="32905cf1-9e12-4fc4-8da4-dda3d142a0a0" Nov 03 20:15:07.529848 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:07.529828 5059 generic.go:356] "Generic (PLEG): container finished" podID="ea54ebf8-4aab-4442-adf0-6930bcb6229b" containerID="0b0635a36e8a232f1f814fabcc60a86cd5df249945d9cf90694feca3ab729d62" exitCode=60 Nov 03 20:15:07.529957 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:07.529913 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft" event={"ID":"ea54ebf8-4aab-4442-adf0-6930bcb6229b","Type":"ContainerDied","Data":"0b0635a36e8a232f1f814fabcc60a86cd5df249945d9cf90694feca3ab729d62"} Nov 03 20:15:07.530110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:07.530097 5059 scope.go:122] "RemoveContainer" containerID="0b0635a36e8a232f1f814fabcc60a86cd5df249945d9cf90694feca3ab729d62" Nov 03 20:15:07.531883 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:07.531860 5059 scope.go:122] "RemoveContainer" containerID="7d11fe0783239bbf8305bb91e55e123815ecb6c3374828d9bfb606cc84fa4df8" Nov 03 20:15:07.532122 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:07.532100 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p_odh-ai-gateway-infra(cb7298dc-a500-4715-a360-bcbdf0c7a298)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p" podUID="cb7298dc-a500-4715-a360-bcbdf0c7a298" Nov 03 20:15:07.533556 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:07.533529 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp" event={"ID":"e3873f10-84bb-489c-9de5-b907f375ff3b","Type":"ContainerStarted","Data":"a3c0cdc5de182ffb64dc9f0b93740ca549919780765f8c433b481f20ebf9f847"} Nov 03 20:15:07.535509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:07.535489 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl" event={"ID":"30d22aa8-3a01-44f1-a59d-b92a70934a7f","Type":"ContainerStarted","Data":"d9df510a86b0006d44422c8dba441a63833b88a5486cf58a51b6fb2d4cb481c5"} Nov 03 20:15:07.537625 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:07.537597 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg" event={"ID":"edb7775d-1e94-40ed-9da3-738ad142ef20","Type":"ContainerStarted","Data":"74a0a6de594c8105435cf5aa3513b6e272c53e1eed665d6c4ec2967c7926770f"} Nov 03 20:15:07.541265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:07.541186 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8" event={"ID":"0e0bbcca-04df-4474-91fc-c5c5e6521732","Type":"ContainerStarted","Data":"46a105c28d2c721125674cf14f8357c899b78479d280b9e47a7775d675fcbc1c"} Nov 03 20:15:07.587203 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:07.587148 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m" podStartSLOduration=7.587133057 podStartE2EDuration="7.587133057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:07.561870746 +0000 UTC m=+2301.819668017" watchObservedRunningTime="2026-11-03 20:15:07.587133057 +0000 UTC m=+2301.844930373" Nov 03 20:15:07.588085 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:07.588051 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp" podStartSLOduration=7.588038706 podStartE2EDuration="7.588038706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:07.58172726 +0000 UTC m=+2301.839524518" watchObservedRunningTime="2026-11-03 20:15:07.588038706 +0000 UTC m=+2301.845835984" Nov 03 20:15:07.618750 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:07.615116 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl" podStartSLOduration=7.615097994 podStartE2EDuration="7.615097994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:07.607352059 +0000 UTC m=+2301.865149317" watchObservedRunningTime="2026-11-03 20:15:07.615097994 +0000 UTC m=+2301.872895253" Nov 03 20:15:07.647772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:07.647709 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9" podStartSLOduration=7.6476900709999995 podStartE2EDuration="7.647690071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:07.63695182 +0000 UTC m=+2301.894749077" watchObservedRunningTime="2026-11-03 20:15:07.647690071 +0000 UTC m=+2301.905487331" Nov 03 20:15:07.684412 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:07.684357 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg" podStartSLOduration=7.68434042 podStartE2EDuration="7.68434042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:07.679348811 +0000 UTC m=+2301.937146081" watchObservedRunningTime="2026-11-03 20:15:07.68434042 +0000 UTC m=+2301.942137740" Nov 03 20:15:07.766715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:07.766646 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8" podStartSLOduration=7.766628098 podStartE2EDuration="7.766628098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:07.764067199 +0000 UTC m=+2302.021864458" watchObservedRunningTime="2026-11-03 20:15:07.766628098 +0000 UTC m=+2302.024425368" Nov 03 20:15:08.552745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.552702 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft" event={"ID":"ea54ebf8-4aab-4442-adf0-6930bcb6229b","Type":"ContainerStarted","Data":"b361175f83d8fc6249aae9c43aa927cb6ee5ab1c7cc52bcebe89a59be47783a9"} Nov 03 20:15:08.555130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.555095 5059 generic.go:356] "Generic (PLEG): container finished" podID="e3873f10-84bb-489c-9de5-b907f375ff3b" containerID="a3c0cdc5de182ffb64dc9f0b93740ca549919780765f8c433b481f20ebf9f847" exitCode=60 Nov 03 20:15:08.555267 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.555157 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp" event={"ID":"e3873f10-84bb-489c-9de5-b907f375ff3b","Type":"ContainerDied","Data":"a3c0cdc5de182ffb64dc9f0b93740ca549919780765f8c433b481f20ebf9f847"} Nov 03 20:15:08.555267 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.555223 5059 scope.go:122] "RemoveContainer" containerID="aff99e33d5b636eb6e448ee05d5fe6d2d297cd59156090e486bb86186fcda8d7" Nov 03 20:15:08.555625 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.555588 5059 scope.go:122] "RemoveContainer" containerID="a3c0cdc5de182ffb64dc9f0b93740ca549919780765f8c433b481f20ebf9f847" Nov 03 20:15:08.556124 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:08.556093 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp_odh-ai-gateway-infra(e3873f10-84bb-489c-9de5-b907f375ff3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp" podUID="e3873f10-84bb-489c-9de5-b907f375ff3b" Nov 03 20:15:08.557734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.557708 5059 generic.go:356] "Generic (PLEG): container finished" podID="30d22aa8-3a01-44f1-a59d-b92a70934a7f" containerID="d9df510a86b0006d44422c8dba441a63833b88a5486cf58a51b6fb2d4cb481c5" exitCode=60 Nov 03 20:15:08.557882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.557743 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl" event={"ID":"30d22aa8-3a01-44f1-a59d-b92a70934a7f","Type":"ContainerDied","Data":"d9df510a86b0006d44422c8dba441a63833b88a5486cf58a51b6fb2d4cb481c5"} Nov 03 20:15:08.558173 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.558153 5059 scope.go:122] "RemoveContainer" containerID="d9df510a86b0006d44422c8dba441a63833b88a5486cf58a51b6fb2d4cb481c5" Nov 03 20:15:08.558633 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:08.558592 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl_odh-ai-gateway-infra(30d22aa8-3a01-44f1-a59d-b92a70934a7f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl" podUID="30d22aa8-3a01-44f1-a59d-b92a70934a7f" Nov 03 20:15:08.560168 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.560141 5059 generic.go:356] "Generic (PLEG): container finished" podID="edb7775d-1e94-40ed-9da3-738ad142ef20" containerID="74a0a6de594c8105435cf5aa3513b6e272c53e1eed665d6c4ec2967c7926770f" exitCode=60 Nov 03 20:15:08.560350 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.560267 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg" event={"ID":"edb7775d-1e94-40ed-9da3-738ad142ef20","Type":"ContainerDied","Data":"74a0a6de594c8105435cf5aa3513b6e272c53e1eed665d6c4ec2967c7926770f"} Nov 03 20:15:08.560521 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.560503 5059 scope.go:122] "RemoveContainer" containerID="74a0a6de594c8105435cf5aa3513b6e272c53e1eed665d6c4ec2967c7926770f" Nov 03 20:15:08.560909 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:08.560888 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg_odh-ai-gateway-infra(edb7775d-1e94-40ed-9da3-738ad142ef20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg" podUID="edb7775d-1e94-40ed-9da3-738ad142ef20" Nov 03 20:15:08.563074 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.563051 5059 generic.go:356] "Generic (PLEG): container finished" podID="0e0bbcca-04df-4474-91fc-c5c5e6521732" containerID="46a105c28d2c721125674cf14f8357c899b78479d280b9e47a7775d675fcbc1c" exitCode=60 Nov 03 20:15:08.563164 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.563074 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8" event={"ID":"0e0bbcca-04df-4474-91fc-c5c5e6521732","Type":"ContainerDied","Data":"46a105c28d2c721125674cf14f8357c899b78479d280b9e47a7775d675fcbc1c"} Nov 03 20:15:08.563478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.563455 5059 scope.go:122] "RemoveContainer" containerID="46a105c28d2c721125674cf14f8357c899b78479d280b9e47a7775d675fcbc1c" Nov 03 20:15:08.563910 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:08.563887 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8_odh-ai-gateway-infra(0e0bbcca-04df-4474-91fc-c5c5e6521732)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8" podUID="0e0bbcca-04df-4474-91fc-c5c5e6521732" Nov 03 20:15:08.565486 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.565465 5059 generic.go:356] "Generic (PLEG): container finished" podID="8f7ed20a-4c3c-4023-b1de-479a8ea346a8" containerID="62db5cf0bb31f7b7d7282599d004697222a80261497bbef3283f170cd0cd87a3" exitCode=60 Nov 03 20:15:08.565589 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.565538 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq" event={"ID":"8f7ed20a-4c3c-4023-b1de-479a8ea346a8","Type":"ContainerDied","Data":"62db5cf0bb31f7b7d7282599d004697222a80261497bbef3283f170cd0cd87a3"} Nov 03 20:15:08.565938 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.565918 5059 scope.go:122] "RemoveContainer" containerID="62db5cf0bb31f7b7d7282599d004697222a80261497bbef3283f170cd0cd87a3" Nov 03 20:15:08.566301 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:08.566277 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq_odh-ai-gateway-infra(8f7ed20a-4c3c-4023-b1de-479a8ea346a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq" podUID="8f7ed20a-4c3c-4023-b1de-479a8ea346a8" Nov 03 20:15:08.567837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.567817 5059 generic.go:356] "Generic (PLEG): container finished" podID="afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9" containerID="3cdda42662f948d6383cab1dc729ed28c7c179dd9a55b7685cde56e67593dfc7" exitCode=60 Nov 03 20:15:08.567918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.567884 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9" event={"ID":"afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9","Type":"ContainerDied","Data":"3cdda42662f948d6383cab1dc729ed28c7c179dd9a55b7685cde56e67593dfc7"} Nov 03 20:15:08.568176 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.568160 5059 scope.go:122] "RemoveContainer" containerID="3cdda42662f948d6383cab1dc729ed28c7c179dd9a55b7685cde56e67593dfc7" Nov 03 20:15:08.568496 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:08.568477 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9_odh-ai-gateway-infra(afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9" podUID="afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9" Nov 03 20:15:08.570246 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.570220 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s" event={"ID":"932670b6-eb80-4970-b092-8682e1daa6f5","Type":"ContainerStarted","Data":"2e0cf38a22a0523c5647f89e6a438290f8aaf64cfc728ffd56d51ef3ee205c02"} Nov 03 20:15:08.573247 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.573221 5059 generic.go:356] "Generic (PLEG): container finished" podID="545cc204-5445-437d-b58e-9b6494f1a3b7" containerID="a2d590c159c646b7bc90c66c7292004a5c54a7b78cad668d2cd0be92bd7864fd" exitCode=60 Nov 03 20:15:08.573381 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.573366 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m" event={"ID":"545cc204-5445-437d-b58e-9b6494f1a3b7","Type":"ContainerDied","Data":"a2d590c159c646b7bc90c66c7292004a5c54a7b78cad668d2cd0be92bd7864fd"} Nov 03 20:15:08.573750 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.573734 5059 scope.go:122] "RemoveContainer" containerID="a2d590c159c646b7bc90c66c7292004a5c54a7b78cad668d2cd0be92bd7864fd" Nov 03 20:15:08.574069 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:08.574051 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m_odh-ai-gateway-infra(545cc204-5445-437d-b58e-9b6494f1a3b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m" podUID="545cc204-5445-437d-b58e-9b6494f1a3b7" Nov 03 20:15:08.574167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.574150 5059 scope.go:122] "RemoveContainer" containerID="8d7d2b586c54f646194ae892c4cc4ecb93d0c609c3d9cb9d84b684b95151a961" Nov 03 20:15:08.576677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.576644 5059 scope.go:122] "RemoveContainer" containerID="a107bc168a164406d6ea61ce0a988930867021ec1584134e9bba99cca682cb88" Nov 03 20:15:08.577040 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:08.577014 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r_odh-ai-gateway-infra(32905cf1-9e12-4fc4-8da4-dda3d142a0a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r" podUID="32905cf1-9e12-4fc4-8da4-dda3d142a0a0" Nov 03 20:15:08.592206 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.592173 5059 scope.go:122] "RemoveContainer" containerID="97fba9d6d0460a49eada10cf35f50ef277d07fca427ae47751192defed3250e4" Nov 03 20:15:08.611743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.611644 5059 scope.go:122] "RemoveContainer" containerID="dfdb566211057e29c8badd5b8f5e6430f80f97d478afd97cdaf962ed153dad94" Nov 03 20:15:08.651901 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.651863 5059 scope.go:122] "RemoveContainer" containerID="4f60f7480a57f3caa3f1d5c2e2d92abbf8eb21a07100fc9813002036efb75287" Nov 03 20:15:08.741785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.741756 5059 scope.go:122] "RemoveContainer" containerID="d7f628c75a9cc807cec172fa5fc6287a7dad7d176c66589c409c76f394f7e034" Nov 03 20:15:08.848937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:08.848904 5059 scope.go:122] "RemoveContainer" containerID="acf38b9c201cfc558dda46642bc6e28ea89cb07642ca5a974d0778e0b1c85880" Nov 03 20:15:09.587948 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:09.587905 5059 scope.go:122] "RemoveContainer" containerID="62db5cf0bb31f7b7d7282599d004697222a80261497bbef3283f170cd0cd87a3" Nov 03 20:15:09.588397 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:09.588370 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq_odh-ai-gateway-infra(8f7ed20a-4c3c-4023-b1de-479a8ea346a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq" podUID="8f7ed20a-4c3c-4023-b1de-479a8ea346a8" Nov 03 20:15:09.589566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:09.589546 5059 scope.go:122] "RemoveContainer" containerID="3cdda42662f948d6383cab1dc729ed28c7c179dd9a55b7685cde56e67593dfc7" Nov 03 20:15:09.589963 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:09.589939 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9_odh-ai-gateway-infra(afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9" podUID="afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9" Nov 03 20:15:09.591383 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:09.591347 5059 generic.go:356] "Generic (PLEG): container finished" podID="932670b6-eb80-4970-b092-8682e1daa6f5" containerID="2e0cf38a22a0523c5647f89e6a438290f8aaf64cfc728ffd56d51ef3ee205c02" exitCode=60 Nov 03 20:15:09.591558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:09.591431 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s" event={"ID":"932670b6-eb80-4970-b092-8682e1daa6f5","Type":"ContainerDied","Data":"2e0cf38a22a0523c5647f89e6a438290f8aaf64cfc728ffd56d51ef3ee205c02"} Nov 03 20:15:09.591558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:09.591523 5059 scope.go:122] "RemoveContainer" containerID="bd7bcb8e314b81ca67b052cfe0fca6f72a4eeb556cf793496a60745dd66aeacb" Nov 03 20:15:09.591764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:09.591645 5059 scope.go:122] "RemoveContainer" containerID="2e0cf38a22a0523c5647f89e6a438290f8aaf64cfc728ffd56d51ef3ee205c02" Nov 03 20:15:09.592091 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:09.592064 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s_odh-ai-gateway-infra(932670b6-eb80-4970-b092-8682e1daa6f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s" podUID="932670b6-eb80-4970-b092-8682e1daa6f5" Nov 03 20:15:09.593468 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:09.593451 5059 scope.go:122] "RemoveContainer" containerID="a2d590c159c646b7bc90c66c7292004a5c54a7b78cad668d2cd0be92bd7864fd" Nov 03 20:15:09.593784 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:09.593764 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m_odh-ai-gateway-infra(545cc204-5445-437d-b58e-9b6494f1a3b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m" podUID="545cc204-5445-437d-b58e-9b6494f1a3b7" Nov 03 20:15:09.595331 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:09.595308 5059 generic.go:356] "Generic (PLEG): container finished" podID="ea54ebf8-4aab-4442-adf0-6930bcb6229b" containerID="b361175f83d8fc6249aae9c43aa927cb6ee5ab1c7cc52bcebe89a59be47783a9" exitCode=60 Nov 03 20:15:09.595425 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:09.595383 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft" event={"ID":"ea54ebf8-4aab-4442-adf0-6930bcb6229b","Type":"ContainerDied","Data":"b361175f83d8fc6249aae9c43aa927cb6ee5ab1c7cc52bcebe89a59be47783a9"} Nov 03 20:15:09.595616 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:09.595599 5059 scope.go:122] "RemoveContainer" containerID="b361175f83d8fc6249aae9c43aa927cb6ee5ab1c7cc52bcebe89a59be47783a9" Nov 03 20:15:09.595924 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:09.595899 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft_odh-ai-gateway-infra(ea54ebf8-4aab-4442-adf0-6930bcb6229b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft" podUID="ea54ebf8-4aab-4442-adf0-6930bcb6229b" Nov 03 20:15:09.597583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:09.597568 5059 scope.go:122] "RemoveContainer" containerID="a3c0cdc5de182ffb64dc9f0b93740ca549919780765f8c433b481f20ebf9f847" Nov 03 20:15:09.597866 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:09.597847 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp_odh-ai-gateway-infra(e3873f10-84bb-489c-9de5-b907f375ff3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp" podUID="e3873f10-84bb-489c-9de5-b907f375ff3b" Nov 03 20:15:09.599356 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:09.599333 5059 scope.go:122] "RemoveContainer" containerID="d9df510a86b0006d44422c8dba441a63833b88a5486cf58a51b6fb2d4cb481c5" Nov 03 20:15:09.599615 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:09.599595 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl_odh-ai-gateway-infra(30d22aa8-3a01-44f1-a59d-b92a70934a7f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl" podUID="30d22aa8-3a01-44f1-a59d-b92a70934a7f" Nov 03 20:15:09.601042 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:09.601021 5059 scope.go:122] "RemoveContainer" containerID="74a0a6de594c8105435cf5aa3513b6e272c53e1eed665d6c4ec2967c7926770f" Nov 03 20:15:09.601282 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:09.601255 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg_odh-ai-gateway-infra(edb7775d-1e94-40ed-9da3-738ad142ef20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg" podUID="edb7775d-1e94-40ed-9da3-738ad142ef20" Nov 03 20:15:09.602865 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:09.602846 5059 scope.go:122] "RemoveContainer" containerID="46a105c28d2c721125674cf14f8357c899b78479d280b9e47a7775d675fcbc1c" Nov 03 20:15:09.603081 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:09.603058 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8_odh-ai-gateway-infra(0e0bbcca-04df-4474-91fc-c5c5e6521732)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8" podUID="0e0bbcca-04df-4474-91fc-c5c5e6521732" Nov 03 20:15:09.607131 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:09.607111 5059 scope.go:122] "RemoveContainer" containerID="0b0635a36e8a232f1f814fabcc60a86cd5df249945d9cf90694feca3ab729d62" Nov 03 20:15:10.622756 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:10.621022 5059 scope.go:122] "RemoveContainer" containerID="2e0cf38a22a0523c5647f89e6a438290f8aaf64cfc728ffd56d51ef3ee205c02" Nov 03 20:15:10.622756 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:10.621624 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s_odh-ai-gateway-infra(932670b6-eb80-4970-b092-8682e1daa6f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s" podUID="932670b6-eb80-4970-b092-8682e1daa6f5" Nov 03 20:15:10.628206 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:10.628181 5059 scope.go:122] "RemoveContainer" containerID="b361175f83d8fc6249aae9c43aa927cb6ee5ab1c7cc52bcebe89a59be47783a9" Nov 03 20:15:10.628625 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:10.628588 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft_odh-ai-gateway-infra(ea54ebf8-4aab-4442-adf0-6930bcb6229b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft" podUID="ea54ebf8-4aab-4442-adf0-6930bcb6229b" Nov 03 20:15:10.636827 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:10.636782 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:15:15.875381 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:15.875352 5059 scope.go:122] "RemoveContainer" containerID="27c5d4aab2ceeb8f6dcc2fa3afdeac510698928a98e9dfce1c397485f66dd355" Nov 03 20:15:16.694170 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:16.694134 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc" event={"ID":"1774de9e-305c-4b0f-b194-a09995fb25bf","Type":"ContainerStarted","Data":"1193939b16e300bbf09f9a9ad9499fd4ec5bfd8591c1c663f4106b4750795f90"} Nov 03 20:15:16.850897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:16.850853 5059 scope.go:122] "RemoveContainer" containerID="d3c79ea4543dd1915df8c46633fc8e2d807a8d10570b2cb61f85bc02cd46eee0" Nov 03 20:15:16.851117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:16.851089 5059 scope.go:122] "RemoveContainer" containerID="2a791fe1424f5e3f7f01347833a3d0c452d0d885d27375eac99aaed96a99aba1" Nov 03 20:15:16.851234 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:16.851153 5059 scope.go:122] "RemoveContainer" containerID="d0ca647ad2a0bb692e9f226eacc2341615eee4d1712d1925878723228313dd6f" Nov 03 20:15:16.851298 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:16.851262 5059 scope.go:122] "RemoveContainer" containerID="344db2809ca423c43d7ba4570f981c986e2be112e1c054ff4d7c69f78a2e4da5" Nov 03 20:15:17.707261 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:17.707028 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt" event={"ID":"f1477a68-b92a-4138-b649-fdcd3b17f693","Type":"ContainerStarted","Data":"319bb529863c075c1ff15187e656a7b32680603ea86273abb13e4cadfccf2151"} Nov 03 20:15:17.710368 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:17.710340 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc" event={"ID":"3a1e6b3a-7887-42c5-b23d-b226cc6d0641","Type":"ContainerStarted","Data":"1aa609000036d90f9226a4aef57f32111d3a5b229ebacc10cc05bb8ab5616130"} Nov 03 20:15:17.712716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:17.712690 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh" event={"ID":"ffb6bb03-2f3c-43f0-871a-1140b53f0fb9","Type":"ContainerStarted","Data":"3c41a492dd335502a3885b51fd2f9a83ea0ce4c2c1c501711e46e143395d9d17"} Nov 03 20:15:17.714643 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:17.714619 5059 generic.go:356] "Generic (PLEG): container finished" podID="1774de9e-305c-4b0f-b194-a09995fb25bf" containerID="1193939b16e300bbf09f9a9ad9499fd4ec5bfd8591c1c663f4106b4750795f90" exitCode=60 Nov 03 20:15:17.714765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:17.714728 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc" event={"ID":"1774de9e-305c-4b0f-b194-a09995fb25bf","Type":"ContainerDied","Data":"1193939b16e300bbf09f9a9ad9499fd4ec5bfd8591c1c663f4106b4750795f90"} Nov 03 20:15:17.714834 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:17.714794 5059 scope.go:122] "RemoveContainer" containerID="27c5d4aab2ceeb8f6dcc2fa3afdeac510698928a98e9dfce1c397485f66dd355" Nov 03 20:15:17.715161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:17.715141 5059 scope.go:122] "RemoveContainer" containerID="1193939b16e300bbf09f9a9ad9499fd4ec5bfd8591c1c663f4106b4750795f90" Nov 03 20:15:17.715561 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:17.715533 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc_odh-ai-gateway-infra(1774de9e-305c-4b0f-b194-a09995fb25bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc" podUID="1774de9e-305c-4b0f-b194-a09995fb25bf" Nov 03 20:15:17.717970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:17.717947 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59" event={"ID":"0b18a5b0-0fc4-4dc8-88b7-affec66830d8","Type":"ContainerStarted","Data":"613626463cb6ef0eb3b4f7d116501779c87e91578038a814c17c378d42164486"} Nov 03 20:15:17.728435 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:17.728411 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc"] Nov 03 20:15:17.863511 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:17.863478 5059 scope.go:122] "RemoveContainer" containerID="b31c48815a514c04876a3d2b2b4c979280141874041103d2903c3f2f0c08868b" Nov 03 20:15:17.863695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:17.863669 5059 scope.go:122] "RemoveContainer" containerID="7d11fe0783239bbf8305bb91e55e123815ecb6c3374828d9bfb606cc84fa4df8" Nov 03 20:15:17.863908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:17.863878 5059 scope.go:122] "RemoveContainer" containerID="a107bc168a164406d6ea61ce0a988930867021ec1584134e9bba99cca682cb88" Nov 03 20:15:18.729849 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.729817 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm" event={"ID":"8588fce3-f04c-472e-a796-ed5314845eb5","Type":"ContainerStarted","Data":"6723f36ac35723bbae65457d1e42f88ad86a0bee98091ecf327e591f500a2eef"} Nov 03 20:15:18.731958 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.731931 5059 generic.go:356] "Generic (PLEG): container finished" podID="ffb6bb03-2f3c-43f0-871a-1140b53f0fb9" containerID="3c41a492dd335502a3885b51fd2f9a83ea0ce4c2c1c501711e46e143395d9d17" exitCode=60 Nov 03 20:15:18.732085 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.731997 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh" event={"ID":"ffb6bb03-2f3c-43f0-871a-1140b53f0fb9","Type":"ContainerDied","Data":"3c41a492dd335502a3885b51fd2f9a83ea0ce4c2c1c501711e46e143395d9d17"} Nov 03 20:15:18.732085 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.732049 5059 scope.go:122] "RemoveContainer" containerID="d0ca647ad2a0bb692e9f226eacc2341615eee4d1712d1925878723228313dd6f" Nov 03 20:15:18.732474 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.732450 5059 scope.go:122] "RemoveContainer" containerID="3c41a492dd335502a3885b51fd2f9a83ea0ce4c2c1c501711e46e143395d9d17" Nov 03 20:15:18.733135 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:18.733112 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895615-ghpwh_odh-ai-gateway-infra(ffb6bb03-2f3c-43f0-871a-1140b53f0fb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh" podUID="ffb6bb03-2f3c-43f0-871a-1140b53f0fb9" Nov 03 20:15:18.736064 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.736045 5059 generic.go:356] "Generic (PLEG): container finished" podID="0b18a5b0-0fc4-4dc8-88b7-affec66830d8" containerID="613626463cb6ef0eb3b4f7d116501779c87e91578038a814c17c378d42164486" exitCode=60 Nov 03 20:15:18.736165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.736109 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59" event={"ID":"0b18a5b0-0fc4-4dc8-88b7-affec66830d8","Type":"ContainerDied","Data":"613626463cb6ef0eb3b4f7d116501779c87e91578038a814c17c378d42164486"} Nov 03 20:15:18.736472 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.736439 5059 scope.go:122] "RemoveContainer" containerID="613626463cb6ef0eb3b4f7d116501779c87e91578038a814c17c378d42164486" Nov 03 20:15:18.736801 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:18.736778 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59_odh-ai-gateway-infra(0b18a5b0-0fc4-4dc8-88b7-affec66830d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59" podUID="0b18a5b0-0fc4-4dc8-88b7-affec66830d8" Nov 03 20:15:18.738407 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.738383 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r" event={"ID":"32905cf1-9e12-4fc4-8da4-dda3d142a0a0","Type":"ContainerStarted","Data":"d5645d4de166254ec4419354b6daee448fe4f7efa4f8a9d97d48ca979ad1c0d1"} Nov 03 20:15:18.740993 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.740972 5059 generic.go:356] "Generic (PLEG): container finished" podID="f1477a68-b92a-4138-b649-fdcd3b17f693" containerID="319bb529863c075c1ff15187e656a7b32680603ea86273abb13e4cadfccf2151" exitCode=60 Nov 03 20:15:18.741107 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.741054 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt" event={"ID":"f1477a68-b92a-4138-b649-fdcd3b17f693","Type":"ContainerDied","Data":"319bb529863c075c1ff15187e656a7b32680603ea86273abb13e4cadfccf2151"} Nov 03 20:15:18.741519 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.741496 5059 scope.go:122] "RemoveContainer" containerID="319bb529863c075c1ff15187e656a7b32680603ea86273abb13e4cadfccf2151" Nov 03 20:15:18.741932 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:18.741906 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt_odh-ai-gateway-infra(f1477a68-b92a-4138-b649-fdcd3b17f693)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt" podUID="f1477a68-b92a-4138-b649-fdcd3b17f693" Nov 03 20:15:18.743378 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.743351 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p" event={"ID":"cb7298dc-a500-4715-a360-bcbdf0c7a298","Type":"ContainerStarted","Data":"08b1ab9b89a17043a6560aebf738c39bab16d69e4a15d643df5200646e8eafea"} Nov 03 20:15:18.747678 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.747633 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt"] Nov 03 20:15:18.756001 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.755168 5059 generic.go:356] "Generic (PLEG): container finished" podID="3a1e6b3a-7887-42c5-b23d-b226cc6d0641" containerID="1aa609000036d90f9226a4aef57f32111d3a5b229ebacc10cc05bb8ab5616130" exitCode=60 Nov 03 20:15:18.756001 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.755233 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc" event={"ID":"3a1e6b3a-7887-42c5-b23d-b226cc6d0641","Type":"ContainerDied","Data":"1aa609000036d90f9226a4aef57f32111d3a5b229ebacc10cc05bb8ab5616130"} Nov 03 20:15:18.756001 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.755931 5059 scope.go:122] "RemoveContainer" containerID="1aa609000036d90f9226a4aef57f32111d3a5b229ebacc10cc05bb8ab5616130" Nov 03 20:15:18.756345 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:18.756321 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc_odh-ai-gateway-infra(3a1e6b3a-7887-42c5-b23d-b226cc6d0641)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc" podUID="3a1e6b3a-7887-42c5-b23d-b226cc6d0641" Nov 03 20:15:18.758929 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.758586 5059 scope.go:122] "RemoveContainer" containerID="344db2809ca423c43d7ba4570f981c986e2be112e1c054ff4d7c69f78a2e4da5" Nov 03 20:15:18.770315 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.770263 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59"] Nov 03 20:15:18.781813 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.781737 5059 scope.go:122] "RemoveContainer" containerID="2a791fe1424f5e3f7f01347833a3d0c452d0d885d27375eac99aaed96a99aba1" Nov 03 20:15:18.797543 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.797482 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc"] Nov 03 20:15:18.800719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.800674 5059 scope.go:122] "RemoveContainer" containerID="d3c79ea4543dd1915df8c46633fc8e2d807a8d10570b2cb61f85bc02cd46eee0" Nov 03 20:15:18.856905 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.854710 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh"] Nov 03 20:15:18.856905 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.855462 5059 scope.go:122] "RemoveContainer" containerID="d9df510a86b0006d44422c8dba441a63833b88a5486cf58a51b6fb2d4cb481c5" Nov 03 20:15:18.865093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.857283 5059 scope.go:122] "RemoveContainer" containerID="62db5cf0bb31f7b7d7282599d004697222a80261497bbef3283f170cd0cd87a3" Nov 03 20:15:18.865093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.857421 5059 scope.go:122] "RemoveContainer" containerID="a3c0cdc5de182ffb64dc9f0b93740ca549919780765f8c433b481f20ebf9f847" Nov 03 20:15:18.865093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.858054 5059 scope.go:122] "RemoveContainer" containerID="6d124e9a3e506b7312a6889ad995a1dba8e0e21475f34dc893d8ef6818c32f43" Nov 03 20:15:18.990826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:18.990796 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc" Nov 03 20:15:19.096091 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.096057 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1774de9e-305c-4b0f-b194-a09995fb25bf-kube-api-access-jmf2s\" (UniqueName: \"kubernetes.io/projected/1774de9e-305c-4b0f-b194-a09995fb25bf-kube-api-access-jmf2s\") pod \"1774de9e-305c-4b0f-b194-a09995fb25bf\" (UID: \"1774de9e-305c-4b0f-b194-a09995fb25bf\") " Nov 03 20:15:19.096365 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.096344 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1774de9e-305c-4b0f-b194-a09995fb25bf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1774de9e-305c-4b0f-b194-a09995fb25bf-maas-api-service-ca\") pod \"1774de9e-305c-4b0f-b194-a09995fb25bf\" (UID: \"1774de9e-305c-4b0f-b194-a09995fb25bf\") " Nov 03 20:15:19.096694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.096654 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1774de9e-305c-4b0f-b194-a09995fb25bf-maas-api-service-ca" pod "1774de9e-305c-4b0f-b194-a09995fb25bf" (UID: "1774de9e-305c-4b0f-b194-a09995fb25bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:19.097223 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.097199 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1774de9e-305c-4b0f-b194-a09995fb25bf-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:19.163835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.163482 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1774de9e-305c-4b0f-b194-a09995fb25bf-kube-api-access-jmf2s" pod "1774de9e-305c-4b0f-b194-a09995fb25bf" (UID: "1774de9e-305c-4b0f-b194-a09995fb25bf"). InnerVolumeSpecName "kube-api-access-jmf2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:19.198865 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.198844 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmf2s\" (UniqueName: \"kubernetes.io/projected/1774de9e-305c-4b0f-b194-a09995fb25bf-kube-api-access-jmf2s\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:19.767213 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.767088 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm"] Nov 03 20:15:19.773599 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.773570 5059 generic.go:356] "Generic (PLEG): container finished" podID="cb7298dc-a500-4715-a360-bcbdf0c7a298" containerID="08b1ab9b89a17043a6560aebf738c39bab16d69e4a15d643df5200646e8eafea" exitCode=60 Nov 03 20:15:19.773739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.773623 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p" event={"ID":"cb7298dc-a500-4715-a360-bcbdf0c7a298","Type":"ContainerDied","Data":"08b1ab9b89a17043a6560aebf738c39bab16d69e4a15d643df5200646e8eafea"} Nov 03 20:15:19.773739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.773722 5059 scope.go:122] "RemoveContainer" containerID="7d11fe0783239bbf8305bb91e55e123815ecb6c3374828d9bfb606cc84fa4df8" Nov 03 20:15:19.774103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.774080 5059 scope.go:122] "RemoveContainer" containerID="08b1ab9b89a17043a6560aebf738c39bab16d69e4a15d643df5200646e8eafea" Nov 03 20:15:19.774492 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:19.774463 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p_odh-ai-gateway-infra(cb7298dc-a500-4715-a360-bcbdf0c7a298)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p" podUID="cb7298dc-a500-4715-a360-bcbdf0c7a298" Nov 03 20:15:19.782724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.782651 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp" event={"ID":"e3873f10-84bb-489c-9de5-b907f375ff3b","Type":"ContainerStarted","Data":"07640b251c8146ecf4614f5bd1de77521ce022831e712315222ee9bc748126c3"} Nov 03 20:15:19.786846 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.785906 5059 generic.go:356] "Generic (PLEG): container finished" podID="8588fce3-f04c-472e-a796-ed5314845eb5" containerID="6723f36ac35723bbae65457d1e42f88ad86a0bee98091ecf327e591f500a2eef" exitCode=60 Nov 03 20:15:19.786846 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.786384 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm" event={"ID":"8588fce3-f04c-472e-a796-ed5314845eb5","Type":"ContainerDied","Data":"6723f36ac35723bbae65457d1e42f88ad86a0bee98091ecf327e591f500a2eef"} Nov 03 20:15:19.791015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.790652 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl" event={"ID":"30d22aa8-3a01-44f1-a59d-b92a70934a7f","Type":"ContainerStarted","Data":"9a8c7d844489e753ccc4601294b547e9204d12806e2c1eb94b01145190422e90"} Nov 03 20:15:19.793063 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.793037 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp" event={"ID":"1be9592b-18d0-4ce7-9322-92898161c6d8","Type":"ContainerStarted","Data":"5d3a5c445733466dbe85af09e1b21c3373edd40bb5f86fac69f87e7839dac943"} Nov 03 20:15:19.798609 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.797935 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq" event={"ID":"8f7ed20a-4c3c-4023-b1de-479a8ea346a8","Type":"ContainerStarted","Data":"2187a2fca756d2acefddda2ab8eccbf1df38ae0598c0c5a4e7f07c6fdc160b95"} Nov 03 20:15:19.800774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.800756 5059 scope.go:122] "RemoveContainer" containerID="b31c48815a514c04876a3d2b2b4c979280141874041103d2903c3f2f0c08868b" Nov 03 20:15:19.804422 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.804391 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc" Nov 03 20:15:19.804605 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.804439 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc" event={"ID":"1774de9e-305c-4b0f-b194-a09995fb25bf","Type":"ContainerDied","Data":"0151b6dcaef645f4d2056752410a26e86127b7d0ac96168f0dbe37b3c4cf0c54"} Nov 03 20:15:19.810700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.809470 5059 generic.go:356] "Generic (PLEG): container finished" podID="32905cf1-9e12-4fc4-8da4-dda3d142a0a0" containerID="d5645d4de166254ec4419354b6daee448fe4f7efa4f8a9d97d48ca979ad1c0d1" exitCode=60 Nov 03 20:15:19.810700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.809536 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r" event={"ID":"32905cf1-9e12-4fc4-8da4-dda3d142a0a0","Type":"ContainerDied","Data":"d5645d4de166254ec4419354b6daee448fe4f7efa4f8a9d97d48ca979ad1c0d1"} Nov 03 20:15:19.810700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.809994 5059 scope.go:122] "RemoveContainer" containerID="d5645d4de166254ec4419354b6daee448fe4f7efa4f8a9d97d48ca979ad1c0d1" Nov 03 20:15:19.810700 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:19.810347 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r_odh-ai-gateway-infra(32905cf1-9e12-4fc4-8da4-dda3d142a0a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r" podUID="32905cf1-9e12-4fc4-8da4-dda3d142a0a0" Nov 03 20:15:19.842610 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.842498 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p"] Nov 03 20:15:19.852069 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.849422 5059 scope.go:122] "RemoveContainer" containerID="1193939b16e300bbf09f9a9ad9499fd4ec5bfd8591c1c663f4106b4750795f90" Nov 03 20:15:19.942016 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.940904 5059 scope.go:122] "RemoveContainer" containerID="46a105c28d2c721125674cf14f8357c899b78479d280b9e47a7775d675fcbc1c" Nov 03 20:15:19.984560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.984528 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r"] Nov 03 20:15:19.987855 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:19.987826 5059 scope.go:122] "RemoveContainer" containerID="a107bc168a164406d6ea61ce0a988930867021ec1584134e9bba99cca682cb88" Nov 03 20:15:20.041895 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.041864 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc"] Nov 03 20:15:20.049852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.049796 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895615-7prxc"] Nov 03 20:15:20.311497 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:20.311463 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3873f10_84bb_489c_9de5_b907f375ff3b.slice/crio-07640b251c8146ecf4614f5bd1de77521ce022831e712315222ee9bc748126c3.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:15:20.315033 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:20.311612 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3873f10_84bb_489c_9de5_b907f375ff3b.slice/crio-conmon-07640b251c8146ecf4614f5bd1de77521ce022831e712315222ee9bc748126c3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d22aa8_3a01_44f1_a59d_b92a70934a7f.slice/crio-conmon-9a8c7d844489e753ccc4601294b547e9204d12806e2c1eb94b01145190422e90.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3873f10_84bb_489c_9de5_b907f375ff3b.slice/crio-07640b251c8146ecf4614f5bd1de77521ce022831e712315222ee9bc748126c3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d22aa8_3a01_44f1_a59d_b92a70934a7f.slice/crio-9a8c7d844489e753ccc4601294b547e9204d12806e2c1eb94b01145190422e90.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:15:20.419339 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.419306 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc" Nov 03 20:15:20.426619 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.426589 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt" Nov 03 20:15:20.432434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.432410 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh" Nov 03 20:15:20.446861 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.446841 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59" Nov 03 20:15:20.457944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.457910 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm" Nov 03 20:15:20.548858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.548769 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8588fce3-f04c-472e-a796-ed5314845eb5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8588fce3-f04c-472e-a796-ed5314845eb5-maas-api-service-ca\") pod \"8588fce3-f04c-472e-a796-ed5314845eb5\" (UID: \"8588fce3-f04c-472e-a796-ed5314845eb5\") " Nov 03 20:15:20.549017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.548910 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8588fce3-f04c-472e-a796-ed5314845eb5-kube-api-access-c7cvn\" (UniqueName: \"kubernetes.io/projected/8588fce3-f04c-472e-a796-ed5314845eb5-kube-api-access-c7cvn\") pod \"8588fce3-f04c-472e-a796-ed5314845eb5\" (UID: \"8588fce3-f04c-472e-a796-ed5314845eb5\") " Nov 03 20:15:20.549017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.548984 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1477a68-b92a-4138-b649-fdcd3b17f693-kube-api-access-ql7sk\" (UniqueName: \"kubernetes.io/projected/f1477a68-b92a-4138-b649-fdcd3b17f693-kube-api-access-ql7sk\") pod \"f1477a68-b92a-4138-b649-fdcd3b17f693\" (UID: \"f1477a68-b92a-4138-b649-fdcd3b17f693\") " Nov 03 20:15:20.549166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.549050 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a1e6b3a-7887-42c5-b23d-b226cc6d0641-kube-api-access-xkd6f\" (UniqueName: \"kubernetes.io/projected/3a1e6b3a-7887-42c5-b23d-b226cc6d0641-kube-api-access-xkd6f\") pod \"3a1e6b3a-7887-42c5-b23d-b226cc6d0641\" (UID: \"3a1e6b3a-7887-42c5-b23d-b226cc6d0641\") " Nov 03 20:15:20.549728 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.549411 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b18a5b0-0fc4-4dc8-88b7-affec66830d8-kube-api-access-tlmgg\" (UniqueName: \"kubernetes.io/projected/0b18a5b0-0fc4-4dc8-88b7-affec66830d8-kube-api-access-tlmgg\") pod \"0b18a5b0-0fc4-4dc8-88b7-affec66830d8\" (UID: \"0b18a5b0-0fc4-4dc8-88b7-affec66830d8\") " Nov 03 20:15:20.550334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.549877 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffb6bb03-2f3c-43f0-871a-1140b53f0fb9-kube-api-access-5h9c2\" (UniqueName: \"kubernetes.io/projected/ffb6bb03-2f3c-43f0-871a-1140b53f0fb9-kube-api-access-5h9c2\") pod \"ffb6bb03-2f3c-43f0-871a-1140b53f0fb9\" (UID: \"ffb6bb03-2f3c-43f0-871a-1140b53f0fb9\") " Nov 03 20:15:20.550334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.549947 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1477a68-b92a-4138-b649-fdcd3b17f693-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1477a68-b92a-4138-b649-fdcd3b17f693-maas-api-service-ca\") pod \"f1477a68-b92a-4138-b649-fdcd3b17f693\" (UID: \"f1477a68-b92a-4138-b649-fdcd3b17f693\") " Nov 03 20:15:20.550334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.550015 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b18a5b0-0fc4-4dc8-88b7-affec66830d8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b18a5b0-0fc4-4dc8-88b7-affec66830d8-maas-api-service-ca\") pod \"0b18a5b0-0fc4-4dc8-88b7-affec66830d8\" (UID: \"0b18a5b0-0fc4-4dc8-88b7-affec66830d8\") " Nov 03 20:15:20.550334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.549139 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8588fce3-f04c-472e-a796-ed5314845eb5-maas-api-service-ca" pod "8588fce3-f04c-472e-a796-ed5314845eb5" (UID: "8588fce3-f04c-472e-a796-ed5314845eb5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:20.550334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.550126 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a1e6b3a-7887-42c5-b23d-b226cc6d0641-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1e6b3a-7887-42c5-b23d-b226cc6d0641-maas-api-service-ca\") pod \"3a1e6b3a-7887-42c5-b23d-b226cc6d0641\" (UID: \"3a1e6b3a-7887-42c5-b23d-b226cc6d0641\") " Nov 03 20:15:20.550334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.550180 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffb6bb03-2f3c-43f0-871a-1140b53f0fb9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffb6bb03-2f3c-43f0-871a-1140b53f0fb9-maas-api-service-ca\") pod \"ffb6bb03-2f3c-43f0-871a-1140b53f0fb9\" (UID: \"ffb6bb03-2f3c-43f0-871a-1140b53f0fb9\") " Nov 03 20:15:20.551284 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.550488 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1477a68-b92a-4138-b649-fdcd3b17f693-maas-api-service-ca" pod "f1477a68-b92a-4138-b649-fdcd3b17f693" (UID: "f1477a68-b92a-4138-b649-fdcd3b17f693"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:20.551284 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.550762 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b18a5b0-0fc4-4dc8-88b7-affec66830d8-maas-api-service-ca" pod "0b18a5b0-0fc4-4dc8-88b7-affec66830d8" (UID: "0b18a5b0-0fc4-4dc8-88b7-affec66830d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:20.551284 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.551088 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb6bb03-2f3c-43f0-871a-1140b53f0fb9-maas-api-service-ca" pod "ffb6bb03-2f3c-43f0-871a-1140b53f0fb9" (UID: "ffb6bb03-2f3c-43f0-871a-1140b53f0fb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:20.551772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.551731 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a1e6b3a-7887-42c5-b23d-b226cc6d0641-maas-api-service-ca" pod "3a1e6b3a-7887-42c5-b23d-b226cc6d0641" (UID: "3a1e6b3a-7887-42c5-b23d-b226cc6d0641"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:20.551947 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.551926 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8588fce3-f04c-472e-a796-ed5314845eb5-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:20.552034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.551951 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1477a68-b92a-4138-b649-fdcd3b17f693-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:20.552034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.551967 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b18a5b0-0fc4-4dc8-88b7-affec66830d8-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:20.552034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.551981 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1e6b3a-7887-42c5-b23d-b226cc6d0641-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:20.552034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.551996 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffb6bb03-2f3c-43f0-871a-1140b53f0fb9-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:20.552244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.552088 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1477a68-b92a-4138-b649-fdcd3b17f693-kube-api-access-ql7sk" pod "f1477a68-b92a-4138-b649-fdcd3b17f693" (UID: "f1477a68-b92a-4138-b649-fdcd3b17f693"). InnerVolumeSpecName "kube-api-access-ql7sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:20.552958 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.552929 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb6bb03-2f3c-43f0-871a-1140b53f0fb9-kube-api-access-5h9c2" pod "ffb6bb03-2f3c-43f0-871a-1140b53f0fb9" (UID: "ffb6bb03-2f3c-43f0-871a-1140b53f0fb9"). InnerVolumeSpecName "kube-api-access-5h9c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:20.553128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.553006 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1e6b3a-7887-42c5-b23d-b226cc6d0641-kube-api-access-xkd6f" pod "3a1e6b3a-7887-42c5-b23d-b226cc6d0641" (UID: "3a1e6b3a-7887-42c5-b23d-b226cc6d0641"). InnerVolumeSpecName "kube-api-access-xkd6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:20.554025 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.553997 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8588fce3-f04c-472e-a796-ed5314845eb5-kube-api-access-c7cvn" pod "8588fce3-f04c-472e-a796-ed5314845eb5" (UID: "8588fce3-f04c-472e-a796-ed5314845eb5"). InnerVolumeSpecName "kube-api-access-c7cvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:20.554277 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.554251 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b18a5b0-0fc4-4dc8-88b7-affec66830d8-kube-api-access-tlmgg" pod "0b18a5b0-0fc4-4dc8-88b7-affec66830d8" (UID: "0b18a5b0-0fc4-4dc8-88b7-affec66830d8"). InnerVolumeSpecName "kube-api-access-tlmgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:20.656525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.656482 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7cvn\" (UniqueName: \"kubernetes.io/projected/8588fce3-f04c-472e-a796-ed5314845eb5-kube-api-access-c7cvn\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:20.656525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.656521 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql7sk\" (UniqueName: \"kubernetes.io/projected/f1477a68-b92a-4138-b649-fdcd3b17f693-kube-api-access-ql7sk\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:20.656809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.656535 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkd6f\" (UniqueName: \"kubernetes.io/projected/3a1e6b3a-7887-42c5-b23d-b226cc6d0641-kube-api-access-xkd6f\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:20.656809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.656550 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlmgg\" (UniqueName: \"kubernetes.io/projected/0b18a5b0-0fc4-4dc8-88b7-affec66830d8-kube-api-access-tlmgg\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:20.656809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.656563 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h9c2\" (UniqueName: \"kubernetes.io/projected/ffb6bb03-2f3c-43f0-871a-1140b53f0fb9-kube-api-access-5h9c2\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:20.855697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.850097 5059 scope.go:122] "RemoveContainer" containerID="74a0a6de594c8105435cf5aa3513b6e272c53e1eed665d6c4ec2967c7926770f" Nov 03 20:15:20.876701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.875468 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp"] Nov 03 20:15:20.898705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.898038 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm" Nov 03 20:15:20.898705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.898117 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm" event={"ID":"8588fce3-f04c-472e-a796-ed5314845eb5","Type":"ContainerDied","Data":"83337441d5fc8aa3f9f21c6f388bd3717e0a48766933d358c20c3b06f2c975f0"} Nov 03 20:15:20.898705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.898218 5059 scope.go:122] "RemoveContainer" containerID="6723f36ac35723bbae65457d1e42f88ad86a0bee98091ecf327e591f500a2eef" Nov 03 20:15:20.905849 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.905761 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl"] Nov 03 20:15:20.909771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.909744 5059 generic.go:356] "Generic (PLEG): container finished" podID="30d22aa8-3a01-44f1-a59d-b92a70934a7f" containerID="9a8c7d844489e753ccc4601294b547e9204d12806e2c1eb94b01145190422e90" exitCode=60 Nov 03 20:15:20.909921 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.909881 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl" event={"ID":"30d22aa8-3a01-44f1-a59d-b92a70934a7f","Type":"ContainerDied","Data":"9a8c7d844489e753ccc4601294b547e9204d12806e2c1eb94b01145190422e90"} Nov 03 20:15:20.920593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.919901 5059 generic.go:356] "Generic (PLEG): container finished" podID="1be9592b-18d0-4ce7-9322-92898161c6d8" containerID="5d3a5c445733466dbe85af09e1b21c3373edd40bb5f86fac69f87e7839dac943" exitCode=60 Nov 03 20:15:20.920593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.919993 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp" event={"ID":"1be9592b-18d0-4ce7-9322-92898161c6d8","Type":"ContainerDied","Data":"5d3a5c445733466dbe85af09e1b21c3373edd40bb5f86fac69f87e7839dac943"} Nov 03 20:15:20.920593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.920460 5059 scope.go:122] "RemoveContainer" containerID="5d3a5c445733466dbe85af09e1b21c3373edd40bb5f86fac69f87e7839dac943" Nov 03 20:15:20.921772 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:20.921425 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp_odh-ai-gateway-infra(1be9592b-18d0-4ce7-9322-92898161c6d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp" podUID="1be9592b-18d0-4ce7-9322-92898161c6d8" Nov 03 20:15:20.925137 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.924520 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8" event={"ID":"0e0bbcca-04df-4474-91fc-c5c5e6521732","Type":"ContainerStarted","Data":"ea9d9c0f68fb82425f437fb627d0937c5c4ce735145569ae74e7e2858fb221cf"} Nov 03 20:15:20.925938 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.925710 5059 scope.go:122] "RemoveContainer" containerID="d9df510a86b0006d44422c8dba441a63833b88a5486cf58a51b6fb2d4cb481c5" Nov 03 20:15:20.927704 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.927589 5059 generic.go:356] "Generic (PLEG): container finished" podID="8f7ed20a-4c3c-4023-b1de-479a8ea346a8" containerID="2187a2fca756d2acefddda2ab8eccbf1df38ae0598c0c5a4e7f07c6fdc160b95" exitCode=60 Nov 03 20:15:20.927792 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.927730 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq" event={"ID":"8f7ed20a-4c3c-4023-b1de-479a8ea346a8","Type":"ContainerDied","Data":"2187a2fca756d2acefddda2ab8eccbf1df38ae0598c0c5a4e7f07c6fdc160b95"} Nov 03 20:15:20.928349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.928191 5059 scope.go:122] "RemoveContainer" containerID="2187a2fca756d2acefddda2ab8eccbf1df38ae0598c0c5a4e7f07c6fdc160b95" Nov 03 20:15:20.928775 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:20.928571 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq_odh-ai-gateway-infra(8f7ed20a-4c3c-4023-b1de-479a8ea346a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq" podUID="8f7ed20a-4c3c-4023-b1de-479a8ea346a8" Nov 03 20:15:20.929356 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.929289 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh" Nov 03 20:15:20.929831 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.929739 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh" event={"ID":"ffb6bb03-2f3c-43f0-871a-1140b53f0fb9","Type":"ContainerDied","Data":"ad52f985b75b77f67ea55389f8cb90df283d76b36b98979e8212d7270e641313"} Nov 03 20:15:20.933511 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.933476 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59" event={"ID":"0b18a5b0-0fc4-4dc8-88b7-affec66830d8","Type":"ContainerDied","Data":"ddafb52d39e712193525e5596a1ee622603944f333153a926b3dae4e067b4fad"} Nov 03 20:15:20.933611 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.933573 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59" Nov 03 20:15:20.941991 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.941974 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt" event={"ID":"f1477a68-b92a-4138-b649-fdcd3b17f693","Type":"ContainerDied","Data":"f8b9a57d221b8140062f07fbdd5376564c8e3ed3a7029541d50b221d9d74e72e"} Nov 03 20:15:20.942134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.942123 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt" Nov 03 20:15:20.956514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.955871 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc" event={"ID":"3a1e6b3a-7887-42c5-b23d-b226cc6d0641","Type":"ContainerDied","Data":"e7047c3a50e0e5af63c3aad56c3df9eed9fdbb01524dbc1e1aee5d0a30c1949b"} Nov 03 20:15:20.956514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.955967 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc" Nov 03 20:15:20.971940 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.970058 5059 scope.go:122] "RemoveContainer" containerID="6d124e9a3e506b7312a6889ad995a1dba8e0e21475f34dc893d8ef6818c32f43" Nov 03 20:15:20.987741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.987707 5059 generic.go:356] "Generic (PLEG): container finished" podID="e3873f10-84bb-489c-9de5-b907f375ff3b" containerID="07640b251c8146ecf4614f5bd1de77521ce022831e712315222ee9bc748126c3" exitCode=60 Nov 03 20:15:20.987956 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.987935 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp" event={"ID":"e3873f10-84bb-489c-9de5-b907f375ff3b","Type":"ContainerDied","Data":"07640b251c8146ecf4614f5bd1de77521ce022831e712315222ee9bc748126c3"} Nov 03 20:15:20.989730 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:20.988947 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm"] Nov 03 20:15:21.014112 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.013844 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4fb1d0-29895615-9bnkm"] Nov 03 20:15:21.086481 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.081793 5059 scope.go:122] "RemoveContainer" containerID="62db5cf0bb31f7b7d7282599d004697222a80261497bbef3283f170cd0cd87a3" Nov 03 20:15:21.086697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.086652 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp"] Nov 03 20:15:21.110524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.110493 5059 scope.go:122] "RemoveContainer" containerID="3c41a492dd335502a3885b51fd2f9a83ea0ce4c2c1c501711e46e143395d9d17" Nov 03 20:15:21.120838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.119147 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh"] Nov 03 20:15:21.124831 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.124810 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-ghpwh"] Nov 03 20:15:21.138903 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.138870 5059 scope.go:122] "RemoveContainer" containerID="613626463cb6ef0eb3b4f7d116501779c87e91578038a814c17c378d42164486" Nov 03 20:15:21.150567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.149951 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq"] Nov 03 20:15:21.154007 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.153985 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59"] Nov 03 20:15:21.161075 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.161057 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895615-9nz59"] Nov 03 20:15:21.206500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.206440 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc"] Nov 03 20:15:21.211735 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.209855 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl" Nov 03 20:15:21.214907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.213049 5059 scope.go:122] "RemoveContainer" containerID="319bb529863c075c1ff15187e656a7b32680603ea86273abb13e4cadfccf2151" Nov 03 20:15:21.219915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.219395 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895615-4wftc"] Nov 03 20:15:21.239638 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.239620 5059 scope.go:122] "RemoveContainer" containerID="1aa609000036d90f9226a4aef57f32111d3a5b229ebacc10cc05bb8ab5616130" Nov 03 20:15:21.241623 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.241609 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt"] Nov 03 20:15:21.250711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.249947 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-43c674-29895615-8x5kt"] Nov 03 20:15:21.256904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.256881 5059 scope.go:122] "RemoveContainer" containerID="a3c0cdc5de182ffb64dc9f0b93740ca549919780765f8c433b481f20ebf9f847" Nov 03 20:15:21.293656 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.293629 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d22aa8-3a01-44f1-a59d-b92a70934a7f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30d22aa8-3a01-44f1-a59d-b92a70934a7f-maas-api-service-ca\") pod \"30d22aa8-3a01-44f1-a59d-b92a70934a7f\" (UID: \"30d22aa8-3a01-44f1-a59d-b92a70934a7f\") " Nov 03 20:15:21.293842 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.293810 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d22aa8-3a01-44f1-a59d-b92a70934a7f-kube-api-access-c9tmz\" (UniqueName: \"kubernetes.io/projected/30d22aa8-3a01-44f1-a59d-b92a70934a7f-kube-api-access-c9tmz\") pod \"30d22aa8-3a01-44f1-a59d-b92a70934a7f\" (UID: \"30d22aa8-3a01-44f1-a59d-b92a70934a7f\") " Nov 03 20:15:21.294298 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.294276 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d22aa8-3a01-44f1-a59d-b92a70934a7f-maas-api-service-ca" pod "30d22aa8-3a01-44f1-a59d-b92a70934a7f" (UID: "30d22aa8-3a01-44f1-a59d-b92a70934a7f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:21.294733 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.294711 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30d22aa8-3a01-44f1-a59d-b92a70934a7f-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:21.359540 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.358235 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d22aa8-3a01-44f1-a59d-b92a70934a7f-kube-api-access-c9tmz" pod "30d22aa8-3a01-44f1-a59d-b92a70934a7f" (UID: "30d22aa8-3a01-44f1-a59d-b92a70934a7f"). InnerVolumeSpecName "kube-api-access-c9tmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:21.396717 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.396656 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9tmz\" (UniqueName: \"kubernetes.io/projected/30d22aa8-3a01-44f1-a59d-b92a70934a7f-kube-api-access-c9tmz\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:21.417055 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.417029 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r" Nov 03 20:15:21.429595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.429569 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p" Nov 03 20:15:21.451313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.451283 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp" Nov 03 20:15:21.496459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.496423 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn"] Nov 03 20:15:21.497824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.497798 5059 cpu_manager.go:414] "containerMap: removing container" podUID="8588fce3-f04c-472e-a796-ed5314845eb5" containerName="cleanup" Nov 03 20:15:21.497914 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.497826 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8588fce3-f04c-472e-a796-ed5314845eb5" containerName="cleanup" Nov 03 20:15:21.497914 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.497853 5059 cpu_manager.go:414] "containerMap: removing container" podUID="32905cf1-9e12-4fc4-8da4-dda3d142a0a0" containerName="cleanup" Nov 03 20:15:21.497914 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.497862 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32905cf1-9e12-4fc4-8da4-dda3d142a0a0" containerName="cleanup" Nov 03 20:15:21.497914 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.497874 5059 cpu_manager.go:414] "containerMap: removing container" podUID="cb7298dc-a500-4715-a360-bcbdf0c7a298" containerName="cleanup" Nov 03 20:15:21.497914 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.497882 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb7298dc-a500-4715-a360-bcbdf0c7a298" containerName="cleanup" Nov 03 20:15:21.497914 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.497896 5059 cpu_manager.go:414] "containerMap: removing container" podUID="32905cf1-9e12-4fc4-8da4-dda3d142a0a0" containerName="cleanup" Nov 03 20:15:21.497914 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.497903 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32905cf1-9e12-4fc4-8da4-dda3d142a0a0" containerName="cleanup" Nov 03 20:15:21.498122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.497926 5059 cpu_manager.go:414] "containerMap: removing container" podUID="3a1e6b3a-7887-42c5-b23d-b226cc6d0641" containerName="cleanup" Nov 03 20:15:21.498122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.497932 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1e6b3a-7887-42c5-b23d-b226cc6d0641" containerName="cleanup" Nov 03 20:15:21.498122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.497946 5059 cpu_manager.go:414] "containerMap: removing container" podUID="f1477a68-b92a-4138-b649-fdcd3b17f693" containerName="cleanup" Nov 03 20:15:21.498122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.497953 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1477a68-b92a-4138-b649-fdcd3b17f693" containerName="cleanup" Nov 03 20:15:21.498122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.497961 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1774de9e-305c-4b0f-b194-a09995fb25bf" containerName="cleanup" Nov 03 20:15:21.498122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.497968 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1774de9e-305c-4b0f-b194-a09995fb25bf" containerName="cleanup" Nov 03 20:15:21.498122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.497976 5059 cpu_manager.go:414] "containerMap: removing container" podUID="30d22aa8-3a01-44f1-a59d-b92a70934a7f" containerName="cleanup" Nov 03 20:15:21.498122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.497983 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d22aa8-3a01-44f1-a59d-b92a70934a7f" containerName="cleanup" Nov 03 20:15:21.498122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498002 5059 cpu_manager.go:414] "containerMap: removing container" podUID="30d22aa8-3a01-44f1-a59d-b92a70934a7f" containerName="cleanup" Nov 03 20:15:21.498122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498011 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d22aa8-3a01-44f1-a59d-b92a70934a7f" containerName="cleanup" Nov 03 20:15:21.498122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498023 5059 cpu_manager.go:414] "containerMap: removing container" podUID="ffb6bb03-2f3c-43f0-871a-1140b53f0fb9" containerName="cleanup" Nov 03 20:15:21.498122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498030 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb6bb03-2f3c-43f0-871a-1140b53f0fb9" containerName="cleanup" Nov 03 20:15:21.498122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498043 5059 cpu_manager.go:414] "containerMap: removing container" podUID="e3873f10-84bb-489c-9de5-b907f375ff3b" containerName="cleanup" Nov 03 20:15:21.498122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498051 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3873f10-84bb-489c-9de5-b907f375ff3b" containerName="cleanup" Nov 03 20:15:21.498122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498071 5059 cpu_manager.go:414] "containerMap: removing container" podUID="3a1e6b3a-7887-42c5-b23d-b226cc6d0641" containerName="cleanup" Nov 03 20:15:21.498122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498077 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1e6b3a-7887-42c5-b23d-b226cc6d0641" containerName="cleanup" Nov 03 20:15:21.498122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498100 5059 cpu_manager.go:414] "containerMap: removing container" podUID="8588fce3-f04c-472e-a796-ed5314845eb5" containerName="cleanup" Nov 03 20:15:21.498122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498108 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8588fce3-f04c-472e-a796-ed5314845eb5" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498141 5059 cpu_manager.go:414] "containerMap: removing container" podUID="3a1e6b3a-7887-42c5-b23d-b226cc6d0641" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498149 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1e6b3a-7887-42c5-b23d-b226cc6d0641" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498172 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1774de9e-305c-4b0f-b194-a09995fb25bf" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498179 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1774de9e-305c-4b0f-b194-a09995fb25bf" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498201 5059 cpu_manager.go:414] "containerMap: removing container" podUID="f1477a68-b92a-4138-b649-fdcd3b17f693" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498208 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1477a68-b92a-4138-b649-fdcd3b17f693" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498219 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1774de9e-305c-4b0f-b194-a09995fb25bf" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498225 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1774de9e-305c-4b0f-b194-a09995fb25bf" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498235 5059 cpu_manager.go:414] "containerMap: removing container" podUID="ffb6bb03-2f3c-43f0-871a-1140b53f0fb9" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498242 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb6bb03-2f3c-43f0-871a-1140b53f0fb9" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498255 5059 cpu_manager.go:414] "containerMap: removing container" podUID="0b18a5b0-0fc4-4dc8-88b7-affec66830d8" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498263 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b18a5b0-0fc4-4dc8-88b7-affec66830d8" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498284 5059 cpu_manager.go:414] "containerMap: removing container" podUID="e3873f10-84bb-489c-9de5-b907f375ff3b" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498292 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3873f10-84bb-489c-9de5-b907f375ff3b" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498318 5059 cpu_manager.go:414] "containerMap: removing container" podUID="cb7298dc-a500-4715-a360-bcbdf0c7a298" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498325 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb7298dc-a500-4715-a360-bcbdf0c7a298" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498335 5059 cpu_manager.go:414] "containerMap: removing container" podUID="0b18a5b0-0fc4-4dc8-88b7-affec66830d8" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498342 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b18a5b0-0fc4-4dc8-88b7-affec66830d8" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498356 5059 cpu_manager.go:414] "containerMap: removing container" podUID="0b18a5b0-0fc4-4dc8-88b7-affec66830d8" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498362 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b18a5b0-0fc4-4dc8-88b7-affec66830d8" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498566 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1774de9e-305c-4b0f-b194-a09995fb25bf" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498582 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1477a68-b92a-4138-b649-fdcd3b17f693" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498593 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1774de9e-305c-4b0f-b194-a09995fb25bf" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498601 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb6bb03-2f3c-43f0-871a-1140b53f0fb9" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498610 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b18a5b0-0fc4-4dc8-88b7-affec66830d8" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498621 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3873f10-84bb-489c-9de5-b907f375ff3b" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498632 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb7298dc-a500-4715-a360-bcbdf0c7a298" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498640 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b18a5b0-0fc4-4dc8-88b7-affec66830d8" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498650 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8588fce3-f04c-472e-a796-ed5314845eb5" containerName="cleanup" Nov 03 20:15:21.498670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498680 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="32905cf1-9e12-4fc4-8da4-dda3d142a0a0" containerName="cleanup" Nov 03 20:15:21.499852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498689 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb7298dc-a500-4715-a360-bcbdf0c7a298" containerName="cleanup" Nov 03 20:15:21.499852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498698 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="32905cf1-9e12-4fc4-8da4-dda3d142a0a0" containerName="cleanup" Nov 03 20:15:21.499852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498705 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1e6b3a-7887-42c5-b23d-b226cc6d0641" containerName="cleanup" Nov 03 20:15:21.499852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498714 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1477a68-b92a-4138-b649-fdcd3b17f693" containerName="cleanup" Nov 03 20:15:21.499852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498723 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d22aa8-3a01-44f1-a59d-b92a70934a7f" containerName="cleanup" Nov 03 20:15:21.499852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498734 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d22aa8-3a01-44f1-a59d-b92a70934a7f" containerName="cleanup" Nov 03 20:15:21.499852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498742 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb6bb03-2f3c-43f0-871a-1140b53f0fb9" containerName="cleanup" Nov 03 20:15:21.499852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498751 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3873f10-84bb-489c-9de5-b907f375ff3b" containerName="cleanup" Nov 03 20:15:21.499852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498763 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1e6b3a-7887-42c5-b23d-b226cc6d0641" containerName="cleanup" Nov 03 20:15:21.499852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498775 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8588fce3-f04c-472e-a796-ed5314845eb5" containerName="cleanup" Nov 03 20:15:21.499852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498790 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8588fce3-f04c-472e-a796-ed5314845eb5" containerName="cleanup" Nov 03 20:15:21.499852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.498801 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="32905cf1-9e12-4fc4-8da4-dda3d142a0a0" containerName="cleanup" Nov 03 20:15:21.503634 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.503606 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.516402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.515029 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn"] Nov 03 20:15:21.608341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.608301 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3873f10-84bb-489c-9de5-b907f375ff3b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3873f10-84bb-489c-9de5-b907f375ff3b-maas-api-service-ca\") pod \"e3873f10-84bb-489c-9de5-b907f375ff3b\" (UID: \"e3873f10-84bb-489c-9de5-b907f375ff3b\") " Nov 03 20:15:21.659954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.631912 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3873f10-84bb-489c-9de5-b907f375ff3b-maas-api-service-ca" pod "e3873f10-84bb-489c-9de5-b907f375ff3b" (UID: "e3873f10-84bb-489c-9de5-b907f375ff3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:21.660204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.660186 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32905cf1-9e12-4fc4-8da4-dda3d142a0a0-kube-api-access-hh8fh\" (UniqueName: \"kubernetes.io/projected/32905cf1-9e12-4fc4-8da4-dda3d142a0a0-kube-api-access-hh8fh\") pod \"32905cf1-9e12-4fc4-8da4-dda3d142a0a0\" (UID: \"32905cf1-9e12-4fc4-8da4-dda3d142a0a0\") " Nov 03 20:15:21.660384 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.660367 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32905cf1-9e12-4fc4-8da4-dda3d142a0a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32905cf1-9e12-4fc4-8da4-dda3d142a0a0-maas-api-service-ca\") pod \"32905cf1-9e12-4fc4-8da4-dda3d142a0a0\" (UID: \"32905cf1-9e12-4fc4-8da4-dda3d142a0a0\") " Nov 03 20:15:21.661319 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.660793 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32905cf1-9e12-4fc4-8da4-dda3d142a0a0-maas-api-service-ca" pod "32905cf1-9e12-4fc4-8da4-dda3d142a0a0" (UID: "32905cf1-9e12-4fc4-8da4-dda3d142a0a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:21.661319 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.660962 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3873f10-84bb-489c-9de5-b907f375ff3b-kube-api-access-lpn6l\" (UniqueName: \"kubernetes.io/projected/e3873f10-84bb-489c-9de5-b907f375ff3b-kube-api-access-lpn6l\") pod \"e3873f10-84bb-489c-9de5-b907f375ff3b\" (UID: \"e3873f10-84bb-489c-9de5-b907f375ff3b\") " Nov 03 20:15:21.661319 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.661056 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb7298dc-a500-4715-a360-bcbdf0c7a298-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb7298dc-a500-4715-a360-bcbdf0c7a298-maas-api-service-ca\") pod \"cb7298dc-a500-4715-a360-bcbdf0c7a298\" (UID: \"cb7298dc-a500-4715-a360-bcbdf0c7a298\") " Nov 03 20:15:21.661579 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.661399 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb7298dc-a500-4715-a360-bcbdf0c7a298-kube-api-access-bzjjb\" (UniqueName: \"kubernetes.io/projected/cb7298dc-a500-4715-a360-bcbdf0c7a298-kube-api-access-bzjjb\") pod \"cb7298dc-a500-4715-a360-bcbdf0c7a298\" (UID: \"cb7298dc-a500-4715-a360-bcbdf0c7a298\") " Nov 03 20:15:21.663008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.661780 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.663008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.662197 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb7298dc-a500-4715-a360-bcbdf0c7a298-maas-api-service-ca" pod "cb7298dc-a500-4715-a360-bcbdf0c7a298" (UID: "cb7298dc-a500-4715-a360-bcbdf0c7a298"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:21.663008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.662180 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.663008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.662280 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.663008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.662410 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.663008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.662572 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8153e018-7f9a-4645-b102-06ff3913f682-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.663008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.662755 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.663008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.662784 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8153e018-7f9a-4645-b102-06ff3913f682-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.663008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.662809 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8153e018-7f9a-4645-b102-06ff3913f682-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.663008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.662891 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55dfj\" (UniqueName: \"kubernetes.io/projected/8153e018-7f9a-4645-b102-06ff3913f682-kube-api-access-55dfj\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.663542 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.663107 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32905cf1-9e12-4fc4-8da4-dda3d142a0a0-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:21.663542 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.663126 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb7298dc-a500-4715-a360-bcbdf0c7a298-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:21.663542 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.663140 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3873f10-84bb-489c-9de5-b907f375ff3b-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:21.666064 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.666024 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb7298dc-a500-4715-a360-bcbdf0c7a298-kube-api-access-bzjjb" pod "cb7298dc-a500-4715-a360-bcbdf0c7a298" (UID: "cb7298dc-a500-4715-a360-bcbdf0c7a298"). InnerVolumeSpecName "kube-api-access-bzjjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:21.671102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.671074 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32905cf1-9e12-4fc4-8da4-dda3d142a0a0-kube-api-access-hh8fh" pod "32905cf1-9e12-4fc4-8da4-dda3d142a0a0" (UID: "32905cf1-9e12-4fc4-8da4-dda3d142a0a0"). InnerVolumeSpecName "kube-api-access-hh8fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:21.671218 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.671123 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3873f10-84bb-489c-9de5-b907f375ff3b-kube-api-access-lpn6l" pod "e3873f10-84bb-489c-9de5-b907f375ff3b" (UID: "e3873f10-84bb-489c-9de5-b907f375ff3b"). InnerVolumeSpecName "kube-api-access-lpn6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:21.764090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.764056 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.764225 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.764096 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8153e018-7f9a-4645-b102-06ff3913f682-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.764225 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.764113 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8153e018-7f9a-4645-b102-06ff3913f682-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.764225 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.764170 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55dfj\" (UniqueName: \"kubernetes.io/projected/8153e018-7f9a-4645-b102-06ff3913f682-kube-api-access-55dfj\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.764225 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.764207 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.764459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.764253 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.764459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.764292 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.764459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.764416 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.764682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.764588 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8153e018-7f9a-4645-b102-06ff3913f682-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.764682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.764606 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.764806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.764708 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.764806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.764745 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.764933 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.764847 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzjjb\" (UniqueName: \"kubernetes.io/projected/cb7298dc-a500-4715-a360-bcbdf0c7a298-kube-api-access-bzjjb\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:21.764933 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.764861 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh8fh\" (UniqueName: \"kubernetes.io/projected/32905cf1-9e12-4fc4-8da4-dda3d142a0a0-kube-api-access-hh8fh\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:21.764933 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.764871 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpn6l\" (UniqueName: \"kubernetes.io/projected/e3873f10-84bb-489c-9de5-b907f375ff3b-kube-api-access-lpn6l\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:21.765068 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.764952 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.765359 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.765334 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8153e018-7f9a-4645-b102-06ff3913f682-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.766411 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.766388 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.766768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.766751 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8153e018-7f9a-4645-b102-06ff3913f682-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.776941 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.776918 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8153e018-7f9a-4645-b102-06ff3913f682-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.777047 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.777029 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55dfj\" (UniqueName: \"kubernetes.io/projected/8153e018-7f9a-4645-b102-06ff3913f682-kube-api-access-55dfj\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.832957 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.832937 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:21.850496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.850472 5059 scope.go:122] "RemoveContainer" containerID="b361175f83d8fc6249aae9c43aa927cb6ee5ab1c7cc52bcebe89a59be47783a9" Nov 03 20:15:21.850611 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.850593 5059 scope.go:122] "RemoveContainer" containerID="a2d590c159c646b7bc90c66c7292004a5c54a7b78cad668d2cd0be92bd7864fd" Nov 03 20:15:21.850785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.850762 5059 scope.go:122] "RemoveContainer" containerID="2e0cf38a22a0523c5647f89e6a438290f8aaf64cfc728ffd56d51ef3ee205c02" Nov 03 20:15:21.870826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.870800 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b18a5b0-0fc4-4dc8-88b7-affec66830d8" path="/var/lib/kubelet/pods/0b18a5b0-0fc4-4dc8-88b7-affec66830d8/volumes" Nov 03 20:15:21.871429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.871406 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1774de9e-305c-4b0f-b194-a09995fb25bf" path="/var/lib/kubelet/pods/1774de9e-305c-4b0f-b194-a09995fb25bf/volumes" Nov 03 20:15:21.872211 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.872184 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a1e6b3a-7887-42c5-b23d-b226cc6d0641" path="/var/lib/kubelet/pods/3a1e6b3a-7887-42c5-b23d-b226cc6d0641/volumes" Nov 03 20:15:21.873500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.873472 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8588fce3-f04c-472e-a796-ed5314845eb5" path="/var/lib/kubelet/pods/8588fce3-f04c-472e-a796-ed5314845eb5/volumes" Nov 03 20:15:21.874190 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.874164 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1477a68-b92a-4138-b649-fdcd3b17f693" path="/var/lib/kubelet/pods/f1477a68-b92a-4138-b649-fdcd3b17f693/volumes" Nov 03 20:15:21.874795 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:21.874775 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffb6bb03-2f3c-43f0-871a-1140b53f0fb9" path="/var/lib/kubelet/pods/ffb6bb03-2f3c-43f0-871a-1140b53f0fb9/volumes" Nov 03 20:15:22.011488 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.010748 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r" Nov 03 20:15:22.011488 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.010722 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r" event={"ID":"32905cf1-9e12-4fc4-8da4-dda3d142a0a0","Type":"ContainerDied","Data":"7c8856108be7e89fce01d0e65487bc2f6750e128e01dbeedcd21f05f4e887a76"} Nov 03 20:15:22.011488 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.010968 5059 scope.go:122] "RemoveContainer" containerID="d5645d4de166254ec4419354b6daee448fe4f7efa4f8a9d97d48ca979ad1c0d1" Nov 03 20:15:22.013582 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.013555 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p" event={"ID":"cb7298dc-a500-4715-a360-bcbdf0c7a298","Type":"ContainerDied","Data":"d87d71a5a5c9be19d71b4e2b3ad4ab28202b3b84f458132cae3b6777927887db"} Nov 03 20:15:22.013745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.013648 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p" Nov 03 20:15:22.017395 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.017368 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp" event={"ID":"e3873f10-84bb-489c-9de5-b907f375ff3b","Type":"ContainerDied","Data":"65538fb8d9a44370fd7c667232b6fa9352b996a2b8afb7322c8ac62772b76dcc"} Nov 03 20:15:22.017479 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.017464 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp" Nov 03 20:15:22.020096 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.020078 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl" Nov 03 20:15:22.020182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.020095 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl" event={"ID":"30d22aa8-3a01-44f1-a59d-b92a70934a7f","Type":"ContainerDied","Data":"f5ab6fc16c6c116db960954179c62e7333d0f8b7cfb5208e68e0f033ca807e33"} Nov 03 20:15:22.022227 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.022206 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg" event={"ID":"edb7775d-1e94-40ed-9da3-738ad142ef20","Type":"ContainerStarted","Data":"0dadb369a968981ad851f4eedd24a02e45d575fab14ccdd71a94d1dbca807fa1"} Nov 03 20:15:22.026161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.026039 5059 generic.go:356] "Generic (PLEG): container finished" podID="0e0bbcca-04df-4474-91fc-c5c5e6521732" containerID="ea9d9c0f68fb82425f437fb627d0937c5c4ce735145569ae74e7e2858fb221cf" exitCode=60 Nov 03 20:15:22.026161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.026074 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8" event={"ID":"0e0bbcca-04df-4474-91fc-c5c5e6521732","Type":"ContainerDied","Data":"ea9d9c0f68fb82425f437fb627d0937c5c4ce735145569ae74e7e2858fb221cf"} Nov 03 20:15:22.026463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.026445 5059 scope.go:122] "RemoveContainer" containerID="ea9d9c0f68fb82425f437fb627d0937c5c4ce735145569ae74e7e2858fb221cf" Nov 03 20:15:22.026737 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:22.026718 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8_odh-ai-gateway-infra(0e0bbcca-04df-4474-91fc-c5c5e6521732)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8" podUID="0e0bbcca-04df-4474-91fc-c5c5e6521732" Nov 03 20:15:22.078613 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.078502 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8"] Nov 03 20:15:22.162642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.160980 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn"] Nov 03 20:15:22.187538 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:15:22.187510 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8153e018_7f9a_4645_b102_06ff3913f682.slice/crio-93c48ae9de73afe2cd0d526c232d81ef7b3edc4ee33e2b54bfcc9f04887312a8 WatchSource:0}: Error finding container 93c48ae9de73afe2cd0d526c232d81ef7b3edc4ee33e2b54bfcc9f04887312a8: Status 404 returned error can't find the container with id 93c48ae9de73afe2cd0d526c232d81ef7b3edc4ee33e2b54bfcc9f04887312a8 Nov 03 20:15:22.192458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.192414 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:15:22.192724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.192606 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:15:22.192922 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.192904 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:15:22.219253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.219227 5059 scope.go:122] "RemoveContainer" containerID="08b1ab9b89a17043a6560aebf738c39bab16d69e4a15d643df5200646e8eafea" Nov 03 20:15:22.288998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.288974 5059 scope.go:122] "RemoveContainer" containerID="07640b251c8146ecf4614f5bd1de77521ce022831e712315222ee9bc748126c3" Nov 03 20:15:22.350082 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.350059 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r"] Nov 03 20:15:22.353386 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.353359 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp" Nov 03 20:15:22.355812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.355789 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895615-6c46r"] Nov 03 20:15:22.370066 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.370016 5059 scope.go:122] "RemoveContainer" containerID="9a8c7d844489e753ccc4601294b547e9204d12806e2c1eb94b01145190422e90" Nov 03 20:15:22.372449 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.372414 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p"] Nov 03 20:15:22.383871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.383775 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895615-5xv8p"] Nov 03 20:15:22.395574 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.395462 5059 scope.go:122] "RemoveContainer" containerID="46a105c28d2c721125674cf14f8357c899b78479d280b9e47a7775d675fcbc1c" Nov 03 20:15:22.395849 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.395831 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq" Nov 03 20:15:22.423161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.420903 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp"] Nov 03 20:15:22.427079 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.427046 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895615-sknbp"] Nov 03 20:15:22.436228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.436159 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl"] Nov 03 20:15:22.442997 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.442969 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895615-kk4gl"] Nov 03 20:15:22.496076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.491262 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1be9592b-18d0-4ce7-9322-92898161c6d8-kube-api-access-kv7cg\" (UniqueName: \"kubernetes.io/projected/1be9592b-18d0-4ce7-9322-92898161c6d8-kube-api-access-kv7cg\") pod \"1be9592b-18d0-4ce7-9322-92898161c6d8\" (UID: \"1be9592b-18d0-4ce7-9322-92898161c6d8\") " Nov 03 20:15:22.496076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.491508 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f7ed20a-4c3c-4023-b1de-479a8ea346a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7ed20a-4c3c-4023-b1de-479a8ea346a8-maas-api-service-ca\") pod \"8f7ed20a-4c3c-4023-b1de-479a8ea346a8\" (UID: \"8f7ed20a-4c3c-4023-b1de-479a8ea346a8\") " Nov 03 20:15:22.496076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.491567 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1be9592b-18d0-4ce7-9322-92898161c6d8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1be9592b-18d0-4ce7-9322-92898161c6d8-maas-api-service-ca\") pod \"1be9592b-18d0-4ce7-9322-92898161c6d8\" (UID: \"1be9592b-18d0-4ce7-9322-92898161c6d8\") " Nov 03 20:15:22.496076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.491712 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f7ed20a-4c3c-4023-b1de-479a8ea346a8-kube-api-access-kpvtn\" (UniqueName: \"kubernetes.io/projected/8f7ed20a-4c3c-4023-b1de-479a8ea346a8-kube-api-access-kpvtn\") pod \"8f7ed20a-4c3c-4023-b1de-479a8ea346a8\" (UID: \"8f7ed20a-4c3c-4023-b1de-479a8ea346a8\") " Nov 03 20:15:22.514987 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.514946 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1be9592b-18d0-4ce7-9322-92898161c6d8-maas-api-service-ca" pod "1be9592b-18d0-4ce7-9322-92898161c6d8" (UID: "1be9592b-18d0-4ce7-9322-92898161c6d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:22.548041 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.547983 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f7ed20a-4c3c-4023-b1de-479a8ea346a8-maas-api-service-ca" pod "8f7ed20a-4c3c-4023-b1de-479a8ea346a8" (UID: "8f7ed20a-4c3c-4023-b1de-479a8ea346a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:22.591798 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.591723 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7ed20a-4c3c-4023-b1de-479a8ea346a8-kube-api-access-kpvtn" pod "8f7ed20a-4c3c-4023-b1de-479a8ea346a8" (UID: "8f7ed20a-4c3c-4023-b1de-479a8ea346a8"). InnerVolumeSpecName "kube-api-access-kpvtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:22.593320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.593289 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be9592b-18d0-4ce7-9322-92898161c6d8-kube-api-access-kv7cg" pod "1be9592b-18d0-4ce7-9322-92898161c6d8" (UID: "1be9592b-18d0-4ce7-9322-92898161c6d8"). InnerVolumeSpecName "kube-api-access-kv7cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:22.594287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.593996 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1be9592b-18d0-4ce7-9322-92898161c6d8-kube-api-access-kv7cg\" (UniqueName: \"kubernetes.io/projected/1be9592b-18d0-4ce7-9322-92898161c6d8-kube-api-access-kv7cg\") pod \"1be9592b-18d0-4ce7-9322-92898161c6d8\" (UID: \"1be9592b-18d0-4ce7-9322-92898161c6d8\") " Nov 03 20:15:22.594287 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:15:22.594134 5059 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/1be9592b-18d0-4ce7-9322-92898161c6d8/volumes/kubernetes.io~projected/kube-api-access-kv7cg Nov 03 20:15:22.594287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.594176 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be9592b-18d0-4ce7-9322-92898161c6d8-kube-api-access-kv7cg" pod "1be9592b-18d0-4ce7-9322-92898161c6d8" (UID: "1be9592b-18d0-4ce7-9322-92898161c6d8"). InnerVolumeSpecName "kube-api-access-kv7cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:22.594833 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.594808 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7ed20a-4c3c-4023-b1de-479a8ea346a8-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:22.594926 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.594839 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1be9592b-18d0-4ce7-9322-92898161c6d8-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:22.594926 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.594856 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpvtn\" (UniqueName: \"kubernetes.io/projected/8f7ed20a-4c3c-4023-b1de-479a8ea346a8-kube-api-access-kpvtn\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:22.594926 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.594871 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv7cg\" (UniqueName: \"kubernetes.io/projected/1be9592b-18d0-4ce7-9322-92898161c6d8-kube-api-access-kv7cg\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:22.850252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:22.850213 5059 scope.go:122] "RemoveContainer" containerID="3cdda42662f948d6383cab1dc729ed28c7c179dd9a55b7685cde56e67593dfc7" Nov 03 20:15:23.038555 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.038519 5059 generic.go:356] "Generic (PLEG): container finished" podID="edb7775d-1e94-40ed-9da3-738ad142ef20" containerID="0dadb369a968981ad851f4eedd24a02e45d575fab14ccdd71a94d1dbca807fa1" exitCode=60 Nov 03 20:15:23.038990 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.038591 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg" event={"ID":"edb7775d-1e94-40ed-9da3-738ad142ef20","Type":"ContainerDied","Data":"0dadb369a968981ad851f4eedd24a02e45d575fab14ccdd71a94d1dbca807fa1"} Nov 03 20:15:23.038990 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.038676 5059 scope.go:122] "RemoveContainer" containerID="74a0a6de594c8105435cf5aa3513b6e272c53e1eed665d6c4ec2967c7926770f" Nov 03 20:15:23.039141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.039120 5059 scope.go:122] "RemoveContainer" containerID="0dadb369a968981ad851f4eedd24a02e45d575fab14ccdd71a94d1dbca807fa1" Nov 03 20:15:23.039538 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:23.039516 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg_odh-ai-gateway-infra(edb7775d-1e94-40ed-9da3-738ad142ef20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg" podUID="edb7775d-1e94-40ed-9da3-738ad142ef20" Nov 03 20:15:23.040156 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.040136 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp" Nov 03 20:15:23.040223 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.040134 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp" event={"ID":"1be9592b-18d0-4ce7-9322-92898161c6d8","Type":"ContainerDied","Data":"b4f18413a134ddf59191a98736526c70cb01aedb4cb0a51e4737d8566d235d53"} Nov 03 20:15:23.043792 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.043762 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" event={"ID":"8153e018-7f9a-4645-b102-06ff3913f682","Type":"ContainerStarted","Data":"3a0b8447bfda8a58216cc853554baf15762b4ccc77d0d98ce90bfe2224c3b122"} Nov 03 20:15:23.043898 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.043816 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" event={"ID":"8153e018-7f9a-4645-b102-06ff3913f682","Type":"ContainerStarted","Data":"93c48ae9de73afe2cd0d526c232d81ef7b3edc4ee33e2b54bfcc9f04887312a8"} Nov 03 20:15:23.045403 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.045381 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq" Nov 03 20:15:23.045518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.045386 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq" event={"ID":"8f7ed20a-4c3c-4023-b1de-479a8ea346a8","Type":"ContainerDied","Data":"4d9a6aa9c74c714b28ceb76d60fbf0e49b07d9badbbaf661f59b14302eab46cf"} Nov 03 20:15:23.047458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.047433 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s" event={"ID":"932670b6-eb80-4970-b092-8682e1daa6f5","Type":"ContainerStarted","Data":"bf7c5b1702efb40539d3f1b699d494e50a3f38d48de9162694a6346548a52c63"} Nov 03 20:15:23.049363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.049341 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m" event={"ID":"545cc204-5445-437d-b58e-9b6494f1a3b7","Type":"ContainerStarted","Data":"1e270ea671ea68e24b742b41e2e803929f3b188e42b8dba7cceec8527a74dec6"} Nov 03 20:15:23.052246 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.052226 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft" event={"ID":"ea54ebf8-4aab-4442-adf0-6930bcb6229b","Type":"ContainerStarted","Data":"961f4d3286dc09d683fbf6bb96375ffabb369ae0bd9d5662a1a4586f0768c076"} Nov 03 20:15:23.324016 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.317226 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg"] Nov 03 20:15:23.324016 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.319001 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" podStartSLOduration=10.318985634 podStartE2EDuration="10.318985634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:23.193077499 +0000 UTC m=+2317.450874758" watchObservedRunningTime="2026-11-03 20:15:23.318985634 +0000 UTC m=+2317.576782890" Nov 03 20:15:23.324451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.324426 5059 scope.go:122] "RemoveContainer" containerID="5d3a5c445733466dbe85af09e1b21c3373edd40bb5f86fac69f87e7839dac943" Nov 03 20:15:23.376139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.376103 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8" Nov 03 20:15:23.388499 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.388473 5059 scope.go:122] "RemoveContainer" containerID="2187a2fca756d2acefddda2ab8eccbf1df38ae0598c0c5a4e7f07c6fdc160b95" Nov 03 20:15:23.399381 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.399330 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq"] Nov 03 20:15:23.410157 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.410087 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895615-gkztq"] Nov 03 20:15:23.422753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.422536 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp"] Nov 03 20:15:23.428864 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.428841 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bcfde8-29895615-fwzlp"] Nov 03 20:15:23.520201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.520167 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e0bbcca-04df-4474-91fc-c5c5e6521732-kube-api-access-jd6zd\" (UniqueName: \"kubernetes.io/projected/0e0bbcca-04df-4474-91fc-c5c5e6521732-kube-api-access-jd6zd\") pod \"0e0bbcca-04df-4474-91fc-c5c5e6521732\" (UID: \"0e0bbcca-04df-4474-91fc-c5c5e6521732\") " Nov 03 20:15:23.520618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.520585 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e0bbcca-04df-4474-91fc-c5c5e6521732-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e0bbcca-04df-4474-91fc-c5c5e6521732-maas-api-service-ca\") pod \"0e0bbcca-04df-4474-91fc-c5c5e6521732\" (UID: \"0e0bbcca-04df-4474-91fc-c5c5e6521732\") " Nov 03 20:15:23.521048 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.521021 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e0bbcca-04df-4474-91fc-c5c5e6521732-maas-api-service-ca" pod "0e0bbcca-04df-4474-91fc-c5c5e6521732" (UID: "0e0bbcca-04df-4474-91fc-c5c5e6521732"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:23.521657 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.521622 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e0bbcca-04df-4474-91fc-c5c5e6521732-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:23.527111 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.527071 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e0bbcca-04df-4474-91fc-c5c5e6521732-kube-api-access-jd6zd" pod "0e0bbcca-04df-4474-91fc-c5c5e6521732" (UID: "0e0bbcca-04df-4474-91fc-c5c5e6521732"). InnerVolumeSpecName "kube-api-access-jd6zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:23.623792 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.623698 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd6zd\" (UniqueName: \"kubernetes.io/projected/0e0bbcca-04df-4474-91fc-c5c5e6521732-kube-api-access-jd6zd\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:23.833843 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.833806 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:23.835841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.835812 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 20:15:23.835966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.835869 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" podUID="8153e018-7f9a-4645-b102-06ff3913f682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 20:15:23.881982 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.881891 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1be9592b-18d0-4ce7-9322-92898161c6d8" path="/var/lib/kubelet/pods/1be9592b-18d0-4ce7-9322-92898161c6d8/volumes" Nov 03 20:15:23.883084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.883055 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30d22aa8-3a01-44f1-a59d-b92a70934a7f" path="/var/lib/kubelet/pods/30d22aa8-3a01-44f1-a59d-b92a70934a7f/volumes" Nov 03 20:15:23.884065 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.884041 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32905cf1-9e12-4fc4-8da4-dda3d142a0a0" path="/var/lib/kubelet/pods/32905cf1-9e12-4fc4-8da4-dda3d142a0a0/volumes" Nov 03 20:15:23.889782 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.889758 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f7ed20a-4c3c-4023-b1de-479a8ea346a8" path="/var/lib/kubelet/pods/8f7ed20a-4c3c-4023-b1de-479a8ea346a8/volumes" Nov 03 20:15:23.890575 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.890553 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb7298dc-a500-4715-a360-bcbdf0c7a298" path="/var/lib/kubelet/pods/cb7298dc-a500-4715-a360-bcbdf0c7a298/volumes" Nov 03 20:15:23.891330 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:23.891310 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3873f10-84bb-489c-9de5-b907f375ff3b" path="/var/lib/kubelet/pods/e3873f10-84bb-489c-9de5-b907f375ff3b/volumes" Nov 03 20:15:24.093969 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.093927 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9" event={"ID":"afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9","Type":"ContainerStarted","Data":"290198ee62130ad9e84fd7afd1a7b1571bd2db6e8d61874c036256de2d1c6cfe"} Nov 03 20:15:24.096322 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.096290 5059 generic.go:356] "Generic (PLEG): container finished" podID="932670b6-eb80-4970-b092-8682e1daa6f5" containerID="bf7c5b1702efb40539d3f1b699d494e50a3f38d48de9162694a6346548a52c63" exitCode=60 Nov 03 20:15:24.096470 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.096367 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s" event={"ID":"932670b6-eb80-4970-b092-8682e1daa6f5","Type":"ContainerDied","Data":"bf7c5b1702efb40539d3f1b699d494e50a3f38d48de9162694a6346548a52c63"} Nov 03 20:15:24.096553 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.096466 5059 scope.go:122] "RemoveContainer" containerID="2e0cf38a22a0523c5647f89e6a438290f8aaf64cfc728ffd56d51ef3ee205c02" Nov 03 20:15:24.096907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.096882 5059 scope.go:122] "RemoveContainer" containerID="bf7c5b1702efb40539d3f1b699d494e50a3f38d48de9162694a6346548a52c63" Nov 03 20:15:24.097364 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:24.097340 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s_odh-ai-gateway-infra(932670b6-eb80-4970-b092-8682e1daa6f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s" podUID="932670b6-eb80-4970-b092-8682e1daa6f5" Nov 03 20:15:24.098913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.098893 5059 generic.go:356] "Generic (PLEG): container finished" podID="545cc204-5445-437d-b58e-9b6494f1a3b7" containerID="1e270ea671ea68e24b742b41e2e803929f3b188e42b8dba7cceec8527a74dec6" exitCode=60 Nov 03 20:15:24.098992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.098961 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m" event={"ID":"545cc204-5445-437d-b58e-9b6494f1a3b7","Type":"ContainerDied","Data":"1e270ea671ea68e24b742b41e2e803929f3b188e42b8dba7cceec8527a74dec6"} Nov 03 20:15:24.099380 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.099362 5059 scope.go:122] "RemoveContainer" containerID="1e270ea671ea68e24b742b41e2e803929f3b188e42b8dba7cceec8527a74dec6" Nov 03 20:15:24.099759 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:24.099736 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m_odh-ai-gateway-infra(545cc204-5445-437d-b58e-9b6494f1a3b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m" podUID="545cc204-5445-437d-b58e-9b6494f1a3b7" Nov 03 20:15:24.101197 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.101177 5059 generic.go:356] "Generic (PLEG): container finished" podID="ea54ebf8-4aab-4442-adf0-6930bcb6229b" containerID="961f4d3286dc09d683fbf6bb96375ffabb369ae0bd9d5662a1a4586f0768c076" exitCode=60 Nov 03 20:15:24.101307 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.101246 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft" event={"ID":"ea54ebf8-4aab-4442-adf0-6930bcb6229b","Type":"ContainerDied","Data":"961f4d3286dc09d683fbf6bb96375ffabb369ae0bd9d5662a1a4586f0768c076"} Nov 03 20:15:24.101684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.101637 5059 scope.go:122] "RemoveContainer" containerID="961f4d3286dc09d683fbf6bb96375ffabb369ae0bd9d5662a1a4586f0768c076" Nov 03 20:15:24.102039 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:24.102013 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft_odh-ai-gateway-infra(ea54ebf8-4aab-4442-adf0-6930bcb6229b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft" podUID="ea54ebf8-4aab-4442-adf0-6930bcb6229b" Nov 03 20:15:24.110313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.109803 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8" Nov 03 20:15:24.110313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.110091 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8" event={"ID":"0e0bbcca-04df-4474-91fc-c5c5e6521732","Type":"ContainerDied","Data":"9dbc2829e04680ce84fa93cadf2abbe8c17c35e9687b862bc3da1d0ad2b37254"} Nov 03 20:15:24.114490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.114173 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m"] Nov 03 20:15:24.119224 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.119204 5059 scope.go:122] "RemoveContainer" containerID="a2d590c159c646b7bc90c66c7292004a5c54a7b78cad668d2cd0be92bd7864fd" Nov 03 20:15:24.138201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.138176 5059 scope.go:122] "RemoveContainer" containerID="b361175f83d8fc6249aae9c43aa927cb6ee5ab1c7cc52bcebe89a59be47783a9" Nov 03 20:15:24.145053 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.145028 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s"] Nov 03 20:15:24.188639 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.187937 5059 scope.go:122] "RemoveContainer" containerID="ea9d9c0f68fb82425f437fb627d0937c5c4ce735145569ae74e7e2858fb221cf" Nov 03 20:15:24.197637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.197601 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft"] Nov 03 20:15:24.249267 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.246693 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8"] Nov 03 20:15:24.299480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.299447 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895615-pv7w8"] Nov 03 20:15:24.394200 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.394119 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg" Nov 03 20:15:24.447023 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.446989 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edb7775d-1e94-40ed-9da3-738ad142ef20-kube-api-access-qgszl\" (UniqueName: \"kubernetes.io/projected/edb7775d-1e94-40ed-9da3-738ad142ef20-kube-api-access-qgszl\") pod \"edb7775d-1e94-40ed-9da3-738ad142ef20\" (UID: \"edb7775d-1e94-40ed-9da3-738ad142ef20\") " Nov 03 20:15:24.447293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.447269 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edb7775d-1e94-40ed-9da3-738ad142ef20-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edb7775d-1e94-40ed-9da3-738ad142ef20-maas-api-service-ca\") pod \"edb7775d-1e94-40ed-9da3-738ad142ef20\" (UID: \"edb7775d-1e94-40ed-9da3-738ad142ef20\") " Nov 03 20:15:24.447720 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.447689 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb7775d-1e94-40ed-9da3-738ad142ef20-maas-api-service-ca" pod "edb7775d-1e94-40ed-9da3-738ad142ef20" (UID: "edb7775d-1e94-40ed-9da3-738ad142ef20"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:24.448227 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.448203 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edb7775d-1e94-40ed-9da3-738ad142ef20-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:24.449826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.449797 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb7775d-1e94-40ed-9da3-738ad142ef20-kube-api-access-qgszl" pod "edb7775d-1e94-40ed-9da3-738ad142ef20" (UID: "edb7775d-1e94-40ed-9da3-738ad142ef20"). InnerVolumeSpecName "kube-api-access-qgszl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:24.549810 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.549771 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgszl\" (UniqueName: \"kubernetes.io/projected/edb7775d-1e94-40ed-9da3-738ad142ef20-kube-api-access-qgszl\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:24.842679 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.839266 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 20:15:24.842679 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:24.839324 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" podUID="8153e018-7f9a-4645-b102-06ff3913f682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 20:15:25.136525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.136431 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9"] Nov 03 20:15:25.139622 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.139594 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg" Nov 03 20:15:25.139790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.139612 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg" event={"ID":"edb7775d-1e94-40ed-9da3-738ad142ef20","Type":"ContainerDied","Data":"8624b23a907e6fec32e99f3f1e511f5333507b9d493e122bd70694eced8bdaf2"} Nov 03 20:15:25.139790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.139729 5059 scope.go:122] "RemoveContainer" containerID="0dadb369a968981ad851f4eedd24a02e45d575fab14ccdd71a94d1dbca807fa1" Nov 03 20:15:25.143444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.143414 5059 generic.go:356] "Generic (PLEG): container finished" podID="afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9" containerID="290198ee62130ad9e84fd7afd1a7b1571bd2db6e8d61874c036256de2d1c6cfe" exitCode=60 Nov 03 20:15:25.143567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.143504 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9" event={"ID":"afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9","Type":"ContainerDied","Data":"290198ee62130ad9e84fd7afd1a7b1571bd2db6e8d61874c036256de2d1c6cfe"} Nov 03 20:15:25.156208 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.156183 5059 scope.go:122] "RemoveContainer" containerID="3cdda42662f948d6383cab1dc729ed28c7c179dd9a55b7685cde56e67593dfc7" Nov 03 20:15:25.205507 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.205470 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg"] Nov 03 20:15:25.216802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.216533 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895615-5dkcg"] Nov 03 20:15:25.512347 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.506926 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft" Nov 03 20:15:25.600346 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.600314 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea54ebf8-4aab-4442-adf0-6930bcb6229b-kube-api-access-tv5xn\" (UniqueName: \"kubernetes.io/projected/ea54ebf8-4aab-4442-adf0-6930bcb6229b-kube-api-access-tv5xn\") pod \"ea54ebf8-4aab-4442-adf0-6930bcb6229b\" (UID: \"ea54ebf8-4aab-4442-adf0-6930bcb6229b\") " Nov 03 20:15:25.600878 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.600851 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea54ebf8-4aab-4442-adf0-6930bcb6229b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea54ebf8-4aab-4442-adf0-6930bcb6229b-maas-api-service-ca\") pod \"ea54ebf8-4aab-4442-adf0-6930bcb6229b\" (UID: \"ea54ebf8-4aab-4442-adf0-6930bcb6229b\") " Nov 03 20:15:25.601397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.601359 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea54ebf8-4aab-4442-adf0-6930bcb6229b-maas-api-service-ca" pod "ea54ebf8-4aab-4442-adf0-6930bcb6229b" (UID: "ea54ebf8-4aab-4442-adf0-6930bcb6229b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:25.602167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.602145 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea54ebf8-4aab-4442-adf0-6930bcb6229b-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:25.604777 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.604610 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea54ebf8-4aab-4442-adf0-6930bcb6229b-kube-api-access-tv5xn" pod "ea54ebf8-4aab-4442-adf0-6930bcb6229b" (UID: "ea54ebf8-4aab-4442-adf0-6930bcb6229b"). InnerVolumeSpecName "kube-api-access-tv5xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:25.627399 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.627360 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s" Nov 03 20:15:25.685131 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.685052 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9" Nov 03 20:15:25.735120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.735055 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv5xn\" (UniqueName: \"kubernetes.io/projected/ea54ebf8-4aab-4442-adf0-6930bcb6229b-kube-api-access-tv5xn\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:25.833568 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.833529 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 20:15:25.833783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.833600 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" podUID="8153e018-7f9a-4645-b102-06ff3913f682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 20:15:25.837532 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.837504 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m" Nov 03 20:15:25.839405 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.839372 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9-kube-api-access-fmq9v\" (UniqueName: \"kubernetes.io/projected/afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9-kube-api-access-fmq9v\") pod \"afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9\" (UID: \"afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9\") " Nov 03 20:15:25.839625 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.839602 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/932670b6-eb80-4970-b092-8682e1daa6f5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932670b6-eb80-4970-b092-8682e1daa6f5-maas-api-service-ca\") pod \"932670b6-eb80-4970-b092-8682e1daa6f5\" (UID: \"932670b6-eb80-4970-b092-8682e1daa6f5\") " Nov 03 20:15:25.839720 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.839703 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/932670b6-eb80-4970-b092-8682e1daa6f5-kube-api-access-gcqkz\" (UniqueName: \"kubernetes.io/projected/932670b6-eb80-4970-b092-8682e1daa6f5-kube-api-access-gcqkz\") pod \"932670b6-eb80-4970-b092-8682e1daa6f5\" (UID: \"932670b6-eb80-4970-b092-8682e1daa6f5\") " Nov 03 20:15:25.839836 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.839815 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9-maas-api-service-ca\") pod \"afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9\" (UID: \"afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9\") " Nov 03 20:15:25.840553 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.840523 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932670b6-eb80-4970-b092-8682e1daa6f5-maas-api-service-ca" pod "932670b6-eb80-4970-b092-8682e1daa6f5" (UID: "932670b6-eb80-4970-b092-8682e1daa6f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:25.840757 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.840726 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932670b6-eb80-4970-b092-8682e1daa6f5-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:25.840934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.840898 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9-maas-api-service-ca" pod "afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9" (UID: "afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:25.842200 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.842169 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9-kube-api-access-fmq9v" pod "afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9" (UID: "afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9"). InnerVolumeSpecName "kube-api-access-fmq9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:25.842476 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.842454 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932670b6-eb80-4970-b092-8682e1daa6f5-kube-api-access-gcqkz" pod "932670b6-eb80-4970-b092-8682e1daa6f5" (UID: "932670b6-eb80-4970-b092-8682e1daa6f5"). InnerVolumeSpecName "kube-api-access-gcqkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:25.897910 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.897879 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e0bbcca-04df-4474-91fc-c5c5e6521732" path="/var/lib/kubelet/pods/0e0bbcca-04df-4474-91fc-c5c5e6521732/volumes" Nov 03 20:15:25.898857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.898829 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edb7775d-1e94-40ed-9da3-738ad142ef20" path="/var/lib/kubelet/pods/edb7775d-1e94-40ed-9da3-738ad142ef20/volumes" Nov 03 20:15:25.941576 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.941512 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/545cc204-5445-437d-b58e-9b6494f1a3b7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/545cc204-5445-437d-b58e-9b6494f1a3b7-maas-api-service-ca\") pod \"545cc204-5445-437d-b58e-9b6494f1a3b7\" (UID: \"545cc204-5445-437d-b58e-9b6494f1a3b7\") " Nov 03 20:15:25.941576 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.941559 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/545cc204-5445-437d-b58e-9b6494f1a3b7-kube-api-access-6r4s2\" (UniqueName: \"kubernetes.io/projected/545cc204-5445-437d-b58e-9b6494f1a3b7-kube-api-access-6r4s2\") pod \"545cc204-5445-437d-b58e-9b6494f1a3b7\" (UID: \"545cc204-5445-437d-b58e-9b6494f1a3b7\") " Nov 03 20:15:25.941939 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.941916 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/545cc204-5445-437d-b58e-9b6494f1a3b7-maas-api-service-ca" pod "545cc204-5445-437d-b58e-9b6494f1a3b7" (UID: "545cc204-5445-437d-b58e-9b6494f1a3b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:25.942308 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.942288 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/545cc204-5445-437d-b58e-9b6494f1a3b7-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:25.942376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.942312 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcqkz\" (UniqueName: \"kubernetes.io/projected/932670b6-eb80-4970-b092-8682e1daa6f5-kube-api-access-gcqkz\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:25.942376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.942327 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9-maas-api-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:25.942376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.942340 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmq9v\" (UniqueName: \"kubernetes.io/projected/afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9-kube-api-access-fmq9v\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:25.943654 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:25.943633 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545cc204-5445-437d-b58e-9b6494f1a3b7-kube-api-access-6r4s2" pod "545cc204-5445-437d-b58e-9b6494f1a3b7" (UID: "545cc204-5445-437d-b58e-9b6494f1a3b7"). InnerVolumeSpecName "kube-api-access-6r4s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:26.044099 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:26.044063 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r4s2\" (UniqueName: \"kubernetes.io/projected/545cc204-5445-437d-b58e-9b6494f1a3b7-kube-api-access-6r4s2\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:26.182687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:26.182036 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft" event={"ID":"ea54ebf8-4aab-4442-adf0-6930bcb6229b","Type":"ContainerDied","Data":"763b4652cf02c03cf287b4d600ed5ff7430bcf7dbc6610ac1134c602aff9b73f"} Nov 03 20:15:26.182687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:26.182115 5059 scope.go:122] "RemoveContainer" containerID="961f4d3286dc09d683fbf6bb96375ffabb369ae0bd9d5662a1a4586f0768c076" Nov 03 20:15:26.182687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:26.182216 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft" Nov 03 20:15:26.186506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:26.186485 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9" Nov 03 20:15:26.186652 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:26.186491 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9" event={"ID":"afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9","Type":"ContainerDied","Data":"3fe1a2cef26d035e5c43f6701ff9c787904ab6d185c34c939c7a3346236b8068"} Nov 03 20:15:26.188119 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:26.188098 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s" Nov 03 20:15:26.188230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:26.188131 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s" event={"ID":"932670b6-eb80-4970-b092-8682e1daa6f5","Type":"ContainerDied","Data":"e3d2f41c255b5346cae2519e6623612f5606b6131525fc2593e624ffc59f03cd"} Nov 03 20:15:26.190042 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:26.190023 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m" Nov 03 20:15:26.190121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:26.190059 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m" event={"ID":"545cc204-5445-437d-b58e-9b6494f1a3b7","Type":"ContainerDied","Data":"fc9d1eb7fe91d2a484c35c68b0cf16395c040ceb653bfb10a71dd3d1d5fd6787"} Nov 03 20:15:26.197506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:26.197488 5059 scope.go:122] "RemoveContainer" containerID="290198ee62130ad9e84fd7afd1a7b1571bd2db6e8d61874c036256de2d1c6cfe" Nov 03 20:15:26.209288 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:26.209263 5059 scope.go:122] "RemoveContainer" containerID="bf7c5b1702efb40539d3f1b699d494e50a3f38d48de9162694a6346548a52c63" Nov 03 20:15:26.216294 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:26.216274 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft"] Nov 03 20:15:26.223874 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:26.223858 5059 scope.go:122] "RemoveContainer" containerID="1e270ea671ea68e24b742b41e2e803929f3b188e42b8dba7cceec8527a74dec6" Nov 03 20:15:26.224379 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:26.224359 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895615-2kpft"] Nov 03 20:15:26.238781 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:26.238760 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s"] Nov 03 20:15:26.243634 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:26.243612 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895615-s2x5s"] Nov 03 20:15:26.300041 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:26.300009 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m"] Nov 03 20:15:26.408757 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:26.408719 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895615-ds44m"] Nov 03 20:15:26.412146 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:26.412118 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9"] Nov 03 20:15:26.417062 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:26.417041 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895615-t2rp9"] Nov 03 20:15:26.834220 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:26.834178 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 20:15:26.834385 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:26.834245 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" podUID="8153e018-7f9a-4645-b102-06ff3913f682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 20:15:27.845510 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:27.845471 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 20:15:27.846111 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:27.846074 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" podUID="8153e018-7f9a-4645-b102-06ff3913f682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 20:15:28.004880 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:28.004837 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="545cc204-5445-437d-b58e-9b6494f1a3b7" path="/var/lib/kubelet/pods/545cc204-5445-437d-b58e-9b6494f1a3b7/volumes" Nov 03 20:15:28.016120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:28.016084 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="932670b6-eb80-4970-b092-8682e1daa6f5" path="/var/lib/kubelet/pods/932670b6-eb80-4970-b092-8682e1daa6f5/volumes" Nov 03 20:15:28.024108 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:28.024073 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9" path="/var/lib/kubelet/pods/afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9/volumes" Nov 03 20:15:28.033337 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:28.033306 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea54ebf8-4aab-4442-adf0-6930bcb6229b" path="/var/lib/kubelet/pods/ea54ebf8-4aab-4442-adf0-6930bcb6229b/volumes" Nov 03 20:15:28.467966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:28.467925 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 03 20:15:28.640259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:28.640214 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:28.785970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:28.784909 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:28.836479 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:28.836044 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 20:15:28.836479 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:28.836107 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" podUID="8153e018-7f9a-4645-b102-06ff3913f682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 20:15:29.122554 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:29.122508 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:15:29.248826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:29.248774 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:29.471900 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:29.471793 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:29.472738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:29.472713 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:15:29.738743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:29.738438 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:15:29.821015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:29.820972 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:29.834102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:29.833911 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 20:15:29.834313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:29.834096 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 20:15:29.834313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:29.834150 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" podUID="8153e018-7f9a-4645-b102-06ff3913f682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 20:15:29.870684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:29.870138 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 20:15:30.834451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.834420 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 20:15:30.834872 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.834474 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" podUID="8153e018-7f9a-4645-b102-06ff3913f682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 20:15:30.948764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.948723 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l"] Nov 03 20:15:30.961786 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.950448 5059 cpu_manager.go:414] "containerMap: removing container" podUID="cb7298dc-a500-4715-a360-bcbdf0c7a298" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.970715 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb7298dc-a500-4715-a360-bcbdf0c7a298" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.970774 5059 cpu_manager.go:414] "containerMap: removing container" podUID="32905cf1-9e12-4fc4-8da4-dda3d142a0a0" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.970781 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32905cf1-9e12-4fc4-8da4-dda3d142a0a0" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.970810 5059 cpu_manager.go:414] "containerMap: removing container" podUID="8f7ed20a-4c3c-4023-b1de-479a8ea346a8" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.970818 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f7ed20a-4c3c-4023-b1de-479a8ea346a8" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.970832 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1be9592b-18d0-4ce7-9322-92898161c6d8" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.970840 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1be9592b-18d0-4ce7-9322-92898161c6d8" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.970851 5059 cpu_manager.go:414] "containerMap: removing container" podUID="30d22aa8-3a01-44f1-a59d-b92a70934a7f" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.970858 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d22aa8-3a01-44f1-a59d-b92a70934a7f" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.970869 5059 cpu_manager.go:414] "containerMap: removing container" podUID="545cc204-5445-437d-b58e-9b6494f1a3b7" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.970876 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="545cc204-5445-437d-b58e-9b6494f1a3b7" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.970887 5059 cpu_manager.go:414] "containerMap: removing container" podUID="ea54ebf8-4aab-4442-adf0-6930bcb6229b" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.970893 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea54ebf8-4aab-4442-adf0-6930bcb6229b" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.970905 5059 cpu_manager.go:414] "containerMap: removing container" podUID="545cc204-5445-437d-b58e-9b6494f1a3b7" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.970912 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="545cc204-5445-437d-b58e-9b6494f1a3b7" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.970925 5059 cpu_manager.go:414] "containerMap: removing container" podUID="e3873f10-84bb-489c-9de5-b907f375ff3b" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.970931 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3873f10-84bb-489c-9de5-b907f375ff3b" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.970947 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1be9592b-18d0-4ce7-9322-92898161c6d8" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.970953 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1be9592b-18d0-4ce7-9322-92898161c6d8" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.970968 5059 cpu_manager.go:414] "containerMap: removing container" podUID="932670b6-eb80-4970-b092-8682e1daa6f5" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.970976 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="932670b6-eb80-4970-b092-8682e1daa6f5" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.970986 5059 cpu_manager.go:414] "containerMap: removing container" podUID="8f7ed20a-4c3c-4023-b1de-479a8ea346a8" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.970992 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f7ed20a-4c3c-4023-b1de-479a8ea346a8" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971002 5059 cpu_manager.go:414] "containerMap: removing container" podUID="0e0bbcca-04df-4474-91fc-c5c5e6521732" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971010 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e0bbcca-04df-4474-91fc-c5c5e6521732" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971024 5059 cpu_manager.go:414] "containerMap: removing container" podUID="edb7775d-1e94-40ed-9da3-738ad142ef20" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971030 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb7775d-1e94-40ed-9da3-738ad142ef20" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971043 5059 cpu_manager.go:414] "containerMap: removing container" podUID="545cc204-5445-437d-b58e-9b6494f1a3b7" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971050 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="545cc204-5445-437d-b58e-9b6494f1a3b7" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971069 5059 cpu_manager.go:414] "containerMap: removing container" podUID="8588fce3-f04c-472e-a796-ed5314845eb5" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971076 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8588fce3-f04c-472e-a796-ed5314845eb5" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971112 5059 cpu_manager.go:414] "containerMap: removing container" podUID="932670b6-eb80-4970-b092-8682e1daa6f5" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971120 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="932670b6-eb80-4970-b092-8682e1daa6f5" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971131 5059 cpu_manager.go:414] "containerMap: removing container" podUID="0e0bbcca-04df-4474-91fc-c5c5e6521732" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971138 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e0bbcca-04df-4474-91fc-c5c5e6521732" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971150 5059 cpu_manager.go:414] "containerMap: removing container" podUID="0e0bbcca-04df-4474-91fc-c5c5e6521732" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971157 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e0bbcca-04df-4474-91fc-c5c5e6521732" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971165 5059 cpu_manager.go:414] "containerMap: removing container" podUID="ea54ebf8-4aab-4442-adf0-6930bcb6229b" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971172 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea54ebf8-4aab-4442-adf0-6930bcb6229b" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971190 5059 cpu_manager.go:414] "containerMap: removing container" podUID="edb7775d-1e94-40ed-9da3-738ad142ef20" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971197 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb7775d-1e94-40ed-9da3-738ad142ef20" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971206 5059 cpu_manager.go:414] "containerMap: removing container" podUID="ea54ebf8-4aab-4442-adf0-6930bcb6229b" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971212 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea54ebf8-4aab-4442-adf0-6930bcb6229b" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971228 5059 cpu_manager.go:414] "containerMap: removing container" podUID="afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971235 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971243 5059 cpu_manager.go:414] "containerMap: removing container" podUID="f1477a68-b92a-4138-b649-fdcd3b17f693" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971250 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1477a68-b92a-4138-b649-fdcd3b17f693" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971263 5059 cpu_manager.go:414] "containerMap: removing container" podUID="ffb6bb03-2f3c-43f0-871a-1140b53f0fb9" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971269 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb6bb03-2f3c-43f0-871a-1140b53f0fb9" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971292 5059 cpu_manager.go:414] "containerMap: removing container" podUID="932670b6-eb80-4970-b092-8682e1daa6f5" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971299 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="932670b6-eb80-4970-b092-8682e1daa6f5" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971324 5059 cpu_manager.go:414] "containerMap: removing container" podUID="afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971330 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971340 5059 cpu_manager.go:414] "containerMap: removing container" podUID="afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9" containerName="cleanup" Nov 03 20:15:30.973685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.971347 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991362 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="edb7775d-1e94-40ed-9da3-738ad142ef20" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991401 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="545cc204-5445-437d-b58e-9b6494f1a3b7" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991425 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="932670b6-eb80-4970-b092-8682e1daa6f5" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991434 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e0bbcca-04df-4474-91fc-c5c5e6521732" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991445 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e0bbcca-04df-4474-91fc-c5c5e6521732" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991452 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea54ebf8-4aab-4442-adf0-6930bcb6229b" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991461 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f7ed20a-4c3c-4023-b1de-479a8ea346a8" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991471 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="edb7775d-1e94-40ed-9da3-738ad142ef20" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991480 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea54ebf8-4aab-4442-adf0-6930bcb6229b" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991490 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991497 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1774de9e-305c-4b0f-b194-a09995fb25bf" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991512 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="932670b6-eb80-4970-b092-8682e1daa6f5" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991525 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991534 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b18a5b0-0fc4-4dc8-88b7-affec66830d8" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991546 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1be9592b-18d0-4ce7-9322-92898161c6d8" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991554 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="edb7775d-1e94-40ed-9da3-738ad142ef20" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991562 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="afef7cbf-c76b-4cb5-ad55-0f16a2bf0ac9" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991574 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb7298dc-a500-4715-a360-bcbdf0c7a298" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991586 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1477a68-b92a-4138-b649-fdcd3b17f693" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991597 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1be9592b-18d0-4ce7-9322-92898161c6d8" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991605 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d22aa8-3a01-44f1-a59d-b92a70934a7f" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991614 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea54ebf8-4aab-4442-adf0-6930bcb6229b" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991624 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb6bb03-2f3c-43f0-871a-1140b53f0fb9" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991633 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3873f10-84bb-489c-9de5-b907f375ff3b" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991643 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1be9592b-18d0-4ce7-9322-92898161c6d8" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991654 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="545cc204-5445-437d-b58e-9b6494f1a3b7" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991681 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f7ed20a-4c3c-4023-b1de-479a8ea346a8" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991689 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e0bbcca-04df-4474-91fc-c5c5e6521732" containerName="cleanup" Nov 03 20:15:31.012506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:30.991698 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1e6b3a-7887-42c5-b23d-b226cc6d0641" containerName="cleanup" Nov 03 20:15:31.119731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.119004 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l" Nov 03 20:15:31.260077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.148463 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"agc-cred-f5fb49f604b6\"" Nov 03 20:15:31.260077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.155889 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"kube-root-ca.crt\"" Nov 03 20:15:31.260077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.155992 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"payload-processing-external-model-e2e--088821c0-dockercfg-mqh9q\"" Nov 03 20:15:31.260077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.156094 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"routing-overlay\"" Nov 03 20:15:31.260077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.156157 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"payload-processing-external-model-plugins-e2e-extmod-b-4c2a28\"" Nov 03 20:15:31.260077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.167365 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"openshift-service-ca.crt\"" Nov 03 20:15:31.260077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.204825 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l"] Nov 03 20:15:31.390793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.388954 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a9960a-00a9-4038-9cc6-dea185172090-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l\" (UID: \"47a9960a-00a9-4038-9cc6-dea185172090\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l" Nov 03 20:15:31.390793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.389178 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47a9960a-00a9-4038-9cc6-dea185172090-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l\" (UID: \"47a9960a-00a9-4038-9cc6-dea185172090\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l" Nov 03 20:15:31.390793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.389216 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmd5t\" (UniqueName: \"kubernetes.io/projected/47a9960a-00a9-4038-9cc6-dea185172090-kube-api-access-bmd5t\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l\" (UID: \"47a9960a-00a9-4038-9cc6-dea185172090\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l" Nov 03 20:15:31.390793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.389546 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/47a9960a-00a9-4038-9cc6-dea185172090-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l\" (UID: \"47a9960a-00a9-4038-9cc6-dea185172090\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l" Nov 03 20:15:31.491932 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.491898 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a9960a-00a9-4038-9cc6-dea185172090-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l\" (UID: \"47a9960a-00a9-4038-9cc6-dea185172090\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l" Nov 03 20:15:31.492132 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.492037 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47a9960a-00a9-4038-9cc6-dea185172090-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l\" (UID: \"47a9960a-00a9-4038-9cc6-dea185172090\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l" Nov 03 20:15:31.492132 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.492057 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmd5t\" (UniqueName: \"kubernetes.io/projected/47a9960a-00a9-4038-9cc6-dea185172090-kube-api-access-bmd5t\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l\" (UID: \"47a9960a-00a9-4038-9cc6-dea185172090\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l" Nov 03 20:15:31.492258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.492230 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/47a9960a-00a9-4038-9cc6-dea185172090-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l\" (UID: \"47a9960a-00a9-4038-9cc6-dea185172090\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l" Nov 03 20:15:31.492824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.492794 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47a9960a-00a9-4038-9cc6-dea185172090-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l\" (UID: \"47a9960a-00a9-4038-9cc6-dea185172090\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l" Nov 03 20:15:31.493047 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.492905 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a9960a-00a9-4038-9cc6-dea185172090-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l\" (UID: \"47a9960a-00a9-4038-9cc6-dea185172090\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l" Nov 03 20:15:31.495424 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.495395 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/47a9960a-00a9-4038-9cc6-dea185172090-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l\" (UID: \"47a9960a-00a9-4038-9cc6-dea185172090\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l" Nov 03 20:15:31.504906 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.504871 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmd5t\" (UniqueName: \"kubernetes.io/projected/47a9960a-00a9-4038-9cc6-dea185172090-kube-api-access-bmd5t\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l\" (UID: \"47a9960a-00a9-4038-9cc6-dea185172090\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l" Nov 03 20:15:31.702838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.702743 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l" Nov 03 20:15:31.771410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.762190 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a856579-v5qpb"] Nov 03 20:15:31.771410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.763569 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1be9592b-18d0-4ce7-9322-92898161c6d8" containerName="cleanup" Nov 03 20:15:31.771410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.763591 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1be9592b-18d0-4ce7-9322-92898161c6d8" containerName="cleanup" Nov 03 20:15:31.771410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.763610 5059 cpu_manager.go:414] "containerMap: removing container" podUID="8f7ed20a-4c3c-4023-b1de-479a8ea346a8" containerName="cleanup" Nov 03 20:15:31.771410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.763618 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f7ed20a-4c3c-4023-b1de-479a8ea346a8" containerName="cleanup" Nov 03 20:15:31.771410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.763729 5059 cpu_manager.go:414] "containerMap: removing container" podUID="edb7775d-1e94-40ed-9da3-738ad142ef20" containerName="cleanup" Nov 03 20:15:31.771410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.763738 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb7775d-1e94-40ed-9da3-738ad142ef20" containerName="cleanup" Nov 03 20:15:31.771410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.763924 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="932670b6-eb80-4970-b092-8682e1daa6f5" containerName="cleanup" Nov 03 20:15:31.771410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.763949 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f7ed20a-4c3c-4023-b1de-479a8ea346a8" containerName="cleanup" Nov 03 20:15:31.771410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.763961 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="545cc204-5445-437d-b58e-9b6494f1a3b7" containerName="cleanup" Nov 03 20:15:31.775590 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.774938 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a856579-v5qpb"] Nov 03 20:15:31.775590 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.775053 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a856579-v5qpb" Nov 03 20:15:31.781870 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.781843 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:15:31.781997 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.781918 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:15:31.799803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.798817 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lb2m\" (UniqueName: \"kubernetes.io/projected/d02cb077-de4b-41cc-afeb-ea3b3bdb5728-kube-api-access-2lb2m\") pod \"maas-api-revoke-sub-8a856579-v5qpb\" (UID: \"d02cb077-de4b-41cc-afeb-ea3b3bdb5728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a856579-v5qpb" Nov 03 20:15:31.799803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.798868 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02cb077-de4b-41cc-afeb-ea3b3bdb5728-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a856579-v5qpb\" (UID: \"d02cb077-de4b-41cc-afeb-ea3b3bdb5728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a856579-v5qpb" Nov 03 20:15:31.799803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.798960 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02cb077-de4b-41cc-afeb-ea3b3bdb5728-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a856579-v5qpb\" (UID: \"d02cb077-de4b-41cc-afeb-ea3b3bdb5728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a856579-v5qpb" Nov 03 20:15:31.841094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.841061 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:31.842136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.842111 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:31.843167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.843139 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:15:31.904550 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.904178 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lb2m\" (UniqueName: \"kubernetes.io/projected/d02cb077-de4b-41cc-afeb-ea3b3bdb5728-kube-api-access-2lb2m\") pod \"maas-api-revoke-sub-8a856579-v5qpb\" (UID: \"d02cb077-de4b-41cc-afeb-ea3b3bdb5728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a856579-v5qpb" Nov 03 20:15:31.904550 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.904231 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02cb077-de4b-41cc-afeb-ea3b3bdb5728-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a856579-v5qpb\" (UID: \"d02cb077-de4b-41cc-afeb-ea3b3bdb5728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a856579-v5qpb" Nov 03 20:15:31.907006 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.905767 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02cb077-de4b-41cc-afeb-ea3b3bdb5728-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a856579-v5qpb\" (UID: \"d02cb077-de4b-41cc-afeb-ea3b3bdb5728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a856579-v5qpb" Nov 03 20:15:31.910776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.910747 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02cb077-de4b-41cc-afeb-ea3b3bdb5728-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a856579-v5qpb\" (UID: \"d02cb077-de4b-41cc-afeb-ea3b3bdb5728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a856579-v5qpb" Nov 03 20:15:31.913801 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.913769 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02cb077-de4b-41cc-afeb-ea3b3bdb5728-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a856579-v5qpb\" (UID: \"d02cb077-de4b-41cc-afeb-ea3b3bdb5728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a856579-v5qpb" Nov 03 20:15:31.933681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:31.933639 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lb2m\" (UniqueName: \"kubernetes.io/projected/d02cb077-de4b-41cc-afeb-ea3b3bdb5728-kube-api-access-2lb2m\") pod \"maas-api-revoke-sub-8a856579-v5qpb\" (UID: \"d02cb077-de4b-41cc-afeb-ea3b3bdb5728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a856579-v5qpb" Nov 03 20:15:32.065587 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:15:32.065554 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a9960a_00a9_4038_9cc6_dea185172090.slice/crio-84af315a5ec2150f989587e072760c6441fe62a31bdd49b9b60eaed180470519 WatchSource:0}: Error finding container 84af315a5ec2150f989587e072760c6441fe62a31bdd49b9b60eaed180470519: Status 404 returned error can't find the container with id 84af315a5ec2150f989587e072760c6441fe62a31bdd49b9b60eaed180470519 Nov 03 20:15:32.066258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:32.066233 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l"] Nov 03 20:15:32.107428 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:32.107405 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a856579-v5qpb" Nov 03 20:15:32.419774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:32.419734 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l" event={"ID":"47a9960a-00a9-4038-9cc6-dea185172090","Type":"ContainerStarted","Data":"6b4a4e4b2c1e3a23d85684882ad115e379761964e22132117b5bac24d69c0591"} Nov 03 20:15:32.420217 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:32.420191 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l" Nov 03 20:15:32.420338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:32.420238 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l" event={"ID":"47a9960a-00a9-4038-9cc6-dea185172090","Type":"ContainerStarted","Data":"84af315a5ec2150f989587e072760c6441fe62a31bdd49b9b60eaed180470519"} Nov 03 20:15:32.422423 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:15:32.422387 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02cb077_de4b_41cc_afeb_ea3b3bdb5728.slice/crio-01629f75141c90bc6a2cd5157d7aae750e89c832f345f068e2167ea081feffc5 WatchSource:0}: Error finding container 01629f75141c90bc6a2cd5157d7aae750e89c832f345f068e2167ea081feffc5: Status 404 returned error can't find the container with id 01629f75141c90bc6a2cd5157d7aae750e89c832f345f068e2167ea081feffc5 Nov 03 20:15:32.422861 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:32.422840 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a856579-v5qpb"] Nov 03 20:15:32.445795 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:32.445753 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l" podStartSLOduration=2.445740129 podStartE2EDuration="2.445740129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:32.439391672 +0000 UTC m=+2326.697188930" watchObservedRunningTime="2026-11-03 20:15:32.445740129 +0000 UTC m=+2326.703537383" Nov 03 20:15:33.431434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:33.431387 5059 generic.go:356] "Generic (PLEG): container finished" podID="d02cb077-de4b-41cc-afeb-ea3b3bdb5728" containerID="bca04c565d1eb984fc3c702b2e4a86c4310b2d892273c68fbd595faf6a4a41b5" exitCode=0 Nov 03 20:15:33.431975 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:33.431831 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a856579-v5qpb" event={"ID":"d02cb077-de4b-41cc-afeb-ea3b3bdb5728","Type":"ContainerDied","Data":"bca04c565d1eb984fc3c702b2e4a86c4310b2d892273c68fbd595faf6a4a41b5"} Nov 03 20:15:33.432107 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:33.432016 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a856579-v5qpb" event={"ID":"d02cb077-de4b-41cc-afeb-ea3b3bdb5728","Type":"ContainerStarted","Data":"01629f75141c90bc6a2cd5157d7aae750e89c832f345f068e2167ea081feffc5"} Nov 03 20:15:34.719938 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:34.719911 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a856579-v5qpb" Nov 03 20:15:34.744682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:34.744630 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d02cb077-de4b-41cc-afeb-ea3b3bdb5728-kube-api-access-2lb2m\" (UniqueName: \"kubernetes.io/projected/d02cb077-de4b-41cc-afeb-ea3b3bdb5728-kube-api-access-2lb2m\") pod \"d02cb077-de4b-41cc-afeb-ea3b3bdb5728\" (UID: \"d02cb077-de4b-41cc-afeb-ea3b3bdb5728\") " Nov 03 20:15:34.744939 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:34.744704 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d02cb077-de4b-41cc-afeb-ea3b3bdb5728-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02cb077-de4b-41cc-afeb-ea3b3bdb5728-maas-api-openshift-ca\") pod \"d02cb077-de4b-41cc-afeb-ea3b3bdb5728\" (UID: \"d02cb077-de4b-41cc-afeb-ea3b3bdb5728\") " Nov 03 20:15:34.744939 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:34.744925 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d02cb077-de4b-41cc-afeb-ea3b3bdb5728-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02cb077-de4b-41cc-afeb-ea3b3bdb5728-maas-api-xks-ca\") pod \"d02cb077-de4b-41cc-afeb-ea3b3bdb5728\" (UID: \"d02cb077-de4b-41cc-afeb-ea3b3bdb5728\") " Nov 03 20:15:34.747575 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:34.747543 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d02cb077-de4b-41cc-afeb-ea3b3bdb5728-maas-api-openshift-ca" pod "d02cb077-de4b-41cc-afeb-ea3b3bdb5728" (UID: "d02cb077-de4b-41cc-afeb-ea3b3bdb5728"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:34.768641 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:34.768608 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02cb077-de4b-41cc-afeb-ea3b3bdb5728-kube-api-access-2lb2m" pod "d02cb077-de4b-41cc-afeb-ea3b3bdb5728" (UID: "d02cb077-de4b-41cc-afeb-ea3b3bdb5728"). InnerVolumeSpecName "kube-api-access-2lb2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:34.772474 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:34.771136 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d02cb077-de4b-41cc-afeb-ea3b3bdb5728-maas-api-xks-ca" pod "d02cb077-de4b-41cc-afeb-ea3b3bdb5728" (UID: "d02cb077-de4b-41cc-afeb-ea3b3bdb5728"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:34.847412 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:34.847374 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lb2m\" (UniqueName: \"kubernetes.io/projected/d02cb077-de4b-41cc-afeb-ea3b3bdb5728-kube-api-access-2lb2m\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:34.847412 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:34.847414 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02cb077-de4b-41cc-afeb-ea3b3bdb5728-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:34.847412 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:34.847424 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02cb077-de4b-41cc-afeb-ea3b3bdb5728-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:35.450409 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:35.450371 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a856579-v5qpb" event={"ID":"d02cb077-de4b-41cc-afeb-ea3b3bdb5728","Type":"ContainerDied","Data":"01629f75141c90bc6a2cd5157d7aae750e89c832f345f068e2167ea081feffc5"} Nov 03 20:15:35.450599 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:35.450423 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01629f75141c90bc6a2cd5157d7aae750e89c832f345f068e2167ea081feffc5" Nov 03 20:15:35.450599 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:35.450384 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a856579-v5qpb" Nov 03 20:15:38.447524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:38.447489 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-612e45b8-tq42s"] Nov 03 20:15:38.448331 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:38.448313 5059 cpu_manager.go:414] "containerMap: removing container" podUID="d02cb077-de4b-41cc-afeb-ea3b3bdb5728" containerName="revoke-keys" Nov 03 20:15:38.448377 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:38.448332 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d02cb077-de4b-41cc-afeb-ea3b3bdb5728" containerName="revoke-keys" Nov 03 20:15:38.449147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:38.448489 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d02cb077-de4b-41cc-afeb-ea3b3bdb5728" containerName="revoke-keys" Nov 03 20:15:38.453775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:38.453758 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-612e45b8-tq42s" Nov 03 20:15:38.457923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:38.457883 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:15:38.460144 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:38.459085 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:15:38.461414 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:38.461390 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-612e45b8-tq42s"] Nov 03 20:15:38.522126 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:38.522049 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8ccb587-5732-4719-8aa2-fa63bf33ed4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-612e45b8-tq42s\" (UID: \"e8ccb587-5732-4719-8aa2-fa63bf33ed4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-612e45b8-tq42s" Nov 03 20:15:38.522317 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:38.522135 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8ccb587-5732-4719-8aa2-fa63bf33ed4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-612e45b8-tq42s\" (UID: \"e8ccb587-5732-4719-8aa2-fa63bf33ed4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-612e45b8-tq42s" Nov 03 20:15:38.522461 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:38.522434 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlqj2\" (UniqueName: \"kubernetes.io/projected/e8ccb587-5732-4719-8aa2-fa63bf33ed4a-kube-api-access-rlqj2\") pod \"maas-api-revoke-sub-612e45b8-tq42s\" (UID: \"e8ccb587-5732-4719-8aa2-fa63bf33ed4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-612e45b8-tq42s" Nov 03 20:15:38.624244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:38.624204 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8ccb587-5732-4719-8aa2-fa63bf33ed4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-612e45b8-tq42s\" (UID: \"e8ccb587-5732-4719-8aa2-fa63bf33ed4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-612e45b8-tq42s" Nov 03 20:15:38.624417 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:38.624275 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8ccb587-5732-4719-8aa2-fa63bf33ed4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-612e45b8-tq42s\" (UID: \"e8ccb587-5732-4719-8aa2-fa63bf33ed4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-612e45b8-tq42s" Nov 03 20:15:38.624564 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:38.624545 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlqj2\" (UniqueName: \"kubernetes.io/projected/e8ccb587-5732-4719-8aa2-fa63bf33ed4a-kube-api-access-rlqj2\") pod \"maas-api-revoke-sub-612e45b8-tq42s\" (UID: \"e8ccb587-5732-4719-8aa2-fa63bf33ed4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-612e45b8-tq42s" Nov 03 20:15:38.624995 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:38.624971 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8ccb587-5732-4719-8aa2-fa63bf33ed4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-612e45b8-tq42s\" (UID: \"e8ccb587-5732-4719-8aa2-fa63bf33ed4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-612e45b8-tq42s" Nov 03 20:15:38.626634 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:38.626611 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8ccb587-5732-4719-8aa2-fa63bf33ed4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-612e45b8-tq42s\" (UID: \"e8ccb587-5732-4719-8aa2-fa63bf33ed4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-612e45b8-tq42s" Nov 03 20:15:38.636817 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:38.636795 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlqj2\" (UniqueName: \"kubernetes.io/projected/e8ccb587-5732-4719-8aa2-fa63bf33ed4a-kube-api-access-rlqj2\") pod \"maas-api-revoke-sub-612e45b8-tq42s\" (UID: \"e8ccb587-5732-4719-8aa2-fa63bf33ed4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-612e45b8-tq42s" Nov 03 20:15:38.776897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:38.776815 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-612e45b8-tq42s" Nov 03 20:15:39.097503 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:15:39.097471 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ccb587_5732_4719_8aa2_fa63bf33ed4a.slice/crio-0b29cdd9e2f4c13add7c55538931ab331aa29d00116440f10aa9e151d1d17a45 WatchSource:0}: Error finding container 0b29cdd9e2f4c13add7c55538931ab331aa29d00116440f10aa9e151d1d17a45: Status 404 returned error can't find the container with id 0b29cdd9e2f4c13add7c55538931ab331aa29d00116440f10aa9e151d1d17a45 Nov 03 20:15:39.098904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:39.098869 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-612e45b8-tq42s"] Nov 03 20:15:39.565047 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:39.565006 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-612e45b8-tq42s" event={"ID":"e8ccb587-5732-4719-8aa2-fa63bf33ed4a","Type":"ContainerStarted","Data":"038675880d305af3567685c554e2f182591d8a167a180e771df679b16d57b8d2"} Nov 03 20:15:39.565477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:39.565086 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-612e45b8-tq42s" event={"ID":"e8ccb587-5732-4719-8aa2-fa63bf33ed4a","Type":"ContainerStarted","Data":"0b29cdd9e2f4c13add7c55538931ab331aa29d00116440f10aa9e151d1d17a45"} Nov 03 20:15:39.588432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:39.588377 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-612e45b8-tq42s" podStartSLOduration=1.5883644879999999 podStartE2EDuration="1.588364488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:39.583950785 +0000 UTC m=+2333.841748043" watchObservedRunningTime="2026-11-03 20:15:39.588364488 +0000 UTC m=+2333.846161745" Nov 03 20:15:40.577806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:40.577768 5059 generic.go:356] "Generic (PLEG): container finished" podID="e8ccb587-5732-4719-8aa2-fa63bf33ed4a" containerID="038675880d305af3567685c554e2f182591d8a167a180e771df679b16d57b8d2" exitCode=0 Nov 03 20:15:40.578201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:40.577818 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-612e45b8-tq42s" event={"ID":"e8ccb587-5732-4719-8aa2-fa63bf33ed4a","Type":"ContainerDied","Data":"038675880d305af3567685c554e2f182591d8a167a180e771df679b16d57b8d2"} Nov 03 20:15:40.608687 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:15:40.608639 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:15:41.075514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:41.075478 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f17f5a8b-xmn59"] Nov 03 20:15:41.080272 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:41.080248 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f17f5a8b-xmn59" Nov 03 20:15:41.091737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:41.091708 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f17f5a8b-xmn59"] Nov 03 20:15:41.255653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:41.255610 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcn6n\" (UniqueName: \"kubernetes.io/projected/f16ace24-5836-44cf-a270-faae87720720-kube-api-access-zcn6n\") pod \"maas-api-revoke-sub-f17f5a8b-xmn59\" (UID: \"f16ace24-5836-44cf-a270-faae87720720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f17f5a8b-xmn59" Nov 03 20:15:41.256280 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:41.256245 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16ace24-5836-44cf-a270-faae87720720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f17f5a8b-xmn59\" (UID: \"f16ace24-5836-44cf-a270-faae87720720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f17f5a8b-xmn59" Nov 03 20:15:41.256608 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:41.256586 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16ace24-5836-44cf-a270-faae87720720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f17f5a8b-xmn59\" (UID: \"f16ace24-5836-44cf-a270-faae87720720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f17f5a8b-xmn59" Nov 03 20:15:41.358724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:41.358606 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcn6n\" (UniqueName: \"kubernetes.io/projected/f16ace24-5836-44cf-a270-faae87720720-kube-api-access-zcn6n\") pod \"maas-api-revoke-sub-f17f5a8b-xmn59\" (UID: \"f16ace24-5836-44cf-a270-faae87720720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f17f5a8b-xmn59" Nov 03 20:15:41.358877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:41.358792 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16ace24-5836-44cf-a270-faae87720720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f17f5a8b-xmn59\" (UID: \"f16ace24-5836-44cf-a270-faae87720720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f17f5a8b-xmn59" Nov 03 20:15:41.358930 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:41.358885 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16ace24-5836-44cf-a270-faae87720720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f17f5a8b-xmn59\" (UID: \"f16ace24-5836-44cf-a270-faae87720720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f17f5a8b-xmn59" Nov 03 20:15:41.359497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:41.359465 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16ace24-5836-44cf-a270-faae87720720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f17f5a8b-xmn59\" (UID: \"f16ace24-5836-44cf-a270-faae87720720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f17f5a8b-xmn59" Nov 03 20:15:41.361537 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:41.361510 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16ace24-5836-44cf-a270-faae87720720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f17f5a8b-xmn59\" (UID: \"f16ace24-5836-44cf-a270-faae87720720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f17f5a8b-xmn59" Nov 03 20:15:41.372037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:41.372015 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcn6n\" (UniqueName: \"kubernetes.io/projected/f16ace24-5836-44cf-a270-faae87720720-kube-api-access-zcn6n\") pod \"maas-api-revoke-sub-f17f5a8b-xmn59\" (UID: \"f16ace24-5836-44cf-a270-faae87720720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f17f5a8b-xmn59" Nov 03 20:15:41.396798 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:41.396752 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f17f5a8b-xmn59" Nov 03 20:15:41.708969 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:41.708938 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f17f5a8b-xmn59"] Nov 03 20:15:41.759476 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:41.759454 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-612e45b8-tq42s" Nov 03 20:15:41.867514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:41.866924 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8ccb587-5732-4719-8aa2-fa63bf33ed4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8ccb587-5732-4719-8aa2-fa63bf33ed4a-maas-api-openshift-ca\") pod \"e8ccb587-5732-4719-8aa2-fa63bf33ed4a\" (UID: \"e8ccb587-5732-4719-8aa2-fa63bf33ed4a\") " Nov 03 20:15:41.867514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:41.867028 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8ccb587-5732-4719-8aa2-fa63bf33ed4a-kube-api-access-rlqj2\" (UniqueName: \"kubernetes.io/projected/e8ccb587-5732-4719-8aa2-fa63bf33ed4a-kube-api-access-rlqj2\") pod \"e8ccb587-5732-4719-8aa2-fa63bf33ed4a\" (UID: \"e8ccb587-5732-4719-8aa2-fa63bf33ed4a\") " Nov 03 20:15:41.867514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:41.867161 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8ccb587-5732-4719-8aa2-fa63bf33ed4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8ccb587-5732-4719-8aa2-fa63bf33ed4a-maas-api-xks-ca\") pod \"e8ccb587-5732-4719-8aa2-fa63bf33ed4a\" (UID: \"e8ccb587-5732-4719-8aa2-fa63bf33ed4a\") " Nov 03 20:15:41.868975 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:41.867990 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ccb587-5732-4719-8aa2-fa63bf33ed4a-maas-api-openshift-ca" pod "e8ccb587-5732-4719-8aa2-fa63bf33ed4a" (UID: "e8ccb587-5732-4719-8aa2-fa63bf33ed4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:41.870248 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:41.870221 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ccb587-5732-4719-8aa2-fa63bf33ed4a-kube-api-access-rlqj2" pod "e8ccb587-5732-4719-8aa2-fa63bf33ed4a" (UID: "e8ccb587-5732-4719-8aa2-fa63bf33ed4a"). InnerVolumeSpecName "kube-api-access-rlqj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:41.870393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:41.870375 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ccb587-5732-4719-8aa2-fa63bf33ed4a-maas-api-xks-ca" pod "e8ccb587-5732-4719-8aa2-fa63bf33ed4a" (UID: "e8ccb587-5732-4719-8aa2-fa63bf33ed4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:42.007776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:41.970803 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8ccb587-5732-4719-8aa2-fa63bf33ed4a-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:42.007776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:41.970847 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlqj2\" (UniqueName: \"kubernetes.io/projected/e8ccb587-5732-4719-8aa2-fa63bf33ed4a-kube-api-access-rlqj2\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:42.007776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:41.970864 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8ccb587-5732-4719-8aa2-fa63bf33ed4a-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:42.596215 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:42.596177 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-612e45b8-tq42s" event={"ID":"e8ccb587-5732-4719-8aa2-fa63bf33ed4a","Type":"ContainerDied","Data":"0b29cdd9e2f4c13add7c55538931ab331aa29d00116440f10aa9e151d1d17a45"} Nov 03 20:15:42.596400 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:42.596239 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b29cdd9e2f4c13add7c55538931ab331aa29d00116440f10aa9e151d1d17a45" Nov 03 20:15:42.596400 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:42.596187 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-612e45b8-tq42s" Nov 03 20:15:42.598034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:42.598007 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f17f5a8b-xmn59" event={"ID":"f16ace24-5836-44cf-a270-faae87720720","Type":"ContainerStarted","Data":"d137e44245117ded1f273201b8436175e3e6bbccc5434833ab4b39c6cbafe2da"} Nov 03 20:15:42.598143 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:42.598057 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f17f5a8b-xmn59" event={"ID":"f16ace24-5836-44cf-a270-faae87720720","Type":"ContainerStarted","Data":"69f36063117b11224dfd22c37fe808b81d78542e2a5ad847979ce5823320a21c"} Nov 03 20:15:42.621454 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:42.621400 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f17f5a8b-xmn59" podStartSLOduration=1.621383061 podStartE2EDuration="1.621383061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:42.617401084 +0000 UTC m=+2336.875198342" watchObservedRunningTime="2026-11-03 20:15:42.621383061 +0000 UTC m=+2336.879180320" Nov 03 20:15:43.459693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:43.459011 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l" Nov 03 20:15:44.045476 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:44.045425 5059 generic.go:356] "Generic (PLEG): container finished" podID="f16ace24-5836-44cf-a270-faae87720720" containerID="d137e44245117ded1f273201b8436175e3e6bbccc5434833ab4b39c6cbafe2da" exitCode=0 Nov 03 20:15:44.045755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:44.045593 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f17f5a8b-xmn59" event={"ID":"f16ace24-5836-44cf-a270-faae87720720","Type":"ContainerDied","Data":"d137e44245117ded1f273201b8436175e3e6bbccc5434833ab4b39c6cbafe2da"} Nov 03 20:15:45.374135 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:45.374110 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f17f5a8b-xmn59" Nov 03 20:15:45.449750 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:45.442456 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f16ace24-5836-44cf-a270-faae87720720-kube-api-access-zcn6n\" (UniqueName: \"kubernetes.io/projected/f16ace24-5836-44cf-a270-faae87720720-kube-api-access-zcn6n\") pod \"f16ace24-5836-44cf-a270-faae87720720\" (UID: \"f16ace24-5836-44cf-a270-faae87720720\") " Nov 03 20:15:45.449750 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:45.442649 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f16ace24-5836-44cf-a270-faae87720720-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16ace24-5836-44cf-a270-faae87720720-maas-api-xks-ca\") pod \"f16ace24-5836-44cf-a270-faae87720720\" (UID: \"f16ace24-5836-44cf-a270-faae87720720\") " Nov 03 20:15:45.449750 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:45.442799 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f16ace24-5836-44cf-a270-faae87720720-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16ace24-5836-44cf-a270-faae87720720-maas-api-openshift-ca\") pod \"f16ace24-5836-44cf-a270-faae87720720\" (UID: \"f16ace24-5836-44cf-a270-faae87720720\") " Nov 03 20:15:45.480120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:45.474535 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16ace24-5836-44cf-a270-faae87720720-maas-api-openshift-ca" pod "f16ace24-5836-44cf-a270-faae87720720" (UID: "f16ace24-5836-44cf-a270-faae87720720"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:45.601047 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:45.590273 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16ace24-5836-44cf-a270-faae87720720-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:45.601907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:45.601846 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16ace24-5836-44cf-a270-faae87720720-kube-api-access-zcn6n" pod "f16ace24-5836-44cf-a270-faae87720720" (UID: "f16ace24-5836-44cf-a270-faae87720720"). InnerVolumeSpecName "kube-api-access-zcn6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:45.608681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:45.602227 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f16ace24-5836-44cf-a270-faae87720720-maas-api-xks-ca" pod "f16ace24-5836-44cf-a270-faae87720720" (UID: "f16ace24-5836-44cf-a270-faae87720720"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:45.706239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:45.704996 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16ace24-5836-44cf-a270-faae87720720-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:45.706239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:45.705032 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcn6n\" (UniqueName: \"kubernetes.io/projected/f16ace24-5836-44cf-a270-faae87720720-kube-api-access-zcn6n\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:46.196023 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:46.195993 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f17f5a8b-xmn59" Nov 03 20:15:46.196197 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:46.195992 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f17f5a8b-xmn59" event={"ID":"f16ace24-5836-44cf-a270-faae87720720","Type":"ContainerDied","Data":"69f36063117b11224dfd22c37fe808b81d78542e2a5ad847979ce5823320a21c"} Nov 03 20:15:46.196197 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:46.196138 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69f36063117b11224dfd22c37fe808b81d78542e2a5ad847979ce5823320a21c" Nov 03 20:15:54.802685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:54.783824 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8f82f8-sr4nk"] Nov 03 20:15:54.802685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:54.785145 5059 cpu_manager.go:414] "containerMap: removing container" podUID="e8ccb587-5732-4719-8aa2-fa63bf33ed4a" containerName="revoke-keys" Nov 03 20:15:54.802685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:54.785169 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8ccb587-5732-4719-8aa2-fa63bf33ed4a" containerName="revoke-keys" Nov 03 20:15:54.802685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:54.785223 5059 cpu_manager.go:414] "containerMap: removing container" podUID="f16ace24-5836-44cf-a270-faae87720720" containerName="revoke-keys" Nov 03 20:15:54.802685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:54.785231 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f16ace24-5836-44cf-a270-faae87720720" containerName="revoke-keys" Nov 03 20:15:54.802685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:54.785434 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8ccb587-5732-4719-8aa2-fa63bf33ed4a" containerName="revoke-keys" Nov 03 20:15:54.802685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:54.785454 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f16ace24-5836-44cf-a270-faae87720720" containerName="revoke-keys" Nov 03 20:15:54.977740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:54.861011 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8f82f8-sr4nk" Nov 03 20:15:54.978473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:54.953709 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8f82f8-sr4nk"] Nov 03 20:15:54.998698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:54.991952 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:15:54.998698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:54.992079 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:15:55.050998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:55.050959 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4767ab09-f06f-4090-a767-2032eafa40d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed8f82f8-sr4nk\" (UID: \"4767ab09-f06f-4090-a767-2032eafa40d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8f82f8-sr4nk" Nov 03 20:15:55.051373 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:55.051347 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lljb\" (UniqueName: \"kubernetes.io/projected/4767ab09-f06f-4090-a767-2032eafa40d2-kube-api-access-5lljb\") pod \"maas-api-revoke-sub-ed8f82f8-sr4nk\" (UID: \"4767ab09-f06f-4090-a767-2032eafa40d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8f82f8-sr4nk" Nov 03 20:15:55.051641 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:55.051608 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4767ab09-f06f-4090-a767-2032eafa40d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed8f82f8-sr4nk\" (UID: \"4767ab09-f06f-4090-a767-2032eafa40d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8f82f8-sr4nk" Nov 03 20:15:55.153552 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:55.153517 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4767ab09-f06f-4090-a767-2032eafa40d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed8f82f8-sr4nk\" (UID: \"4767ab09-f06f-4090-a767-2032eafa40d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8f82f8-sr4nk" Nov 03 20:15:55.153788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:55.153649 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lljb\" (UniqueName: \"kubernetes.io/projected/4767ab09-f06f-4090-a767-2032eafa40d2-kube-api-access-5lljb\") pod \"maas-api-revoke-sub-ed8f82f8-sr4nk\" (UID: \"4767ab09-f06f-4090-a767-2032eafa40d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8f82f8-sr4nk" Nov 03 20:15:55.153788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:55.153756 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4767ab09-f06f-4090-a767-2032eafa40d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed8f82f8-sr4nk\" (UID: \"4767ab09-f06f-4090-a767-2032eafa40d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8f82f8-sr4nk" Nov 03 20:15:55.154422 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:55.154402 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4767ab09-f06f-4090-a767-2032eafa40d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed8f82f8-sr4nk\" (UID: \"4767ab09-f06f-4090-a767-2032eafa40d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8f82f8-sr4nk" Nov 03 20:15:55.156045 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:55.156020 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4767ab09-f06f-4090-a767-2032eafa40d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed8f82f8-sr4nk\" (UID: \"4767ab09-f06f-4090-a767-2032eafa40d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8f82f8-sr4nk" Nov 03 20:15:55.165953 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:55.165931 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lljb\" (UniqueName: \"kubernetes.io/projected/4767ab09-f06f-4090-a767-2032eafa40d2-kube-api-access-5lljb\") pod \"maas-api-revoke-sub-ed8f82f8-sr4nk\" (UID: \"4767ab09-f06f-4090-a767-2032eafa40d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8f82f8-sr4nk" Nov 03 20:15:55.340011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:55.339975 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8f82f8-sr4nk" Nov 03 20:15:55.634341 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:15:55.634302 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4767ab09_f06f_4090_a767_2032eafa40d2.slice/crio-51e5900a340afc8891abbf9d67e05558e1a2dd6e44a298392e12124002508645 WatchSource:0}: Error finding container 51e5900a340afc8891abbf9d67e05558e1a2dd6e44a298392e12124002508645: Status 404 returned error can't find the container with id 51e5900a340afc8891abbf9d67e05558e1a2dd6e44a298392e12124002508645 Nov 03 20:15:55.635827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:55.635741 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8f82f8-sr4nk"] Nov 03 20:15:56.586122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:56.586080 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8f82f8-sr4nk" event={"ID":"4767ab09-f06f-4090-a767-2032eafa40d2","Type":"ContainerStarted","Data":"c8729ee43ddc112c927ac777a234b409d8fc809ee5610afce202321b3de775ce"} Nov 03 20:15:56.586493 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:56.586144 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8f82f8-sr4nk" event={"ID":"4767ab09-f06f-4090-a767-2032eafa40d2","Type":"ContainerStarted","Data":"51e5900a340afc8891abbf9d67e05558e1a2dd6e44a298392e12124002508645"} Nov 03 20:15:56.609163 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:56.609113 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8f82f8-sr4nk" podStartSLOduration=2.60909976 podStartE2EDuration="2.60909976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:56.604209384 +0000 UTC m=+2350.862006642" watchObservedRunningTime="2026-11-03 20:15:56.60909976 +0000 UTC m=+2350.866897016" Nov 03 20:15:57.182385 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:57.178972 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l"] Nov 03 20:15:57.706481 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:57.706446 5059 generic.go:356] "Generic (PLEG): container finished" podID="4767ab09-f06f-4090-a767-2032eafa40d2" containerID="c8729ee43ddc112c927ac777a234b409d8fc809ee5610afce202321b3de775ce" exitCode=0 Nov 03 20:15:57.706964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:57.706749 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8f82f8-sr4nk" event={"ID":"4767ab09-f06f-4090-a767-2032eafa40d2","Type":"ContainerDied","Data":"c8729ee43ddc112c927ac777a234b409d8fc809ee5610afce202321b3de775ce"} Nov 03 20:15:58.899886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:58.899857 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8f82f8-sr4nk" Nov 03 20:15:59.031017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:59.030940 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4767ab09-f06f-4090-a767-2032eafa40d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4767ab09-f06f-4090-a767-2032eafa40d2-maas-api-openshift-ca\") pod \"4767ab09-f06f-4090-a767-2032eafa40d2\" (UID: \"4767ab09-f06f-4090-a767-2032eafa40d2\") " Nov 03 20:15:59.031248 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:59.031235 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4767ab09-f06f-4090-a767-2032eafa40d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4767ab09-f06f-4090-a767-2032eafa40d2-maas-api-xks-ca\") pod \"4767ab09-f06f-4090-a767-2032eafa40d2\" (UID: \"4767ab09-f06f-4090-a767-2032eafa40d2\") " Nov 03 20:15:59.031304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:59.031246 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4767ab09-f06f-4090-a767-2032eafa40d2-maas-api-openshift-ca" pod "4767ab09-f06f-4090-a767-2032eafa40d2" (UID: "4767ab09-f06f-4090-a767-2032eafa40d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:59.031304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:59.031263 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4767ab09-f06f-4090-a767-2032eafa40d2-kube-api-access-5lljb\" (UniqueName: \"kubernetes.io/projected/4767ab09-f06f-4090-a767-2032eafa40d2-kube-api-access-5lljb\") pod \"4767ab09-f06f-4090-a767-2032eafa40d2\" (UID: \"4767ab09-f06f-4090-a767-2032eafa40d2\") " Nov 03 20:15:59.031946 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:59.031918 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4767ab09-f06f-4090-a767-2032eafa40d2-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:59.033387 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:59.033354 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4767ab09-f06f-4090-a767-2032eafa40d2-maas-api-xks-ca" pod "4767ab09-f06f-4090-a767-2032eafa40d2" (UID: "4767ab09-f06f-4090-a767-2032eafa40d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:59.033509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:59.033415 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4767ab09-f06f-4090-a767-2032eafa40d2-kube-api-access-5lljb" pod "4767ab09-f06f-4090-a767-2032eafa40d2" (UID: "4767ab09-f06f-4090-a767-2032eafa40d2"). InnerVolumeSpecName "kube-api-access-5lljb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:59.134877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:59.134837 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4767ab09-f06f-4090-a767-2032eafa40d2-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:59.134877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:59.134879 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lljb\" (UniqueName: \"kubernetes.io/projected/4767ab09-f06f-4090-a767-2032eafa40d2-kube-api-access-5lljb\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:59.724285 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:59.724248 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8f82f8-sr4nk" event={"ID":"4767ab09-f06f-4090-a767-2032eafa40d2","Type":"ContainerDied","Data":"51e5900a340afc8891abbf9d67e05558e1a2dd6e44a298392e12124002508645"} Nov 03 20:15:59.724462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:59.724332 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51e5900a340afc8891abbf9d67e05558e1a2dd6e44a298392e12124002508645" Nov 03 20:15:59.724462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:15:59.724262 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8f82f8-sr4nk" Nov 03 20:16:02.185689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:02.183804 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l" podUID="47a9960a-00a9-4038-9cc6-dea185172090" containerName="payload-processing" containerID="cri-o://6b4a4e4b2c1e3a23d85684882ad115e379761964e22132117b5bac24d69c0591" gracePeriod=25 Nov 03 20:16:02.543410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:02.543382 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l" Nov 03 20:16:02.614681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:02.614635 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a9960a-00a9-4038-9cc6-dea185172090-kube-api-access-bmd5t\" (UniqueName: \"kubernetes.io/projected/47a9960a-00a9-4038-9cc6-dea185172090-kube-api-access-bmd5t\") pod \"47a9960a-00a9-4038-9cc6-dea185172090\" (UID: \"47a9960a-00a9-4038-9cc6-dea185172090\") " Nov 03 20:16:02.614865 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:02.614824 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47a9960a-00a9-4038-9cc6-dea185172090-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a9960a-00a9-4038-9cc6-dea185172090-plugins-config-volume\") pod \"47a9960a-00a9-4038-9cc6-dea185172090\" (UID: \"47a9960a-00a9-4038-9cc6-dea185172090\") " Nov 03 20:16:02.614943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:02.614869 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47a9960a-00a9-4038-9cc6-dea185172090-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47a9960a-00a9-4038-9cc6-dea185172090-routing-overlay-volume\") pod \"47a9960a-00a9-4038-9cc6-dea185172090\" (UID: \"47a9960a-00a9-4038-9cc6-dea185172090\") " Nov 03 20:16:02.615111 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:02.615092 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a9960a-00a9-4038-9cc6-dea185172090-provider-credentials\" (UniqueName: \"kubernetes.io/projected/47a9960a-00a9-4038-9cc6-dea185172090-provider-credentials\") pod \"47a9960a-00a9-4038-9cc6-dea185172090\" (UID: \"47a9960a-00a9-4038-9cc6-dea185172090\") " Nov 03 20:16:02.615323 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:02.615274 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47a9960a-00a9-4038-9cc6-dea185172090-plugins-config-volume" pod "47a9960a-00a9-4038-9cc6-dea185172090" (UID: "47a9960a-00a9-4038-9cc6-dea185172090"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:02.615458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:02.615394 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47a9960a-00a9-4038-9cc6-dea185172090-routing-overlay-volume" pod "47a9960a-00a9-4038-9cc6-dea185172090" (UID: "47a9960a-00a9-4038-9cc6-dea185172090"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:02.616133 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:02.616104 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a9960a-00a9-4038-9cc6-dea185172090-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:02.616225 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:02.616138 5059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47a9960a-00a9-4038-9cc6-dea185172090-routing-overlay-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:02.617315 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:02.617292 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a9960a-00a9-4038-9cc6-dea185172090-kube-api-access-bmd5t" pod "47a9960a-00a9-4038-9cc6-dea185172090" (UID: "47a9960a-00a9-4038-9cc6-dea185172090"). InnerVolumeSpecName "kube-api-access-bmd5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:02.618215 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:02.618192 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a9960a-00a9-4038-9cc6-dea185172090-provider-credentials" pod "47a9960a-00a9-4038-9cc6-dea185172090" (UID: "47a9960a-00a9-4038-9cc6-dea185172090"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:02.717757 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:02.717725 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmd5t\" (UniqueName: \"kubernetes.io/projected/47a9960a-00a9-4038-9cc6-dea185172090-kube-api-access-bmd5t\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:02.717757 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:02.717752 5059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/47a9960a-00a9-4038-9cc6-dea185172090-provider-credentials\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:02.751358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:02.751328 5059 generic.go:356] "Generic (PLEG): container finished" podID="47a9960a-00a9-4038-9cc6-dea185172090" containerID="6b4a4e4b2c1e3a23d85684882ad115e379761964e22132117b5bac24d69c0591" exitCode=0 Nov 03 20:16:02.751478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:02.751373 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l" Nov 03 20:16:02.751478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:02.751405 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l" event={"ID":"47a9960a-00a9-4038-9cc6-dea185172090","Type":"ContainerDied","Data":"6b4a4e4b2c1e3a23d85684882ad115e379761964e22132117b5bac24d69c0591"} Nov 03 20:16:02.751478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:02.751459 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l" event={"ID":"47a9960a-00a9-4038-9cc6-dea185172090","Type":"ContainerDied","Data":"84af315a5ec2150f989587e072760c6441fe62a31bdd49b9b60eaed180470519"} Nov 03 20:16:02.751615 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:02.751480 5059 scope.go:122] "RemoveContainer" containerID="6b4a4e4b2c1e3a23d85684882ad115e379761964e22132117b5bac24d69c0591" Nov 03 20:16:02.764489 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:02.764469 5059 scope.go:122] "RemoveContainer" containerID="6b4a4e4b2c1e3a23d85684882ad115e379761964e22132117b5bac24d69c0591" Nov 03 20:16:02.764953 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:16:02.764884 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b4a4e4b2c1e3a23d85684882ad115e379761964e22132117b5bac24d69c0591\": container with ID starting with 6b4a4e4b2c1e3a23d85684882ad115e379761964e22132117b5bac24d69c0591 not found: ID does not exist" containerID="6b4a4e4b2c1e3a23d85684882ad115e379761964e22132117b5bac24d69c0591" Nov 03 20:16:02.764953 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:02.764919 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b4a4e4b2c1e3a23d85684882ad115e379761964e22132117b5bac24d69c0591"} err="failed to get container status \"6b4a4e4b2c1e3a23d85684882ad115e379761964e22132117b5bac24d69c0591\": rpc error: code = NotFound desc = could not find container \"6b4a4e4b2c1e3a23d85684882ad115e379761964e22132117b5bac24d69c0591\": container with ID starting with 6b4a4e4b2c1e3a23d85684882ad115e379761964e22132117b5bac24d69c0591 not found: ID does not exist" Nov 03 20:16:02.779960 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:02.779937 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l"] Nov 03 20:16:02.784832 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:02.784811 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d4gpk7l"] Nov 03 20:16:03.880496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:03.880456 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47a9960a-00a9-4038-9cc6-dea185172090" path="/var/lib/kubelet/pods/47a9960a-00a9-4038-9cc6-dea185172090/volumes" Nov 03 20:16:03.881006 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:03.880989 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8735403-hn7lg"] Nov 03 20:16:03.881768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:03.881748 5059 cpu_manager.go:414] "containerMap: removing container" podUID="4767ab09-f06f-4090-a767-2032eafa40d2" containerName="revoke-keys" Nov 03 20:16:03.881768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:03.881768 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4767ab09-f06f-4090-a767-2032eafa40d2" containerName="revoke-keys" Nov 03 20:16:03.881958 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:03.881791 5059 cpu_manager.go:414] "containerMap: removing container" podUID="47a9960a-00a9-4038-9cc6-dea185172090" containerName="payload-processing" Nov 03 20:16:03.881958 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:03.881799 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47a9960a-00a9-4038-9cc6-dea185172090" containerName="payload-processing" Nov 03 20:16:03.882090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:03.882075 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4767ab09-f06f-4090-a767-2032eafa40d2" containerName="revoke-keys" Nov 03 20:16:03.882166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:03.882096 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="47a9960a-00a9-4038-9cc6-dea185172090" containerName="payload-processing" Nov 03 20:16:03.887762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:03.887738 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8735403-hn7lg"] Nov 03 20:16:03.887879 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:03.887832 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8735403-hn7lg" Nov 03 20:16:03.891992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:03.891966 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:16:03.892879 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:03.892856 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:16:03.978562 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:03.978531 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d6c21d5-1599-440b-8342-1664fcc7594c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8735403-hn7lg\" (UID: \"6d6c21d5-1599-440b-8342-1664fcc7594c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8735403-hn7lg" Nov 03 20:16:03.979124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:03.979101 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d6c21d5-1599-440b-8342-1664fcc7594c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8735403-hn7lg\" (UID: \"6d6c21d5-1599-440b-8342-1664fcc7594c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8735403-hn7lg" Nov 03 20:16:03.979294 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:03.979273 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hm9p\" (UniqueName: \"kubernetes.io/projected/6d6c21d5-1599-440b-8342-1664fcc7594c-kube-api-access-7hm9p\") pod \"maas-api-revoke-sub-a8735403-hn7lg\" (UID: \"6d6c21d5-1599-440b-8342-1664fcc7594c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8735403-hn7lg" Nov 03 20:16:04.080712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:04.080644 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d6c21d5-1599-440b-8342-1664fcc7594c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8735403-hn7lg\" (UID: \"6d6c21d5-1599-440b-8342-1664fcc7594c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8735403-hn7lg" Nov 03 20:16:04.080852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:04.080742 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hm9p\" (UniqueName: \"kubernetes.io/projected/6d6c21d5-1599-440b-8342-1664fcc7594c-kube-api-access-7hm9p\") pod \"maas-api-revoke-sub-a8735403-hn7lg\" (UID: \"6d6c21d5-1599-440b-8342-1664fcc7594c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8735403-hn7lg" Nov 03 20:16:04.080852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:04.080768 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d6c21d5-1599-440b-8342-1664fcc7594c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8735403-hn7lg\" (UID: \"6d6c21d5-1599-440b-8342-1664fcc7594c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8735403-hn7lg" Nov 03 20:16:04.081472 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:04.081449 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d6c21d5-1599-440b-8342-1664fcc7594c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8735403-hn7lg\" (UID: \"6d6c21d5-1599-440b-8342-1664fcc7594c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8735403-hn7lg" Nov 03 20:16:04.083122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:04.083091 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d6c21d5-1599-440b-8342-1664fcc7594c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8735403-hn7lg\" (UID: \"6d6c21d5-1599-440b-8342-1664fcc7594c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8735403-hn7lg" Nov 03 20:16:04.093207 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:04.093182 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hm9p\" (UniqueName: \"kubernetes.io/projected/6d6c21d5-1599-440b-8342-1664fcc7594c-kube-api-access-7hm9p\") pod \"maas-api-revoke-sub-a8735403-hn7lg\" (UID: \"6d6c21d5-1599-440b-8342-1664fcc7594c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8735403-hn7lg" Nov 03 20:16:04.208954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:04.208883 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8735403-hn7lg" Nov 03 20:16:04.502838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:04.502774 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8735403-hn7lg"] Nov 03 20:16:04.502954 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:16:04.502879 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6c21d5_1599_440b_8342_1664fcc7594c.slice/crio-f3b7eb02156b4686eb999ac75587cbe4b8ab73278a0c07f95b2698b51d7f9b5b WatchSource:0}: Error finding container f3b7eb02156b4686eb999ac75587cbe4b8ab73278a0c07f95b2698b51d7f9b5b: Status 404 returned error can't find the container with id f3b7eb02156b4686eb999ac75587cbe4b8ab73278a0c07f95b2698b51d7f9b5b Nov 03 20:16:04.779337 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:04.779262 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8735403-hn7lg" event={"ID":"6d6c21d5-1599-440b-8342-1664fcc7594c","Type":"ContainerStarted","Data":"f3b7eb02156b4686eb999ac75587cbe4b8ab73278a0c07f95b2698b51d7f9b5b"} Nov 03 20:16:05.790204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:05.790105 5059 generic.go:356] "Generic (PLEG): container finished" podID="6d6c21d5-1599-440b-8342-1664fcc7594c" containerID="ea7d947176a43903416157e70e7189267dc836e12d224da3cf26ca6db31bddd6" exitCode=0 Nov 03 20:16:05.790204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:05.790145 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8735403-hn7lg" event={"ID":"6d6c21d5-1599-440b-8342-1664fcc7594c","Type":"ContainerDied","Data":"ea7d947176a43903416157e70e7189267dc836e12d224da3cf26ca6db31bddd6"} Nov 03 20:16:07.261910 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:07.261879 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8735403-hn7lg" Nov 03 20:16:07.425559 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:07.425509 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d6c21d5-1599-440b-8342-1664fcc7594c-kube-api-access-7hm9p\" (UniqueName: \"kubernetes.io/projected/6d6c21d5-1599-440b-8342-1664fcc7594c-kube-api-access-7hm9p\") pod \"6d6c21d5-1599-440b-8342-1664fcc7594c\" (UID: \"6d6c21d5-1599-440b-8342-1664fcc7594c\") " Nov 03 20:16:07.425955 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:07.425924 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d6c21d5-1599-440b-8342-1664fcc7594c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d6c21d5-1599-440b-8342-1664fcc7594c-maas-api-openshift-ca\") pod \"6d6c21d5-1599-440b-8342-1664fcc7594c\" (UID: \"6d6c21d5-1599-440b-8342-1664fcc7594c\") " Nov 03 20:16:07.426459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:07.426435 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d6c21d5-1599-440b-8342-1664fcc7594c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d6c21d5-1599-440b-8342-1664fcc7594c-maas-api-xks-ca\") pod \"6d6c21d5-1599-440b-8342-1664fcc7594c\" (UID: \"6d6c21d5-1599-440b-8342-1664fcc7594c\") " Nov 03 20:16:07.433059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:07.433035 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d6c21d5-1599-440b-8342-1664fcc7594c-maas-api-openshift-ca" pod "6d6c21d5-1599-440b-8342-1664fcc7594c" (UID: "6d6c21d5-1599-440b-8342-1664fcc7594c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:07.466924 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:07.466849 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d6c21d5-1599-440b-8342-1664fcc7594c-maas-api-xks-ca" pod "6d6c21d5-1599-440b-8342-1664fcc7594c" (UID: "6d6c21d5-1599-440b-8342-1664fcc7594c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:07.484836 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:07.484756 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d6c21d5-1599-440b-8342-1664fcc7594c-kube-api-access-7hm9p" pod "6d6c21d5-1599-440b-8342-1664fcc7594c" (UID: "6d6c21d5-1599-440b-8342-1664fcc7594c"). InnerVolumeSpecName "kube-api-access-7hm9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:07.575681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:07.550297 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d6c21d5-1599-440b-8342-1664fcc7594c-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:07.575681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:07.550342 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hm9p\" (UniqueName: \"kubernetes.io/projected/6d6c21d5-1599-440b-8342-1664fcc7594c-kube-api-access-7hm9p\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:07.575681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:07.550359 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d6c21d5-1599-440b-8342-1664fcc7594c-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:08.269741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:08.269706 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8735403-hn7lg" Nov 03 20:16:08.342840 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:08.342804 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8735403-hn7lg" event={"ID":"6d6c21d5-1599-440b-8342-1664fcc7594c","Type":"ContainerDied","Data":"f3b7eb02156b4686eb999ac75587cbe4b8ab73278a0c07f95b2698b51d7f9b5b"} Nov 03 20:16:08.343760 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:08.343733 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3b7eb02156b4686eb999ac75587cbe4b8ab73278a0c07f95b2698b51d7f9b5b" Nov 03 20:16:10.606930 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:16:10.606895 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:16:14.734075 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:14.733975 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1830c39-6vrr7"] Nov 03 20:16:14.737866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:14.737798 5059 cpu_manager.go:414] "containerMap: removing container" podUID="6d6c21d5-1599-440b-8342-1664fcc7594c" containerName="revoke-keys" Nov 03 20:16:14.737866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:14.737824 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d6c21d5-1599-440b-8342-1664fcc7594c" containerName="revoke-keys" Nov 03 20:16:14.739211 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:14.738327 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d6c21d5-1599-440b-8342-1664fcc7594c" containerName="revoke-keys" Nov 03 20:16:14.782220 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:14.782188 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1830c39-6vrr7" Nov 03 20:16:14.808693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:14.796885 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:16:14.816738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:14.809219 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:16:14.908186 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:14.908153 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb5bg\" (UniqueName: \"kubernetes.io/projected/da37c287-ec4f-47c5-8319-1891003a6b89-kube-api-access-tb5bg\") pod \"maas-api-revoke-sub-c1830c39-6vrr7\" (UID: \"da37c287-ec4f-47c5-8319-1891003a6b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1830c39-6vrr7" Nov 03 20:16:14.908559 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:14.908533 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da37c287-ec4f-47c5-8319-1891003a6b89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1830c39-6vrr7\" (UID: \"da37c287-ec4f-47c5-8319-1891003a6b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1830c39-6vrr7" Nov 03 20:16:14.908887 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:14.908864 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da37c287-ec4f-47c5-8319-1891003a6b89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1830c39-6vrr7\" (UID: \"da37c287-ec4f-47c5-8319-1891003a6b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1830c39-6vrr7" Nov 03 20:16:14.927958 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:14.927464 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1830c39-6vrr7"] Nov 03 20:16:15.011154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:15.011057 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb5bg\" (UniqueName: \"kubernetes.io/projected/da37c287-ec4f-47c5-8319-1891003a6b89-kube-api-access-tb5bg\") pod \"maas-api-revoke-sub-c1830c39-6vrr7\" (UID: \"da37c287-ec4f-47c5-8319-1891003a6b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1830c39-6vrr7" Nov 03 20:16:15.025686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:15.022793 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da37c287-ec4f-47c5-8319-1891003a6b89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1830c39-6vrr7\" (UID: \"da37c287-ec4f-47c5-8319-1891003a6b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1830c39-6vrr7" Nov 03 20:16:15.025686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:15.022870 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da37c287-ec4f-47c5-8319-1891003a6b89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1830c39-6vrr7\" (UID: \"da37c287-ec4f-47c5-8319-1891003a6b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1830c39-6vrr7" Nov 03 20:16:15.025686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:15.023813 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da37c287-ec4f-47c5-8319-1891003a6b89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1830c39-6vrr7\" (UID: \"da37c287-ec4f-47c5-8319-1891003a6b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1830c39-6vrr7" Nov 03 20:16:15.116479 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:15.116430 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb5bg\" (UniqueName: \"kubernetes.io/projected/da37c287-ec4f-47c5-8319-1891003a6b89-kube-api-access-tb5bg\") pod \"maas-api-revoke-sub-c1830c39-6vrr7\" (UID: \"da37c287-ec4f-47c5-8319-1891003a6b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1830c39-6vrr7" Nov 03 20:16:15.117076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:15.117050 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da37c287-ec4f-47c5-8319-1891003a6b89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1830c39-6vrr7\" (UID: \"da37c287-ec4f-47c5-8319-1891003a6b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1830c39-6vrr7" Nov 03 20:16:15.252462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:15.252422 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1830c39-6vrr7" Nov 03 20:16:16.145406 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:16.142584 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:16.648678 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:16.647398 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:16.954842 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:16.947786 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:17.042112 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:17.042063 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:17.350017 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:16:17.349971 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda37c287_ec4f_47c5_8319_1891003a6b89.slice/crio-e659f7c8efc45bc86242a50e65d96a106664ffd1bc2f67a3eb6b4f3ffb2265fd WatchSource:0}: Error finding container e659f7c8efc45bc86242a50e65d96a106664ffd1bc2f67a3eb6b4f3ffb2265fd: Status 404 returned error can't find the container with id e659f7c8efc45bc86242a50e65d96a106664ffd1bc2f67a3eb6b4f3ffb2265fd Nov 03 20:16:17.364274 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:17.364238 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1830c39-6vrr7"] Nov 03 20:16:18.221318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:18.221257 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1830c39-6vrr7" event={"ID":"da37c287-ec4f-47c5-8319-1891003a6b89","Type":"ContainerStarted","Data":"3441d537f563b84dc27ed266da6bc868378f8c7da2dc25adc1c8b82b015cbcd8"} Nov 03 20:16:18.227455 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:18.227412 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1830c39-6vrr7" event={"ID":"da37c287-ec4f-47c5-8319-1891003a6b89","Type":"ContainerStarted","Data":"e659f7c8efc45bc86242a50e65d96a106664ffd1bc2f67a3eb6b4f3ffb2265fd"} Nov 03 20:16:18.321163 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:18.320985 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1830c39-6vrr7" podStartSLOduration=4.3209654650000004 podStartE2EDuration="4.320965465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:18.319554152 +0000 UTC m=+2372.577351411" watchObservedRunningTime="2026-11-03 20:16:18.320965465 +0000 UTC m=+2372.578762723" Nov 03 20:16:19.273707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:19.273645 5059 generic.go:356] "Generic (PLEG): container finished" podID="da37c287-ec4f-47c5-8319-1891003a6b89" containerID="3441d537f563b84dc27ed266da6bc868378f8c7da2dc25adc1c8b82b015cbcd8" exitCode=0 Nov 03 20:16:19.274834 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:19.274805 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1830c39-6vrr7" event={"ID":"da37c287-ec4f-47c5-8319-1891003a6b89","Type":"ContainerDied","Data":"3441d537f563b84dc27ed266da6bc868378f8c7da2dc25adc1c8b82b015cbcd8"} Nov 03 20:16:19.428846 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:19.428793 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vznnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:19.429048 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:19.428866 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" podUID="171c7288-3157-4fe0-80ee-94cf5b2ca2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:20.486485 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:20.486450 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1830c39-6vrr7" Nov 03 20:16:20.614191 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:20.614138 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da37c287-ec4f-47c5-8319-1891003a6b89-kube-api-access-tb5bg\" (UniqueName: \"kubernetes.io/projected/da37c287-ec4f-47c5-8319-1891003a6b89-kube-api-access-tb5bg\") pod \"da37c287-ec4f-47c5-8319-1891003a6b89\" (UID: \"da37c287-ec4f-47c5-8319-1891003a6b89\") " Nov 03 20:16:20.614408 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:20.614346 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da37c287-ec4f-47c5-8319-1891003a6b89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da37c287-ec4f-47c5-8319-1891003a6b89-maas-api-xks-ca\") pod \"da37c287-ec4f-47c5-8319-1891003a6b89\" (UID: \"da37c287-ec4f-47c5-8319-1891003a6b89\") " Nov 03 20:16:20.614408 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:20.614383 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da37c287-ec4f-47c5-8319-1891003a6b89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da37c287-ec4f-47c5-8319-1891003a6b89-maas-api-openshift-ca\") pod \"da37c287-ec4f-47c5-8319-1891003a6b89\" (UID: \"da37c287-ec4f-47c5-8319-1891003a6b89\") " Nov 03 20:16:20.614837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:20.614809 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da37c287-ec4f-47c5-8319-1891003a6b89-maas-api-openshift-ca" pod "da37c287-ec4f-47c5-8319-1891003a6b89" (UID: "da37c287-ec4f-47c5-8319-1891003a6b89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:20.615311 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:20.615285 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da37c287-ec4f-47c5-8319-1891003a6b89-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:20.616931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:20.616906 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da37c287-ec4f-47c5-8319-1891003a6b89-kube-api-access-tb5bg" pod "da37c287-ec4f-47c5-8319-1891003a6b89" (UID: "da37c287-ec4f-47c5-8319-1891003a6b89"). InnerVolumeSpecName "kube-api-access-tb5bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:20.617099 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:20.617072 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da37c287-ec4f-47c5-8319-1891003a6b89-maas-api-xks-ca" pod "da37c287-ec4f-47c5-8319-1891003a6b89" (UID: "da37c287-ec4f-47c5-8319-1891003a6b89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:20.716404 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:20.716370 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb5bg\" (UniqueName: \"kubernetes.io/projected/da37c287-ec4f-47c5-8319-1891003a6b89-kube-api-access-tb5bg\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:20.716404 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:20.716402 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da37c287-ec4f-47c5-8319-1891003a6b89-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:21.018523 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:21.018434 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3777cca-rq2wv"] Nov 03 20:16:21.019360 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:21.019335 5059 cpu_manager.go:414] "containerMap: removing container" podUID="da37c287-ec4f-47c5-8319-1891003a6b89" containerName="revoke-keys" Nov 03 20:16:21.019488 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:21.019360 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da37c287-ec4f-47c5-8319-1891003a6b89" containerName="revoke-keys" Nov 03 20:16:21.019607 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:21.019592 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="da37c287-ec4f-47c5-8319-1891003a6b89" containerName="revoke-keys" Nov 03 20:16:21.023821 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:21.023798 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3777cca-rq2wv" Nov 03 20:16:21.034578 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:21.034550 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3777cca-rq2wv"] Nov 03 20:16:21.142391 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:21.142257 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w264g\" (UniqueName: \"kubernetes.io/projected/aafca12e-5345-4985-b452-0d661e655802-kube-api-access-w264g\") pod \"maas-api-revoke-sub-e3777cca-rq2wv\" (UID: \"aafca12e-5345-4985-b452-0d661e655802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3777cca-rq2wv" Nov 03 20:16:21.142391 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:21.142323 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aafca12e-5345-4985-b452-0d661e655802-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3777cca-rq2wv\" (UID: \"aafca12e-5345-4985-b452-0d661e655802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3777cca-rq2wv" Nov 03 20:16:21.142391 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:21.142350 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aafca12e-5345-4985-b452-0d661e655802-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3777cca-rq2wv\" (UID: \"aafca12e-5345-4985-b452-0d661e655802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3777cca-rq2wv" Nov 03 20:16:21.244676 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:21.244639 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aafca12e-5345-4985-b452-0d661e655802-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3777cca-rq2wv\" (UID: \"aafca12e-5345-4985-b452-0d661e655802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3777cca-rq2wv" Nov 03 20:16:21.245538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:21.244694 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aafca12e-5345-4985-b452-0d661e655802-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3777cca-rq2wv\" (UID: \"aafca12e-5345-4985-b452-0d661e655802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3777cca-rq2wv" Nov 03 20:16:21.245538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:21.245168 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w264g\" (UniqueName: \"kubernetes.io/projected/aafca12e-5345-4985-b452-0d661e655802-kube-api-access-w264g\") pod \"maas-api-revoke-sub-e3777cca-rq2wv\" (UID: \"aafca12e-5345-4985-b452-0d661e655802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3777cca-rq2wv" Nov 03 20:16:21.245538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:21.245475 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aafca12e-5345-4985-b452-0d661e655802-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3777cca-rq2wv\" (UID: \"aafca12e-5345-4985-b452-0d661e655802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3777cca-rq2wv" Nov 03 20:16:21.247933 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:21.247905 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aafca12e-5345-4985-b452-0d661e655802-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3777cca-rq2wv\" (UID: \"aafca12e-5345-4985-b452-0d661e655802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3777cca-rq2wv" Nov 03 20:16:21.257635 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:21.257611 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w264g\" (UniqueName: \"kubernetes.io/projected/aafca12e-5345-4985-b452-0d661e655802-kube-api-access-w264g\") pod \"maas-api-revoke-sub-e3777cca-rq2wv\" (UID: \"aafca12e-5345-4985-b452-0d661e655802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3777cca-rq2wv" Nov 03 20:16:21.309107 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:21.309028 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1830c39-6vrr7" event={"ID":"da37c287-ec4f-47c5-8319-1891003a6b89","Type":"ContainerDied","Data":"e659f7c8efc45bc86242a50e65d96a106664ffd1bc2f67a3eb6b4f3ffb2265fd"} Nov 03 20:16:21.309257 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:21.309118 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e659f7c8efc45bc86242a50e65d96a106664ffd1bc2f67a3eb6b4f3ffb2265fd" Nov 03 20:16:21.309257 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:21.309194 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1830c39-6vrr7" Nov 03 20:16:21.345733 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:21.345698 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3777cca-rq2wv" Nov 03 20:16:21.646444 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:16:21.642393 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaafca12e_5345_4985_b452_0d661e655802.slice/crio-ea707f6693e97e872409473502f39ef4c471f5c3abc760ae9b457c3c83a6996f WatchSource:0}: Error finding container ea707f6693e97e872409473502f39ef4c471f5c3abc760ae9b457c3c83a6996f: Status 404 returned error can't find the container with id ea707f6693e97e872409473502f39ef4c471f5c3abc760ae9b457c3c83a6996f Nov 03 20:16:21.646444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:21.644394 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3777cca-rq2wv"] Nov 03 20:16:22.326027 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:22.325992 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3777cca-rq2wv" event={"ID":"aafca12e-5345-4985-b452-0d661e655802","Type":"ContainerStarted","Data":"275d675106b85b74d23e458c800730f3bbecf7283214b15237e01a1bf1419cbf"} Nov 03 20:16:22.326222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:22.326047 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3777cca-rq2wv" event={"ID":"aafca12e-5345-4985-b452-0d661e655802","Type":"ContainerStarted","Data":"ea707f6693e97e872409473502f39ef4c471f5c3abc760ae9b457c3c83a6996f"} Nov 03 20:16:22.346354 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:22.346303 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3777cca-rq2wv" podStartSLOduration=2.346286153 podStartE2EDuration="2.346286153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:22.346192498 +0000 UTC m=+2376.603989757" watchObservedRunningTime="2026-11-03 20:16:22.346286153 +0000 UTC m=+2376.604083407" Nov 03 20:16:23.396654 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:23.396601 5059 generic.go:356] "Generic (PLEG): container finished" podID="aafca12e-5345-4985-b452-0d661e655802" containerID="275d675106b85b74d23e458c800730f3bbecf7283214b15237e01a1bf1419cbf" exitCode=0 Nov 03 20:16:23.398098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:23.398062 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3777cca-rq2wv" event={"ID":"aafca12e-5345-4985-b452-0d661e655802","Type":"ContainerDied","Data":"275d675106b85b74d23e458c800730f3bbecf7283214b15237e01a1bf1419cbf"} Nov 03 20:16:24.595034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:24.595006 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3777cca-rq2wv" Nov 03 20:16:24.695187 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:24.695095 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aafca12e-5345-4985-b452-0d661e655802-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aafca12e-5345-4985-b452-0d661e655802-maas-api-openshift-ca\") pod \"aafca12e-5345-4985-b452-0d661e655802\" (UID: \"aafca12e-5345-4985-b452-0d661e655802\") " Nov 03 20:16:24.695367 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:24.695231 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aafca12e-5345-4985-b452-0d661e655802-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aafca12e-5345-4985-b452-0d661e655802-maas-api-xks-ca\") pod \"aafca12e-5345-4985-b452-0d661e655802\" (UID: \"aafca12e-5345-4985-b452-0d661e655802\") " Nov 03 20:16:24.695506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:24.695482 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aafca12e-5345-4985-b452-0d661e655802-maas-api-openshift-ca" pod "aafca12e-5345-4985-b452-0d661e655802" (UID: "aafca12e-5345-4985-b452-0d661e655802"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:24.695571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:24.695495 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aafca12e-5345-4985-b452-0d661e655802-kube-api-access-w264g\" (UniqueName: \"kubernetes.io/projected/aafca12e-5345-4985-b452-0d661e655802-kube-api-access-w264g\") pod \"aafca12e-5345-4985-b452-0d661e655802\" (UID: \"aafca12e-5345-4985-b452-0d661e655802\") " Nov 03 20:16:24.696636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:24.696605 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aafca12e-5345-4985-b452-0d661e655802-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:24.698098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:24.698071 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aafca12e-5345-4985-b452-0d661e655802-kube-api-access-w264g" pod "aafca12e-5345-4985-b452-0d661e655802" (UID: "aafca12e-5345-4985-b452-0d661e655802"). InnerVolumeSpecName "kube-api-access-w264g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:24.702306 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:24.702277 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aafca12e-5345-4985-b452-0d661e655802-maas-api-xks-ca" pod "aafca12e-5345-4985-b452-0d661e655802" (UID: "aafca12e-5345-4985-b452-0d661e655802"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:24.798533 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:24.798493 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aafca12e-5345-4985-b452-0d661e655802-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:24.798533 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:24.798522 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w264g\" (UniqueName: \"kubernetes.io/projected/aafca12e-5345-4985-b452-0d661e655802-kube-api-access-w264g\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:25.417754 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:25.417722 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3777cca-rq2wv" Nov 03 20:16:25.417960 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:25.417722 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3777cca-rq2wv" event={"ID":"aafca12e-5345-4985-b452-0d661e655802","Type":"ContainerDied","Data":"ea707f6693e97e872409473502f39ef4c471f5c3abc760ae9b457c3c83a6996f"} Nov 03 20:16:25.417960 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:25.417883 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea707f6693e97e872409473502f39ef4c471f5c3abc760ae9b457c3c83a6996f" Nov 03 20:16:25.657751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:25.657712 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93391ed7-gc8c4"] Nov 03 20:16:25.659246 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:25.659218 5059 cpu_manager.go:414] "containerMap: removing container" podUID="aafca12e-5345-4985-b452-0d661e655802" containerName="revoke-keys" Nov 03 20:16:25.659246 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:25.659244 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aafca12e-5345-4985-b452-0d661e655802" containerName="revoke-keys" Nov 03 20:16:25.659583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:25.659564 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="aafca12e-5345-4985-b452-0d661e655802" containerName="revoke-keys" Nov 03 20:16:25.663851 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:25.663830 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93391ed7-gc8c4" Nov 03 20:16:25.668475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:25.668417 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:16:25.669095 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:25.669074 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:16:25.671206 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:25.671186 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93391ed7-gc8c4"] Nov 03 20:16:25.720086 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:25.719996 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0323f1ca-e8f8-4015-993d-c2b2229c485b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93391ed7-gc8c4\" (UID: \"0323f1ca-e8f8-4015-993d-c2b2229c485b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93391ed7-gc8c4" Nov 03 20:16:25.720318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:25.720259 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0323f1ca-e8f8-4015-993d-c2b2229c485b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93391ed7-gc8c4\" (UID: \"0323f1ca-e8f8-4015-993d-c2b2229c485b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93391ed7-gc8c4" Nov 03 20:16:25.720387 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:25.720330 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6x6s\" (UniqueName: \"kubernetes.io/projected/0323f1ca-e8f8-4015-993d-c2b2229c485b-kube-api-access-s6x6s\") pod \"maas-api-revoke-sub-93391ed7-gc8c4\" (UID: \"0323f1ca-e8f8-4015-993d-c2b2229c485b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93391ed7-gc8c4" Nov 03 20:16:25.821738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:25.821706 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0323f1ca-e8f8-4015-993d-c2b2229c485b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93391ed7-gc8c4\" (UID: \"0323f1ca-e8f8-4015-993d-c2b2229c485b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93391ed7-gc8c4" Nov 03 20:16:25.821911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:25.821781 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6x6s\" (UniqueName: \"kubernetes.io/projected/0323f1ca-e8f8-4015-993d-c2b2229c485b-kube-api-access-s6x6s\") pod \"maas-api-revoke-sub-93391ed7-gc8c4\" (UID: \"0323f1ca-e8f8-4015-993d-c2b2229c485b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93391ed7-gc8c4" Nov 03 20:16:25.822029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:25.822009 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0323f1ca-e8f8-4015-993d-c2b2229c485b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93391ed7-gc8c4\" (UID: \"0323f1ca-e8f8-4015-993d-c2b2229c485b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93391ed7-gc8c4" Nov 03 20:16:25.822342 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:25.822321 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0323f1ca-e8f8-4015-993d-c2b2229c485b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93391ed7-gc8c4\" (UID: \"0323f1ca-e8f8-4015-993d-c2b2229c485b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93391ed7-gc8c4" Nov 03 20:16:25.824519 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:25.824497 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0323f1ca-e8f8-4015-993d-c2b2229c485b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93391ed7-gc8c4\" (UID: \"0323f1ca-e8f8-4015-993d-c2b2229c485b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93391ed7-gc8c4" Nov 03 20:16:25.867034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:25.867003 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6x6s\" (UniqueName: \"kubernetes.io/projected/0323f1ca-e8f8-4015-993d-c2b2229c485b-kube-api-access-s6x6s\") pod \"maas-api-revoke-sub-93391ed7-gc8c4\" (UID: \"0323f1ca-e8f8-4015-993d-c2b2229c485b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93391ed7-gc8c4" Nov 03 20:16:25.985273 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:25.985210 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93391ed7-gc8c4" Nov 03 20:16:26.287600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:26.287568 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93391ed7-gc8c4"] Nov 03 20:16:26.287934 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:16:26.287911 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0323f1ca_e8f8_4015_993d_c2b2229c485b.slice/crio-d480f92bc28dce08fdd09160b4746bfbbaf80325041200c3e079bdc8961fa042 WatchSource:0}: Error finding container d480f92bc28dce08fdd09160b4746bfbbaf80325041200c3e079bdc8961fa042: Status 404 returned error can't find the container with id d480f92bc28dce08fdd09160b4746bfbbaf80325041200c3e079bdc8961fa042 Nov 03 20:16:26.427929 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:26.427900 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93391ed7-gc8c4" event={"ID":"0323f1ca-e8f8-4015-993d-c2b2229c485b","Type":"ContainerStarted","Data":"d480f92bc28dce08fdd09160b4746bfbbaf80325041200c3e079bdc8961fa042"} Nov 03 20:16:27.537685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:27.513148 5059 generic.go:356] "Generic (PLEG): container finished" podID="0323f1ca-e8f8-4015-993d-c2b2229c485b" containerID="9342254650cedc9bcaea3549b02ad08e1cbc29ba8f7299a5e40b49daba6753ae" exitCode=0 Nov 03 20:16:27.537685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:27.513365 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93391ed7-gc8c4" event={"ID":"0323f1ca-e8f8-4015-993d-c2b2229c485b","Type":"ContainerDied","Data":"9342254650cedc9bcaea3549b02ad08e1cbc29ba8f7299a5e40b49daba6753ae"} Nov 03 20:16:28.712854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:28.712825 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93391ed7-gc8c4" Nov 03 20:16:28.734804 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:28.734776 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0323f1ca-e8f8-4015-993d-c2b2229c485b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0323f1ca-e8f8-4015-993d-c2b2229c485b-maas-api-xks-ca\") pod \"0323f1ca-e8f8-4015-993d-c2b2229c485b\" (UID: \"0323f1ca-e8f8-4015-993d-c2b2229c485b\") " Nov 03 20:16:28.734964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:28.734830 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0323f1ca-e8f8-4015-993d-c2b2229c485b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0323f1ca-e8f8-4015-993d-c2b2229c485b-maas-api-openshift-ca\") pod \"0323f1ca-e8f8-4015-993d-c2b2229c485b\" (UID: \"0323f1ca-e8f8-4015-993d-c2b2229c485b\") " Nov 03 20:16:28.735235 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:28.735206 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0323f1ca-e8f8-4015-993d-c2b2229c485b-kube-api-access-s6x6s\" (UniqueName: \"kubernetes.io/projected/0323f1ca-e8f8-4015-993d-c2b2229c485b-kube-api-access-s6x6s\") pod \"0323f1ca-e8f8-4015-993d-c2b2229c485b\" (UID: \"0323f1ca-e8f8-4015-993d-c2b2229c485b\") " Nov 03 20:16:28.735931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:28.735849 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0323f1ca-e8f8-4015-993d-c2b2229c485b-maas-api-openshift-ca" pod "0323f1ca-e8f8-4015-993d-c2b2229c485b" (UID: "0323f1ca-e8f8-4015-993d-c2b2229c485b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:28.736189 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:28.736062 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0323f1ca-e8f8-4015-993d-c2b2229c485b-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:28.738020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:28.737994 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0323f1ca-e8f8-4015-993d-c2b2229c485b-maas-api-xks-ca" pod "0323f1ca-e8f8-4015-993d-c2b2229c485b" (UID: "0323f1ca-e8f8-4015-993d-c2b2229c485b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:28.738182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:28.738147 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0323f1ca-e8f8-4015-993d-c2b2229c485b-kube-api-access-s6x6s" pod "0323f1ca-e8f8-4015-993d-c2b2229c485b" (UID: "0323f1ca-e8f8-4015-993d-c2b2229c485b"). InnerVolumeSpecName "kube-api-access-s6x6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:28.837272 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:28.837241 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6x6s\" (UniqueName: \"kubernetes.io/projected/0323f1ca-e8f8-4015-993d-c2b2229c485b-kube-api-access-s6x6s\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:28.837272 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:28.837270 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0323f1ca-e8f8-4015-993d-c2b2229c485b-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:29.533273 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:29.533237 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93391ed7-gc8c4" event={"ID":"0323f1ca-e8f8-4015-993d-c2b2229c485b","Type":"ContainerDied","Data":"d480f92bc28dce08fdd09160b4746bfbbaf80325041200c3e079bdc8961fa042"} Nov 03 20:16:29.533273 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:29.533259 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93391ed7-gc8c4" Nov 03 20:16:29.533517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:29.533297 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d480f92bc28dce08fdd09160b4746bfbbaf80325041200c3e079bdc8961fa042" Nov 03 20:16:32.681683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:32.677772 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94176598-95bzq"] Nov 03 20:16:32.681683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:32.679168 5059 cpu_manager.go:414] "containerMap: removing container" podUID="0323f1ca-e8f8-4015-993d-c2b2229c485b" containerName="revoke-keys" Nov 03 20:16:32.681683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:32.679192 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0323f1ca-e8f8-4015-993d-c2b2229c485b" containerName="revoke-keys" Nov 03 20:16:32.681683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:32.679429 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0323f1ca-e8f8-4015-993d-c2b2229c485b" containerName="revoke-keys" Nov 03 20:16:32.768686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:32.767566 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94176598-95bzq" Nov 03 20:16:32.802827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:32.796083 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:16:32.808279 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:32.808252 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:16:32.989316 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:32.989273 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvkvx\" (UniqueName: \"kubernetes.io/projected/6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530-kube-api-access-jvkvx\") pod \"maas-api-revoke-sub-94176598-95bzq\" (UID: \"6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94176598-95bzq" Nov 03 20:16:32.989579 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:32.989565 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94176598-95bzq\" (UID: \"6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94176598-95bzq" Nov 03 20:16:32.989751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:32.989735 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94176598-95bzq\" (UID: \"6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94176598-95bzq" Nov 03 20:16:32.993616 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:32.993590 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94176598-95bzq"] Nov 03 20:16:33.091531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:33.091499 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94176598-95bzq\" (UID: \"6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94176598-95bzq" Nov 03 20:16:33.091707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:33.091549 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvkvx\" (UniqueName: \"kubernetes.io/projected/6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530-kube-api-access-jvkvx\") pod \"maas-api-revoke-sub-94176598-95bzq\" (UID: \"6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94176598-95bzq" Nov 03 20:16:33.091788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:33.091760 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94176598-95bzq\" (UID: \"6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94176598-95bzq" Nov 03 20:16:33.092160 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:33.092137 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94176598-95bzq\" (UID: \"6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94176598-95bzq" Nov 03 20:16:33.094074 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:33.094050 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94176598-95bzq\" (UID: \"6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94176598-95bzq" Nov 03 20:16:33.103869 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:33.103848 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvkvx\" (UniqueName: \"kubernetes.io/projected/6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530-kube-api-access-jvkvx\") pod \"maas-api-revoke-sub-94176598-95bzq\" (UID: \"6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94176598-95bzq" Nov 03 20:16:33.297483 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:33.297401 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94176598-95bzq" Nov 03 20:16:33.704738 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:16:33.704701 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6051fbf1_1d0b_4e44_af7f_ecf6b9dd3530.slice/crio-c6ad6d1f204da42a6dc4468a4e01231b63255e0ee6af32b4e15129b8d36d686e WatchSource:0}: Error finding container c6ad6d1f204da42a6dc4468a4e01231b63255e0ee6af32b4e15129b8d36d686e: Status 404 returned error can't find the container with id c6ad6d1f204da42a6dc4468a4e01231b63255e0ee6af32b4e15129b8d36d686e Nov 03 20:16:33.705565 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:33.705542 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94176598-95bzq"] Nov 03 20:16:34.003864 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:34.003833 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94176598-95bzq" event={"ID":"6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530","Type":"ContainerStarted","Data":"a9ea8abd23c9fc68664ed01ad3f064efcecd25de6bfa9eb3e95b9cef8938bcaa"} Nov 03 20:16:34.004031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:34.003892 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94176598-95bzq" event={"ID":"6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530","Type":"ContainerStarted","Data":"c6ad6d1f204da42a6dc4468a4e01231b63255e0ee6af32b4e15129b8d36d686e"} Nov 03 20:16:34.029941 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:34.029891 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94176598-95bzq" podStartSLOduration=2.029877792 podStartE2EDuration="2.029877792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:34.022445664 +0000 UTC m=+2388.280242938" watchObservedRunningTime="2026-11-03 20:16:34.029877792 +0000 UTC m=+2388.287675048" Nov 03 20:16:35.013987 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:35.013899 5059 generic.go:356] "Generic (PLEG): container finished" podID="6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530" containerID="a9ea8abd23c9fc68664ed01ad3f064efcecd25de6bfa9eb3e95b9cef8938bcaa" exitCode=0 Nov 03 20:16:35.013987 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:35.013972 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94176598-95bzq" event={"ID":"6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530","Type":"ContainerDied","Data":"a9ea8abd23c9fc68664ed01ad3f064efcecd25de6bfa9eb3e95b9cef8938bcaa"} Nov 03 20:16:36.237855 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.237828 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94176598-95bzq" Nov 03 20:16:36.399321 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.399283 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530-maas-api-xks-ca\") pod \"6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530\" (UID: \"6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530\") " Nov 03 20:16:36.399536 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.399350 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530-kube-api-access-jvkvx\" (UniqueName: \"kubernetes.io/projected/6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530-kube-api-access-jvkvx\") pod \"6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530\" (UID: \"6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530\") " Nov 03 20:16:36.399536 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.399467 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530-maas-api-openshift-ca\") pod \"6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530\" (UID: \"6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530\") " Nov 03 20:16:36.399841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.399816 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530-maas-api-openshift-ca" pod "6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530" (UID: "6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:36.400210 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.400192 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:36.401403 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.401380 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530-maas-api-xks-ca" pod "6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530" (UID: "6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:36.401603 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.401578 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530-kube-api-access-jvkvx" pod "6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530" (UID: "6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530"). InnerVolumeSpecName "kube-api-access-jvkvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:36.501517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.501487 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:36.501517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.501515 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvkvx\" (UniqueName: \"kubernetes.io/projected/6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530-kube-api-access-jvkvx\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:36.615115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.615079 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-65759456-sb7cs"] Nov 03 20:16:36.616076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.616050 5059 cpu_manager.go:414] "containerMap: removing container" podUID="6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530" containerName="revoke-keys" Nov 03 20:16:36.616076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.616074 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530" containerName="revoke-keys" Nov 03 20:16:36.616282 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.616268 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530" containerName="revoke-keys" Nov 03 20:16:36.620464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.620443 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65759456-sb7cs" Nov 03 20:16:36.626183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.626126 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 20:16:36.626320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.626226 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 20:16:36.626320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.626238 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-16659cd0d969\"" Nov 03 20:16:36.626506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.626356 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 20:16:36.626506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.626422 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-5lpr4\"" Nov 03 20:16:36.626873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.626851 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 20:16:36.634940 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.634914 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-65759456-sb7cs"] Nov 03 20:16:36.742743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.719086 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/737a2b7e-1e59-4f55-b680-34c1fa1977c2-routing-overlay-volume\") pod \"payload-processing-external-model-65759456-sb7cs\" (UID: \"737a2b7e-1e59-4f55-b680-34c1fa1977c2\") " pod="models-as-a-service/payload-processing-external-model-65759456-sb7cs" Nov 03 20:16:36.749990 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.749927 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/737a2b7e-1e59-4f55-b680-34c1fa1977c2-provider-credentials\") pod \"payload-processing-external-model-65759456-sb7cs\" (UID: \"737a2b7e-1e59-4f55-b680-34c1fa1977c2\") " pod="models-as-a-service/payload-processing-external-model-65759456-sb7cs" Nov 03 20:16:36.761846 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.761801 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl8np\" (UniqueName: \"kubernetes.io/projected/737a2b7e-1e59-4f55-b680-34c1fa1977c2-kube-api-access-zl8np\") pod \"payload-processing-external-model-65759456-sb7cs\" (UID: \"737a2b7e-1e59-4f55-b680-34c1fa1977c2\") " pod="models-as-a-service/payload-processing-external-model-65759456-sb7cs" Nov 03 20:16:36.762281 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.762247 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/737a2b7e-1e59-4f55-b680-34c1fa1977c2-plugins-config-volume\") pod \"payload-processing-external-model-65759456-sb7cs\" (UID: \"737a2b7e-1e59-4f55-b680-34c1fa1977c2\") " pod="models-as-a-service/payload-processing-external-model-65759456-sb7cs" Nov 03 20:16:36.864121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.864074 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/737a2b7e-1e59-4f55-b680-34c1fa1977c2-provider-credentials\") pod \"payload-processing-external-model-65759456-sb7cs\" (UID: \"737a2b7e-1e59-4f55-b680-34c1fa1977c2\") " pod="models-as-a-service/payload-processing-external-model-65759456-sb7cs" Nov 03 20:16:36.890810 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.890772 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl8np\" (UniqueName: \"kubernetes.io/projected/737a2b7e-1e59-4f55-b680-34c1fa1977c2-kube-api-access-zl8np\") pod \"payload-processing-external-model-65759456-sb7cs\" (UID: \"737a2b7e-1e59-4f55-b680-34c1fa1977c2\") " pod="models-as-a-service/payload-processing-external-model-65759456-sb7cs" Nov 03 20:16:36.891224 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.891195 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/737a2b7e-1e59-4f55-b680-34c1fa1977c2-plugins-config-volume\") pod \"payload-processing-external-model-65759456-sb7cs\" (UID: \"737a2b7e-1e59-4f55-b680-34c1fa1977c2\") " pod="models-as-a-service/payload-processing-external-model-65759456-sb7cs" Nov 03 20:16:36.891635 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.891615 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/737a2b7e-1e59-4f55-b680-34c1fa1977c2-routing-overlay-volume\") pod \"payload-processing-external-model-65759456-sb7cs\" (UID: \"737a2b7e-1e59-4f55-b680-34c1fa1977c2\") " pod="models-as-a-service/payload-processing-external-model-65759456-sb7cs" Nov 03 20:16:36.892719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.892693 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/737a2b7e-1e59-4f55-b680-34c1fa1977c2-routing-overlay-volume\") pod \"payload-processing-external-model-65759456-sb7cs\" (UID: \"737a2b7e-1e59-4f55-b680-34c1fa1977c2\") " pod="models-as-a-service/payload-processing-external-model-65759456-sb7cs" Nov 03 20:16:36.892877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.887415 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/737a2b7e-1e59-4f55-b680-34c1fa1977c2-provider-credentials\") pod \"payload-processing-external-model-65759456-sb7cs\" (UID: \"737a2b7e-1e59-4f55-b680-34c1fa1977c2\") " pod="models-as-a-service/payload-processing-external-model-65759456-sb7cs" Nov 03 20:16:36.894025 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.893986 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/737a2b7e-1e59-4f55-b680-34c1fa1977c2-plugins-config-volume\") pod \"payload-processing-external-model-65759456-sb7cs\" (UID: \"737a2b7e-1e59-4f55-b680-34c1fa1977c2\") " pod="models-as-a-service/payload-processing-external-model-65759456-sb7cs" Nov 03 20:16:36.911683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.908026 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl8np\" (UniqueName: \"kubernetes.io/projected/737a2b7e-1e59-4f55-b680-34c1fa1977c2-kube-api-access-zl8np\") pod \"payload-processing-external-model-65759456-sb7cs\" (UID: \"737a2b7e-1e59-4f55-b680-34c1fa1977c2\") " pod="models-as-a-service/payload-processing-external-model-65759456-sb7cs" Nov 03 20:16:36.944017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:36.941995 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65759456-sb7cs" Nov 03 20:16:37.318628 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:37.318433 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94176598-95bzq" Nov 03 20:16:37.327440 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:37.326573 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94176598-95bzq" event={"ID":"6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530","Type":"ContainerDied","Data":"c6ad6d1f204da42a6dc4468a4e01231b63255e0ee6af32b4e15129b8d36d686e"} Nov 03 20:16:37.329475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:37.329445 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6ad6d1f204da42a6dc4468a4e01231b63255e0ee6af32b4e15129b8d36d686e" Nov 03 20:16:37.496352 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:16:37.496319 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6051fbf1_1d0b_4e44_af7f_ecf6b9dd3530.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:16:37.672986 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:16:37.672950 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod737a2b7e_1e59_4f55_b680_34c1fa1977c2.slice/crio-9a068854632b3cbb4c24bb081aee76fbdde7e5c072c1eb149f2e5aff023d8f65 WatchSource:0}: Error finding container 9a068854632b3cbb4c24bb081aee76fbdde7e5c072c1eb149f2e5aff023d8f65: Status 404 returned error can't find the container with id 9a068854632b3cbb4c24bb081aee76fbdde7e5c072c1eb149f2e5aff023d8f65 Nov 03 20:16:37.674878 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:37.674854 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-65759456-sb7cs"] Nov 03 20:16:38.337854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:38.337812 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65759456-sb7cs" event={"ID":"737a2b7e-1e59-4f55-b680-34c1fa1977c2","Type":"ContainerStarted","Data":"5d1862b4b20c3a5a68f0dafd29b91509f6ee5a3c40d4ee6ce829bfb13afeaa8c"} Nov 03 20:16:38.338252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:38.337881 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65759456-sb7cs" event={"ID":"737a2b7e-1e59-4f55-b680-34c1fa1977c2","Type":"ContainerStarted","Data":"9a068854632b3cbb4c24bb081aee76fbdde7e5c072c1eb149f2e5aff023d8f65"} Nov 03 20:16:38.338252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:38.337915 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-65759456-sb7cs" Nov 03 20:16:38.369332 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:38.369281 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-65759456-sb7cs" podStartSLOduration=2.3692665489999998 podStartE2EDuration="2.369266549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:38.366993306 +0000 UTC m=+2392.624790563" watchObservedRunningTime="2026-11-03 20:16:38.369266549 +0000 UTC m=+2392.627063853" Nov 03 20:16:39.333848 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:39.333811 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-65759456-sb7cs"] Nov 03 20:16:40.760686 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:16:40.755883 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:16:43.628102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:43.628056 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2bb2534-tf72r"] Nov 03 20:16:43.633139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:43.633121 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2bb2534-tf72r" Nov 03 20:16:43.637652 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:43.637467 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:16:43.638568 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:43.638541 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:16:43.642439 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:43.642388 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2bb2534-tf72r"] Nov 03 20:16:43.669505 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:43.669465 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxddm\" (UniqueName: \"kubernetes.io/projected/35fc7c23-900c-41ca-ae62-90df054d185c-kube-api-access-pxddm\") pod \"maas-api-revoke-sub-b2bb2534-tf72r\" (UID: \"35fc7c23-900c-41ca-ae62-90df054d185c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2bb2534-tf72r" Nov 03 20:16:43.801684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:43.787741 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35fc7c23-900c-41ca-ae62-90df054d185c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2bb2534-tf72r\" (UID: \"35fc7c23-900c-41ca-ae62-90df054d185c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2bb2534-tf72r" Nov 03 20:16:43.801684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:43.788259 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35fc7c23-900c-41ca-ae62-90df054d185c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2bb2534-tf72r\" (UID: \"35fc7c23-900c-41ca-ae62-90df054d185c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2bb2534-tf72r" Nov 03 20:16:43.991754 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:43.991420 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35fc7c23-900c-41ca-ae62-90df054d185c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2bb2534-tf72r\" (UID: \"35fc7c23-900c-41ca-ae62-90df054d185c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2bb2534-tf72r" Nov 03 20:16:43.991754 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:43.991686 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxddm\" (UniqueName: \"kubernetes.io/projected/35fc7c23-900c-41ca-ae62-90df054d185c-kube-api-access-pxddm\") pod \"maas-api-revoke-sub-b2bb2534-tf72r\" (UID: \"35fc7c23-900c-41ca-ae62-90df054d185c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2bb2534-tf72r" Nov 03 20:16:43.991754 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:43.991726 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35fc7c23-900c-41ca-ae62-90df054d185c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2bb2534-tf72r\" (UID: \"35fc7c23-900c-41ca-ae62-90df054d185c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2bb2534-tf72r" Nov 03 20:16:44.033687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:44.029038 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35fc7c23-900c-41ca-ae62-90df054d185c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2bb2534-tf72r\" (UID: \"35fc7c23-900c-41ca-ae62-90df054d185c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2bb2534-tf72r" Nov 03 20:16:44.106615 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:44.100003 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35fc7c23-900c-41ca-ae62-90df054d185c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2bb2534-tf72r\" (UID: \"35fc7c23-900c-41ca-ae62-90df054d185c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2bb2534-tf72r" Nov 03 20:16:44.107989 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:44.103511 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxddm\" (UniqueName: \"kubernetes.io/projected/35fc7c23-900c-41ca-ae62-90df054d185c-kube-api-access-pxddm\") pod \"maas-api-revoke-sub-b2bb2534-tf72r\" (UID: \"35fc7c23-900c-41ca-ae62-90df054d185c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2bb2534-tf72r" Nov 03 20:16:44.135271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:44.134968 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2bb2534-tf72r" Nov 03 20:16:44.433559 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:16:44.433527 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35fc7c23_900c_41ca_ae62_90df054d185c.slice/crio-e9148998b4b9cf9d6a29bab5e3c62227fc5215dd7258ea29d4b5bda6dcd1c041 WatchSource:0}: Error finding container e9148998b4b9cf9d6a29bab5e3c62227fc5215dd7258ea29d4b5bda6dcd1c041: Status 404 returned error can't find the container with id e9148998b4b9cf9d6a29bab5e3c62227fc5215dd7258ea29d4b5bda6dcd1c041 Nov 03 20:16:44.434735 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:44.434709 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2bb2534-tf72r"] Nov 03 20:16:44.595688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:44.588941 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2bb2534-tf72r" event={"ID":"35fc7c23-900c-41ca-ae62-90df054d185c","Type":"ContainerStarted","Data":"e9148998b4b9cf9d6a29bab5e3c62227fc5215dd7258ea29d4b5bda6dcd1c041"} Nov 03 20:16:45.405692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:45.399272 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-65759456-sb7cs" podUID="737a2b7e-1e59-4f55-b680-34c1fa1977c2" containerName="payload-processing" containerID="cri-o://5d1862b4b20c3a5a68f0dafd29b91509f6ee5a3c40d4ee6ce829bfb13afeaa8c" gracePeriod=25 Nov 03 20:16:45.736792 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:45.734929 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:45.750728 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:45.745219 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:45.884403 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:45.884359 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:16:45.969943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:45.969881 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2bb2534-tf72r" podStartSLOduration=2.969861609 podStartE2EDuration="2.969861609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:45.95281736 +0000 UTC m=+2400.210614618" watchObservedRunningTime="2026-11-03 20:16:45.969861609 +0000 UTC m=+2400.227658867" Nov 03 20:16:46.184253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.184228 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65759456-sb7cs" Nov 03 20:16:46.227260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.227198 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh"] Nov 03 20:16:46.228713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.228646 5059 cpu_manager.go:414] "containerMap: removing container" podUID="737a2b7e-1e59-4f55-b680-34c1fa1977c2" containerName="payload-processing" Nov 03 20:16:46.228713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.228688 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="737a2b7e-1e59-4f55-b680-34c1fa1977c2" containerName="payload-processing" Nov 03 20:16:46.228948 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.228931 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="737a2b7e-1e59-4f55-b680-34c1fa1977c2" containerName="payload-processing" Nov 03 20:16:46.240982 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.240953 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" Nov 03 20:16:46.246323 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.246284 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh"] Nov 03 20:16:46.247245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.247078 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"agc-cred-3eafb068aa07\"" Nov 03 20:16:46.247245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.247232 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"payload-processing-external-model-e2e--aa51d60f-dockercfg-wfnqm\"" Nov 03 20:16:46.247781 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.247272 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"kube-root-ca.crt\"" Nov 03 20:16:46.247781 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.247237 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"payload-processing-external-model-plugins-e2e-extmod-a-33e267\"" Nov 03 20:16:46.247781 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.247289 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"openshift-service-ca.crt\"" Nov 03 20:16:46.247781 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.247360 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"routing-overlay\"" Nov 03 20:16:46.272325 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.272294 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/737a2b7e-1e59-4f55-b680-34c1fa1977c2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/737a2b7e-1e59-4f55-b680-34c1fa1977c2-provider-credentials\") pod \"737a2b7e-1e59-4f55-b680-34c1fa1977c2\" (UID: \"737a2b7e-1e59-4f55-b680-34c1fa1977c2\") " Nov 03 20:16:46.272544 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.272519 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/737a2b7e-1e59-4f55-b680-34c1fa1977c2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/737a2b7e-1e59-4f55-b680-34c1fa1977c2-routing-overlay-volume\") pod \"737a2b7e-1e59-4f55-b680-34c1fa1977c2\" (UID: \"737a2b7e-1e59-4f55-b680-34c1fa1977c2\") " Nov 03 20:16:46.272742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.272711 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/737a2b7e-1e59-4f55-b680-34c1fa1977c2-kube-api-access-zl8np\" (UniqueName: \"kubernetes.io/projected/737a2b7e-1e59-4f55-b680-34c1fa1977c2-kube-api-access-zl8np\") pod \"737a2b7e-1e59-4f55-b680-34c1fa1977c2\" (UID: \"737a2b7e-1e59-4f55-b680-34c1fa1977c2\") " Nov 03 20:16:46.272839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.272807 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/737a2b7e-1e59-4f55-b680-34c1fa1977c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/737a2b7e-1e59-4f55-b680-34c1fa1977c2-plugins-config-volume\") pod \"737a2b7e-1e59-4f55-b680-34c1fa1977c2\" (UID: \"737a2b7e-1e59-4f55-b680-34c1fa1977c2\") " Nov 03 20:16:46.273016 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.272989 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/737a2b7e-1e59-4f55-b680-34c1fa1977c2-routing-overlay-volume" pod "737a2b7e-1e59-4f55-b680-34c1fa1977c2" (UID: "737a2b7e-1e59-4f55-b680-34c1fa1977c2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:46.273515 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.273487 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcnk8\" (UniqueName: \"kubernetes.io/projected/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-kube-api-access-hcnk8\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh\" (UID: \"c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" Nov 03 20:16:46.273632 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.273603 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh\" (UID: \"c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" Nov 03 20:16:46.273762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.273684 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/737a2b7e-1e59-4f55-b680-34c1fa1977c2-plugins-config-volume" pod "737a2b7e-1e59-4f55-b680-34c1fa1977c2" (UID: "737a2b7e-1e59-4f55-b680-34c1fa1977c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:46.273848 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.273757 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh\" (UID: \"c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" Nov 03 20:16:46.273915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.273849 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh\" (UID: \"c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" Nov 03 20:16:46.274134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.274110 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/737a2b7e-1e59-4f55-b680-34c1fa1977c2-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:46.274227 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.274142 5059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/737a2b7e-1e59-4f55-b680-34c1fa1977c2-routing-overlay-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:46.274903 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.274864 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/737a2b7e-1e59-4f55-b680-34c1fa1977c2-provider-credentials" pod "737a2b7e-1e59-4f55-b680-34c1fa1977c2" (UID: "737a2b7e-1e59-4f55-b680-34c1fa1977c2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:46.275459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.275435 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/737a2b7e-1e59-4f55-b680-34c1fa1977c2-kube-api-access-zl8np" pod "737a2b7e-1e59-4f55-b680-34c1fa1977c2" (UID: "737a2b7e-1e59-4f55-b680-34c1fa1977c2"). InnerVolumeSpecName "kube-api-access-zl8np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:46.375674 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.375622 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcnk8\" (UniqueName: \"kubernetes.io/projected/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-kube-api-access-hcnk8\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh\" (UID: \"c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" Nov 03 20:16:46.375853 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.375733 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh\" (UID: \"c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" Nov 03 20:16:46.375853 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.375808 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh\" (UID: \"c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" Nov 03 20:16:46.375984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.375881 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh\" (UID: \"c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" Nov 03 20:16:46.376196 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.376174 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl8np\" (UniqueName: \"kubernetes.io/projected/737a2b7e-1e59-4f55-b680-34c1fa1977c2-kube-api-access-zl8np\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:46.376234 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.376204 5059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/737a2b7e-1e59-4f55-b680-34c1fa1977c2-provider-credentials\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:46.376526 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.376502 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh\" (UID: \"c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" Nov 03 20:16:46.376571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.376553 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh\" (UID: \"c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" Nov 03 20:16:46.378398 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.378374 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh\" (UID: \"c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" Nov 03 20:16:46.388592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.388571 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcnk8\" (UniqueName: \"kubernetes.io/projected/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-kube-api-access-hcnk8\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh\" (UID: \"c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" Nov 03 20:16:46.564384 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.563014 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" Nov 03 20:16:46.890001 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.889912 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 20:16:46.897626 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.897600 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 20:16:46.939799 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:16:46.939765 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ba500a_c13f_43a7_a0d3_78a6c9c2b8bb.slice/crio-1c60b547cee42a8e2c7c57363ccf205083aba47b66ad6a104df2db73e0fff249 WatchSource:0}: Error finding container 1c60b547cee42a8e2c7c57363ccf205083aba47b66ad6a104df2db73e0fff249: Status 404 returned error can't find the container with id 1c60b547cee42a8e2c7c57363ccf205083aba47b66ad6a104df2db73e0fff249 Nov 03 20:16:46.942356 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.942330 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh"] Nov 03 20:16:46.946288 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.946264 5059 generic.go:356] "Generic (PLEG): container finished" podID="35fc7c23-900c-41ca-ae62-90df054d185c" containerID="6f00779d178bbb7df7d0a1c42362a3f0bd0b7d1a7b5cf5eac297d3a24b472712" exitCode=0 Nov 03 20:16:46.946409 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.946322 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2bb2534-tf72r" event={"ID":"35fc7c23-900c-41ca-ae62-90df054d185c","Type":"ContainerDied","Data":"6f00779d178bbb7df7d0a1c42362a3f0bd0b7d1a7b5cf5eac297d3a24b472712"} Nov 03 20:16:46.947780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.947755 5059 generic.go:356] "Generic (PLEG): container finished" podID="737a2b7e-1e59-4f55-b680-34c1fa1977c2" containerID="5d1862b4b20c3a5a68f0dafd29b91509f6ee5a3c40d4ee6ce829bfb13afeaa8c" exitCode=0 Nov 03 20:16:46.947873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.947801 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65759456-sb7cs" Nov 03 20:16:46.947873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.947834 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65759456-sb7cs" event={"ID":"737a2b7e-1e59-4f55-b680-34c1fa1977c2","Type":"ContainerDied","Data":"5d1862b4b20c3a5a68f0dafd29b91509f6ee5a3c40d4ee6ce829bfb13afeaa8c"} Nov 03 20:16:46.947958 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.947883 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65759456-sb7cs" event={"ID":"737a2b7e-1e59-4f55-b680-34c1fa1977c2","Type":"ContainerDied","Data":"9a068854632b3cbb4c24bb081aee76fbdde7e5c072c1eb149f2e5aff023d8f65"} Nov 03 20:16:46.947958 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.947911 5059 scope.go:122] "RemoveContainer" containerID="5d1862b4b20c3a5a68f0dafd29b91509f6ee5a3c40d4ee6ce829bfb13afeaa8c" Nov 03 20:16:46.963673 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.963640 5059 scope.go:122] "RemoveContainer" containerID="5d1862b4b20c3a5a68f0dafd29b91509f6ee5a3c40d4ee6ce829bfb13afeaa8c" Nov 03 20:16:46.964002 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:16:46.963973 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d1862b4b20c3a5a68f0dafd29b91509f6ee5a3c40d4ee6ce829bfb13afeaa8c\": container with ID starting with 5d1862b4b20c3a5a68f0dafd29b91509f6ee5a3c40d4ee6ce829bfb13afeaa8c not found: ID does not exist" containerID="5d1862b4b20c3a5a68f0dafd29b91509f6ee5a3c40d4ee6ce829bfb13afeaa8c" Nov 03 20:16:46.964098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.964003 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1862b4b20c3a5a68f0dafd29b91509f6ee5a3c40d4ee6ce829bfb13afeaa8c"} err="failed to get container status \"5d1862b4b20c3a5a68f0dafd29b91509f6ee5a3c40d4ee6ce829bfb13afeaa8c\": rpc error: code = NotFound desc = could not find container \"5d1862b4b20c3a5a68f0dafd29b91509f6ee5a3c40d4ee6ce829bfb13afeaa8c\": container with ID starting with 5d1862b4b20c3a5a68f0dafd29b91509f6ee5a3c40d4ee6ce829bfb13afeaa8c not found: ID does not exist" Nov 03 20:16:46.985996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.985973 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-65759456-sb7cs"] Nov 03 20:16:46.991303 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:46.991281 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-65759456-sb7cs"] Nov 03 20:16:47.861195 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:47.861158 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="737a2b7e-1e59-4f55-b680-34c1fa1977c2" path="/var/lib/kubelet/pods/737a2b7e-1e59-4f55-b680-34c1fa1977c2/volumes" Nov 03 20:16:47.999695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:47.995321 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" event={"ID":"c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb","Type":"ContainerStarted","Data":"2a0e6dbb090024b1ff098aead685df3f93a0db69e17be84b75cd9fda7eb86653"} Nov 03 20:16:47.999695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:47.995435 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" event={"ID":"c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb","Type":"ContainerStarted","Data":"1c60b547cee42a8e2c7c57363ccf205083aba47b66ad6a104df2db73e0fff249"} Nov 03 20:16:47.999695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:47.995518 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" Nov 03 20:16:48.074878 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:48.074810 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" podStartSLOduration=4.074787676 podStartE2EDuration="4.074787676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:48.036255395 +0000 UTC m=+2402.294052678" watchObservedRunningTime="2026-11-03 20:16:48.074787676 +0000 UTC m=+2402.332584934" Nov 03 20:16:48.347764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:48.347736 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2bb2534-tf72r" Nov 03 20:16:48.445175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:48.445084 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35fc7c23-900c-41ca-ae62-90df054d185c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35fc7c23-900c-41ca-ae62-90df054d185c-maas-api-xks-ca\") pod \"35fc7c23-900c-41ca-ae62-90df054d185c\" (UID: \"35fc7c23-900c-41ca-ae62-90df054d185c\") " Nov 03 20:16:48.445355 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:48.445188 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35fc7c23-900c-41ca-ae62-90df054d185c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35fc7c23-900c-41ca-ae62-90df054d185c-maas-api-openshift-ca\") pod \"35fc7c23-900c-41ca-ae62-90df054d185c\" (UID: \"35fc7c23-900c-41ca-ae62-90df054d185c\") " Nov 03 20:16:48.445514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:48.445495 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35fc7c23-900c-41ca-ae62-90df054d185c-kube-api-access-pxddm\" (UniqueName: \"kubernetes.io/projected/35fc7c23-900c-41ca-ae62-90df054d185c-kube-api-access-pxddm\") pod \"35fc7c23-900c-41ca-ae62-90df054d185c\" (UID: \"35fc7c23-900c-41ca-ae62-90df054d185c\") " Nov 03 20:16:48.445606 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:48.445580 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35fc7c23-900c-41ca-ae62-90df054d185c-maas-api-openshift-ca" pod "35fc7c23-900c-41ca-ae62-90df054d185c" (UID: "35fc7c23-900c-41ca-ae62-90df054d185c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:48.446225 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:48.446195 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35fc7c23-900c-41ca-ae62-90df054d185c-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:48.447396 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:48.447370 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35fc7c23-900c-41ca-ae62-90df054d185c-maas-api-xks-ca" pod "35fc7c23-900c-41ca-ae62-90df054d185c" (UID: "35fc7c23-900c-41ca-ae62-90df054d185c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:48.447629 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:48.447605 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35fc7c23-900c-41ca-ae62-90df054d185c-kube-api-access-pxddm" pod "35fc7c23-900c-41ca-ae62-90df054d185c" (UID: "35fc7c23-900c-41ca-ae62-90df054d185c"). InnerVolumeSpecName "kube-api-access-pxddm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:48.548149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:48.548115 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxddm\" (UniqueName: \"kubernetes.io/projected/35fc7c23-900c-41ca-ae62-90df054d185c-kube-api-access-pxddm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:48.548149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:48.548148 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35fc7c23-900c-41ca-ae62-90df054d185c-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:49.002492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:49.002464 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2bb2534-tf72r" Nov 03 20:16:49.002904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:49.002509 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2bb2534-tf72r" event={"ID":"35fc7c23-900c-41ca-ae62-90df054d185c","Type":"ContainerDied","Data":"e9148998b4b9cf9d6a29bab5e3c62227fc5215dd7258ea29d4b5bda6dcd1c041"} Nov 03 20:16:49.002904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:49.002562 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9148998b4b9cf9d6a29bab5e3c62227fc5215dd7258ea29d4b5bda6dcd1c041" Nov 03 20:16:57.747697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:57.745869 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:57.885684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:57.876956 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:58.124692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:58.123780 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:58.251456 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:58.250443 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:58.752822 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:58.752775 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:58.759709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:58.754616 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:16:58.875830 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:58.875796 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:16:59.047604 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:16:59.047475 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" podUID="c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:17:09.004400 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:09.004351 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" podUID="c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:17:10.625471 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:17:10.625435 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:17:13.677037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:13.676991 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh"] Nov 03 20:17:13.679903 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:13.679868 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" podUID="c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:17:18.680017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:18.679974 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" podUID="c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb" containerName="payload-processing" containerID="cri-o://2a0e6dbb090024b1ff098aead685df3f93a0db69e17be84b75cd9fda7eb86653" gracePeriod=25 Nov 03 20:17:19.546840 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:19.546811 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" Nov 03 20:17:19.621171 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:19.621076 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-provider-credentials\") pod \"c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb\" (UID: \"c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb\") " Nov 03 20:17:19.621361 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:19.621248 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-routing-overlay-volume\") pod \"c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb\" (UID: \"c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb\") " Nov 03 20:17:19.621361 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:19.621285 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-kube-api-access-hcnk8\" (UniqueName: \"kubernetes.io/projected/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-kube-api-access-hcnk8\") pod \"c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb\" (UID: \"c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb\") " Nov 03 20:17:19.621481 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:19.621448 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-plugins-config-volume\") pod \"c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb\" (UID: \"c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb\") " Nov 03 20:17:19.622601 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:19.621904 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-routing-overlay-volume" pod "c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb" (UID: "c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:17:19.622601 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:19.622505 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-plugins-config-volume" pod "c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb" (UID: "c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:17:19.624541 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:19.624148 5059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-routing-overlay-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:19.624541 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:19.624175 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:19.636911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:19.636886 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-provider-credentials" pod "c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb" (UID: "c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:19.637017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:19.636929 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-kube-api-access-hcnk8" pod "c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb" (UID: "c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb"). InnerVolumeSpecName "kube-api-access-hcnk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:19.745186 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:19.745151 5059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-provider-credentials\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:19.745186 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:19.745187 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcnk8\" (UniqueName: \"kubernetes.io/projected/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb-kube-api-access-hcnk8\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:19.874407 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:19.874304 5059 generic.go:356] "Generic (PLEG): container finished" podID="c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb" containerID="2a0e6dbb090024b1ff098aead685df3f93a0db69e17be84b75cd9fda7eb86653" exitCode=0 Nov 03 20:17:19.874572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:19.874432 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" Nov 03 20:17:19.874970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:19.874938 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" event={"ID":"c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb","Type":"ContainerDied","Data":"2a0e6dbb090024b1ff098aead685df3f93a0db69e17be84b75cd9fda7eb86653"} Nov 03 20:17:19.875186 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:19.874997 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh" event={"ID":"c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb","Type":"ContainerDied","Data":"1c60b547cee42a8e2c7c57363ccf205083aba47b66ad6a104df2db73e0fff249"} Nov 03 20:17:19.875186 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:19.875024 5059 scope.go:122] "RemoveContainer" containerID="2a0e6dbb090024b1ff098aead685df3f93a0db69e17be84b75cd9fda7eb86653" Nov 03 20:17:19.958807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:19.958766 5059 scope.go:122] "RemoveContainer" containerID="2a0e6dbb090024b1ff098aead685df3f93a0db69e17be84b75cd9fda7eb86653" Nov 03 20:17:19.980698 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:17:19.979870 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a0e6dbb090024b1ff098aead685df3f93a0db69e17be84b75cd9fda7eb86653\": container with ID starting with 2a0e6dbb090024b1ff098aead685df3f93a0db69e17be84b75cd9fda7eb86653 not found: ID does not exist" containerID="2a0e6dbb090024b1ff098aead685df3f93a0db69e17be84b75cd9fda7eb86653" Nov 03 20:17:19.980698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:19.979949 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a0e6dbb090024b1ff098aead685df3f93a0db69e17be84b75cd9fda7eb86653"} err="failed to get container status \"2a0e6dbb090024b1ff098aead685df3f93a0db69e17be84b75cd9fda7eb86653\": rpc error: code = NotFound desc = could not find container \"2a0e6dbb090024b1ff098aead685df3f93a0db69e17be84b75cd9fda7eb86653\": container with ID starting with 2a0e6dbb090024b1ff098aead685df3f93a0db69e17be84b75cd9fda7eb86653 not found: ID does not exist" Nov 03 20:17:19.999687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:19.992710 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh"] Nov 03 20:17:20.004385 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:20.004331 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6b8wgmh"] Nov 03 20:17:21.860902 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:21.860862 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb" path="/var/lib/kubelet/pods/c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb/volumes" Nov 03 20:17:33.740338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:33.740300 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9"] Nov 03 20:17:33.743134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:33.740560 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" podUID="04b5af1a-f4e4-45a3-a7ee-59037e562b8c" containerName="istio-proxy" containerID="cri-o://14facf3af6388868103be0c86ac9a7621270e968cb1494f92df19d7fbcf7b809" gracePeriod=30 Nov 03 20:17:38.982927 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:38.981564 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-366aef64-jxpvk"] Nov 03 20:17:38.988796 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:38.987213 5059 cpu_manager.go:414] "containerMap: removing container" podUID="35fc7c23-900c-41ca-ae62-90df054d185c" containerName="revoke-keys" Nov 03 20:17:38.988796 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:38.987929 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35fc7c23-900c-41ca-ae62-90df054d185c" containerName="revoke-keys" Nov 03 20:17:38.988796 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:38.988031 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb" containerName="payload-processing" Nov 03 20:17:38.988796 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:38.988041 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb" containerName="payload-processing" Nov 03 20:17:38.988796 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:38.988299 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="35fc7c23-900c-41ca-ae62-90df054d185c" containerName="revoke-keys" Nov 03 20:17:38.988796 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:38.988329 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5ba500a-c13f-43a7-a0d3-78a6c9c2b8bb" containerName="payload-processing" Nov 03 20:17:39.003273 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.003244 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366aef64-jxpvk" Nov 03 20:17:39.005036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.005007 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-366aef64-jxpvk"] Nov 03 20:17:39.041722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.039903 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:17:39.041722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.041374 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:17:39.074759 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.073374 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c930c184-40d2-4b5a-ac89-a24e4e7a6590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-366aef64-jxpvk\" (UID: \"c930c184-40d2-4b5a-ac89-a24e4e7a6590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366aef64-jxpvk" Nov 03 20:17:39.074759 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.073550 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c930c184-40d2-4b5a-ac89-a24e4e7a6590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-366aef64-jxpvk\" (UID: \"c930c184-40d2-4b5a-ac89-a24e4e7a6590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366aef64-jxpvk" Nov 03 20:17:39.074759 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.073589 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n7jd\" (UniqueName: \"kubernetes.io/projected/c930c184-40d2-4b5a-ac89-a24e4e7a6590-kube-api-access-6n7jd\") pod \"maas-api-revoke-sub-366aef64-jxpvk\" (UID: \"c930c184-40d2-4b5a-ac89-a24e4e7a6590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366aef64-jxpvk" Nov 03 20:17:39.175178 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.175130 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c930c184-40d2-4b5a-ac89-a24e4e7a6590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-366aef64-jxpvk\" (UID: \"c930c184-40d2-4b5a-ac89-a24e4e7a6590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366aef64-jxpvk" Nov 03 20:17:39.175373 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.175281 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c930c184-40d2-4b5a-ac89-a24e4e7a6590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-366aef64-jxpvk\" (UID: \"c930c184-40d2-4b5a-ac89-a24e4e7a6590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366aef64-jxpvk" Nov 03 20:17:39.175373 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.175312 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n7jd\" (UniqueName: \"kubernetes.io/projected/c930c184-40d2-4b5a-ac89-a24e4e7a6590-kube-api-access-6n7jd\") pod \"maas-api-revoke-sub-366aef64-jxpvk\" (UID: \"c930c184-40d2-4b5a-ac89-a24e4e7a6590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366aef64-jxpvk" Nov 03 20:17:39.177833 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.177802 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c930c184-40d2-4b5a-ac89-a24e4e7a6590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-366aef64-jxpvk\" (UID: \"c930c184-40d2-4b5a-ac89-a24e4e7a6590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366aef64-jxpvk" Nov 03 20:17:39.220712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.220358 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c930c184-40d2-4b5a-ac89-a24e4e7a6590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-366aef64-jxpvk\" (UID: \"c930c184-40d2-4b5a-ac89-a24e4e7a6590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366aef64-jxpvk" Nov 03 20:17:39.221288 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.221238 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n7jd\" (UniqueName: \"kubernetes.io/projected/c930c184-40d2-4b5a-ac89-a24e4e7a6590-kube-api-access-6n7jd\") pod \"maas-api-revoke-sub-366aef64-jxpvk\" (UID: \"c930c184-40d2-4b5a-ac89-a24e4e7a6590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-366aef64-jxpvk" Nov 03 20:17:39.429732 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.428626 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366aef64-jxpvk" Nov 03 20:17:39.492640 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.492269 5059 generic.go:356] "Generic (PLEG): container finished" podID="04b5af1a-f4e4-45a3-a7ee-59037e562b8c" containerID="14facf3af6388868103be0c86ac9a7621270e968cb1494f92df19d7fbcf7b809" exitCode=0 Nov 03 20:17:39.492640 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.492444 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" event={"ID":"04b5af1a-f4e4-45a3-a7ee-59037e562b8c","Type":"ContainerDied","Data":"14facf3af6388868103be0c86ac9a7621270e968cb1494f92df19d7fbcf7b809"} Nov 03 20:17:39.492640 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.492537 5059 scope.go:122] "RemoveContainer" containerID="d2ed43f59e564d0a049376f043011d192ceaea1235be3c18c1ae208bbb3f3aa9" Nov 03 20:17:39.593464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.593173 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:17:39.642614 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.639078 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7"] Nov 03 20:17:39.642614 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.639966 5059 cpu_manager.go:414] "containerMap: removing container" podUID="04b5af1a-f4e4-45a3-a7ee-59037e562b8c" containerName="istio-proxy" Nov 03 20:17:39.642614 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.639984 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b5af1a-f4e4-45a3-a7ee-59037e562b8c" containerName="istio-proxy" Nov 03 20:17:39.642614 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.640007 5059 cpu_manager.go:414] "containerMap: removing container" podUID="04b5af1a-f4e4-45a3-a7ee-59037e562b8c" containerName="istio-proxy" Nov 03 20:17:39.642614 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.640012 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b5af1a-f4e4-45a3-a7ee-59037e562b8c" containerName="istio-proxy" Nov 03 20:17:39.642614 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.640165 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="04b5af1a-f4e4-45a3-a7ee-59037e562b8c" containerName="istio-proxy" Nov 03 20:17:39.655758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.648097 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.670716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.655759 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7"] Nov 03 20:17:39.754492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.752923 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istiod-ca-cert\") pod \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " Nov 03 20:17:39.754492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.753029 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-workload-socket\") pod \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " Nov 03 20:17:39.754492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.753100 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-envoy\") pod \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " Nov 03 20:17:39.754492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.753193 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-data\") pod \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " Nov 03 20:17:39.754492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.753265 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-podinfo\") pod \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " Nov 03 20:17:39.754492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.753318 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-token\" (UniqueName: \"kubernetes.io/projected/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-token\") pod \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " Nov 03 20:17:39.754492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.753340 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-workload-certs\") pod \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " Nov 03 20:17:39.754492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.753377 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-kube-api-access-wwpmv\" (UniqueName: \"kubernetes.io/projected/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-kube-api-access-wwpmv\") pod \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " Nov 03 20:17:39.754492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.753416 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-credential-socket\") pod \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\" (UID: \"04b5af1a-f4e4-45a3-a7ee-59037e562b8c\") " Nov 03 20:17:39.754492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.753747 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.754492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.753816 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.754492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.753969 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.754492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.754018 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.754492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.754084 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.754492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.754226 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f32ecf7-06f7-4098-9793-acf761a30c09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.754492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.754298 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-workload-socket" pod "04b5af1a-f4e4-45a3-a7ee-59037e562b8c" (UID: "04b5af1a-f4e4-45a3-a7ee-59037e562b8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:39.754492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.754489 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.756097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.754573 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.756097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.754618 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kztkj\" (UniqueName: \"kubernetes.io/projected/0f32ecf7-06f7-4098-9793-acf761a30c09-kube-api-access-kztkj\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.756097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.754759 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:39.756097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.754762 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-workload-certs" pod "04b5af1a-f4e4-45a3-a7ee-59037e562b8c" (UID: "04b5af1a-f4e4-45a3-a7ee-59037e562b8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:39.756097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.755000 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-credential-socket" pod "04b5af1a-f4e4-45a3-a7ee-59037e562b8c" (UID: "04b5af1a-f4e4-45a3-a7ee-59037e562b8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:39.756097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.755540 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istiod-ca-cert" pod "04b5af1a-f4e4-45a3-a7ee-59037e562b8c" (UID: "04b5af1a-f4e4-45a3-a7ee-59037e562b8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:17:39.757301 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.756480 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-data" pod "04b5af1a-f4e4-45a3-a7ee-59037e562b8c" (UID: "04b5af1a-f4e4-45a3-a7ee-59037e562b8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:39.758102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.757845 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-kube-api-access-wwpmv" pod "04b5af1a-f4e4-45a3-a7ee-59037e562b8c" (UID: "04b5af1a-f4e4-45a3-a7ee-59037e562b8c"). InnerVolumeSpecName "kube-api-access-wwpmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:39.758102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.757905 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-podinfo" pod "04b5af1a-f4e4-45a3-a7ee-59037e562b8c" (UID: "04b5af1a-f4e4-45a3-a7ee-59037e562b8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:17:39.758915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.758885 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-envoy" pod "04b5af1a-f4e4-45a3-a7ee-59037e562b8c" (UID: "04b5af1a-f4e4-45a3-a7ee-59037e562b8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:17:39.759137 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.759117 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-token" pod "04b5af1a-f4e4-45a3-a7ee-59037e562b8c" (UID: "04b5af1a-f4e4-45a3-a7ee-59037e562b8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:39.855846 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.855815 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.856018 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.855870 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.856018 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.855910 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kztkj\" (UniqueName: \"kubernetes.io/projected/0f32ecf7-06f7-4098-9793-acf761a30c09-kube-api-access-kztkj\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.856018 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.855949 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.856018 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.855994 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.856249 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.856114 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.856249 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.856230 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.856396 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.856322 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.856396 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.856340 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.856514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.856435 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.856514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.856483 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.856635 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.856561 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f32ecf7-06f7-4098-9793-acf761a30c09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.856839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.856784 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.857033 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.857009 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:39.857033 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.857039 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:39.858059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.857056 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:39.858059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.857071 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:39.858059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.857084 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:39.858059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.857098 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:39.858059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.857113 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwpmv\" (UniqueName: \"kubernetes.io/projected/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-kube-api-access-wwpmv\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:39.858059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.857129 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b5af1a-f4e4-45a3-a7ee-59037e562b8c-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:39.858059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.857968 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f32ecf7-06f7-4098-9793-acf761a30c09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.859139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.858693 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.859139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.858724 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.868973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.868944 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.869089 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.869064 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kztkj\" (UniqueName: \"kubernetes.io/projected/0f32ecf7-06f7-4098-9793-acf761a30c09-kube-api-access-kztkj\") pod \"maas-default-gateway-openshift-default-855f9574cf-g6xz7\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:39.902833 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.902809 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-366aef64-jxpvk"] Nov 03 20:17:39.983598 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:39.983572 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:40.279219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:40.279180 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7"] Nov 03 20:17:40.360217 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:17:40.360183 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f32ecf7_06f7_4098_9793_acf761a30c09.slice/crio-58f0c433bf4f049cef0f62c8ae114894d5f9997d1c364fb2311ace0372d8fe70 WatchSource:0}: Error finding container 58f0c433bf4f049cef0f62c8ae114894d5f9997d1c364fb2311ace0372d8fe70: Status 404 returned error can't find the container with id 58f0c433bf4f049cef0f62c8ae114894d5f9997d1c364fb2311ace0372d8fe70 Nov 03 20:17:40.363330 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:40.363293 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:17:40.363424 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:40.363376 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:17:40.363424 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:40.363406 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:17:40.503637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:40.503600 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" event={"ID":"0f32ecf7-06f7-4098-9793-acf761a30c09","Type":"ContainerStarted","Data":"5ac8799b34596375075829e9ebddeeb7da35d54820c64c585e69600c30f1d815"} Nov 03 20:17:40.503826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:40.503712 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" event={"ID":"0f32ecf7-06f7-4098-9793-acf761a30c09","Type":"ContainerStarted","Data":"58f0c433bf4f049cef0f62c8ae114894d5f9997d1c364fb2311ace0372d8fe70"} Nov 03 20:17:40.505592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:40.505564 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366aef64-jxpvk" event={"ID":"c930c184-40d2-4b5a-ac89-a24e4e7a6590","Type":"ContainerStarted","Data":"8965713e7b67fd786a3f2eb079b3bf700a4c843fc6da3de604aa79b0f7c40b3e"} Nov 03 20:17:40.505767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:40.505612 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366aef64-jxpvk" event={"ID":"c930c184-40d2-4b5a-ac89-a24e4e7a6590","Type":"ContainerStarted","Data":"65b06af4a9f16758e99690205d79b69f397bc8ed5ead57e9e7ade32bba2977a0"} Nov 03 20:17:40.507535 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:40.507519 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" Nov 03 20:17:40.507630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:40.507564 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9" event={"ID":"04b5af1a-f4e4-45a3-a7ee-59037e562b8c","Type":"ContainerDied","Data":"d12731bc11fea58f19d6f8edbc419ac2df5d460c37339f819d94e14eb97ebaf2"} Nov 03 20:17:40.507630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:40.507620 5059 scope.go:122] "RemoveContainer" containerID="14facf3af6388868103be0c86ac9a7621270e968cb1494f92df19d7fbcf7b809" Nov 03 20:17:40.539000 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:40.538895 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" podStartSLOduration=58.538875924 podStartE2EDuration="58.538875924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:40.529468935 +0000 UTC m=+2454.787266193" watchObservedRunningTime="2026-11-03 20:17:40.538875924 +0000 UTC m=+2454.796673182" Nov 03 20:17:40.559224 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:40.559174 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366aef64-jxpvk" podStartSLOduration=2.5591565960000002 podStartE2EDuration="2.559156596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:40.550597247 +0000 UTC m=+2454.808394505" watchObservedRunningTime="2026-11-03 20:17:40.559156596 +0000 UTC m=+2454.816953858" Nov 03 20:17:40.569950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:40.569925 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9"] Nov 03 20:17:40.578085 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:40.578063 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-pt2n9"] Nov 03 20:17:40.657976 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:17:40.657924 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:17:40.984388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:40.984358 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:41.631234 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:41.630494 5059 generic.go:356] "Generic (PLEG): container finished" podID="c930c184-40d2-4b5a-ac89-a24e4e7a6590" containerID="8965713e7b67fd786a3f2eb079b3bf700a4c843fc6da3de604aa79b0f7c40b3e" exitCode=0 Nov 03 20:17:41.631234 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:41.630763 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366aef64-jxpvk" event={"ID":"c930c184-40d2-4b5a-ac89-a24e4e7a6590","Type":"ContainerDied","Data":"8965713e7b67fd786a3f2eb079b3bf700a4c843fc6da3de604aa79b0f7c40b3e"} Nov 03 20:17:41.882743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:41.881349 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04b5af1a-f4e4-45a3-a7ee-59037e562b8c" path="/var/lib/kubelet/pods/04b5af1a-f4e4-45a3-a7ee-59037e562b8c/volumes" Nov 03 20:17:41.989477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:41.989418 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-g6xz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body= Nov 03 20:17:41.989927 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:41.989494 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" podUID="0f32ecf7-06f7-4098-9793-acf761a30c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" Nov 03 20:17:42.996795 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:42.985212 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-g6xz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body= Nov 03 20:17:42.996795 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:42.985272 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" podUID="0f32ecf7-06f7-4098-9793-acf761a30c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" Nov 03 20:17:43.046955 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:43.046932 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366aef64-jxpvk" Nov 03 20:17:43.088722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:43.088134 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c930c184-40d2-4b5a-ac89-a24e4e7a6590-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c930c184-40d2-4b5a-ac89-a24e4e7a6590-maas-api-openshift-ca\") pod \"c930c184-40d2-4b5a-ac89-a24e4e7a6590\" (UID: \"c930c184-40d2-4b5a-ac89-a24e4e7a6590\") " Nov 03 20:17:43.088722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:43.088364 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c930c184-40d2-4b5a-ac89-a24e4e7a6590-kube-api-access-6n7jd\" (UniqueName: \"kubernetes.io/projected/c930c184-40d2-4b5a-ac89-a24e4e7a6590-kube-api-access-6n7jd\") pod \"c930c184-40d2-4b5a-ac89-a24e4e7a6590\" (UID: \"c930c184-40d2-4b5a-ac89-a24e4e7a6590\") " Nov 03 20:17:43.088722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:43.088624 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c930c184-40d2-4b5a-ac89-a24e4e7a6590-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c930c184-40d2-4b5a-ac89-a24e4e7a6590-maas-api-xks-ca\") pod \"c930c184-40d2-4b5a-ac89-a24e4e7a6590\" (UID: \"c930c184-40d2-4b5a-ac89-a24e4e7a6590\") " Nov 03 20:17:43.089076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:43.088869 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c930c184-40d2-4b5a-ac89-a24e4e7a6590-maas-api-openshift-ca" pod "c930c184-40d2-4b5a-ac89-a24e4e7a6590" (UID: "c930c184-40d2-4b5a-ac89-a24e4e7a6590"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:17:43.089712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:43.089681 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c930c184-40d2-4b5a-ac89-a24e4e7a6590-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:43.094670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:43.094540 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c930c184-40d2-4b5a-ac89-a24e4e7a6590-kube-api-access-6n7jd" pod "c930c184-40d2-4b5a-ac89-a24e4e7a6590" (UID: "c930c184-40d2-4b5a-ac89-a24e4e7a6590"). InnerVolumeSpecName "kube-api-access-6n7jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:43.094670 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:43.094643 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c930c184-40d2-4b5a-ac89-a24e4e7a6590-maas-api-xks-ca" pod "c930c184-40d2-4b5a-ac89-a24e4e7a6590" (UID: "c930c184-40d2-4b5a-ac89-a24e4e7a6590"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:17:43.191883 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:43.191841 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n7jd\" (UniqueName: \"kubernetes.io/projected/c930c184-40d2-4b5a-ac89-a24e4e7a6590-kube-api-access-6n7jd\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:43.191883 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:43.191884 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c930c184-40d2-4b5a-ac89-a24e4e7a6590-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:43.691033 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:43.690989 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366aef64-jxpvk" event={"ID":"c930c184-40d2-4b5a-ac89-a24e4e7a6590","Type":"ContainerDied","Data":"65b06af4a9f16758e99690205d79b69f397bc8ed5ead57e9e7ade32bba2977a0"} Nov 03 20:17:43.691250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:43.691049 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65b06af4a9f16758e99690205d79b69f397bc8ed5ead57e9e7ade32bba2977a0" Nov 03 20:17:43.691250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:43.691101 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-366aef64-jxpvk" Nov 03 20:17:43.984208 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:43.984119 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-g6xz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body= Nov 03 20:17:43.984208 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:43.984187 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" podUID="0f32ecf7-06f7-4098-9793-acf761a30c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" Nov 03 20:17:44.993187 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:44.993145 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-g6xz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body= Nov 03 20:17:44.993838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:44.993802 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" podUID="0f32ecf7-06f7-4098-9793-acf761a30c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" Nov 03 20:17:45.984146 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:45.984105 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-g6xz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body= Nov 03 20:17:45.984366 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:45.984173 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" podUID="0f32ecf7-06f7-4098-9793-acf761a30c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" Nov 03 20:17:46.984185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:46.984137 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-g6xz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body= Nov 03 20:17:46.984763 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:46.984207 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" podUID="0f32ecf7-06f7-4098-9793-acf761a30c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" Nov 03 20:17:47.029135 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:47.029101 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cebde73-pqrbc"] Nov 03 20:17:47.034433 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:47.034405 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cebde73-pqrbc"] Nov 03 20:17:47.864532 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:47.864495 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4dd321e-0d3f-4295-bd95-c2ef50971afc" path="/var/lib/kubelet/pods/a4dd321e-0d3f-4295-bd95-c2ef50971afc/volumes" Nov 03 20:17:47.984219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:47.984178 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-g6xz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body= Nov 03 20:17:47.984674 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:47.984245 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" podUID="0f32ecf7-06f7-4098-9793-acf761a30c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" Nov 03 20:17:48.984608 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:48.984576 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-g6xz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body= Nov 03 20:17:48.985026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:48.984646 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" podUID="0f32ecf7-06f7-4098-9793-acf761a30c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" Nov 03 20:17:50.137826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:50.137793 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:50.138637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:50.138616 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:17:50.139415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:17:50.139384 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:18:05.680245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.680202 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f"] Nov 03 20:18:05.681301 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.681275 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c930c184-40d2-4b5a-ac89-a24e4e7a6590" containerName="revoke-keys" Nov 03 20:18:05.681425 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.681302 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c930c184-40d2-4b5a-ac89-a24e4e7a6590" containerName="revoke-keys" Nov 03 20:18:05.681537 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.681522 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c930c184-40d2-4b5a-ac89-a24e4e7a6590" containerName="revoke-keys" Nov 03 20:18:05.681600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.681545 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="04b5af1a-f4e4-45a3-a7ee-59037e562b8c" containerName="istio-proxy" Nov 03 20:18:05.688074 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.688050 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f" Nov 03 20:18:05.693032 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.693004 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f"] Nov 03 20:18:05.693487 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.693456 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ee9d78131bed\"" Nov 03 20:18:05.693632 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.693485 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-16659cd0d969\"" Nov 03 20:18:05.693632 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.693564 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 20:18:05.694890 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.694867 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-51f23d5569d8\"" Nov 03 20:18:05.695060 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.694869 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 20:18:05.695060 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.694877 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-5lpr4\"" Nov 03 20:18:05.695060 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.694941 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 20:18:05.697059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.694980 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 20:18:05.817529 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.817493 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crh5w\" (UniqueName: \"kubernetes.io/projected/84c8f0ee-a739-4315-9578-f8e61a0e4f67-kube-api-access-crh5w\") pod \"payload-processing-external-model-788fcb4f8c-r8p9f\" (UID: \"84c8f0ee-a739-4315-9578-f8e61a0e4f67\") " pod="models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f" Nov 03 20:18:05.817720 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.817628 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84c8f0ee-a739-4315-9578-f8e61a0e4f67-provider-credentials\") pod \"payload-processing-external-model-788fcb4f8c-r8p9f\" (UID: \"84c8f0ee-a739-4315-9578-f8e61a0e4f67\") " pod="models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f" Nov 03 20:18:05.818839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.817742 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84c8f0ee-a739-4315-9578-f8e61a0e4f67-routing-overlay-volume\") pod \"payload-processing-external-model-788fcb4f8c-r8p9f\" (UID: \"84c8f0ee-a739-4315-9578-f8e61a0e4f67\") " pod="models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f" Nov 03 20:18:05.818839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.817864 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c8f0ee-a739-4315-9578-f8e61a0e4f67-plugins-config-volume\") pod \"payload-processing-external-model-788fcb4f8c-r8p9f\" (UID: \"84c8f0ee-a739-4315-9578-f8e61a0e4f67\") " pod="models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f" Nov 03 20:18:05.919835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.919802 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c8f0ee-a739-4315-9578-f8e61a0e4f67-plugins-config-volume\") pod \"payload-processing-external-model-788fcb4f8c-r8p9f\" (UID: \"84c8f0ee-a739-4315-9578-f8e61a0e4f67\") " pod="models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f" Nov 03 20:18:05.920038 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.920015 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crh5w\" (UniqueName: \"kubernetes.io/projected/84c8f0ee-a739-4315-9578-f8e61a0e4f67-kube-api-access-crh5w\") pod \"payload-processing-external-model-788fcb4f8c-r8p9f\" (UID: \"84c8f0ee-a739-4315-9578-f8e61a0e4f67\") " pod="models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f" Nov 03 20:18:05.920193 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.920169 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84c8f0ee-a739-4315-9578-f8e61a0e4f67-provider-credentials\") pod \"payload-processing-external-model-788fcb4f8c-r8p9f\" (UID: \"84c8f0ee-a739-4315-9578-f8e61a0e4f67\") " pod="models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f" Nov 03 20:18:05.920306 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.920272 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84c8f0ee-a739-4315-9578-f8e61a0e4f67-routing-overlay-volume\") pod \"payload-processing-external-model-788fcb4f8c-r8p9f\" (UID: \"84c8f0ee-a739-4315-9578-f8e61a0e4f67\") " pod="models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f" Nov 03 20:18:05.924576 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.924553 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84c8f0ee-a739-4315-9578-f8e61a0e4f67-routing-overlay-volume\") pod \"payload-processing-external-model-788fcb4f8c-r8p9f\" (UID: \"84c8f0ee-a739-4315-9578-f8e61a0e4f67\") " pod="models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f" Nov 03 20:18:05.925981 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.924570 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c8f0ee-a739-4315-9578-f8e61a0e4f67-plugins-config-volume\") pod \"payload-processing-external-model-788fcb4f8c-r8p9f\" (UID: \"84c8f0ee-a739-4315-9578-f8e61a0e4f67\") " pod="models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f" Nov 03 20:18:05.925981 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.925307 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84c8f0ee-a739-4315-9578-f8e61a0e4f67-provider-credentials\") pod \"payload-processing-external-model-788fcb4f8c-r8p9f\" (UID: \"84c8f0ee-a739-4315-9578-f8e61a0e4f67\") " pod="models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f" Nov 03 20:18:05.932231 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:05.932178 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crh5w\" (UniqueName: \"kubernetes.io/projected/84c8f0ee-a739-4315-9578-f8e61a0e4f67-kube-api-access-crh5w\") pod \"payload-processing-external-model-788fcb4f8c-r8p9f\" (UID: \"84c8f0ee-a739-4315-9578-f8e61a0e4f67\") " pod="models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f" Nov 03 20:18:06.019134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:06.019094 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f" Nov 03 20:18:06.395198 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:18:06.395164 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c8f0ee_a739_4315_9578_f8e61a0e4f67.slice/crio-1f66152db277af4cfc12a0fd739747178e9459ce1fc5d09434da6f3f92739b93 WatchSource:0}: Error finding container 1f66152db277af4cfc12a0fd739747178e9459ce1fc5d09434da6f3f92739b93: Status 404 returned error can't find the container with id 1f66152db277af4cfc12a0fd739747178e9459ce1fc5d09434da6f3f92739b93 Nov 03 20:18:06.395324 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:06.395292 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f"] Nov 03 20:18:07.448930 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:07.448889 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f" event={"ID":"84c8f0ee-a739-4315-9578-f8e61a0e4f67","Type":"ContainerStarted","Data":"ac1c795b78380b6665eca11a2375a02a49880f00380c34bd86b8771507fe476c"} Nov 03 20:18:07.450367 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:07.449407 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f" event={"ID":"84c8f0ee-a739-4315-9578-f8e61a0e4f67","Type":"ContainerStarted","Data":"1f66152db277af4cfc12a0fd739747178e9459ce1fc5d09434da6f3f92739b93"} Nov 03 20:18:07.450367 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:07.449466 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f" Nov 03 20:18:07.476049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:07.475519 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f" podStartSLOduration=2.475501535 podStartE2EDuration="2.475501535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:07.469903057 +0000 UTC m=+2481.727700314" watchObservedRunningTime="2026-11-03 20:18:07.475501535 +0000 UTC m=+2481.733298786" Nov 03 20:18:10.619234 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:18:10.619189 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:18:13.026776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:13.026732 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a55f0974-wxtpc"] Nov 03 20:18:13.031996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:13.031968 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a55f0974-wxtpc"] Nov 03 20:18:13.861935 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:13.861899 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0a6ca31-3be0-4803-bcdb-998ffa9636e2" path="/var/lib/kubelet/pods/c0a6ca31-3be0-4803-bcdb-998ffa9636e2/volumes" Nov 03 20:18:18.505284 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:18.505250 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f" Nov 03 20:18:25.027614 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:25.027576 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-605cf807-5dt4l"] Nov 03 20:18:25.033026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:25.032998 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-605cf807-5dt4l"] Nov 03 20:18:25.862640 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:25.862602 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2942b7cd-a9df-4503-90c2-4299e5a5ed4f" path="/var/lib/kubelet/pods/2942b7cd-a9df-4503-90c2-4299e5a5ed4f/volumes" Nov 03 20:18:31.026625 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:31.026589 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d28fd70-dm69h"] Nov 03 20:18:31.031702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:31.031678 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d28fd70-dm69h"] Nov 03 20:18:31.860888 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:31.860854 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e962f76-40b4-459c-b600-c1cd72ddd43a" path="/var/lib/kubelet/pods/7e962f76-40b4-459c-b600-c1cd72ddd43a/volumes" Nov 03 20:18:32.031291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:32.031259 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca7e8b9b-jtg7k"] Nov 03 20:18:32.036369 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:32.036341 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca7e8b9b-jtg7k"] Nov 03 20:18:33.860230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:33.860197 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c79062a1-763f-43de-9c97-6bf3b9d27fb9" path="/var/lib/kubelet/pods/c79062a1-763f-43de-9c97-6bf3b9d27fb9/volumes" Nov 03 20:18:36.955066 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:36.955031 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9"] Nov 03 20:18:36.957743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:36.957717 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" podUID="63fb9260-c216-4e29-99c9-95dd0d4e4c69" containerName="istio-proxy" containerID="cri-o://832030defdd41fcfdac7fdd33897d9aafc05fd3cb9f6ae3535e2570a1aab65ed" gracePeriod=30 Nov 03 20:18:37.031114 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:37.031082 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aced2fa-t7d6c"] Nov 03 20:18:37.036241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:37.036216 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aced2fa-t7d6c"] Nov 03 20:18:37.874982 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:37.874925 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a00812c-0595-4045-8d46-735a667dc32e" path="/var/lib/kubelet/pods/1a00812c-0595-4045-8d46-735a667dc32e/volumes" Nov 03 20:18:40.627899 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:18:40.627863 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:18:41.039595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:41.039513 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdaa2d49-5b9ms"] Nov 03 20:18:41.044429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:41.044403 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdaa2d49-5b9ms"] Nov 03 20:18:41.862243 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:41.862211 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd71f6f2-b3f5-438c-917a-6645c43c6fd2" path="/var/lib/kubelet/pods/cd71f6f2-b3f5-438c-917a-6645c43c6fd2/volumes" Nov 03 20:18:42.374189 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.374166 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 20:18:42.415391 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.415320 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7"] Nov 03 20:18:42.416637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.416211 5059 cpu_manager.go:414] "containerMap: removing container" podUID="63fb9260-c216-4e29-99c9-95dd0d4e4c69" containerName="istio-proxy" Nov 03 20:18:42.416637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.416228 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63fb9260-c216-4e29-99c9-95dd0d4e4c69" containerName="istio-proxy" Nov 03 20:18:42.416637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.416406 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="63fb9260-c216-4e29-99c9-95dd0d4e4c69" containerName="istio-proxy" Nov 03 20:18:42.420448 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.420428 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.436923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.436899 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7"] Nov 03 20:18:42.470237 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.470208 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-podinfo\") pod \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " Nov 03 20:18:42.470399 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.470278 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-workload-socket\") pod \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " Nov 03 20:18:42.470399 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.470328 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63fb9260-c216-4e29-99c9-95dd0d4e4c69-kube-api-access-qhmgb\" (UniqueName: \"kubernetes.io/projected/63fb9260-c216-4e29-99c9-95dd0d4e4c69-kube-api-access-qhmgb\") pod \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " Nov 03 20:18:42.470399 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.470393 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istiod-ca-cert\") pod \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " Nov 03 20:18:42.471375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.470572 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-workload-certs\") pod \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " Nov 03 20:18:42.471375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.470623 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-workload-socket" pod "63fb9260-c216-4e29-99c9-95dd0d4e4c69" (UID: "63fb9260-c216-4e29-99c9-95dd0d4e4c69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:42.471375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.470751 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-token\" (UniqueName: \"kubernetes.io/projected/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-token\") pod \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " Nov 03 20:18:42.471375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.470878 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-data\") pod \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " Nov 03 20:18:42.471375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.470941 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-credential-socket\") pod \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " Nov 03 20:18:42.471375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.470958 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-envoy\") pod \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\" (UID: \"63fb9260-c216-4e29-99c9-95dd0d4e4c69\") " Nov 03 20:18:42.472600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.471619 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:42.472600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.472135 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-workload-certs" pod "63fb9260-c216-4e29-99c9-95dd0d4e4c69" (UID: "63fb9260-c216-4e29-99c9-95dd0d4e4c69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:42.472600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.472196 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-data" pod "63fb9260-c216-4e29-99c9-95dd0d4e4c69" (UID: "63fb9260-c216-4e29-99c9-95dd0d4e4c69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:42.472600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.472207 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istiod-ca-cert" pod "63fb9260-c216-4e29-99c9-95dd0d4e4c69" (UID: "63fb9260-c216-4e29-99c9-95dd0d4e4c69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:18:42.472600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.472295 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-credential-socket" pod "63fb9260-c216-4e29-99c9-95dd0d4e4c69" (UID: "63fb9260-c216-4e29-99c9-95dd0d4e4c69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:42.479649 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.479626 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-token" pod "63fb9260-c216-4e29-99c9-95dd0d4e4c69" (UID: "63fb9260-c216-4e29-99c9-95dd0d4e4c69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:42.479783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.479703 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63fb9260-c216-4e29-99c9-95dd0d4e4c69-kube-api-access-qhmgb" pod "63fb9260-c216-4e29-99c9-95dd0d4e4c69" (UID: "63fb9260-c216-4e29-99c9-95dd0d4e4c69"). InnerVolumeSpecName "kube-api-access-qhmgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:42.479783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.479735 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-podinfo" pod "63fb9260-c216-4e29-99c9-95dd0d4e4c69" (UID: "63fb9260-c216-4e29-99c9-95dd0d4e4c69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:18:42.480002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.479978 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-envoy" pod "63fb9260-c216-4e29-99c9-95dd0d4e4c69" (UID: "63fb9260-c216-4e29-99c9-95dd0d4e4c69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:42.573474 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.573439 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.573673 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.573500 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8683a65-9898-4b4f-a692-f2da325a93b4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.574458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.573758 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.574458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.573818 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.574458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.573886 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.574458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.573937 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.574458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.574097 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67fhl\" (UniqueName: \"kubernetes.io/projected/c8683a65-9898-4b4f-a692-f2da325a93b4-kube-api-access-67fhl\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.574458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.574131 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.574458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.574161 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.574458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.574337 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:42.574458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.574351 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:42.574458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.574362 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:42.574458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.574372 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:42.574458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.574381 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:42.574458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.574389 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhmgb\" (UniqueName: \"kubernetes.io/projected/63fb9260-c216-4e29-99c9-95dd0d4e4c69-kube-api-access-qhmgb\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:42.574458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.574398 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fb9260-c216-4e29-99c9-95dd0d4e4c69-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:42.574458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.574408 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fb9260-c216-4e29-99c9-95dd0d4e4c69-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:42.676232 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.676167 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67fhl\" (UniqueName: \"kubernetes.io/projected/c8683a65-9898-4b4f-a692-f2da325a93b4-kube-api-access-67fhl\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.676232 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.676208 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.676387 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.676231 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.676387 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.676288 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.676387 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.676344 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8683a65-9898-4b4f-a692-f2da325a93b4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.678858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.676485 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.678858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.676536 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.678858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.676560 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.678858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.676608 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.678858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.677163 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.678858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.677209 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.678858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.677332 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.678858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.677369 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.678858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.677531 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8683a65-9898-4b4f-a692-f2da325a93b4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.683214 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.680638 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.683214 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.680884 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.688831 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.688808 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.689617 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.689599 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67fhl\" (UniqueName: \"kubernetes.io/projected/c8683a65-9898-4b4f-a692-f2da325a93b4-kube-api-access-67fhl\") pod \"maas-default-gateway-openshift-default-855f9574cf-464r7\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:42.747609 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:42.747585 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:43.088822 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:43.088779 5059 generic.go:356] "Generic (PLEG): container finished" podID="63fb9260-c216-4e29-99c9-95dd0d4e4c69" containerID="832030defdd41fcfdac7fdd33897d9aafc05fd3cb9f6ae3535e2570a1aab65ed" exitCode=0 Nov 03 20:18:43.090088 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:43.088842 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" Nov 03 20:18:43.090088 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:43.088841 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" event={"ID":"63fb9260-c216-4e29-99c9-95dd0d4e4c69","Type":"ContainerDied","Data":"832030defdd41fcfdac7fdd33897d9aafc05fd3cb9f6ae3535e2570a1aab65ed"} Nov 03 20:18:43.090088 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:43.088994 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9" event={"ID":"63fb9260-c216-4e29-99c9-95dd0d4e4c69","Type":"ContainerDied","Data":"7abed323dd86db25ce994b774fd90800d0bc1b8deb9422fbb88d78811a4c23c6"} Nov 03 20:18:43.090088 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:43.089841 5059 scope.go:122] "RemoveContainer" containerID="832030defdd41fcfdac7fdd33897d9aafc05fd3cb9f6ae3535e2570a1aab65ed" Nov 03 20:18:43.100436 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:43.100410 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7"] Nov 03 20:18:43.104496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:43.104462 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:18:43.104611 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:43.104538 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:18:43.104611 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:43.104601 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:18:43.137430 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:43.137413 5059 scope.go:122] "RemoveContainer" containerID="832030defdd41fcfdac7fdd33897d9aafc05fd3cb9f6ae3535e2570a1aab65ed" Nov 03 20:18:43.138219 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:18:43.138197 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"832030defdd41fcfdac7fdd33897d9aafc05fd3cb9f6ae3535e2570a1aab65ed\": container with ID starting with 832030defdd41fcfdac7fdd33897d9aafc05fd3cb9f6ae3535e2570a1aab65ed not found: ID does not exist" containerID="832030defdd41fcfdac7fdd33897d9aafc05fd3cb9f6ae3535e2570a1aab65ed" Nov 03 20:18:43.138285 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:43.138229 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"832030defdd41fcfdac7fdd33897d9aafc05fd3cb9f6ae3535e2570a1aab65ed"} err="failed to get container status \"832030defdd41fcfdac7fdd33897d9aafc05fd3cb9f6ae3535e2570a1aab65ed\": rpc error: code = NotFound desc = could not find container \"832030defdd41fcfdac7fdd33897d9aafc05fd3cb9f6ae3535e2570a1aab65ed\": container with ID starting with 832030defdd41fcfdac7fdd33897d9aafc05fd3cb9f6ae3535e2570a1aab65ed not found: ID does not exist" Nov 03 20:18:43.147509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:43.147482 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9"] Nov 03 20:18:43.153088 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:43.153058 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-pknk9"] Nov 03 20:18:43.895440 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:43.895401 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63fb9260-c216-4e29-99c9-95dd0d4e4c69" path="/var/lib/kubelet/pods/63fb9260-c216-4e29-99c9-95dd0d4e4c69/volumes" Nov 03 20:18:44.083701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:44.082999 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87ed69a2-2w485"] Nov 03 20:18:44.099179 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:44.099141 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87ed69a2-2w485"] Nov 03 20:18:44.117156 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:44.117112 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" event={"ID":"c8683a65-9898-4b4f-a692-f2da325a93b4","Type":"ContainerStarted","Data":"d36cbc617302444ca97907469c12558d0fc4a6b46d78b847f1d5ccbc843b16e6"} Nov 03 20:18:44.117382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:44.117182 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" event={"ID":"c8683a65-9898-4b4f-a692-f2da325a93b4","Type":"ContainerStarted","Data":"58ce0025686f1eadbf4bc094248fadb9cc6bcf4ba93427714d48ab1519673f95"} Nov 03 20:18:44.146415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:44.146307 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" podStartSLOduration=122.146287826 podStartE2EDuration="2m2.146287826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:44.144697502 +0000 UTC m=+2518.402494760" watchObservedRunningTime="2026-11-03 20:18:44.146287826 +0000 UTC m=+2518.404085085" Nov 03 20:18:44.749703 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:44.748405 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:44.763687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:44.750235 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-464r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 03 20:18:44.763687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:44.750292 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" podUID="c8683a65-9898-4b4f-a692-f2da325a93b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 03 20:18:45.753685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:45.753029 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-464r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 03 20:18:45.753685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:45.753090 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" podUID="c8683a65-9898-4b4f-a692-f2da325a93b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 03 20:18:45.917687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:45.914324 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e98c550f-1c06-4553-b266-979be0615bd3" path="/var/lib/kubelet/pods/e98c550f-1c06-4553-b266-979be0615bd3/volumes" Nov 03 20:18:46.748624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:46.748572 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-464r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 03 20:18:46.748898 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:46.748641 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" podUID="c8683a65-9898-4b4f-a692-f2da325a93b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 03 20:18:47.751688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:47.751126 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-464r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 03 20:18:47.751688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:47.751205 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" podUID="c8683a65-9898-4b4f-a692-f2da325a93b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 03 20:18:47.911698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:47.885799 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:48.254687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:48.253880 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 20:18:48.748687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:48.748195 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-464r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 03 20:18:48.748687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:48.748277 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" podUID="c8683a65-9898-4b4f-a692-f2da325a93b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 03 20:18:48.776689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:48.774338 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:48.870697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:48.870290 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:49.420772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:49.420715 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vznnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:18:49.421118 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:49.421063 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" podUID="171c7288-3157-4fe0-80ee-94cf5b2ca2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:49.749180 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:49.749084 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-464r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 03 20:18:49.757976 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:49.757931 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" podUID="c8683a65-9898-4b4f-a692-f2da325a93b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 03 20:18:50.778505 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:50.778455 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-464r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 03 20:18:50.779275 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:50.779232 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" podUID="c8683a65-9898-4b4f-a692-f2da325a93b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 03 20:18:50.825792 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:50.825530 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25401a7d-6m5ws"] Nov 03 20:18:50.842551 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:50.842176 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25401a7d-6m5ws"] Nov 03 20:18:50.842551 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:50.842320 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25401a7d-6m5ws" Nov 03 20:18:50.854725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:50.850566 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:18:50.854725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:50.850789 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:18:50.882226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:50.882177 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96b45e9-ce7d-43cf-b2ac-4df96acca191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25401a7d-6m5ws\" (UID: \"b96b45e9-ce7d-43cf-b2ac-4df96acca191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25401a7d-6m5ws" Nov 03 20:18:50.883722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:50.882696 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82rg8\" (UniqueName: \"kubernetes.io/projected/b96b45e9-ce7d-43cf-b2ac-4df96acca191-kube-api-access-82rg8\") pod \"maas-api-revoke-sub-25401a7d-6m5ws\" (UID: \"b96b45e9-ce7d-43cf-b2ac-4df96acca191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25401a7d-6m5ws" Nov 03 20:18:50.883722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:50.882790 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96b45e9-ce7d-43cf-b2ac-4df96acca191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25401a7d-6m5ws\" (UID: \"b96b45e9-ce7d-43cf-b2ac-4df96acca191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25401a7d-6m5ws" Nov 03 20:18:50.985971 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:50.985920 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96b45e9-ce7d-43cf-b2ac-4df96acca191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25401a7d-6m5ws\" (UID: \"b96b45e9-ce7d-43cf-b2ac-4df96acca191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25401a7d-6m5ws" Nov 03 20:18:50.986167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:50.986121 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82rg8\" (UniqueName: \"kubernetes.io/projected/b96b45e9-ce7d-43cf-b2ac-4df96acca191-kube-api-access-82rg8\") pod \"maas-api-revoke-sub-25401a7d-6m5ws\" (UID: \"b96b45e9-ce7d-43cf-b2ac-4df96acca191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25401a7d-6m5ws" Nov 03 20:18:50.986267 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:50.986186 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96b45e9-ce7d-43cf-b2ac-4df96acca191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25401a7d-6m5ws\" (UID: \"b96b45e9-ce7d-43cf-b2ac-4df96acca191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25401a7d-6m5ws" Nov 03 20:18:50.987088 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:50.987054 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96b45e9-ce7d-43cf-b2ac-4df96acca191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25401a7d-6m5ws\" (UID: \"b96b45e9-ce7d-43cf-b2ac-4df96acca191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25401a7d-6m5ws" Nov 03 20:18:50.989139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:50.989112 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96b45e9-ce7d-43cf-b2ac-4df96acca191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25401a7d-6m5ws\" (UID: \"b96b45e9-ce7d-43cf-b2ac-4df96acca191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25401a7d-6m5ws" Nov 03 20:18:50.998600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:50.998571 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82rg8\" (UniqueName: \"kubernetes.io/projected/b96b45e9-ce7d-43cf-b2ac-4df96acca191-kube-api-access-82rg8\") pod \"maas-api-revoke-sub-25401a7d-6m5ws\" (UID: \"b96b45e9-ce7d-43cf-b2ac-4df96acca191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25401a7d-6m5ws" Nov 03 20:18:51.177042 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:51.177004 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25401a7d-6m5ws" Nov 03 20:18:51.749057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:51.749019 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-464r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 03 20:18:51.749212 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:51.749093 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" podUID="c8683a65-9898-4b4f-a692-f2da325a93b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 03 20:18:51.983734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:51.983697 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25401a7d-6m5ws"] Nov 03 20:18:51.986834 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:18:51.986792 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96b45e9_ce7d_43cf_b2ac_4df96acca191.slice/crio-0e9006389b1e41777d0073785f7d1afa0323bdd14564b878558ce71a0d3ee496 WatchSource:0}: Error finding container 0e9006389b1e41777d0073785f7d1afa0323bdd14564b878558ce71a0d3ee496: Status 404 returned error can't find the container with id 0e9006389b1e41777d0073785f7d1afa0323bdd14564b878558ce71a0d3ee496 Nov 03 20:18:52.717891 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:52.717857 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25401a7d-6m5ws" event={"ID":"b96b45e9-ce7d-43cf-b2ac-4df96acca191","Type":"ContainerStarted","Data":"94821d8bafaa395235edfb3518b58bbfa0a80434927536272c057fe1ccbaa4ad"} Nov 03 20:18:52.718092 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:52.717917 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25401a7d-6m5ws" event={"ID":"b96b45e9-ce7d-43cf-b2ac-4df96acca191","Type":"ContainerStarted","Data":"0e9006389b1e41777d0073785f7d1afa0323bdd14564b878558ce71a0d3ee496"} Nov 03 20:18:52.742897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:52.742851 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25401a7d-6m5ws" podStartSLOduration=2.742836661 podStartE2EDuration="2.742836661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:52.737957987 +0000 UTC m=+2526.995755245" watchObservedRunningTime="2026-11-03 20:18:52.742836661 +0000 UTC m=+2527.000633958" Nov 03 20:18:52.748734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:52.748699 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-464r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 03 20:18:52.748875 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:52.748762 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" podUID="c8683a65-9898-4b4f-a692-f2da325a93b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 03 20:18:53.735959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:53.735901 5059 generic.go:356] "Generic (PLEG): container finished" podID="b96b45e9-ce7d-43cf-b2ac-4df96acca191" containerID="94821d8bafaa395235edfb3518b58bbfa0a80434927536272c057fe1ccbaa4ad" exitCode=0 Nov 03 20:18:53.738854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:53.736014 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25401a7d-6m5ws" event={"ID":"b96b45e9-ce7d-43cf-b2ac-4df96acca191","Type":"ContainerDied","Data":"94821d8bafaa395235edfb3518b58bbfa0a80434927536272c057fe1ccbaa4ad"} Nov 03 20:18:54.749767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:54.749717 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-464r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:18:54.750238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:54.749785 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" podUID="c8683a65-9898-4b4f-a692-f2da325a93b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:55.144326 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:55.144301 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25401a7d-6m5ws" Nov 03 20:18:55.189866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:55.189831 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b96b45e9-ce7d-43cf-b2ac-4df96acca191-kube-api-access-82rg8\" (UniqueName: \"kubernetes.io/projected/b96b45e9-ce7d-43cf-b2ac-4df96acca191-kube-api-access-82rg8\") pod \"b96b45e9-ce7d-43cf-b2ac-4df96acca191\" (UID: \"b96b45e9-ce7d-43cf-b2ac-4df96acca191\") " Nov 03 20:18:55.190848 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:55.190190 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b96b45e9-ce7d-43cf-b2ac-4df96acca191-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96b45e9-ce7d-43cf-b2ac-4df96acca191-maas-api-openshift-ca\") pod \"b96b45e9-ce7d-43cf-b2ac-4df96acca191\" (UID: \"b96b45e9-ce7d-43cf-b2ac-4df96acca191\") " Nov 03 20:18:55.190848 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:55.190227 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b96b45e9-ce7d-43cf-b2ac-4df96acca191-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96b45e9-ce7d-43cf-b2ac-4df96acca191-maas-api-xks-ca\") pod \"b96b45e9-ce7d-43cf-b2ac-4df96acca191\" (UID: \"b96b45e9-ce7d-43cf-b2ac-4df96acca191\") " Nov 03 20:18:55.190848 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:55.190606 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96b45e9-ce7d-43cf-b2ac-4df96acca191-maas-api-openshift-ca" pod "b96b45e9-ce7d-43cf-b2ac-4df96acca191" (UID: "b96b45e9-ce7d-43cf-b2ac-4df96acca191"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:18:55.191337 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:55.191301 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96b45e9-ce7d-43cf-b2ac-4df96acca191-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:55.214873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:55.214839 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b96b45e9-ce7d-43cf-b2ac-4df96acca191-maas-api-xks-ca" pod "b96b45e9-ce7d-43cf-b2ac-4df96acca191" (UID: "b96b45e9-ce7d-43cf-b2ac-4df96acca191"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:18:55.215002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:55.214877 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96b45e9-ce7d-43cf-b2ac-4df96acca191-kube-api-access-82rg8" pod "b96b45e9-ce7d-43cf-b2ac-4df96acca191" (UID: "b96b45e9-ce7d-43cf-b2ac-4df96acca191"). InnerVolumeSpecName "kube-api-access-82rg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:55.293692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:55.293645 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82rg8\" (UniqueName: \"kubernetes.io/projected/b96b45e9-ce7d-43cf-b2ac-4df96acca191-kube-api-access-82rg8\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:55.294364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:55.293692 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96b45e9-ce7d-43cf-b2ac-4df96acca191-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:55.765688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:55.764179 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-464r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:18:55.765688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:55.764250 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" podUID="c8683a65-9898-4b4f-a692-f2da325a93b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:55.825928 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:55.825887 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25401a7d-6m5ws" event={"ID":"b96b45e9-ce7d-43cf-b2ac-4df96acca191","Type":"ContainerDied","Data":"0e9006389b1e41777d0073785f7d1afa0323bdd14564b878558ce71a0d3ee496"} Nov 03 20:18:55.826432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:55.825997 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e9006389b1e41777d0073785f7d1afa0323bdd14564b878558ce71a0d3ee496" Nov 03 20:18:55.826432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:55.826072 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25401a7d-6m5ws" Nov 03 20:18:56.281025 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:56.280991 5059 scope.go:122] "RemoveContainer" containerID="42779f04df1e4721774b3b54da0dbcfc59a67d87a90b4bd237140e7d01ff843b" Nov 03 20:18:56.805898 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:56.805851 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-464r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:18:56.807322 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:56.805922 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" podUID="c8683a65-9898-4b4f-a692-f2da325a93b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:57.319125 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:57.319087 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:57.320191 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:57.320162 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:18:57.321283 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:18:57.321200 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:19:10.707263 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:19:10.707216 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:19:21.322716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:21.261406 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 20:19:21.973739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:21.870766 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:21.973739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:21.945180 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded" Nov 03 20:19:21.973739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:21.955333 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:22.055962 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:21.975262 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 03 20:19:22.235626 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:22.235536 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:22.261826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:22.261776 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:22.608690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:22.602251 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 20:19:22.734367 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:22.732171 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 20:19:29.472900 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:29.472857 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf"] Nov 03 20:19:29.475855 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:29.473117 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" podUID="4df279a7-3d4b-4bad-9fd0-675243359616" containerName="istio-proxy" containerID="cri-o://906a4c87ed7c84ea2bc89825e56653fdc587d0ae53e1811d7698a2d74f9cd601" gracePeriod=30 Nov 03 20:19:29.481336 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:29.481305 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk"] Nov 03 20:19:29.482699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:29.481702 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" podUID="5b30b4c4-debd-4919-a4e6-67f7ee38eb90" containerName="istio-proxy" containerID="cri-o://6c77ae8b3fd95f803333003c66104b729e78fdae56d3aa26d441db19a235aa60" gracePeriod=30 Nov 03 20:19:29.487865 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:29.487840 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m"] Nov 03 20:19:29.488139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:29.488097 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" podUID="c15cd101-19a6-4a3c-8381-b175e1182bce" containerName="istio-proxy" containerID="cri-o://80953a1b9a91f162514cd4bebdcb490c46dfb0ce230b8b37a318d9925a83f540" gracePeriod=30 Nov 03 20:19:33.920171 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:33.920137 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:19:33.921121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:33.920190 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" podUID="5b30b4c4-debd-4919-a4e6-67f7ee38eb90" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:19:34.992918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:34.992891 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:19:35.034607 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.034342 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:19:35.040567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.040359 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:19:35.042443 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.042414 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h"] Nov 03 20:19:35.043577 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.043541 5059 cpu_manager.go:414] "containerMap: removing container" podUID="b96b45e9-ce7d-43cf-b2ac-4df96acca191" containerName="revoke-keys" Nov 03 20:19:35.043577 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.043569 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96b45e9-ce7d-43cf-b2ac-4df96acca191" containerName="revoke-keys" Nov 03 20:19:35.045218 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.043588 5059 cpu_manager.go:414] "containerMap: removing container" podUID="5b30b4c4-debd-4919-a4e6-67f7ee38eb90" containerName="istio-proxy" Nov 03 20:19:35.045218 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.043595 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b30b4c4-debd-4919-a4e6-67f7ee38eb90" containerName="istio-proxy" Nov 03 20:19:35.045218 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.043640 5059 cpu_manager.go:414] "containerMap: removing container" podUID="5b30b4c4-debd-4919-a4e6-67f7ee38eb90" containerName="istio-proxy" Nov 03 20:19:35.045218 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.043647 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b30b4c4-debd-4919-a4e6-67f7ee38eb90" containerName="istio-proxy" Nov 03 20:19:35.045218 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.043719 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c15cd101-19a6-4a3c-8381-b175e1182bce" containerName="istio-proxy" Nov 03 20:19:35.045218 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.043728 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15cd101-19a6-4a3c-8381-b175e1182bce" containerName="istio-proxy" Nov 03 20:19:35.045218 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.043772 5059 cpu_manager.go:414] "containerMap: removing container" podUID="4df279a7-3d4b-4bad-9fd0-675243359616" containerName="istio-proxy" Nov 03 20:19:35.045218 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.043816 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4df279a7-3d4b-4bad-9fd0-675243359616" containerName="istio-proxy" Nov 03 20:19:35.045218 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.044117 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4df279a7-3d4b-4bad-9fd0-675243359616" containerName="istio-proxy" Nov 03 20:19:35.045218 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.044173 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96b45e9-ce7d-43cf-b2ac-4df96acca191" containerName="revoke-keys" Nov 03 20:19:35.045218 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.044184 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b30b4c4-debd-4919-a4e6-67f7ee38eb90" containerName="istio-proxy" Nov 03 20:19:35.045218 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.044237 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15cd101-19a6-4a3c-8381-b175e1182bce" containerName="istio-proxy" Nov 03 20:19:35.052257 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.052237 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h"] Nov 03 20:19:35.053031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.052352 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h" Nov 03 20:19:35.058357 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.058331 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b503a8\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b503a8\"" Nov 03 20:19:35.061862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.061329 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b503a8\"/\"openshift-service-ca.crt\"" Nov 03 20:19:35.062656 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.062021 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b503a8\"/\"routing-overlay\"" Nov 03 20:19:35.062656 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.062118 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b503a8\"/\"agc-cred-4979dc0a8338\"" Nov 03 20:19:35.062656 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.062149 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b503a8\"/\"kube-root-ca.crt\"" Nov 03 20:19:35.066601 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.066568 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b503a8\"/\"payload-processing-external-model-e2e--834dd25f-dockercfg-n9wr5\"" Nov 03 20:19:35.072153 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.072126 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4df279a7-3d4b-4bad-9fd0-675243359616-istio-token\" (UniqueName: \"kubernetes.io/projected/4df279a7-3d4b-4bad-9fd0-675243359616-istio-token\") pod \"4df279a7-3d4b-4bad-9fd0-675243359616\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " Nov 03 20:19:35.072285 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.072270 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-istio-data\") pod \"4df279a7-3d4b-4bad-9fd0-675243359616\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " Nov 03 20:19:35.072402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.072371 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-credential-socket\") pod \"4df279a7-3d4b-4bad-9fd0-675243359616\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " Nov 03 20:19:35.072492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.072400 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-workload-certs\") pod \"4df279a7-3d4b-4bad-9fd0-675243359616\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " Nov 03 20:19:35.072492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.072454 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-workload-socket\") pod \"4df279a7-3d4b-4bad-9fd0-675243359616\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " Nov 03 20:19:35.072641 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.072527 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4df279a7-3d4b-4bad-9fd0-675243359616-kube-api-access-757bf\" (UniqueName: \"kubernetes.io/projected/4df279a7-3d4b-4bad-9fd0-675243359616-kube-api-access-757bf\") pod \"4df279a7-3d4b-4bad-9fd0-675243359616\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " Nov 03 20:19:35.072804 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.072642 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4df279a7-3d4b-4bad-9fd0-675243359616-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df279a7-3d4b-4bad-9fd0-675243359616-istiod-ca-cert\") pod \"4df279a7-3d4b-4bad-9fd0-675243359616\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " Nov 03 20:19:35.072804 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.072797 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-istio-envoy\") pod \"4df279a7-3d4b-4bad-9fd0-675243359616\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " Nov 03 20:19:35.072946 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.072843 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4df279a7-3d4b-4bad-9fd0-675243359616-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df279a7-3d4b-4bad-9fd0-675243359616-istio-podinfo\") pod \"4df279a7-3d4b-4bad-9fd0-675243359616\" (UID: \"4df279a7-3d4b-4bad-9fd0-675243359616\") " Nov 03 20:19:35.073904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.073425 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-workload-certs" pod "4df279a7-3d4b-4bad-9fd0-675243359616" (UID: "4df279a7-3d4b-4bad-9fd0-675243359616"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:35.074416 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.074202 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.074416 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.073431 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-istio-data" pod "4df279a7-3d4b-4bad-9fd0-675243359616" (UID: "4df279a7-3d4b-4bad-9fd0-675243359616"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:35.074416 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.074226 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-workload-socket" pod "4df279a7-3d4b-4bad-9fd0-675243359616" (UID: "4df279a7-3d4b-4bad-9fd0-675243359616"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:35.076369 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.074865 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df279a7-3d4b-4bad-9fd0-675243359616-istiod-ca-cert" pod "4df279a7-3d4b-4bad-9fd0-675243359616" (UID: "4df279a7-3d4b-4bad-9fd0-675243359616"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:35.076369 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.075161 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-credential-socket" pod "4df279a7-3d4b-4bad-9fd0-675243359616" (UID: "4df279a7-3d4b-4bad-9fd0-675243359616"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:35.102770 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.102734 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-istio-envoy" pod "4df279a7-3d4b-4bad-9fd0-675243359616" (UID: "4df279a7-3d4b-4bad-9fd0-675243359616"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:35.102992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.102755 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4df279a7-3d4b-4bad-9fd0-675243359616-istio-podinfo" pod "4df279a7-3d4b-4bad-9fd0-675243359616" (UID: "4df279a7-3d4b-4bad-9fd0-675243359616"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:19:35.103723 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.103648 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df279a7-3d4b-4bad-9fd0-675243359616-kube-api-access-757bf" pod "4df279a7-3d4b-4bad-9fd0-675243359616" (UID: "4df279a7-3d4b-4bad-9fd0-675243359616"). InnerVolumeSpecName "kube-api-access-757bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:35.104792 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.104008 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df279a7-3d4b-4bad-9fd0-675243359616-istio-token" pod "4df279a7-3d4b-4bad-9fd0-675243359616" (UID: "4df279a7-3d4b-4bad-9fd0-675243359616"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:35.175332 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.175295 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-credential-socket\") pod \"c15cd101-19a6-4a3c-8381-b175e1182bce\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " Nov 03 20:19:35.175495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.175346 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istiod-ca-cert\") pod \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " Nov 03 20:19:35.175495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.175435 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-envoy\") pod \"c15cd101-19a6-4a3c-8381-b175e1182bce\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " Nov 03 20:19:35.175633 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.175526 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15cd101-19a6-4a3c-8381-b175e1182bce-kube-api-access-g9zfr\" (UniqueName: \"kubernetes.io/projected/c15cd101-19a6-4a3c-8381-b175e1182bce-kube-api-access-g9zfr\") pod \"c15cd101-19a6-4a3c-8381-b175e1182bce\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " Nov 03 20:19:35.175633 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.175620 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-workload-certs\") pod \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " Nov 03 20:19:35.175769 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.175641 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-credential-socket" pod "c15cd101-19a6-4a3c-8381-b175e1182bce" (UID: "c15cd101-19a6-4a3c-8381-b175e1182bce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:35.175769 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.175700 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c15cd101-19a6-4a3c-8381-b175e1182bce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15cd101-19a6-4a3c-8381-b175e1182bce-istiod-ca-cert\") pod \"c15cd101-19a6-4a3c-8381-b175e1182bce\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " Nov 03 20:19:35.175769 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.175745 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-envoy\") pod \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.175803 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-data\") pod \"c15cd101-19a6-4a3c-8381-b175e1182bce\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.175832 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-token\" (UniqueName: \"kubernetes.io/projected/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-token\") pod \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.175869 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-data\") pod \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.175896 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istiod-ca-cert" pod "5b30b4c4-debd-4919-a4e6-67f7ee38eb90" (UID: "5b30b4c4-debd-4919-a4e6-67f7ee38eb90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.175900 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-workload-socket\") pod \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.175973 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-workload-socket\") pod \"c15cd101-19a6-4a3c-8381-b175e1182bce\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.176075 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-workload-certs\") pod \"c15cd101-19a6-4a3c-8381-b175e1182bce\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.176114 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-kube-api-access-6vrqz\" (UniqueName: \"kubernetes.io/projected/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-kube-api-access-6vrqz\") pod \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.176252 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-workload-socket" pod "5b30b4c4-debd-4919-a4e6-67f7ee38eb90" (UID: "5b30b4c4-debd-4919-a4e6-67f7ee38eb90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.176295 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-credential-socket\") pod \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.176297 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-workload-certs" pod "5b30b4c4-debd-4919-a4e6-67f7ee38eb90" (UID: "5b30b4c4-debd-4919-a4e6-67f7ee38eb90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.176360 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-podinfo\") pod \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\" (UID: \"5b30b4c4-debd-4919-a4e6-67f7ee38eb90\") " Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.176398 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-podinfo\") pod \"c15cd101-19a6-4a3c-8381-b175e1182bce\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.176501 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-token\" (UniqueName: \"kubernetes.io/projected/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-token\") pod \"c15cd101-19a6-4a3c-8381-b175e1182bce\" (UID: \"c15cd101-19a6-4a3c-8381-b175e1182bce\") " Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.176525 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-workload-certs" pod "c15cd101-19a6-4a3c-8381-b175e1182bce" (UID: "c15cd101-19a6-4a3c-8381-b175e1182bce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.176648 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15cd101-19a6-4a3c-8381-b175e1182bce-istiod-ca-cert" pod "c15cd101-19a6-4a3c-8381-b175e1182bce" (UID: "c15cd101-19a6-4a3c-8381-b175e1182bce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.177086 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-data" pod "5b30b4c4-debd-4919-a4e6-67f7ee38eb90" (UID: "5b30b4c4-debd-4919-a4e6-67f7ee38eb90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.177095 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h\" (UID: \"5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56\") " pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.177227 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdsts\" (UniqueName: \"kubernetes.io/projected/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-kube-api-access-sdsts\") pod \"payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h\" (UID: \"5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56\") " pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.177300 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h\" (UID: \"5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56\") " pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.177366 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h\" (UID: \"5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56\") " pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.177391 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-data" pod "c15cd101-19a6-4a3c-8381-b175e1182bce" (UID: "c15cd101-19a6-4a3c-8381-b175e1182bce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.177752 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.177778 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.177794 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.177809 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.177824 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-757bf\" (UniqueName: \"kubernetes.io/projected/4df279a7-3d4b-4bad-9fd0-675243359616-kube-api-access-757bf\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.177840 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df279a7-3d4b-4bad-9fd0-675243359616-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.177854 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.177869 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.177883 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15cd101-19a6-4a3c-8381-b175e1182bce-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.177898 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df279a7-3d4b-4bad-9fd0-675243359616-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.177911 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df279a7-3d4b-4bad-9fd0-675243359616-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.177927 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.177941 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.177955 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.177969 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.177982 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df279a7-3d4b-4bad-9fd0-675243359616-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.178439 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15cd101-19a6-4a3c-8381-b175e1182bce-kube-api-access-g9zfr" pod "c15cd101-19a6-4a3c-8381-b175e1182bce" (UID: "c15cd101-19a6-4a3c-8381-b175e1182bce"). InnerVolumeSpecName "kube-api-access-g9zfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.178485 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-workload-socket" pod "c15cd101-19a6-4a3c-8381-b175e1182bce" (UID: "c15cd101-19a6-4a3c-8381-b175e1182bce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:35.179260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.178838 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-credential-socket" pod "5b30b4c4-debd-4919-a4e6-67f7ee38eb90" (UID: "5b30b4c4-debd-4919-a4e6-67f7ee38eb90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:35.186175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.179378 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-envoy" pod "c15cd101-19a6-4a3c-8381-b175e1182bce" (UID: "c15cd101-19a6-4a3c-8381-b175e1182bce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:35.186175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.179510 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-token" pod "5b30b4c4-debd-4919-a4e6-67f7ee38eb90" (UID: "5b30b4c4-debd-4919-a4e6-67f7ee38eb90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:35.186175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.180015 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-token" pod "c15cd101-19a6-4a3c-8381-b175e1182bce" (UID: "c15cd101-19a6-4a3c-8381-b175e1182bce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:35.186175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.180710 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-envoy" pod "5b30b4c4-debd-4919-a4e6-67f7ee38eb90" (UID: "5b30b4c4-debd-4919-a4e6-67f7ee38eb90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:35.186175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.181129 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-podinfo" pod "c15cd101-19a6-4a3c-8381-b175e1182bce" (UID: "c15cd101-19a6-4a3c-8381-b175e1182bce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:19:35.186175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.181255 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-podinfo" pod "5b30b4c4-debd-4919-a4e6-67f7ee38eb90" (UID: "5b30b4c4-debd-4919-a4e6-67f7ee38eb90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:19:35.186175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.182702 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-kube-api-access-6vrqz" pod "5b30b4c4-debd-4919-a4e6-67f7ee38eb90" (UID: "5b30b4c4-debd-4919-a4e6-67f7ee38eb90"). InnerVolumeSpecName "kube-api-access-6vrqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:35.279279 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.279196 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h\" (UID: \"5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56\") " pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h" Nov 03 20:19:35.280462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.279311 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdsts\" (UniqueName: \"kubernetes.io/projected/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-kube-api-access-sdsts\") pod \"payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h\" (UID: \"5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56\") " pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h" Nov 03 20:19:35.280462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.279371 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h\" (UID: \"5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56\") " pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h" Nov 03 20:19:35.280462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.279425 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h\" (UID: \"5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56\") " pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h" Nov 03 20:19:35.280462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.279739 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.280462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.279760 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.280462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.279775 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.280462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.279791 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vrqz\" (UniqueName: \"kubernetes.io/projected/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-kube-api-access-6vrqz\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.280462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.279806 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.280462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.279819 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b30b4c4-debd-4919-a4e6-67f7ee38eb90-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.280462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.279832 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.280462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.279845 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.280462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.279860 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15cd101-19a6-4a3c-8381-b175e1182bce-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.280462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.279875 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9zfr\" (UniqueName: \"kubernetes.io/projected/c15cd101-19a6-4a3c-8381-b175e1182bce-kube-api-access-g9zfr\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.280462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.280078 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h\" (UID: \"5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56\") " pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h" Nov 03 20:19:35.280462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.280149 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h\" (UID: \"5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56\") " pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h" Nov 03 20:19:35.283950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.283928 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h\" (UID: \"5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56\") " pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h" Nov 03 20:19:35.292445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.292423 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdsts\" (UniqueName: \"kubernetes.io/projected/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-kube-api-access-sdsts\") pod \"payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h\" (UID: \"5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56\") " pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h" Nov 03 20:19:35.378081 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.378055 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h" Nov 03 20:19:35.431495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.431466 5059 generic.go:356] "Generic (PLEG): container finished" podID="4df279a7-3d4b-4bad-9fd0-675243359616" containerID="906a4c87ed7c84ea2bc89825e56653fdc587d0ae53e1811d7698a2d74f9cd601" exitCode=0 Nov 03 20:19:35.432460 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.431539 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" Nov 03 20:19:35.432460 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.431543 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" event={"ID":"4df279a7-3d4b-4bad-9fd0-675243359616","Type":"ContainerDied","Data":"906a4c87ed7c84ea2bc89825e56653fdc587d0ae53e1811d7698a2d74f9cd601"} Nov 03 20:19:35.432460 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.431618 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf" event={"ID":"4df279a7-3d4b-4bad-9fd0-675243359616","Type":"ContainerDied","Data":"05d728e5f8be7a486e15b72256103d1e7b63d82681fcecd499794af594a6c750"} Nov 03 20:19:35.432460 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.431650 5059 scope.go:122] "RemoveContainer" containerID="906a4c87ed7c84ea2bc89825e56653fdc587d0ae53e1811d7698a2d74f9cd601" Nov 03 20:19:35.434303 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.434274 5059 generic.go:356] "Generic (PLEG): container finished" podID="c15cd101-19a6-4a3c-8381-b175e1182bce" containerID="80953a1b9a91f162514cd4bebdcb490c46dfb0ce230b8b37a318d9925a83f540" exitCode=0 Nov 03 20:19:35.434415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.434305 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" event={"ID":"c15cd101-19a6-4a3c-8381-b175e1182bce","Type":"ContainerDied","Data":"80953a1b9a91f162514cd4bebdcb490c46dfb0ce230b8b37a318d9925a83f540"} Nov 03 20:19:35.434415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.434346 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" Nov 03 20:19:35.434415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.434368 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m" event={"ID":"c15cd101-19a6-4a3c-8381-b175e1182bce","Type":"ContainerDied","Data":"8031ebfdf68626e4eadfd7cfa669c3bccde1ce227c7c6df6a2760fb6ae1b8f49"} Nov 03 20:19:35.436824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.436803 5059 generic.go:356] "Generic (PLEG): container finished" podID="5b30b4c4-debd-4919-a4e6-67f7ee38eb90" containerID="6c77ae8b3fd95f803333003c66104b729e78fdae56d3aa26d441db19a235aa60" exitCode=0 Nov 03 20:19:35.436918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.436839 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" event={"ID":"5b30b4c4-debd-4919-a4e6-67f7ee38eb90","Type":"ContainerDied","Data":"6c77ae8b3fd95f803333003c66104b729e78fdae56d3aa26d441db19a235aa60"} Nov 03 20:19:35.436918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.436870 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" event={"ID":"5b30b4c4-debd-4919-a4e6-67f7ee38eb90","Type":"ContainerDied","Data":"dec408d6192a31fdba83a1d0a1facdcb6c5238a3754650932a40f4abf48c9e5a"} Nov 03 20:19:35.437806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.436939 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk" Nov 03 20:19:35.448349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.447753 5059 scope.go:122] "RemoveContainer" containerID="906a4c87ed7c84ea2bc89825e56653fdc587d0ae53e1811d7698a2d74f9cd601" Nov 03 20:19:35.451059 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:19:35.448752 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"906a4c87ed7c84ea2bc89825e56653fdc587d0ae53e1811d7698a2d74f9cd601\": container with ID starting with 906a4c87ed7c84ea2bc89825e56653fdc587d0ae53e1811d7698a2d74f9cd601 not found: ID does not exist" containerID="906a4c87ed7c84ea2bc89825e56653fdc587d0ae53e1811d7698a2d74f9cd601" Nov 03 20:19:35.451059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.448784 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906a4c87ed7c84ea2bc89825e56653fdc587d0ae53e1811d7698a2d74f9cd601"} err="failed to get container status \"906a4c87ed7c84ea2bc89825e56653fdc587d0ae53e1811d7698a2d74f9cd601\": rpc error: code = NotFound desc = could not find container \"906a4c87ed7c84ea2bc89825e56653fdc587d0ae53e1811d7698a2d74f9cd601\": container with ID starting with 906a4c87ed7c84ea2bc89825e56653fdc587d0ae53e1811d7698a2d74f9cd601 not found: ID does not exist" Nov 03 20:19:35.451059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.448802 5059 scope.go:122] "RemoveContainer" containerID="80953a1b9a91f162514cd4bebdcb490c46dfb0ce230b8b37a318d9925a83f540" Nov 03 20:19:35.484084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.464361 5059 scope.go:122] "RemoveContainer" containerID="80953a1b9a91f162514cd4bebdcb490c46dfb0ce230b8b37a318d9925a83f540" Nov 03 20:19:35.484084 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:19:35.464799 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80953a1b9a91f162514cd4bebdcb490c46dfb0ce230b8b37a318d9925a83f540\": container with ID starting with 80953a1b9a91f162514cd4bebdcb490c46dfb0ce230b8b37a318d9925a83f540 not found: ID does not exist" containerID="80953a1b9a91f162514cd4bebdcb490c46dfb0ce230b8b37a318d9925a83f540" Nov 03 20:19:35.484084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.464830 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80953a1b9a91f162514cd4bebdcb490c46dfb0ce230b8b37a318d9925a83f540"} err="failed to get container status \"80953a1b9a91f162514cd4bebdcb490c46dfb0ce230b8b37a318d9925a83f540\": rpc error: code = NotFound desc = could not find container \"80953a1b9a91f162514cd4bebdcb490c46dfb0ce230b8b37a318d9925a83f540\": container with ID starting with 80953a1b9a91f162514cd4bebdcb490c46dfb0ce230b8b37a318d9925a83f540 not found: ID does not exist" Nov 03 20:19:35.484084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.464847 5059 scope.go:122] "RemoveContainer" containerID="6c77ae8b3fd95f803333003c66104b729e78fdae56d3aa26d441db19a235aa60" Nov 03 20:19:35.484084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.469512 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf"] Nov 03 20:19:35.484084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.479971 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-mrfkf"] Nov 03 20:19:35.494050 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.494022 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk"] Nov 03 20:19:35.502402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.502369 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-hdmkk"] Nov 03 20:19:35.520776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.513002 5059 scope.go:122] "RemoveContainer" containerID="552ec3f06918ad1d09fc456296125df0f1384648fad63a5f292c3a93c5e6ece9" Nov 03 20:19:35.520776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.517757 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m"] Nov 03 20:19:35.541519 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.524136 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-s7c7m"] Nov 03 20:19:35.556829 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.556584 5059 scope.go:122] "RemoveContainer" containerID="6c77ae8b3fd95f803333003c66104b729e78fdae56d3aa26d441db19a235aa60" Nov 03 20:19:35.557560 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:19:35.557006 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c77ae8b3fd95f803333003c66104b729e78fdae56d3aa26d441db19a235aa60\": container with ID starting with 6c77ae8b3fd95f803333003c66104b729e78fdae56d3aa26d441db19a235aa60 not found: ID does not exist" containerID="6c77ae8b3fd95f803333003c66104b729e78fdae56d3aa26d441db19a235aa60" Nov 03 20:19:35.557560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.557032 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c77ae8b3fd95f803333003c66104b729e78fdae56d3aa26d441db19a235aa60"} err="failed to get container status \"6c77ae8b3fd95f803333003c66104b729e78fdae56d3aa26d441db19a235aa60\": rpc error: code = NotFound desc = could not find container \"6c77ae8b3fd95f803333003c66104b729e78fdae56d3aa26d441db19a235aa60\": container with ID starting with 6c77ae8b3fd95f803333003c66104b729e78fdae56d3aa26d441db19a235aa60 not found: ID does not exist" Nov 03 20:19:35.557560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.557053 5059 scope.go:122] "RemoveContainer" containerID="552ec3f06918ad1d09fc456296125df0f1384648fad63a5f292c3a93c5e6ece9" Nov 03 20:19:35.557560 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:19:35.557324 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"552ec3f06918ad1d09fc456296125df0f1384648fad63a5f292c3a93c5e6ece9\": container with ID starting with 552ec3f06918ad1d09fc456296125df0f1384648fad63a5f292c3a93c5e6ece9 not found: ID does not exist" containerID="552ec3f06918ad1d09fc456296125df0f1384648fad63a5f292c3a93c5e6ece9" Nov 03 20:19:35.557560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.557357 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552ec3f06918ad1d09fc456296125df0f1384648fad63a5f292c3a93c5e6ece9"} err="failed to get container status \"552ec3f06918ad1d09fc456296125df0f1384648fad63a5f292c3a93c5e6ece9\": rpc error: code = NotFound desc = could not find container \"552ec3f06918ad1d09fc456296125df0f1384648fad63a5f292c3a93c5e6ece9\": container with ID starting with 552ec3f06918ad1d09fc456296125df0f1384648fad63a5f292c3a93c5e6ece9 not found: ID does not exist" Nov 03 20:19:35.716422 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:19:35.716364 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5317e6f6_e1b2_4bea_a6a1_6e4439c6ce56.slice/crio-3a7a67940ebd337d1183d5db3111b377ce58c0c2c3b4762a7bfc99a9dc064db3 WatchSource:0}: Error finding container 3a7a67940ebd337d1183d5db3111b377ce58c0c2c3b4762a7bfc99a9dc064db3: Status 404 returned error can't find the container with id 3a7a67940ebd337d1183d5db3111b377ce58c0c2c3b4762a7bfc99a9dc064db3 Nov 03 20:19:35.716594 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.716539 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h"] Nov 03 20:19:35.860728 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.860699 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4df279a7-3d4b-4bad-9fd0-675243359616" path="/var/lib/kubelet/pods/4df279a7-3d4b-4bad-9fd0-675243359616/volumes" Nov 03 20:19:35.861245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.861230 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b30b4c4-debd-4919-a4e6-67f7ee38eb90" path="/var/lib/kubelet/pods/5b30b4c4-debd-4919-a4e6-67f7ee38eb90/volumes" Nov 03 20:19:35.861765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:35.861744 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c15cd101-19a6-4a3c-8381-b175e1182bce" path="/var/lib/kubelet/pods/c15cd101-19a6-4a3c-8381-b175e1182bce/volumes" Nov 03 20:19:36.449702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:36.449624 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h" event={"ID":"5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56","Type":"ContainerStarted","Data":"eba398fcd62354e2f9a8fd1d8c08ddf515bc6ab98059119a9ca256a880a3c73a"} Nov 03 20:19:36.450196 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:36.449729 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h" event={"ID":"5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56","Type":"ContainerStarted","Data":"3a7a67940ebd337d1183d5db3111b377ce58c0c2c3b4762a7bfc99a9dc064db3"} Nov 03 20:19:36.450196 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:36.449844 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h" Nov 03 20:19:36.475558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:36.475512 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h" podStartSLOduration=15.475499216 podStartE2EDuration="15.475499216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:36.473070361 +0000 UTC m=+2570.730867620" watchObservedRunningTime="2026-11-03 20:19:36.475499216 +0000 UTC m=+2570.733296508" Nov 03 20:19:40.607708 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:19:40.607649 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:19:43.760797 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:43.760745 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:46.020801 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:19:46.018707 5059 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-16659cd0d969: secret "agc-cred-16659cd0d969" not found Nov 03 20:19:46.020801 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:19:46.018766 5059 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f: secret "agc-cred-16659cd0d969" not found Nov 03 20:19:46.029823 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:19:46.026111 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84c8f0ee-a739-4315-9578-f8e61a0e4f67-provider-credentials podName:84c8f0ee-a739-4315-9578-f8e61a0e4f67 nodeName:}" failed. No retries permitted until 2026-11-03 20:19:46.526068382 +0000 UTC m=+2580.783865620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/84c8f0ee-a739-4315-9578-f8e61a0e4f67-provider-credentials") pod "payload-processing-external-model-788fcb4f8c-r8p9f" (UID: "84c8f0ee-a739-4315-9578-f8e61a0e4f67") : secret "agc-cred-16659cd0d969" not found Nov 03 20:19:46.157688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:46.157265 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f"] Nov 03 20:19:46.570728 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:19:46.567573 5059 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-16659cd0d969: secret "agc-cred-16659cd0d969" not found Nov 03 20:19:46.570728 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:19:46.567625 5059 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f: secret "agc-cred-16659cd0d969" not found Nov 03 20:19:46.575688 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:19:46.571650 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84c8f0ee-a739-4315-9578-f8e61a0e4f67-provider-credentials podName:84c8f0ee-a739-4315-9578-f8e61a0e4f67 nodeName:}" failed. No retries permitted until 2026-11-03 20:19:47.571444406 +0000 UTC m=+2581.829241654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/84c8f0ee-a739-4315-9578-f8e61a0e4f67-provider-credentials") pod "payload-processing-external-model-788fcb4f8c-r8p9f" (UID: "84c8f0ee-a739-4315-9578-f8e61a0e4f67") : secret "agc-cred-16659cd0d969" not found Nov 03 20:19:46.681841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:46.680736 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:46.757291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:46.756865 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 20:19:46.876960 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:46.875729 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:46.876960 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:46.875964 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:19:47.025234 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:47.025190 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Nov 03 20:19:47.488212 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:47.488179 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h" Nov 03 20:19:47.666647 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:19:47.665193 5059 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-16659cd0d969: secret "agc-cred-16659cd0d969" not found Nov 03 20:19:47.666647 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:19:47.665228 5059 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f: secret "agc-cred-16659cd0d969" not found Nov 03 20:19:47.666647 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:19:47.665291 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84c8f0ee-a739-4315-9578-f8e61a0e4f67-provider-credentials podName:84c8f0ee-a739-4315-9578-f8e61a0e4f67 nodeName:}" failed. No retries permitted until 2026-11-03 20:19:49.665274301 +0000 UTC m=+2583.923071536 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/84c8f0ee-a739-4315-9578-f8e61a0e4f67-provider-credentials") pod "payload-processing-external-model-788fcb4f8c-r8p9f" (UID: "84c8f0ee-a739-4315-9578-f8e61a0e4f67") : secret "agc-cred-16659cd0d969" not found Nov 03 20:19:48.632177 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:48.632139 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae219-5mpbm"] Nov 03 20:19:48.633105 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:48.633084 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b30b4c4-debd-4919-a4e6-67f7ee38eb90" containerName="istio-proxy" Nov 03 20:19:48.637155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:48.637129 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae219-5mpbm" Nov 03 20:19:48.642118 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:48.642095 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:19:48.642702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:48.642642 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:19:48.647619 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:48.647583 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae219-5mpbm"] Nov 03 20:19:48.788676 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:48.788636 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdbfw\" (UniqueName: \"kubernetes.io/projected/ea2b3d82-2b69-485e-a576-c8d805fca8b4-kube-api-access-zdbfw\") pod \"maas-api-revoke-sub-eb4ae219-5mpbm\" (UID: \"ea2b3d82-2b69-485e-a576-c8d805fca8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae219-5mpbm" Nov 03 20:19:48.788889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:48.788875 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea2b3d82-2b69-485e-a576-c8d805fca8b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb4ae219-5mpbm\" (UID: \"ea2b3d82-2b69-485e-a576-c8d805fca8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae219-5mpbm" Nov 03 20:19:48.789128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:48.789102 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea2b3d82-2b69-485e-a576-c8d805fca8b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb4ae219-5mpbm\" (UID: \"ea2b3d82-2b69-485e-a576-c8d805fca8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae219-5mpbm" Nov 03 20:19:48.890985 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:48.890895 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdbfw\" (UniqueName: \"kubernetes.io/projected/ea2b3d82-2b69-485e-a576-c8d805fca8b4-kube-api-access-zdbfw\") pod \"maas-api-revoke-sub-eb4ae219-5mpbm\" (UID: \"ea2b3d82-2b69-485e-a576-c8d805fca8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae219-5mpbm" Nov 03 20:19:48.891165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:48.891103 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea2b3d82-2b69-485e-a576-c8d805fca8b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb4ae219-5mpbm\" (UID: \"ea2b3d82-2b69-485e-a576-c8d805fca8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae219-5mpbm" Nov 03 20:19:48.892053 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:48.891239 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea2b3d82-2b69-485e-a576-c8d805fca8b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb4ae219-5mpbm\" (UID: \"ea2b3d82-2b69-485e-a576-c8d805fca8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae219-5mpbm" Nov 03 20:19:48.892541 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:48.892513 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea2b3d82-2b69-485e-a576-c8d805fca8b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb4ae219-5mpbm\" (UID: \"ea2b3d82-2b69-485e-a576-c8d805fca8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae219-5mpbm" Nov 03 20:19:48.893706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:48.893682 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea2b3d82-2b69-485e-a576-c8d805fca8b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb4ae219-5mpbm\" (UID: \"ea2b3d82-2b69-485e-a576-c8d805fca8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae219-5mpbm" Nov 03 20:19:48.969314 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:48.963559 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdbfw\" (UniqueName: \"kubernetes.io/projected/ea2b3d82-2b69-485e-a576-c8d805fca8b4-kube-api-access-zdbfw\") pod \"maas-api-revoke-sub-eb4ae219-5mpbm\" (UID: \"ea2b3d82-2b69-485e-a576-c8d805fca8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae219-5mpbm" Nov 03 20:19:48.979227 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:48.976993 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae219-5mpbm" Nov 03 20:19:49.322450 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:19:49.322398 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2b3d82_2b69_485e_a576_c8d805fca8b4.slice/crio-588dbb45d5f1a16899c26ed143f5c26bd583ca0cba1758cb71f7e343d6662d69 WatchSource:0}: Error finding container 588dbb45d5f1a16899c26ed143f5c26bd583ca0cba1758cb71f7e343d6662d69: Status 404 returned error can't find the container with id 588dbb45d5f1a16899c26ed143f5c26bd583ca0cba1758cb71f7e343d6662d69 Nov 03 20:19:49.322862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:49.322834 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae219-5mpbm"] Nov 03 20:19:49.803760 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:19:49.803724 5059 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-16659cd0d969: secret "agc-cred-16659cd0d969" not found Nov 03 20:19:49.808817 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:19:49.804281 5059 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f: secret "agc-cred-16659cd0d969" not found Nov 03 20:19:49.808817 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:19:49.804375 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84c8f0ee-a739-4315-9578-f8e61a0e4f67-provider-credentials podName:84c8f0ee-a739-4315-9578-f8e61a0e4f67 nodeName:}" failed. No retries permitted until 2026-11-03 20:19:53.804356121 +0000 UTC m=+2588.062153370 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/84c8f0ee-a739-4315-9578-f8e61a0e4f67-provider-credentials") pod "payload-processing-external-model-788fcb4f8c-r8p9f" (UID: "84c8f0ee-a739-4315-9578-f8e61a0e4f67") : secret "agc-cred-16659cd0d969" not found Nov 03 20:19:49.866393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:49.866348 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae219-5mpbm" event={"ID":"ea2b3d82-2b69-485e-a576-c8d805fca8b4","Type":"ContainerStarted","Data":"588dbb45d5f1a16899c26ed143f5c26bd583ca0cba1758cb71f7e343d6662d69"} Nov 03 20:19:50.867755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:50.867712 5059 generic.go:356] "Generic (PLEG): container finished" podID="ea2b3d82-2b69-485e-a576-c8d805fca8b4" containerID="4047cf868565f81ce559fe77432c44c45df4e1fead0fa2f927393c42875bfb67" exitCode=0 Nov 03 20:19:50.868145 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:50.867753 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae219-5mpbm" event={"ID":"ea2b3d82-2b69-485e-a576-c8d805fca8b4","Type":"ContainerDied","Data":"4047cf868565f81ce559fe77432c44c45df4e1fead0fa2f927393c42875bfb67"} Nov 03 20:19:51.813093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:51.813035 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f" podUID="84c8f0ee-a739-4315-9578-f8e61a0e4f67" containerName="payload-processing" containerID="cri-o://ac1c795b78380b6665eca11a2375a02a49880f00380c34bd86b8771507fe476c" gracePeriod=25 Nov 03 20:19:52.188331 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.188078 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae219-5mpbm" Nov 03 20:19:52.267048 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.267014 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea2b3d82-2b69-485e-a576-c8d805fca8b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea2b3d82-2b69-485e-a576-c8d805fca8b4-maas-api-xks-ca\") pod \"ea2b3d82-2b69-485e-a576-c8d805fca8b4\" (UID: \"ea2b3d82-2b69-485e-a576-c8d805fca8b4\") " Nov 03 20:19:52.267201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.267132 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea2b3d82-2b69-485e-a576-c8d805fca8b4-kube-api-access-zdbfw\" (UniqueName: \"kubernetes.io/projected/ea2b3d82-2b69-485e-a576-c8d805fca8b4-kube-api-access-zdbfw\") pod \"ea2b3d82-2b69-485e-a576-c8d805fca8b4\" (UID: \"ea2b3d82-2b69-485e-a576-c8d805fca8b4\") " Nov 03 20:19:52.268027 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.267210 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea2b3d82-2b69-485e-a576-c8d805fca8b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea2b3d82-2b69-485e-a576-c8d805fca8b4-maas-api-openshift-ca\") pod \"ea2b3d82-2b69-485e-a576-c8d805fca8b4\" (UID: \"ea2b3d82-2b69-485e-a576-c8d805fca8b4\") " Nov 03 20:19:52.268027 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.267728 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2b3d82-2b69-485e-a576-c8d805fca8b4-maas-api-openshift-ca" pod "ea2b3d82-2b69-485e-a576-c8d805fca8b4" (UID: "ea2b3d82-2b69-485e-a576-c8d805fca8b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:52.268410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.268379 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea2b3d82-2b69-485e-a576-c8d805fca8b4-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:52.270177 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.270139 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2b3d82-2b69-485e-a576-c8d805fca8b4-kube-api-access-zdbfw" pod "ea2b3d82-2b69-485e-a576-c8d805fca8b4" (UID: "ea2b3d82-2b69-485e-a576-c8d805fca8b4"). InnerVolumeSpecName "kube-api-access-zdbfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:52.270379 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.270354 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2b3d82-2b69-485e-a576-c8d805fca8b4-maas-api-xks-ca" pod "ea2b3d82-2b69-485e-a576-c8d805fca8b4" (UID: "ea2b3d82-2b69-485e-a576-c8d805fca8b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:19:52.279395 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.279375 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f" Nov 03 20:19:52.369969 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.369935 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84c8f0ee-a739-4315-9578-f8e61a0e4f67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c8f0ee-a739-4315-9578-f8e61a0e4f67-plugins-config-volume\") pod \"84c8f0ee-a739-4315-9578-f8e61a0e4f67\" (UID: \"84c8f0ee-a739-4315-9578-f8e61a0e4f67\") " Nov 03 20:19:52.370139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.370044 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c8f0ee-a739-4315-9578-f8e61a0e4f67-kube-api-access-crh5w\" (UniqueName: \"kubernetes.io/projected/84c8f0ee-a739-4315-9578-f8e61a0e4f67-kube-api-access-crh5w\") pod \"84c8f0ee-a739-4315-9578-f8e61a0e4f67\" (UID: \"84c8f0ee-a739-4315-9578-f8e61a0e4f67\") " Nov 03 20:19:52.370139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.370106 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84c8f0ee-a739-4315-9578-f8e61a0e4f67-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84c8f0ee-a739-4315-9578-f8e61a0e4f67-routing-overlay-volume\") pod \"84c8f0ee-a739-4315-9578-f8e61a0e4f67\" (UID: \"84c8f0ee-a739-4315-9578-f8e61a0e4f67\") " Nov 03 20:19:52.371436 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.370141 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c8f0ee-a739-4315-9578-f8e61a0e4f67-provider-credentials\" (UniqueName: \"kubernetes.io/projected/84c8f0ee-a739-4315-9578-f8e61a0e4f67-provider-credentials\") pod \"84c8f0ee-a739-4315-9578-f8e61a0e4f67\" (UID: \"84c8f0ee-a739-4315-9578-f8e61a0e4f67\") " Nov 03 20:19:52.371436 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.370456 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c8f0ee-a739-4315-9578-f8e61a0e4f67-plugins-config-volume" pod "84c8f0ee-a739-4315-9578-f8e61a0e4f67" (UID: "84c8f0ee-a739-4315-9578-f8e61a0e4f67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:52.371436 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.370501 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c8f0ee-a739-4315-9578-f8e61a0e4f67-routing-overlay-volume" pod "84c8f0ee-a739-4315-9578-f8e61a0e4f67" (UID: "84c8f0ee-a739-4315-9578-f8e61a0e4f67"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:52.371436 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.370818 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c8f0ee-a739-4315-9578-f8e61a0e4f67-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:52.371436 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.370833 5059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84c8f0ee-a739-4315-9578-f8e61a0e4f67-routing-overlay-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:52.371436 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.370843 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea2b3d82-2b69-485e-a576-c8d805fca8b4-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:52.371436 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.370857 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdbfw\" (UniqueName: \"kubernetes.io/projected/ea2b3d82-2b69-485e-a576-c8d805fca8b4-kube-api-access-zdbfw\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:52.372388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.372362 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c8f0ee-a739-4315-9578-f8e61a0e4f67-kube-api-access-crh5w" pod "84c8f0ee-a739-4315-9578-f8e61a0e4f67" (UID: "84c8f0ee-a739-4315-9578-f8e61a0e4f67"). InnerVolumeSpecName "kube-api-access-crh5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:52.372447 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.372414 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c8f0ee-a739-4315-9578-f8e61a0e4f67-provider-credentials" pod "84c8f0ee-a739-4315-9578-f8e61a0e4f67" (UID: "84c8f0ee-a739-4315-9578-f8e61a0e4f67"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:52.472098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.472010 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crh5w\" (UniqueName: \"kubernetes.io/projected/84c8f0ee-a739-4315-9578-f8e61a0e4f67-kube-api-access-crh5w\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:52.472098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.472046 5059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84c8f0ee-a739-4315-9578-f8e61a0e4f67-provider-credentials\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:52.890783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.890743 5059 generic.go:356] "Generic (PLEG): container finished" podID="84c8f0ee-a739-4315-9578-f8e61a0e4f67" containerID="ac1c795b78380b6665eca11a2375a02a49880f00380c34bd86b8771507fe476c" exitCode=0 Nov 03 20:19:52.891700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.890794 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f" Nov 03 20:19:52.891700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.890796 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f" event={"ID":"84c8f0ee-a739-4315-9578-f8e61a0e4f67","Type":"ContainerDied","Data":"ac1c795b78380b6665eca11a2375a02a49880f00380c34bd86b8771507fe476c"} Nov 03 20:19:52.891700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.890855 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f" event={"ID":"84c8f0ee-a739-4315-9578-f8e61a0e4f67","Type":"ContainerDied","Data":"1f66152db277af4cfc12a0fd739747178e9459ce1fc5d09434da6f3f92739b93"} Nov 03 20:19:52.891700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.890878 5059 scope.go:122] "RemoveContainer" containerID="ac1c795b78380b6665eca11a2375a02a49880f00380c34bd86b8771507fe476c" Nov 03 20:19:52.892591 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.892564 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae219-5mpbm" event={"ID":"ea2b3d82-2b69-485e-a576-c8d805fca8b4","Type":"ContainerDied","Data":"588dbb45d5f1a16899c26ed143f5c26bd583ca0cba1758cb71f7e343d6662d69"} Nov 03 20:19:52.892704 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.892617 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="588dbb45d5f1a16899c26ed143f5c26bd583ca0cba1758cb71f7e343d6662d69" Nov 03 20:19:52.892704 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.892577 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae219-5mpbm" Nov 03 20:19:52.911599 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.911580 5059 scope.go:122] "RemoveContainer" containerID="ac1c795b78380b6665eca11a2375a02a49880f00380c34bd86b8771507fe476c" Nov 03 20:19:52.913423 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:19:52.911877 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1c795b78380b6665eca11a2375a02a49880f00380c34bd86b8771507fe476c\": container with ID starting with ac1c795b78380b6665eca11a2375a02a49880f00380c34bd86b8771507fe476c not found: ID does not exist" containerID="ac1c795b78380b6665eca11a2375a02a49880f00380c34bd86b8771507fe476c" Nov 03 20:19:52.913423 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.911902 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1c795b78380b6665eca11a2375a02a49880f00380c34bd86b8771507fe476c"} err="failed to get container status \"ac1c795b78380b6665eca11a2375a02a49880f00380c34bd86b8771507fe476c\": rpc error: code = NotFound desc = could not find container \"ac1c795b78380b6665eca11a2375a02a49880f00380c34bd86b8771507fe476c\": container with ID starting with ac1c795b78380b6665eca11a2375a02a49880f00380c34bd86b8771507fe476c not found: ID does not exist" Nov 03 20:19:52.922837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.922816 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f"] Nov 03 20:19:52.927844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:52.927822 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-788fcb4f8c-r8p9f"] Nov 03 20:19:53.861154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:53.861120 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84c8f0ee-a739-4315-9578-f8e61a0e4f67" path="/var/lib/kubelet/pods/84c8f0ee-a739-4315-9578-f8e61a0e4f67/volumes" Nov 03 20:19:56.353692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:56.351109 5059 scope.go:122] "RemoveContainer" containerID="877ba5b6bb8922da79517f070ebf5ed7acbd22f66190671c82224d47eb7dc8ac" Nov 03 20:19:56.372060 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:56.372035 5059 scope.go:122] "RemoveContainer" containerID="4e655c42295929ba687cf5b5615e912fa9a10072e9f0e29b4771d3d696cb0dcb" Nov 03 20:19:56.383943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:56.383919 5059 scope.go:122] "RemoveContainer" containerID="f699606f369980d8290e3ce1e2cd9be2286f1319b3ed9a3df6365bb1d6c00987" Nov 03 20:19:56.394696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:56.394671 5059 scope.go:122] "RemoveContainer" containerID="b0fb18b05faf1c551b3d319859a3a3fbe61243db3aff2e8f5bf61992514fcd4e" Nov 03 20:19:56.408709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:56.408648 5059 scope.go:122] "RemoveContainer" containerID="ee192b7e2870a7575f388924d9b1bf31233af9ee5f207d6799a544ebe36fc804" Nov 03 20:19:56.419071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:56.419052 5059 scope.go:122] "RemoveContainer" containerID="5ed25179a720c02f6cd6430787db423d5e4ca1a560471a4494160bbc12737947" Nov 03 20:19:56.430173 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:56.430142 5059 scope.go:122] "RemoveContainer" containerID="1c8649494a394f35f22a91b1ef156c134f8777bae822ea21a75e4a1bc8656536" Nov 03 20:19:59.125523 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:59.125482 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20c16c95-k4k62"] Nov 03 20:19:59.127011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:59.126638 5059 cpu_manager.go:414] "containerMap: removing container" podUID="ea2b3d82-2b69-485e-a576-c8d805fca8b4" containerName="revoke-keys" Nov 03 20:19:59.127011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:59.126657 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2b3d82-2b69-485e-a576-c8d805fca8b4" containerName="revoke-keys" Nov 03 20:19:59.127011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:59.126720 5059 cpu_manager.go:414] "containerMap: removing container" podUID="84c8f0ee-a739-4315-9578-f8e61a0e4f67" containerName="payload-processing" Nov 03 20:19:59.127011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:59.126729 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c8f0ee-a739-4315-9578-f8e61a0e4f67" containerName="payload-processing" Nov 03 20:19:59.127011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:59.126910 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2b3d82-2b69-485e-a576-c8d805fca8b4" containerName="revoke-keys" Nov 03 20:19:59.127011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:59.126931 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c8f0ee-a739-4315-9578-f8e61a0e4f67" containerName="payload-processing" Nov 03 20:19:59.133625 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:59.133600 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c16c95-k4k62" Nov 03 20:19:59.137588 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:59.137566 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20c16c95-k4k62"] Nov 03 20:19:59.139419 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:59.139399 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:19:59.139479 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:59.139457 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:19:59.252212 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:59.219440 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c717fa3f-98a4-418a-a857-7425412e727f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20c16c95-k4k62\" (UID: \"c717fa3f-98a4-418a-a857-7425412e727f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c16c95-k4k62" Nov 03 20:19:59.263573 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:59.263529 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c717fa3f-98a4-418a-a857-7425412e727f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20c16c95-k4k62\" (UID: \"c717fa3f-98a4-418a-a857-7425412e727f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c16c95-k4k62" Nov 03 20:19:59.291825 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:59.291777 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6hjd\" (UniqueName: \"kubernetes.io/projected/c717fa3f-98a4-418a-a857-7425412e727f-kube-api-access-r6hjd\") pod \"maas-api-revoke-sub-20c16c95-k4k62\" (UID: \"c717fa3f-98a4-418a-a857-7425412e727f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c16c95-k4k62" Nov 03 20:19:59.393233 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:59.393147 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c717fa3f-98a4-418a-a857-7425412e727f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20c16c95-k4k62\" (UID: \"c717fa3f-98a4-418a-a857-7425412e727f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c16c95-k4k62" Nov 03 20:19:59.393405 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:59.393289 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c717fa3f-98a4-418a-a857-7425412e727f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20c16c95-k4k62\" (UID: \"c717fa3f-98a4-418a-a857-7425412e727f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c16c95-k4k62" Nov 03 20:19:59.393405 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:59.393391 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6hjd\" (UniqueName: \"kubernetes.io/projected/c717fa3f-98a4-418a-a857-7425412e727f-kube-api-access-r6hjd\") pod \"maas-api-revoke-sub-20c16c95-k4k62\" (UID: \"c717fa3f-98a4-418a-a857-7425412e727f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c16c95-k4k62" Nov 03 20:19:59.394469 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:59.393804 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c717fa3f-98a4-418a-a857-7425412e727f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20c16c95-k4k62\" (UID: \"c717fa3f-98a4-418a-a857-7425412e727f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c16c95-k4k62" Nov 03 20:19:59.395707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:59.395688 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c717fa3f-98a4-418a-a857-7425412e727f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20c16c95-k4k62\" (UID: \"c717fa3f-98a4-418a-a857-7425412e727f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c16c95-k4k62" Nov 03 20:19:59.405701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:59.405678 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6hjd\" (UniqueName: \"kubernetes.io/projected/c717fa3f-98a4-418a-a857-7425412e727f-kube-api-access-r6hjd\") pod \"maas-api-revoke-sub-20c16c95-k4k62\" (UID: \"c717fa3f-98a4-418a-a857-7425412e727f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c16c95-k4k62" Nov 03 20:19:59.451995 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:59.451970 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c16c95-k4k62" Nov 03 20:19:59.741258 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:19:59.741228 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc717fa3f_98a4_418a_a857_7425412e727f.slice/crio-ceab1fec65f74ec440b36ea0e406f692ce40c2497891cc4aec80517a42c42511 WatchSource:0}: Error finding container ceab1fec65f74ec440b36ea0e406f692ce40c2497891cc4aec80517a42c42511: Status 404 returned error can't find the container with id ceab1fec65f74ec440b36ea0e406f692ce40c2497891cc4aec80517a42c42511 Nov 03 20:19:59.741740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:59.741713 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20c16c95-k4k62"] Nov 03 20:19:59.980725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:19:59.980155 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c16c95-k4k62" event={"ID":"c717fa3f-98a4-418a-a857-7425412e727f","Type":"ContainerStarted","Data":"ceab1fec65f74ec440b36ea0e406f692ce40c2497891cc4aec80517a42c42511"} Nov 03 20:20:00.989590 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:00.989556 5059 generic.go:356] "Generic (PLEG): container finished" podID="c717fa3f-98a4-418a-a857-7425412e727f" containerID="981dfae86baecb93fdc63c743ed0c803df2f98640cac0233f4bc9661252407bf" exitCode=0 Nov 03 20:20:00.990021 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:00.989649 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c16c95-k4k62" event={"ID":"c717fa3f-98a4-418a-a857-7425412e727f","Type":"ContainerDied","Data":"981dfae86baecb93fdc63c743ed0c803df2f98640cac0233f4bc9661252407bf"} Nov 03 20:20:02.278013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:02.277988 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c16c95-k4k62" Nov 03 20:20:02.355566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:02.355526 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c717fa3f-98a4-418a-a857-7425412e727f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c717fa3f-98a4-418a-a857-7425412e727f-maas-api-openshift-ca\") pod \"c717fa3f-98a4-418a-a857-7425412e727f\" (UID: \"c717fa3f-98a4-418a-a857-7425412e727f\") " Nov 03 20:20:02.355775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:02.355577 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c717fa3f-98a4-418a-a857-7425412e727f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c717fa3f-98a4-418a-a857-7425412e727f-maas-api-xks-ca\") pod \"c717fa3f-98a4-418a-a857-7425412e727f\" (UID: \"c717fa3f-98a4-418a-a857-7425412e727f\") " Nov 03 20:20:02.355775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:02.355627 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c717fa3f-98a4-418a-a857-7425412e727f-kube-api-access-r6hjd\" (UniqueName: \"kubernetes.io/projected/c717fa3f-98a4-418a-a857-7425412e727f-kube-api-access-r6hjd\") pod \"c717fa3f-98a4-418a-a857-7425412e727f\" (UID: \"c717fa3f-98a4-418a-a857-7425412e727f\") " Nov 03 20:20:02.356145 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:02.356114 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c717fa3f-98a4-418a-a857-7425412e727f-maas-api-openshift-ca" pod "c717fa3f-98a4-418a-a857-7425412e727f" (UID: "c717fa3f-98a4-418a-a857-7425412e727f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:02.357072 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:02.357040 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c717fa3f-98a4-418a-a857-7425412e727f-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:02.357926 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:02.357898 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c717fa3f-98a4-418a-a857-7425412e727f-maas-api-xks-ca" pod "c717fa3f-98a4-418a-a857-7425412e727f" (UID: "c717fa3f-98a4-418a-a857-7425412e727f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:20:02.358595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:02.358569 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c717fa3f-98a4-418a-a857-7425412e727f-kube-api-access-r6hjd" pod "c717fa3f-98a4-418a-a857-7425412e727f" (UID: "c717fa3f-98a4-418a-a857-7425412e727f"). InnerVolumeSpecName "kube-api-access-r6hjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:02.458836 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:02.458805 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c717fa3f-98a4-418a-a857-7425412e727f-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:02.458836 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:02.458837 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6hjd\" (UniqueName: \"kubernetes.io/projected/c717fa3f-98a4-418a-a857-7425412e727f-kube-api-access-r6hjd\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:03.007509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:03.007471 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c16c95-k4k62" Nov 03 20:20:03.007743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:03.007509 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20c16c95-k4k62" event={"ID":"c717fa3f-98a4-418a-a857-7425412e727f","Type":"ContainerDied","Data":"ceab1fec65f74ec440b36ea0e406f692ce40c2497891cc4aec80517a42c42511"} Nov 03 20:20:03.007743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:03.007567 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ceab1fec65f74ec440b36ea0e406f692ce40c2497891cc4aec80517a42c42511" Nov 03 20:20:08.058645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:08.058612 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c52bfc6c-d9n4v"] Nov 03 20:20:08.059522 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:08.059504 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c717fa3f-98a4-418a-a857-7425412e727f" containerName="revoke-keys" Nov 03 20:20:08.059569 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:08.059523 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c717fa3f-98a4-418a-a857-7425412e727f" containerName="revoke-keys" Nov 03 20:20:08.059735 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:08.059723 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c717fa3f-98a4-418a-a857-7425412e727f" containerName="revoke-keys" Nov 03 20:20:08.063934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:08.063916 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52bfc6c-d9n4v" Nov 03 20:20:08.069449 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:08.069414 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:20:08.069731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:08.069417 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:20:08.072349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:08.072324 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c52bfc6c-d9n4v"] Nov 03 20:20:08.239799 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:08.239758 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3548c76a-c800-4837-a85c-aa4ffcc97d61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c52bfc6c-d9n4v\" (UID: \"3548c76a-c800-4837-a85c-aa4ffcc97d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52bfc6c-d9n4v" Nov 03 20:20:08.240066 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:08.240043 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j2xf\" (UniqueName: \"kubernetes.io/projected/3548c76a-c800-4837-a85c-aa4ffcc97d61-kube-api-access-6j2xf\") pod \"maas-api-revoke-sub-c52bfc6c-d9n4v\" (UID: \"3548c76a-c800-4837-a85c-aa4ffcc97d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52bfc6c-d9n4v" Nov 03 20:20:08.240358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:08.240337 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3548c76a-c800-4837-a85c-aa4ffcc97d61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c52bfc6c-d9n4v\" (UID: \"3548c76a-c800-4837-a85c-aa4ffcc97d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52bfc6c-d9n4v" Nov 03 20:20:08.342069 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:08.342036 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3548c76a-c800-4837-a85c-aa4ffcc97d61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c52bfc6c-d9n4v\" (UID: \"3548c76a-c800-4837-a85c-aa4ffcc97d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52bfc6c-d9n4v" Nov 03 20:20:08.342262 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:08.342138 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3548c76a-c800-4837-a85c-aa4ffcc97d61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c52bfc6c-d9n4v\" (UID: \"3548c76a-c800-4837-a85c-aa4ffcc97d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52bfc6c-d9n4v" Nov 03 20:20:08.342338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:08.342265 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j2xf\" (UniqueName: \"kubernetes.io/projected/3548c76a-c800-4837-a85c-aa4ffcc97d61-kube-api-access-6j2xf\") pod \"maas-api-revoke-sub-c52bfc6c-d9n4v\" (UID: \"3548c76a-c800-4837-a85c-aa4ffcc97d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52bfc6c-d9n4v" Nov 03 20:20:08.343414 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:08.342873 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3548c76a-c800-4837-a85c-aa4ffcc97d61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c52bfc6c-d9n4v\" (UID: \"3548c76a-c800-4837-a85c-aa4ffcc97d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52bfc6c-d9n4v" Nov 03 20:20:08.344640 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:08.344618 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3548c76a-c800-4837-a85c-aa4ffcc97d61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c52bfc6c-d9n4v\" (UID: \"3548c76a-c800-4837-a85c-aa4ffcc97d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52bfc6c-d9n4v" Nov 03 20:20:08.354909 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:08.354885 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j2xf\" (UniqueName: \"kubernetes.io/projected/3548c76a-c800-4837-a85c-aa4ffcc97d61-kube-api-access-6j2xf\") pod \"maas-api-revoke-sub-c52bfc6c-d9n4v\" (UID: \"3548c76a-c800-4837-a85c-aa4ffcc97d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52bfc6c-d9n4v" Nov 03 20:20:08.383598 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:08.383576 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52bfc6c-d9n4v" Nov 03 20:20:08.679605 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:20:08.679573 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3548c76a_c800_4837_a85c_aa4ffcc97d61.slice/crio-89808ce4f0fd7e785dcc046428d63c5e4e5d5e74bdbfae58ae5a98a40ff443c0 WatchSource:0}: Error finding container 89808ce4f0fd7e785dcc046428d63c5e4e5d5e74bdbfae58ae5a98a40ff443c0: Status 404 returned error can't find the container with id 89808ce4f0fd7e785dcc046428d63c5e4e5d5e74bdbfae58ae5a98a40ff443c0 Nov 03 20:20:08.682853 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:08.682725 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c52bfc6c-d9n4v"] Nov 03 20:20:09.258129 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:09.258095 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52bfc6c-d9n4v" event={"ID":"3548c76a-c800-4837-a85c-aa4ffcc97d61","Type":"ContainerStarted","Data":"7bef2c62c2b96701d07f59fbf898023344294a1924ab4eb7ca9da4476e94db98"} Nov 03 20:20:09.258519 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:09.258151 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52bfc6c-d9n4v" event={"ID":"3548c76a-c800-4837-a85c-aa4ffcc97d61","Type":"ContainerStarted","Data":"89808ce4f0fd7e785dcc046428d63c5e4e5d5e74bdbfae58ae5a98a40ff443c0"} Nov 03 20:20:09.278557 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:09.278509 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52bfc6c-d9n4v" podStartSLOduration=1.27849624 podStartE2EDuration="1.27849624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:09.277827585 +0000 UTC m=+2603.535624842" watchObservedRunningTime="2026-11-03 20:20:09.27849624 +0000 UTC m=+2603.536293496" Nov 03 20:20:10.267963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:10.267930 5059 generic.go:356] "Generic (PLEG): container finished" podID="3548c76a-c800-4837-a85c-aa4ffcc97d61" containerID="7bef2c62c2b96701d07f59fbf898023344294a1924ab4eb7ca9da4476e94db98" exitCode=0 Nov 03 20:20:10.268436 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:10.268015 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52bfc6c-d9n4v" event={"ID":"3548c76a-c800-4837-a85c-aa4ffcc97d61","Type":"ContainerDied","Data":"7bef2c62c2b96701d07f59fbf898023344294a1924ab4eb7ca9da4476e94db98"} Nov 03 20:20:10.649449 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:20:10.649412 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:20:11.623264 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:11.622866 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52bfc6c-d9n4v" Nov 03 20:20:11.743756 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:11.742033 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3548c76a-c800-4837-a85c-aa4ffcc97d61-kube-api-access-6j2xf\" (UniqueName: \"kubernetes.io/projected/3548c76a-c800-4837-a85c-aa4ffcc97d61-kube-api-access-6j2xf\") pod \"3548c76a-c800-4837-a85c-aa4ffcc97d61\" (UID: \"3548c76a-c800-4837-a85c-aa4ffcc97d61\") " Nov 03 20:20:11.743756 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:11.742121 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3548c76a-c800-4837-a85c-aa4ffcc97d61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3548c76a-c800-4837-a85c-aa4ffcc97d61-maas-api-openshift-ca\") pod \"3548c76a-c800-4837-a85c-aa4ffcc97d61\" (UID: \"3548c76a-c800-4837-a85c-aa4ffcc97d61\") " Nov 03 20:20:11.743756 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:11.742537 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3548c76a-c800-4837-a85c-aa4ffcc97d61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3548c76a-c800-4837-a85c-aa4ffcc97d61-maas-api-xks-ca\") pod \"3548c76a-c800-4837-a85c-aa4ffcc97d61\" (UID: \"3548c76a-c800-4837-a85c-aa4ffcc97d61\") " Nov 03 20:20:11.757227 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:11.757199 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3548c76a-c800-4837-a85c-aa4ffcc97d61-maas-api-openshift-ca" pod "3548c76a-c800-4837-a85c-aa4ffcc97d61" (UID: "3548c76a-c800-4837-a85c-aa4ffcc97d61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:12.014745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:12.013684 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3548c76a-c800-4837-a85c-aa4ffcc97d61-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:12.139982 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:12.139933 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3548c76a-c800-4837-a85c-aa4ffcc97d61-maas-api-xks-ca" pod "3548c76a-c800-4837-a85c-aa4ffcc97d61" (UID: "3548c76a-c800-4837-a85c-aa4ffcc97d61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:20:12.139982 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:12.139958 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3548c76a-c800-4837-a85c-aa4ffcc97d61-kube-api-access-6j2xf" pod "3548c76a-c800-4837-a85c-aa4ffcc97d61" (UID: "3548c76a-c800-4837-a85c-aa4ffcc97d61"). InnerVolumeSpecName "kube-api-access-6j2xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:12.229854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:12.229814 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3548c76a-c800-4837-a85c-aa4ffcc97d61-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:12.229854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:12.229849 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j2xf\" (UniqueName: \"kubernetes.io/projected/3548c76a-c800-4837-a85c-aa4ffcc97d61-kube-api-access-6j2xf\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:12.308686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:12.308565 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52bfc6c-d9n4v" event={"ID":"3548c76a-c800-4837-a85c-aa4ffcc97d61","Type":"ContainerDied","Data":"89808ce4f0fd7e785dcc046428d63c5e4e5d5e74bdbfae58ae5a98a40ff443c0"} Nov 03 20:20:12.308985 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:12.308963 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89808ce4f0fd7e785dcc046428d63c5e4e5d5e74bdbfae58ae5a98a40ff443c0" Nov 03 20:20:12.309192 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:12.309177 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52bfc6c-d9n4v" Nov 03 20:20:29.652685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:29.646338 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:20:30.066992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.066904 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf"] Nov 03 20:20:30.068087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.068062 5059 cpu_manager.go:414] "containerMap: removing container" podUID="3548c76a-c800-4837-a85c-aa4ffcc97d61" containerName="revoke-keys" Nov 03 20:20:30.068980 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.068087 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3548c76a-c800-4837-a85c-aa4ffcc97d61" containerName="revoke-keys" Nov 03 20:20:30.068980 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.068289 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3548c76a-c800-4837-a85c-aa4ffcc97d61" containerName="revoke-keys" Nov 03 20:20:30.072228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.072207 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf" Nov 03 20:20:30.079968 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.079943 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"agc-cred-53243bacc083\"" Nov 03 20:20:30.081489 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.081464 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"agc-cred-e5af0b75dc63\"" Nov 03 20:20:30.081624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.081468 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"routing-overlay\"" Nov 03 20:20:30.081740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.081468 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"openshift-service-ca.crt\"" Nov 03 20:20:30.081740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.081473 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"payload-processing-external-model-plugins-e2e-extmod-a-33e267\"" Nov 03 20:20:30.081740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.081503 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"kube-root-ca.crt\"" Nov 03 20:20:30.081740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.081508 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf"] Nov 03 20:20:30.081740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.081608 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"payload-processing-external-model-e2e--aa51d60f-dockercfg-wjs86\"" Nov 03 20:20:30.170406 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.170355 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9413e30-88c7-4379-b598-677e99898b65-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf\" (UID: \"a9413e30-88c7-4379-b598-677e99898b65\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf" Nov 03 20:20:30.170825 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.170798 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9413e30-88c7-4379-b598-677e99898b65-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf\" (UID: \"a9413e30-88c7-4379-b598-677e99898b65\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf" Nov 03 20:20:30.170944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.170875 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9413e30-88c7-4379-b598-677e99898b65-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf\" (UID: \"a9413e30-88c7-4379-b598-677e99898b65\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf" Nov 03 20:20:30.170944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.170930 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zlck\" (UniqueName: \"kubernetes.io/projected/a9413e30-88c7-4379-b598-677e99898b65-kube-api-access-2zlck\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf\" (UID: \"a9413e30-88c7-4379-b598-677e99898b65\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf" Nov 03 20:20:30.273103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.273050 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9413e30-88c7-4379-b598-677e99898b65-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf\" (UID: \"a9413e30-88c7-4379-b598-677e99898b65\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf" Nov 03 20:20:30.273289 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.273147 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9413e30-88c7-4379-b598-677e99898b65-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf\" (UID: \"a9413e30-88c7-4379-b598-677e99898b65\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf" Nov 03 20:20:30.273289 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.273206 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zlck\" (UniqueName: \"kubernetes.io/projected/a9413e30-88c7-4379-b598-677e99898b65-kube-api-access-2zlck\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf\" (UID: \"a9413e30-88c7-4379-b598-677e99898b65\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf" Nov 03 20:20:30.273438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.273318 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9413e30-88c7-4379-b598-677e99898b65-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf\" (UID: \"a9413e30-88c7-4379-b598-677e99898b65\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf" Nov 03 20:20:30.273890 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.273862 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9413e30-88c7-4379-b598-677e99898b65-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf\" (UID: \"a9413e30-88c7-4379-b598-677e99898b65\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf" Nov 03 20:20:30.274577 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.274039 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9413e30-88c7-4379-b598-677e99898b65-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf\" (UID: \"a9413e30-88c7-4379-b598-677e99898b65\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf" Nov 03 20:20:30.275992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.275972 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9413e30-88c7-4379-b598-677e99898b65-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf\" (UID: \"a9413e30-88c7-4379-b598-677e99898b65\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf" Nov 03 20:20:30.285783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.285763 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zlck\" (UniqueName: \"kubernetes.io/projected/a9413e30-88c7-4379-b598-677e99898b65-kube-api-access-2zlck\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf\" (UID: \"a9413e30-88c7-4379-b598-677e99898b65\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf" Nov 03 20:20:30.391841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:30.391815 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf" Nov 03 20:20:31.184018 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:31.183985 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3c252b2-tln54"] Nov 03 20:20:31.227344 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:31.227307 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3c252b2-tln54"] Nov 03 20:20:31.227521 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:31.227434 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c252b2-tln54" Nov 03 20:20:31.236403 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:31.236382 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:20:31.237305 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:31.237110 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:20:31.333606 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:31.333571 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvnwt\" (UniqueName: \"kubernetes.io/projected/197e1bd8-4f7a-4366-b0b9-88bcc9cf515a-kube-api-access-pvnwt\") pod \"maas-api-revoke-sub-d3c252b2-tln54\" (UID: \"197e1bd8-4f7a-4366-b0b9-88bcc9cf515a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c252b2-tln54" Nov 03 20:20:31.333756 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:31.333724 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/197e1bd8-4f7a-4366-b0b9-88bcc9cf515a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3c252b2-tln54\" (UID: \"197e1bd8-4f7a-4366-b0b9-88bcc9cf515a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c252b2-tln54" Nov 03 20:20:31.334062 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:31.334047 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/197e1bd8-4f7a-4366-b0b9-88bcc9cf515a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3c252b2-tln54\" (UID: \"197e1bd8-4f7a-4366-b0b9-88bcc9cf515a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c252b2-tln54" Nov 03 20:20:31.361044 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:31.361015 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf"] Nov 03 20:20:31.361456 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:20:31.361429 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9413e30_88c7_4379_b598_677e99898b65.slice/crio-e683f2ab9f57f64eb8b2eef535b21e24ce68d9f4f3fd58b39e35bbedab76216b WatchSource:0}: Error finding container e683f2ab9f57f64eb8b2eef535b21e24ce68d9f4f3fd58b39e35bbedab76216b: Status 404 returned error can't find the container with id e683f2ab9f57f64eb8b2eef535b21e24ce68d9f4f3fd58b39e35bbedab76216b Nov 03 20:20:31.435712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:31.435648 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/197e1bd8-4f7a-4366-b0b9-88bcc9cf515a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3c252b2-tln54\" (UID: \"197e1bd8-4f7a-4366-b0b9-88bcc9cf515a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c252b2-tln54" Nov 03 20:20:31.435831 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:31.435747 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvnwt\" (UniqueName: \"kubernetes.io/projected/197e1bd8-4f7a-4366-b0b9-88bcc9cf515a-kube-api-access-pvnwt\") pod \"maas-api-revoke-sub-d3c252b2-tln54\" (UID: \"197e1bd8-4f7a-4366-b0b9-88bcc9cf515a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c252b2-tln54" Nov 03 20:20:31.435831 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:31.435808 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/197e1bd8-4f7a-4366-b0b9-88bcc9cf515a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3c252b2-tln54\" (UID: \"197e1bd8-4f7a-4366-b0b9-88bcc9cf515a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c252b2-tln54" Nov 03 20:20:31.436420 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:31.436395 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/197e1bd8-4f7a-4366-b0b9-88bcc9cf515a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3c252b2-tln54\" (UID: \"197e1bd8-4f7a-4366-b0b9-88bcc9cf515a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c252b2-tln54" Nov 03 20:20:31.438219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:31.438196 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/197e1bd8-4f7a-4366-b0b9-88bcc9cf515a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3c252b2-tln54\" (UID: \"197e1bd8-4f7a-4366-b0b9-88bcc9cf515a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c252b2-tln54" Nov 03 20:20:31.464689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:31.449722 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvnwt\" (UniqueName: \"kubernetes.io/projected/197e1bd8-4f7a-4366-b0b9-88bcc9cf515a-kube-api-access-pvnwt\") pod \"maas-api-revoke-sub-d3c252b2-tln54\" (UID: \"197e1bd8-4f7a-4366-b0b9-88bcc9cf515a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c252b2-tln54" Nov 03 20:20:31.564990 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:31.564955 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c252b2-tln54" Nov 03 20:20:31.857908 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:20:31.857874 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod197e1bd8_4f7a_4366_b0b9_88bcc9cf515a.slice/crio-bca6c83a7aed1630b0691ae11d1090bf75a53939ec261704ae83225e334b1150 WatchSource:0}: Error finding container bca6c83a7aed1630b0691ae11d1090bf75a53939ec261704ae83225e334b1150: Status 404 returned error can't find the container with id bca6c83a7aed1630b0691ae11d1090bf75a53939ec261704ae83225e334b1150 Nov 03 20:20:31.862276 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:31.862251 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3c252b2-tln54"] Nov 03 20:20:32.248015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:32.247975 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c252b2-tln54" event={"ID":"197e1bd8-4f7a-4366-b0b9-88bcc9cf515a","Type":"ContainerStarted","Data":"2bb96bf56e8cc17a793ce1ddb74c44ae1dca020bdfeee2d770dda49eb3c3d502"} Nov 03 20:20:32.248423 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:32.248039 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c252b2-tln54" event={"ID":"197e1bd8-4f7a-4366-b0b9-88bcc9cf515a","Type":"ContainerStarted","Data":"bca6c83a7aed1630b0691ae11d1090bf75a53939ec261704ae83225e334b1150"} Nov 03 20:20:32.249801 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:32.249773 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf" event={"ID":"a9413e30-88c7-4379-b598-677e99898b65","Type":"ContainerStarted","Data":"3ac340f939011b76b5f8415b2bef3772f5c72dd74e4301b698de740f839445e8"} Nov 03 20:20:32.249903 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:32.249821 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf" event={"ID":"a9413e30-88c7-4379-b598-677e99898b65","Type":"ContainerStarted","Data":"e683f2ab9f57f64eb8b2eef535b21e24ce68d9f4f3fd58b39e35bbedab76216b"} Nov 03 20:20:32.249903 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:32.249871 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf" Nov 03 20:20:32.272964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:32.272917 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c252b2-tln54" podStartSLOduration=2.272904066 podStartE2EDuration="2.272904066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:32.266425813 +0000 UTC m=+2626.524223070" watchObservedRunningTime="2026-11-03 20:20:32.272904066 +0000 UTC m=+2626.530701355" Nov 03 20:20:32.289702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:32.289647 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf" podStartSLOduration=2.289635763 podStartE2EDuration="2.289635763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:32.28595598 +0000 UTC m=+2626.543753239" watchObservedRunningTime="2026-11-03 20:20:32.289635763 +0000 UTC m=+2626.547433020" Nov 03 20:20:33.262461 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:33.262423 5059 generic.go:356] "Generic (PLEG): container finished" podID="197e1bd8-4f7a-4366-b0b9-88bcc9cf515a" containerID="2bb96bf56e8cc17a793ce1ddb74c44ae1dca020bdfeee2d770dda49eb3c3d502" exitCode=0 Nov 03 20:20:33.262887 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:33.262499 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c252b2-tln54" event={"ID":"197e1bd8-4f7a-4366-b0b9-88bcc9cf515a","Type":"ContainerDied","Data":"2bb96bf56e8cc17a793ce1ddb74c44ae1dca020bdfeee2d770dda49eb3c3d502"} Nov 03 20:20:34.444119 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:34.444090 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c252b2-tln54" Nov 03 20:20:34.533604 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:34.533503 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/197e1bd8-4f7a-4366-b0b9-88bcc9cf515a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/197e1bd8-4f7a-4366-b0b9-88bcc9cf515a-maas-api-openshift-ca\") pod \"197e1bd8-4f7a-4366-b0b9-88bcc9cf515a\" (UID: \"197e1bd8-4f7a-4366-b0b9-88bcc9cf515a\") " Nov 03 20:20:34.533823 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:34.533687 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/197e1bd8-4f7a-4366-b0b9-88bcc9cf515a-kube-api-access-pvnwt\" (UniqueName: \"kubernetes.io/projected/197e1bd8-4f7a-4366-b0b9-88bcc9cf515a-kube-api-access-pvnwt\") pod \"197e1bd8-4f7a-4366-b0b9-88bcc9cf515a\" (UID: \"197e1bd8-4f7a-4366-b0b9-88bcc9cf515a\") " Nov 03 20:20:34.533898 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:34.533865 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/197e1bd8-4f7a-4366-b0b9-88bcc9cf515a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/197e1bd8-4f7a-4366-b0b9-88bcc9cf515a-maas-api-xks-ca\") pod \"197e1bd8-4f7a-4366-b0b9-88bcc9cf515a\" (UID: \"197e1bd8-4f7a-4366-b0b9-88bcc9cf515a\") " Nov 03 20:20:34.534043 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:34.534014 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/197e1bd8-4f7a-4366-b0b9-88bcc9cf515a-maas-api-openshift-ca" pod "197e1bd8-4f7a-4366-b0b9-88bcc9cf515a" (UID: "197e1bd8-4f7a-4366-b0b9-88bcc9cf515a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:34.534608 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:34.534578 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/197e1bd8-4f7a-4366-b0b9-88bcc9cf515a-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:34.536083 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:34.536054 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197e1bd8-4f7a-4366-b0b9-88bcc9cf515a-kube-api-access-pvnwt" pod "197e1bd8-4f7a-4366-b0b9-88bcc9cf515a" (UID: "197e1bd8-4f7a-4366-b0b9-88bcc9cf515a"). InnerVolumeSpecName "kube-api-access-pvnwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:34.536651 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:34.536618 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/197e1bd8-4f7a-4366-b0b9-88bcc9cf515a-maas-api-xks-ca" pod "197e1bd8-4f7a-4366-b0b9-88bcc9cf515a" (UID: "197e1bd8-4f7a-4366-b0b9-88bcc9cf515a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:20:34.642684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:34.642203 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvnwt\" (UniqueName: \"kubernetes.io/projected/197e1bd8-4f7a-4366-b0b9-88bcc9cf515a-kube-api-access-pvnwt\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:34.642684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:34.642245 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/197e1bd8-4f7a-4366-b0b9-88bcc9cf515a-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:35.026090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:35.026050 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a856579-v5qpb"] Nov 03 20:20:35.031160 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:35.031131 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a856579-v5qpb"] Nov 03 20:20:35.344826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:35.342360 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c252b2-tln54" event={"ID":"197e1bd8-4f7a-4366-b0b9-88bcc9cf515a","Type":"ContainerDied","Data":"bca6c83a7aed1630b0691ae11d1090bf75a53939ec261704ae83225e334b1150"} Nov 03 20:20:35.344826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:35.342465 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bca6c83a7aed1630b0691ae11d1090bf75a53939ec261704ae83225e334b1150" Nov 03 20:20:35.344826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:35.342538 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3c252b2-tln54" Nov 03 20:20:35.861731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:35.861699 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d02cb077-de4b-41cc-afeb-ea3b3bdb5728" path="/var/lib/kubelet/pods/d02cb077-de4b-41cc-afeb-ea3b3bdb5728/volumes" Nov 03 20:20:40.723687 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:20:40.708432 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:20:42.047065 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:42.046962 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-612e45b8-tq42s"] Nov 03 20:20:42.077914 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:42.077855 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-612e45b8-tq42s"] Nov 03 20:20:42.873688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:42.860331 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf"] Nov 03 20:20:42.958678 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:42.952834 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf" Nov 03 20:20:43.379266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:43.379222 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-464r7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:20:43.379895 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:43.379856 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" podUID="c8683a65-9898-4b4f-a692-f2da325a93b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:20:43.956969 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:43.956925 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8ccb587-5732-4719-8aa2-fa63bf33ed4a" path="/var/lib/kubelet/pods/e8ccb587-5732-4719-8aa2-fa63bf33ed4a/volumes" Nov 03 20:20:44.213739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:44.213570 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:20:44.818820 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:44.818768 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:20:45.057596 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.057559 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx"] Nov 03 20:20:45.059070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.059043 5059 cpu_manager.go:414] "containerMap: removing container" podUID="197e1bd8-4f7a-4366-b0b9-88bcc9cf515a" containerName="revoke-keys" Nov 03 20:20:45.059212 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.059200 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="197e1bd8-4f7a-4366-b0b9-88bcc9cf515a" containerName="revoke-keys" Nov 03 20:20:45.059528 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.059516 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="197e1bd8-4f7a-4366-b0b9-88bcc9cf515a" containerName="revoke-keys" Nov 03 20:20:45.127696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.127454 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx" Nov 03 20:20:45.135753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.134440 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"payload-processing-external-model-plugins-e2e-extmod-b-4c2a28\"" Nov 03 20:20:45.135753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.134521 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"kube-root-ca.crt\"" Nov 03 20:20:45.135753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.134597 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"agc-cred-0b519286d93f\"" Nov 03 20:20:45.135753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.134725 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"payload-processing-external-model-e2e--088821c0-dockercfg-jwvtb\"" Nov 03 20:20:45.135753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.134782 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"agc-cred-60dec61e44e9\"" Nov 03 20:20:45.135753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.134855 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"agc-cred-bc1cba871723\"" Nov 03 20:20:45.135753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.135106 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"openshift-service-ca.crt\"" Nov 03 20:20:45.135753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.135213 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"routing-overlay\"" Nov 03 20:20:45.145115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.145086 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f17f5a8b-xmn59"] Nov 03 20:20:45.201735 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.184705 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx"] Nov 03 20:20:45.393682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.388278 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e50ede5-4e49-4912-9e00-9a0d37ba0614-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx\" (UID: \"2e50ede5-4e49-4912-9e00-9a0d37ba0614\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx" Nov 03 20:20:45.393682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.388333 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e50ede5-4e49-4912-9e00-9a0d37ba0614-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx\" (UID: \"2e50ede5-4e49-4912-9e00-9a0d37ba0614\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx" Nov 03 20:20:45.393682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.388486 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e50ede5-4e49-4912-9e00-9a0d37ba0614-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx\" (UID: \"2e50ede5-4e49-4912-9e00-9a0d37ba0614\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx" Nov 03 20:20:45.393682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.388516 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkt9c\" (UniqueName: \"kubernetes.io/projected/2e50ede5-4e49-4912-9e00-9a0d37ba0614-kube-api-access-gkt9c\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx\" (UID: \"2e50ede5-4e49-4912-9e00-9a0d37ba0614\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx" Nov 03 20:20:45.585119 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.585083 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e50ede5-4e49-4912-9e00-9a0d37ba0614-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx\" (UID: \"2e50ede5-4e49-4912-9e00-9a0d37ba0614\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx" Nov 03 20:20:45.634711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.609488 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e50ede5-4e49-4912-9e00-9a0d37ba0614-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx\" (UID: \"2e50ede5-4e49-4912-9e00-9a0d37ba0614\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx" Nov 03 20:20:45.652487 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.652388 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e50ede5-4e49-4912-9e00-9a0d37ba0614-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx\" (UID: \"2e50ede5-4e49-4912-9e00-9a0d37ba0614\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx" Nov 03 20:20:45.652926 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.652898 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkt9c\" (UniqueName: \"kubernetes.io/projected/2e50ede5-4e49-4912-9e00-9a0d37ba0614-kube-api-access-gkt9c\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx\" (UID: \"2e50ede5-4e49-4912-9e00-9a0d37ba0614\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx" Nov 03 20:20:45.653922 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.621571 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e50ede5-4e49-4912-9e00-9a0d37ba0614-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx\" (UID: \"2e50ede5-4e49-4912-9e00-9a0d37ba0614\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx" Nov 03 20:20:45.655400 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.655379 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e50ede5-4e49-4912-9e00-9a0d37ba0614-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx\" (UID: \"2e50ede5-4e49-4912-9e00-9a0d37ba0614\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx" Nov 03 20:20:45.663253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.663201 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f17f5a8b-xmn59"] Nov 03 20:20:45.663905 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.663417 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:20:45.933856 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:45.933777 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e50ede5-4e49-4912-9e00-9a0d37ba0614-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx\" (UID: \"2e50ede5-4e49-4912-9e00-9a0d37ba0614\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx" Nov 03 20:20:46.074441 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:46.074403 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkt9c\" (UniqueName: \"kubernetes.io/projected/2e50ede5-4e49-4912-9e00-9a0d37ba0614-kube-api-access-gkt9c\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx\" (UID: \"2e50ede5-4e49-4912-9e00-9a0d37ba0614\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx" Nov 03 20:20:46.150173 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:46.150125 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"payload-processing-external-model-e2e--088821c0-dockercfg-jwvtb\"" Nov 03 20:20:46.162697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:46.151801 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx" Nov 03 20:20:46.506787 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:46.484413 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f16ace24-5836-44cf-a270-faae87720720" path="/var/lib/kubelet/pods/f16ace24-5836-44cf-a270-faae87720720/volumes" Nov 03 20:20:46.988128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:46.987979 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx"] Nov 03 20:20:46.990149 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:20:46.990115 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e50ede5_4e49_4912_9e00_9a0d37ba0614.slice/crio-f6e76b1f4eb653d18779761d5d74f3247407dc1680b4f02f5dc339e76aacda58 WatchSource:0}: Error finding container f6e76b1f4eb653d18779761d5d74f3247407dc1680b4f02f5dc339e76aacda58: Status 404 returned error can't find the container with id f6e76b1f4eb653d18779761d5d74f3247407dc1680b4f02f5dc339e76aacda58 Nov 03 20:20:47.499204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:47.499153 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx" event={"ID":"2e50ede5-4e49-4912-9e00-9a0d37ba0614","Type":"ContainerStarted","Data":"67fa604274d25af74fb4ee60ff533af52d0922d1d3635aceaa63a1b4afd51d87"} Nov 03 20:20:47.499458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:47.499233 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx" event={"ID":"2e50ede5-4e49-4912-9e00-9a0d37ba0614","Type":"ContainerStarted","Data":"f6e76b1f4eb653d18779761d5d74f3247407dc1680b4f02f5dc339e76aacda58"} Nov 03 20:20:47.499458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:47.499320 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx" Nov 03 20:20:47.523266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:47.523214 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx" podStartSLOduration=3.523199435 podStartE2EDuration="3.523199435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:47.519105468 +0000 UTC m=+2641.776902725" watchObservedRunningTime="2026-11-03 20:20:47.523199435 +0000 UTC m=+2641.780996722" Nov 03 20:20:47.867097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:47.867034 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf" podUID="a9413e30-88c7-4379-b598-677e99898b65" containerName="payload-processing" containerID="cri-o://3ac340f939011b76b5f8415b2bef3772f5c72dd74e4301b698de740f839445e8" gracePeriod=25 Nov 03 20:20:48.632596 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:48.632557 5059 generic.go:356] "Generic (PLEG): container finished" podID="a9413e30-88c7-4379-b598-677e99898b65" containerID="3ac340f939011b76b5f8415b2bef3772f5c72dd74e4301b698de740f839445e8" exitCode=0 Nov 03 20:20:48.636344 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:48.636310 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf" event={"ID":"a9413e30-88c7-4379-b598-677e99898b65","Type":"ContainerDied","Data":"3ac340f939011b76b5f8415b2bef3772f5c72dd74e4301b698de740f839445e8"} Nov 03 20:20:48.636637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:48.636610 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf" event={"ID":"a9413e30-88c7-4379-b598-677e99898b65","Type":"ContainerDied","Data":"e683f2ab9f57f64eb8b2eef535b21e24ce68d9f4f3fd58b39e35bbedab76216b"} Nov 03 20:20:48.636801 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:48.636780 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e683f2ab9f57f64eb8b2eef535b21e24ce68d9f4f3fd58b39e35bbedab76216b" Nov 03 20:20:48.640856 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:48.640814 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf" Nov 03 20:20:48.781470 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:48.780817 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9413e30-88c7-4379-b598-677e99898b65-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9413e30-88c7-4379-b598-677e99898b65-routing-overlay-volume\") pod \"a9413e30-88c7-4379-b598-677e99898b65\" (UID: \"a9413e30-88c7-4379-b598-677e99898b65\") " Nov 03 20:20:48.783427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:48.781949 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9413e30-88c7-4379-b598-677e99898b65-kube-api-access-2zlck\" (UniqueName: \"kubernetes.io/projected/a9413e30-88c7-4379-b598-677e99898b65-kube-api-access-2zlck\") pod \"a9413e30-88c7-4379-b598-677e99898b65\" (UID: \"a9413e30-88c7-4379-b598-677e99898b65\") " Nov 03 20:20:48.783427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:48.782189 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9413e30-88c7-4379-b598-677e99898b65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9413e30-88c7-4379-b598-677e99898b65-plugins-config-volume\") pod \"a9413e30-88c7-4379-b598-677e99898b65\" (UID: \"a9413e30-88c7-4379-b598-677e99898b65\") " Nov 03 20:20:48.783427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:48.782355 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9413e30-88c7-4379-b598-677e99898b65-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9413e30-88c7-4379-b598-677e99898b65-provider-credentials\") pod \"a9413e30-88c7-4379-b598-677e99898b65\" (UID: \"a9413e30-88c7-4379-b598-677e99898b65\") " Nov 03 20:20:48.784817 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:48.784397 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9413e30-88c7-4379-b598-677e99898b65-routing-overlay-volume" pod "a9413e30-88c7-4379-b598-677e99898b65" (UID: "a9413e30-88c7-4379-b598-677e99898b65"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:48.789246 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:48.789110 5059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9413e30-88c7-4379-b598-677e99898b65-routing-overlay-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:48.791758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:48.791331 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9413e30-88c7-4379-b598-677e99898b65-plugins-config-volume" pod "a9413e30-88c7-4379-b598-677e99898b65" (UID: "a9413e30-88c7-4379-b598-677e99898b65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:48.794490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:48.794465 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9413e30-88c7-4379-b598-677e99898b65-provider-credentials" pod "a9413e30-88c7-4379-b598-677e99898b65" (UID: "a9413e30-88c7-4379-b598-677e99898b65"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:48.794899 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:48.794874 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9413e30-88c7-4379-b598-677e99898b65-kube-api-access-2zlck" pod "a9413e30-88c7-4379-b598-677e99898b65" (UID: "a9413e30-88c7-4379-b598-677e99898b65"). InnerVolumeSpecName "kube-api-access-2zlck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:48.890478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:48.890444 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9413e30-88c7-4379-b598-677e99898b65-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:48.890478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:48.890473 5059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9413e30-88c7-4379-b598-677e99898b65-provider-credentials\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:48.890478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:48.890484 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zlck\" (UniqueName: \"kubernetes.io/projected/a9413e30-88c7-4379-b598-677e99898b65-kube-api-access-2zlck\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:49.644429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:49.644385 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf" Nov 03 20:20:49.677523 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:49.677493 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf"] Nov 03 20:20:49.682552 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:49.682525 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fjdhpf"] Nov 03 20:20:49.862141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:49.862109 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9413e30-88c7-4379-b598-677e99898b65" path="/var/lib/kubelet/pods/a9413e30-88c7-4379-b598-677e99898b65/volumes" Nov 03 20:20:52.970244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:52.970207 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bec3b13-hb84q"] Nov 03 20:20:52.971014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:52.970999 5059 cpu_manager.go:414] "containerMap: removing container" podUID="a9413e30-88c7-4379-b598-677e99898b65" containerName="payload-processing" Nov 03 20:20:52.971952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:52.971016 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9413e30-88c7-4379-b598-677e99898b65" containerName="payload-processing" Nov 03 20:20:52.971952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:52.971142 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9413e30-88c7-4379-b598-677e99898b65" containerName="payload-processing" Nov 03 20:20:52.975822 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:52.975802 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bec3b13-hb84q" Nov 03 20:20:52.983953 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:52.983928 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bec3b13-hb84q"] Nov 03 20:20:53.036794 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:53.033620 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f29306d-863d-484a-80b7-e05e6effbd51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bec3b13-hb84q\" (UID: \"2f29306d-863d-484a-80b7-e05e6effbd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bec3b13-hb84q" Nov 03 20:20:53.036794 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:53.033788 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmbht\" (UniqueName: \"kubernetes.io/projected/2f29306d-863d-484a-80b7-e05e6effbd51-kube-api-access-kmbht\") pod \"maas-api-revoke-sub-0bec3b13-hb84q\" (UID: \"2f29306d-863d-484a-80b7-e05e6effbd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bec3b13-hb84q" Nov 03 20:20:53.036794 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:53.033836 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f29306d-863d-484a-80b7-e05e6effbd51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bec3b13-hb84q\" (UID: \"2f29306d-863d-484a-80b7-e05e6effbd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bec3b13-hb84q" Nov 03 20:20:53.048560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:53.048527 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:20:53.048719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:53.048606 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:20:53.136406 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:53.136371 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f29306d-863d-484a-80b7-e05e6effbd51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bec3b13-hb84q\" (UID: \"2f29306d-863d-484a-80b7-e05e6effbd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bec3b13-hb84q" Nov 03 20:20:53.137193 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:53.136468 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmbht\" (UniqueName: \"kubernetes.io/projected/2f29306d-863d-484a-80b7-e05e6effbd51-kube-api-access-kmbht\") pod \"maas-api-revoke-sub-0bec3b13-hb84q\" (UID: \"2f29306d-863d-484a-80b7-e05e6effbd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bec3b13-hb84q" Nov 03 20:20:53.137193 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:53.136503 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f29306d-863d-484a-80b7-e05e6effbd51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bec3b13-hb84q\" (UID: \"2f29306d-863d-484a-80b7-e05e6effbd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bec3b13-hb84q" Nov 03 20:20:53.137312 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:53.137264 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f29306d-863d-484a-80b7-e05e6effbd51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bec3b13-hb84q\" (UID: \"2f29306d-863d-484a-80b7-e05e6effbd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bec3b13-hb84q" Nov 03 20:20:53.138862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:53.138842 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f29306d-863d-484a-80b7-e05e6effbd51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bec3b13-hb84q\" (UID: \"2f29306d-863d-484a-80b7-e05e6effbd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bec3b13-hb84q" Nov 03 20:20:53.149674 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:53.149634 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmbht\" (UniqueName: \"kubernetes.io/projected/2f29306d-863d-484a-80b7-e05e6effbd51-kube-api-access-kmbht\") pod \"maas-api-revoke-sub-0bec3b13-hb84q\" (UID: \"2f29306d-863d-484a-80b7-e05e6effbd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bec3b13-hb84q" Nov 03 20:20:53.292778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:53.292696 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bec3b13-hb84q" Nov 03 20:20:53.601946 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:20:53.601912 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f29306d_863d_484a_80b7_e05e6effbd51.slice/crio-4b2f753afaa6a23d743356e98c51f43b22c554f99aed223392d8b5ac561b6814 WatchSource:0}: Error finding container 4b2f753afaa6a23d743356e98c51f43b22c554f99aed223392d8b5ac561b6814: Status 404 returned error can't find the container with id 4b2f753afaa6a23d743356e98c51f43b22c554f99aed223392d8b5ac561b6814 Nov 03 20:20:53.602727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:53.602698 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bec3b13-hb84q"] Nov 03 20:20:53.952132 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:53.952041 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bec3b13-hb84q" event={"ID":"2f29306d-863d-484a-80b7-e05e6effbd51","Type":"ContainerStarted","Data":"4b2f753afaa6a23d743356e98c51f43b22c554f99aed223392d8b5ac561b6814"} Nov 03 20:20:55.014882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:55.002539 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bec3b13-hb84q" event={"ID":"2f29306d-863d-484a-80b7-e05e6effbd51","Type":"ContainerStarted","Data":"502104f017af359d04ca3c8ec6ed2634250c518da5bf724e38ed745f86c73474"} Nov 03 20:20:55.172682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:55.168794 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bec3b13-hb84q" podStartSLOduration=3.168772562 podStartE2EDuration="3.168772562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:55.044728314 +0000 UTC m=+2649.302525597" watchObservedRunningTime="2026-11-03 20:20:55.168772562 +0000 UTC m=+2649.426569825" Nov 03 20:20:56.166691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:56.157354 5059 generic.go:356] "Generic (PLEG): container finished" podID="2f29306d-863d-484a-80b7-e05e6effbd51" containerID="502104f017af359d04ca3c8ec6ed2634250c518da5bf724e38ed745f86c73474" exitCode=0 Nov 03 20:20:56.166691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:56.157731 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bec3b13-hb84q" event={"ID":"2f29306d-863d-484a-80b7-e05e6effbd51","Type":"ContainerDied","Data":"502104f017af359d04ca3c8ec6ed2634250c518da5bf724e38ed745f86c73474"} Nov 03 20:20:57.350918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:57.350892 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bec3b13-hb84q" Nov 03 20:20:57.378559 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:57.378527 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f29306d-863d-484a-80b7-e05e6effbd51-kube-api-access-kmbht\" (UniqueName: \"kubernetes.io/projected/2f29306d-863d-484a-80b7-e05e6effbd51-kube-api-access-kmbht\") pod \"2f29306d-863d-484a-80b7-e05e6effbd51\" (UID: \"2f29306d-863d-484a-80b7-e05e6effbd51\") " Nov 03 20:20:57.379615 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:57.378815 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f29306d-863d-484a-80b7-e05e6effbd51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f29306d-863d-484a-80b7-e05e6effbd51-maas-api-xks-ca\") pod \"2f29306d-863d-484a-80b7-e05e6effbd51\" (UID: \"2f29306d-863d-484a-80b7-e05e6effbd51\") " Nov 03 20:20:57.379615 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:57.378873 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f29306d-863d-484a-80b7-e05e6effbd51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f29306d-863d-484a-80b7-e05e6effbd51-maas-api-openshift-ca\") pod \"2f29306d-863d-484a-80b7-e05e6effbd51\" (UID: \"2f29306d-863d-484a-80b7-e05e6effbd51\") " Nov 03 20:20:57.379615 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:57.379262 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f29306d-863d-484a-80b7-e05e6effbd51-maas-api-openshift-ca" pod "2f29306d-863d-484a-80b7-e05e6effbd51" (UID: "2f29306d-863d-484a-80b7-e05e6effbd51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:57.379615 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:57.379583 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f29306d-863d-484a-80b7-e05e6effbd51-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:57.381566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:57.381539 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f29306d-863d-484a-80b7-e05e6effbd51-kube-api-access-kmbht" pod "2f29306d-863d-484a-80b7-e05e6effbd51" (UID: "2f29306d-863d-484a-80b7-e05e6effbd51"). InnerVolumeSpecName "kube-api-access-kmbht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:57.382327 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:57.382300 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f29306d-863d-484a-80b7-e05e6effbd51-maas-api-xks-ca" pod "2f29306d-863d-484a-80b7-e05e6effbd51" (UID: "2f29306d-863d-484a-80b7-e05e6effbd51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:20:57.481101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:57.481012 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f29306d-863d-484a-80b7-e05e6effbd51-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:57.481101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:57.481041 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmbht\" (UniqueName: \"kubernetes.io/projected/2f29306d-863d-484a-80b7-e05e6effbd51-kube-api-access-kmbht\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:58.175752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:58.175647 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bec3b13-hb84q" Nov 03 20:20:58.175752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:58.175712 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bec3b13-hb84q" event={"ID":"2f29306d-863d-484a-80b7-e05e6effbd51","Type":"ContainerDied","Data":"4b2f753afaa6a23d743356e98c51f43b22c554f99aed223392d8b5ac561b6814"} Nov 03 20:20:58.176752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:58.175770 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b2f753afaa6a23d743356e98c51f43b22c554f99aed223392d8b5ac561b6814" Nov 03 20:20:58.638219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:58.638186 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx" Nov 03 20:20:59.025504 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:59.025414 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8f82f8-sr4nk"] Nov 03 20:20:59.031066 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:59.031032 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8f82f8-sr4nk"] Nov 03 20:20:59.937963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:20:59.937922 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4767ab09-f06f-4090-a767-2032eafa40d2" path="/var/lib/kubelet/pods/4767ab09-f06f-4090-a767-2032eafa40d2/volumes" Nov 03 20:21:03.991631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:03.991591 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5"] Nov 03 20:21:03.992825 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:03.992806 5059 cpu_manager.go:414] "containerMap: removing container" podUID="2f29306d-863d-484a-80b7-e05e6effbd51" containerName="revoke-keys" Nov 03 20:21:03.992883 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:03.992828 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f29306d-863d-484a-80b7-e05e6effbd51" containerName="revoke-keys" Nov 03 20:21:03.993017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:03.993006 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f29306d-863d-484a-80b7-e05e6effbd51" containerName="revoke-keys" Nov 03 20:21:03.998786 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:03.998766 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.014251 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.014226 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5"] Nov 03 20:21:04.029637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.029615 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e7b70d-openshift-default-dockercfg-jjqkc\"" Nov 03 20:21:04.136687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.136412 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.136687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.136537 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d3c590-f523-4b7e-9086-60df5fc585e8-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.136687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.136584 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.136687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.136685 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.137077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.136756 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.137077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.136791 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf55b\" (UniqueName: \"kubernetes.io/projected/f0d3c590-f523-4b7e-9086-60df5fc585e8-kube-api-access-mf55b\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.137077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.136849 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.137077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.136971 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.137077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.137039 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.324388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.314607 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.324388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.314730 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.324388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.314911 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.324388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.315027 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d3c590-f523-4b7e-9086-60df5fc585e8-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.324388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.315081 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.324388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.315173 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.324388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.315248 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.324388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.315290 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf55b\" (UniqueName: \"kubernetes.io/projected/f0d3c590-f523-4b7e-9086-60df5fc585e8-kube-api-access-mf55b\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.324388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.315365 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.324388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.316001 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.381541 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.381503 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.424200 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.382087 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.424429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.383130 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.424534 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.422913 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d3c590-f523-4b7e-9086-60df5fc585e8-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.514109 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.514076 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.546805 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.546759 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.553823 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.553753 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.554708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.554631 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf55b\" (UniqueName: \"kubernetes.io/projected/f0d3c590-f523-4b7e-9086-60df5fc585e8-kube-api-access-mf55b\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:04.618942 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:04.618853 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:05.362006 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:05.361976 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5"] Nov 03 20:21:05.362341 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:21:05.362111 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d3c590_f523_4b7e_9086_60df5fc585e8.slice/crio-af5e6ffa1ae4d694d732a2b4e736590cfab71fec562f7e7542592fa7e65d20bf WatchSource:0}: Error finding container af5e6ffa1ae4d694d732a2b4e736590cfab71fec562f7e7542592fa7e65d20bf: Status 404 returned error can't find the container with id af5e6ffa1ae4d694d732a2b4e736590cfab71fec562f7e7542592fa7e65d20bf Nov 03 20:21:05.365459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:05.365426 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:21:05.366020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:05.365492 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:21:05.366020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:05.365524 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:21:05.668749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:05.668534 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" event={"ID":"f0d3c590-f523-4b7e-9086-60df5fc585e8","Type":"ContainerStarted","Data":"efcfe340defa1e3223df36397fde5c79fbe4b5c7832e90583c68ba8293d3e314"} Nov 03 20:21:05.668749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:05.668600 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" event={"ID":"f0d3c590-f523-4b7e-9086-60df5fc585e8","Type":"ContainerStarted","Data":"af5e6ffa1ae4d694d732a2b4e736590cfab71fec562f7e7542592fa7e65d20bf"} Nov 03 20:21:05.716956 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:05.716887 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" podStartSLOduration=2.716862787 podStartE2EDuration="2.716862787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:05.703963984 +0000 UTC m=+2659.961761245" watchObservedRunningTime="2026-11-03 20:21:05.716862787 +0000 UTC m=+2659.974660047" Nov 03 20:21:06.625720 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:06.623157 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:06.629122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:06.629095 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body= Nov 03 20:21:06.629370 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:06.629341 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" podUID="f0d3c590-f523-4b7e-9086-60df5fc585e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" Nov 03 20:21:07.700689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:07.665053 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:08.057759 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:08.057647 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:08.065913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:08.065889 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:21:08.100685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:08.098707 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8735403-hn7lg"] Nov 03 20:21:08.146348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:08.146276 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8735403-hn7lg"] Nov 03 20:21:09.657885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:09.656820 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:21:09.736641 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:09.736566 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:09.952927 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:09.952821 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:09.981355 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:09.975011 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:10.047953 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:10.047916 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d6c21d5-1599-440b-8342-1664fcc7594c" path="/var/lib/kubelet/pods/6d6c21d5-1599-440b-8342-1664fcc7594c/volumes" Nov 03 20:21:10.802985 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:21:10.802772 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:21:21.062520 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:21.062480 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1830c39-6vrr7"] Nov 03 20:21:21.070671 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:21.070619 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1830c39-6vrr7"] Nov 03 20:21:21.860522 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:21.860486 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da37c287-ec4f-47c5-8319-1891003a6b89" path="/var/lib/kubelet/pods/da37c287-ec4f-47c5-8319-1891003a6b89/volumes" Nov 03 20:21:25.024276 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:25.024235 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3777cca-rq2wv"] Nov 03 20:21:25.030031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:25.030001 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3777cca-rq2wv"] Nov 03 20:21:25.864466 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:25.864426 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aafca12e-5345-4985-b452-0d661e655802" path="/var/lib/kubelet/pods/aafca12e-5345-4985-b452-0d661e655802/volumes" Nov 03 20:21:29.081246 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:29.081204 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2"] Nov 03 20:21:29.149930 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:29.149896 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2"] Nov 03 20:21:29.150187 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:29.150172 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93391ed7-gc8c4"] Nov 03 20:21:29.150301 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:29.150292 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93391ed7-gc8c4"] Nov 03 20:21:29.150398 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:29.150094 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" Nov 03 20:21:29.177685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:29.156022 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea6e69e-3556-4ffe-bb9e-796e8795d747-maas-api-tls\") pod \"maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2\" (UID: \"4ea6e69e-3556-4ffe-bb9e-796e8795d747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" Nov 03 20:21:29.177685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:29.156103 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea6e69e-3556-4ffe-bb9e-796e8795d747-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2\" (UID: \"4ea6e69e-3556-4ffe-bb9e-796e8795d747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" Nov 03 20:21:29.177685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:29.156378 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea6e69e-3556-4ffe-bb9e-796e8795d747-metrics-certs\") pod \"maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2\" (UID: \"4ea6e69e-3556-4ffe-bb9e-796e8795d747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" Nov 03 20:21:29.177685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:29.156408 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf6rz\" (UniqueName: \"kubernetes.io/projected/4ea6e69e-3556-4ffe-bb9e-796e8795d747-kube-api-access-vf6rz\") pod \"maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2\" (UID: \"4ea6e69e-3556-4ffe-bb9e-796e8795d747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" Nov 03 20:21:29.185457 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:29.185423 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e7b70d-serving-cert\"" Nov 03 20:21:29.187024 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:29.186844 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:21:29.187257 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:29.186938 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-942q2\"" Nov 03 20:21:29.277683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:29.277395 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea6e69e-3556-4ffe-bb9e-796e8795d747-metrics-certs\") pod \"maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2\" (UID: \"4ea6e69e-3556-4ffe-bb9e-796e8795d747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" Nov 03 20:21:29.277683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:29.277519 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf6rz\" (UniqueName: \"kubernetes.io/projected/4ea6e69e-3556-4ffe-bb9e-796e8795d747-kube-api-access-vf6rz\") pod \"maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2\" (UID: \"4ea6e69e-3556-4ffe-bb9e-796e8795d747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" Nov 03 20:21:29.294265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:29.294205 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea6e69e-3556-4ffe-bb9e-796e8795d747-maas-api-tls\") pod \"maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2\" (UID: \"4ea6e69e-3556-4ffe-bb9e-796e8795d747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" Nov 03 20:21:29.294701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:29.294649 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea6e69e-3556-4ffe-bb9e-796e8795d747-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2\" (UID: \"4ea6e69e-3556-4ffe-bb9e-796e8795d747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" Nov 03 20:21:29.296754 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:29.296728 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea6e69e-3556-4ffe-bb9e-796e8795d747-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2\" (UID: \"4ea6e69e-3556-4ffe-bb9e-796e8795d747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" Nov 03 20:21:29.322541 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:29.322421 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea6e69e-3556-4ffe-bb9e-796e8795d747-metrics-certs\") pod \"maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2\" (UID: \"4ea6e69e-3556-4ffe-bb9e-796e8795d747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" Nov 03 20:21:29.330024 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:29.329950 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea6e69e-3556-4ffe-bb9e-796e8795d747-maas-api-tls\") pod \"maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2\" (UID: \"4ea6e69e-3556-4ffe-bb9e-796e8795d747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" Nov 03 20:21:29.343468 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:29.343355 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf6rz\" (UniqueName: \"kubernetes.io/projected/4ea6e69e-3556-4ffe-bb9e-796e8795d747-kube-api-access-vf6rz\") pod \"maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2\" (UID: \"4ea6e69e-3556-4ffe-bb9e-796e8795d747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" Nov 03 20:21:29.570098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:29.567561 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" Nov 03 20:21:30.035835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:30.010172 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0323f1ca-e8f8-4015-993d-c2b2229c485b" path="/var/lib/kubelet/pods/0323f1ca-e8f8-4015-993d-c2b2229c485b/volumes" Nov 03 20:21:30.919754 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:30.915876 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:31.341705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:31.341651 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2"] Nov 03 20:21:31.899687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:31.886552 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:31.930686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:31.919943 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 03 20:21:31.930686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:31.920413 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:21:31.948677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:31.948614 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:31.954089 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:31.949118 5059 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:21:31.999715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:31.999657 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:21:32.151849 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:32.151707 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:32.670077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:32.669982 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" event={"ID":"4ea6e69e-3556-4ffe-bb9e-796e8795d747","Type":"ContainerStarted","Data":"ddbc5f90543f820fd5f5ea6b8587849321d7d4c87b3a8a2ea777eb765a071395"} Nov 03 20:21:32.672030 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:32.672004 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:21:33.963485 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:33.963245 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw"] Nov 03 20:21:33.963991 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:33.963834 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" podUID="154d0ad4-2191-4633-af2e-57ca9b458f42" containerName="istio-proxy" containerID="cri-o://049c99efb365c2ff91b8218d5f60551a4bad355fbcc0a7ec9968e46acb798d68" gracePeriod=30 Nov 03 20:21:36.029314 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:36.029275 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94176598-95bzq"] Nov 03 20:21:36.035016 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:36.034983 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94176598-95bzq"] Nov 03 20:21:37.715226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:37.715172 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" event={"ID":"4ea6e69e-3556-4ffe-bb9e-796e8795d747","Type":"ContainerStarted","Data":"c21a21bf6b505e70d50fabd813f6c6546130257160456e17e096e7eab6ae4f7d"} Nov 03 20:21:37.716631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:37.716605 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" Nov 03 20:21:37.752681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:37.749424 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" podStartSLOduration=5.210773244 podStartE2EDuration="9.749404195s" totalImagesPullingTime="4.538630951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:37.748074376 +0000 UTC m=+2692.005871635" watchObservedRunningTime="2026-11-03 20:21:37.749404195 +0000 UTC m=+2692.007201452" Nov 03 20:21:37.915402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:37.915252 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530" path="/var/lib/kubelet/pods/6051fbf1-1d0b-4e44-af7f-ecf6b9dd3530/volumes" Nov 03 20:21:39.374595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.374563 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:21:39.414224 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.414191 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l"] Nov 03 20:21:39.415005 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.414985 5059 cpu_manager.go:414] "containerMap: removing container" podUID="154d0ad4-2191-4633-af2e-57ca9b458f42" containerName="istio-proxy" Nov 03 20:21:39.415005 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.415004 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154d0ad4-2191-4633-af2e-57ca9b458f42" containerName="istio-proxy" Nov 03 20:21:39.415207 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.415193 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="154d0ad4-2191-4633-af2e-57ca9b458f42" containerName="istio-proxy" Nov 03 20:21:39.418991 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.418967 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.432845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.432815 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l"] Nov 03 20:21:39.493377 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.493340 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-data\") pod \"154d0ad4-2191-4633-af2e-57ca9b458f42\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " Nov 03 20:21:39.494265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.493464 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-envoy\") pod \"154d0ad4-2191-4633-af2e-57ca9b458f42\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " Nov 03 20:21:39.494265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.493534 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-credential-socket\") pod \"154d0ad4-2191-4633-af2e-57ca9b458f42\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " Nov 03 20:21:39.494265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.493736 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-token\" (UniqueName: \"kubernetes.io/projected/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-token\") pod \"154d0ad4-2191-4633-af2e-57ca9b458f42\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " Nov 03 20:21:39.494265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.493804 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-podinfo\") pod \"154d0ad4-2191-4633-af2e-57ca9b458f42\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " Nov 03 20:21:39.494265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.493731 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-data" pod "154d0ad4-2191-4633-af2e-57ca9b458f42" (UID: "154d0ad4-2191-4633-af2e-57ca9b458f42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:39.494265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.493869 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/154d0ad4-2191-4633-af2e-57ca9b458f42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154d0ad4-2191-4633-af2e-57ca9b458f42-istiod-ca-cert\") pod \"154d0ad4-2191-4633-af2e-57ca9b458f42\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " Nov 03 20:21:39.494265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.493883 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-credential-socket" pod "154d0ad4-2191-4633-af2e-57ca9b458f42" (UID: "154d0ad4-2191-4633-af2e-57ca9b458f42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:39.494265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.493927 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154d0ad4-2191-4633-af2e-57ca9b458f42-kube-api-access-g6hvf\" (UniqueName: \"kubernetes.io/projected/154d0ad4-2191-4633-af2e-57ca9b458f42-kube-api-access-g6hvf\") pod \"154d0ad4-2191-4633-af2e-57ca9b458f42\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " Nov 03 20:21:39.494265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.493961 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-workload-certs\") pod \"154d0ad4-2191-4633-af2e-57ca9b458f42\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " Nov 03 20:21:39.494265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.494039 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-workload-socket\") pod \"154d0ad4-2191-4633-af2e-57ca9b458f42\" (UID: \"154d0ad4-2191-4633-af2e-57ca9b458f42\") " Nov 03 20:21:39.497464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.494301 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-workload-certs" pod "154d0ad4-2191-4633-af2e-57ca9b458f42" (UID: "154d0ad4-2191-4633-af2e-57ca9b458f42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:39.497464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.494327 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/154d0ad4-2191-4633-af2e-57ca9b458f42-istiod-ca-cert" pod "154d0ad4-2191-4633-af2e-57ca9b458f42" (UID: "154d0ad4-2191-4633-af2e-57ca9b458f42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:39.497464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.494446 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-workload-socket\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.497464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.494531 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-workload-socket" pod "154d0ad4-2191-4633-af2e-57ca9b458f42" (UID: "154d0ad4-2191-4633-af2e-57ca9b458f42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:39.497464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.494653 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05fceb9c-4a04-470f-a94a-055121319f82-istio-token\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.497464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.494813 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-credential-socket\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.497464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.494927 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-istio-envoy\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.497464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.494991 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05fceb9c-4a04-470f-a94a-055121319f82-istio-podinfo\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.497464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.495083 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld86p\" (UniqueName: \"kubernetes.io/projected/05fceb9c-4a04-470f-a94a-055121319f82-kube-api-access-ld86p\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.497464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.495132 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05fceb9c-4a04-470f-a94a-055121319f82-istiod-ca-cert\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.497464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.495208 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-workload-certs\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.497464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.495256 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-istio-data\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.497464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.495390 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154d0ad4-2191-4633-af2e-57ca9b458f42-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:39.497464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.495408 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:39.497464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.495423 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:39.497464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.495456 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:39.497464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.495482 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:39.501101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.499789 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-podinfo" pod "154d0ad4-2191-4633-af2e-57ca9b458f42" (UID: "154d0ad4-2191-4633-af2e-57ca9b458f42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:21:39.501101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.499826 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154d0ad4-2191-4633-af2e-57ca9b458f42-kube-api-access-g6hvf" pod "154d0ad4-2191-4633-af2e-57ca9b458f42" (UID: "154d0ad4-2191-4633-af2e-57ca9b458f42"). InnerVolumeSpecName "kube-api-access-g6hvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:39.501101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.499964 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-envoy" pod "154d0ad4-2191-4633-af2e-57ca9b458f42" (UID: "154d0ad4-2191-4633-af2e-57ca9b458f42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:39.501101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.499984 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-token" pod "154d0ad4-2191-4633-af2e-57ca9b458f42" (UID: "154d0ad4-2191-4633-af2e-57ca9b458f42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:39.597527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.597439 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-workload-socket\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.597758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.597535 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05fceb9c-4a04-470f-a94a-055121319f82-istio-token\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.597758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.597587 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-credential-socket\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.597758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.597638 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-istio-envoy\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.597758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.597687 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05fceb9c-4a04-470f-a94a-055121319f82-istio-podinfo\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.598788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.597778 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld86p\" (UniqueName: \"kubernetes.io/projected/05fceb9c-4a04-470f-a94a-055121319f82-kube-api-access-ld86p\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.598788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.597845 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05fceb9c-4a04-470f-a94a-055121319f82-istiod-ca-cert\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.598788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.597899 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-workload-certs\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.598788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.597944 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-istio-data\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.598788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.597958 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-workload-socket\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.598788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.597995 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-credential-socket\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.598788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.598150 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:39.598788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.598171 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:39.598788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.598183 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154d0ad4-2191-4633-af2e-57ca9b458f42-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:39.598788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.598192 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6hvf\" (UniqueName: \"kubernetes.io/projected/154d0ad4-2191-4633-af2e-57ca9b458f42-kube-api-access-g6hvf\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:39.598788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.598640 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-workload-certs\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.598788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.598744 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-istio-data\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.599596 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.599572 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05fceb9c-4a04-470f-a94a-055121319f82-istiod-ca-cert\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.601148 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.601130 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-istio-envoy\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.601314 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.601295 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05fceb9c-4a04-470f-a94a-055121319f82-istio-podinfo\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.610303 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.610277 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05fceb9c-4a04-470f-a94a-055121319f82-istio-token\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.610494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.610475 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld86p\" (UniqueName: \"kubernetes.io/projected/05fceb9c-4a04-470f-a94a-055121319f82-kube-api-access-ld86p\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.734296 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.734261 5059 generic.go:356] "Generic (PLEG): container finished" podID="154d0ad4-2191-4633-af2e-57ca9b458f42" containerID="049c99efb365c2ff91b8218d5f60551a4bad355fbcc0a7ec9968e46acb798d68" exitCode=0 Nov 03 20:21:39.734509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.734323 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" Nov 03 20:21:39.734509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.734339 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" event={"ID":"154d0ad4-2191-4633-af2e-57ca9b458f42","Type":"ContainerDied","Data":"049c99efb365c2ff91b8218d5f60551a4bad355fbcc0a7ec9968e46acb798d68"} Nov 03 20:21:39.734509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.734393 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw" event={"ID":"154d0ad4-2191-4633-af2e-57ca9b458f42","Type":"ContainerDied","Data":"93f67f4ca8fa5d2065339ba5b1f30d76dd256df96f5bc0b9b581163d2d0afdf0"} Nov 03 20:21:39.734509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.734418 5059 scope.go:122] "RemoveContainer" containerID="049c99efb365c2ff91b8218d5f60551a4bad355fbcc0a7ec9968e46acb798d68" Nov 03 20:21:39.740004 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.739781 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:39.855735 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.851145 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw"] Nov 03 20:21:39.865887 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.863205 5059 scope.go:122] "RemoveContainer" containerID="efd770fcbd173d9ab92d2bc525a9123ed2006ef5998ec9ce1906a3109519a3de" Nov 03 20:21:39.869017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.868993 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-b24gw"] Nov 03 20:21:39.884288 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.884165 5059 scope.go:122] "RemoveContainer" containerID="049c99efb365c2ff91b8218d5f60551a4bad355fbcc0a7ec9968e46acb798d68" Nov 03 20:21:39.884433 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:21:39.884410 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"049c99efb365c2ff91b8218d5f60551a4bad355fbcc0a7ec9968e46acb798d68\": container with ID starting with 049c99efb365c2ff91b8218d5f60551a4bad355fbcc0a7ec9968e46acb798d68 not found: ID does not exist" containerID="049c99efb365c2ff91b8218d5f60551a4bad355fbcc0a7ec9968e46acb798d68" Nov 03 20:21:39.884510 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.884444 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"049c99efb365c2ff91b8218d5f60551a4bad355fbcc0a7ec9968e46acb798d68"} err="failed to get container status \"049c99efb365c2ff91b8218d5f60551a4bad355fbcc0a7ec9968e46acb798d68\": rpc error: code = NotFound desc = could not find container \"049c99efb365c2ff91b8218d5f60551a4bad355fbcc0a7ec9968e46acb798d68\": container with ID starting with 049c99efb365c2ff91b8218d5f60551a4bad355fbcc0a7ec9968e46acb798d68 not found: ID does not exist" Nov 03 20:21:39.884510 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.884470 5059 scope.go:122] "RemoveContainer" containerID="efd770fcbd173d9ab92d2bc525a9123ed2006ef5998ec9ce1906a3109519a3de" Nov 03 20:21:39.884841 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:21:39.884821 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efd770fcbd173d9ab92d2bc525a9123ed2006ef5998ec9ce1906a3109519a3de\": container with ID starting with efd770fcbd173d9ab92d2bc525a9123ed2006ef5998ec9ce1906a3109519a3de not found: ID does not exist" containerID="efd770fcbd173d9ab92d2bc525a9123ed2006ef5998ec9ce1906a3109519a3de" Nov 03 20:21:39.885628 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:39.884848 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efd770fcbd173d9ab92d2bc525a9123ed2006ef5998ec9ce1906a3109519a3de"} err="failed to get container status \"efd770fcbd173d9ab92d2bc525a9123ed2006ef5998ec9ce1906a3109519a3de\": rpc error: code = NotFound desc = could not find container \"efd770fcbd173d9ab92d2bc525a9123ed2006ef5998ec9ce1906a3109519a3de\": container with ID starting with efd770fcbd173d9ab92d2bc525a9123ed2006ef5998ec9ce1906a3109519a3de not found: ID does not exist" Nov 03 20:21:40.171383 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:40.171354 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l"] Nov 03 20:21:40.173840 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:21:40.173813 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05fceb9c_4a04_470f_a94a_055121319f82.slice/crio-9f155b483f4b2676e0c5a139fb683e66a580a3b42972f576a5c67b9f37065425 WatchSource:0}: Error finding container 9f155b483f4b2676e0c5a139fb683e66a580a3b42972f576a5c67b9f37065425: Status 404 returned error can't find the container with id 9f155b483f4b2676e0c5a139fb683e66a580a3b42972f576a5c67b9f37065425 Nov 03 20:21:40.177296 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:40.177259 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:21:40.177418 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:40.177356 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:21:40.177418 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:40.177390 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:21:40.698049 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:21:40.698011 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:21:40.748306 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:40.748270 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" event={"ID":"05fceb9c-4a04-470f-a94a-055121319f82","Type":"ContainerStarted","Data":"8eba0c0d93805e32f585be9b5dffe0df1cf0f78d3928ac2e35a5a76d60ddd0e3"} Nov 03 20:21:40.748512 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:40.748330 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" event={"ID":"05fceb9c-4a04-470f-a94a-055121319f82","Type":"ContainerStarted","Data":"9f155b483f4b2676e0c5a139fb683e66a580a3b42972f576a5c67b9f37065425"} Nov 03 20:21:40.783733 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:40.783680 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" podStartSLOduration=7.783651457 podStartE2EDuration="7.783651457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:40.775837941 +0000 UTC m=+2695.033635206" watchObservedRunningTime="2026-11-03 20:21:40.783651457 +0000 UTC m=+2695.041448714" Nov 03 20:21:41.740416 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:41.740379 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:41.742422 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:41.742393 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" start-of-body= Nov 03 20:21:41.742559 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:41.742464 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" podUID="05fceb9c-4a04-470f-a94a-055121319f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" Nov 03 20:21:41.865688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:41.862638 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="154d0ad4-2191-4633-af2e-57ca9b458f42" path="/var/lib/kubelet/pods/154d0ad4-2191-4633-af2e-57ca9b458f42/volumes" Nov 03 20:21:42.740156 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:42.740114 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" start-of-body= Nov 03 20:21:42.740339 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:42.740195 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" podUID="05fceb9c-4a04-470f-a94a-055121319f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" Nov 03 20:21:43.746763 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:43.745124 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" start-of-body= Nov 03 20:21:43.746763 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:43.745259 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" podUID="05fceb9c-4a04-470f-a94a-055121319f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" Nov 03 20:21:43.753386 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:43.753359 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" Nov 03 20:21:44.740820 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:44.740785 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" start-of-body= Nov 03 20:21:44.741031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:44.740840 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" podUID="05fceb9c-4a04-470f-a94a-055121319f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" Nov 03 20:21:45.740373 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:45.740328 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" start-of-body= Nov 03 20:21:45.740814 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:45.740408 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" podUID="05fceb9c-4a04-470f-a94a-055121319f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" Nov 03 20:21:46.740599 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:46.740565 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" start-of-body= Nov 03 20:21:46.741049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:46.740623 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" podUID="05fceb9c-4a04-470f-a94a-055121319f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" Nov 03 20:21:46.965449 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:46.965418 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 20:21:46.967675 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:46.967637 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 20:21:47.745306 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:47.745269 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:47.745808 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:47.745787 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:47.746694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:47.746642 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:21:48.032475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:48.032382 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2bb2534-tf72r"] Nov 03 20:21:48.037569 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:48.037538 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2bb2534-tf72r"] Nov 03 20:21:49.863898 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:49.863858 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35fc7c23-900c-41ca-ae62-90df054d185c" path="/var/lib/kubelet/pods/35fc7c23-900c-41ca-ae62-90df054d185c/volumes" Nov 03 20:21:56.711686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:56.705595 5059 scope.go:122] "RemoveContainer" containerID="d137e44245117ded1f273201b8436175e3e6bbccc5434833ab4b39c6cbafe2da" Nov 03 20:21:56.780815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:56.780770 5059 scope.go:122] "RemoveContainer" containerID="038675880d305af3567685c554e2f182591d8a167a180e771df679b16d57b8d2" Nov 03 20:21:56.796729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:56.796706 5059 scope.go:122] "RemoveContainer" containerID="bca04c565d1eb984fc3c702b2e4a86c4310b2d892273c68fbd595faf6a4a41b5" Nov 03 20:21:56.811223 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:21:56.810723 5059 scope.go:122] "RemoveContainer" containerID="c8729ee43ddc112c927ac777a234b409d8fc809ee5610afce202321b3de775ce" Nov 03 20:22:10.633212 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:22:10.633163 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:22:32.551339 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:32.551284 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:22:32.655959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:32.654762 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:22:35.518713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.516700 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk"] Nov 03 20:22:35.518713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.518011 5059 cpu_manager.go:414] "containerMap: removing container" podUID="154d0ad4-2191-4633-af2e-57ca9b458f42" containerName="istio-proxy" Nov 03 20:22:35.518713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.518036 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154d0ad4-2191-4633-af2e-57ca9b458f42" containerName="istio-proxy" Nov 03 20:22:35.518713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.518314 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="154d0ad4-2191-4633-af2e-57ca9b458f42" containerName="istio-proxy" Nov 03 20:22:35.565769 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.534792 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" Nov 03 20:22:35.567365 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.566912 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"agc-cred-70adf96ae88c\"" Nov 03 20:22:35.568528 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.568084 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"kube-root-ca.crt\"" Nov 03 20:22:35.568528 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.568167 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"routing-overlay\"" Nov 03 20:22:35.568528 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.568252 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"payload-processing-external-model-plugins-e2e-extmod-a-98529d\"" Nov 03 20:22:35.568528 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.568312 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"openshift-service-ca.crt\"" Nov 03 20:22:35.568528 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.568371 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"agc-cred-f535493e5746\"" Nov 03 20:22:35.568528 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.568483 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"payload-processing-external-model-e2e--c4b4e61e-dockercfg-p7bnm\"" Nov 03 20:22:35.573049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.573023 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk"] Nov 03 20:22:35.658336 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.658304 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd65f7b-2f44-4b57-9675-358d64774fbc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk\" (UID: \"6fd65f7b-2f44-4b57-9675-358d64774fbc\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" Nov 03 20:22:35.658598 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.658571 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmk59\" (UniqueName: \"kubernetes.io/projected/6fd65f7b-2f44-4b57-9675-358d64774fbc-kube-api-access-gmk59\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk\" (UID: \"6fd65f7b-2f44-4b57-9675-358d64774fbc\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" Nov 03 20:22:35.659618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.658606 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fd65f7b-2f44-4b57-9675-358d64774fbc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk\" (UID: \"6fd65f7b-2f44-4b57-9675-358d64774fbc\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" Nov 03 20:22:35.659618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.658892 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fd65f7b-2f44-4b57-9675-358d64774fbc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk\" (UID: \"6fd65f7b-2f44-4b57-9675-358d64774fbc\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" Nov 03 20:22:35.760215 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.760177 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fd65f7b-2f44-4b57-9675-358d64774fbc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk\" (UID: \"6fd65f7b-2f44-4b57-9675-358d64774fbc\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" Nov 03 20:22:35.760413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.760292 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd65f7b-2f44-4b57-9675-358d64774fbc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk\" (UID: \"6fd65f7b-2f44-4b57-9675-358d64774fbc\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" Nov 03 20:22:35.760413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.760336 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmk59\" (UniqueName: \"kubernetes.io/projected/6fd65f7b-2f44-4b57-9675-358d64774fbc-kube-api-access-gmk59\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk\" (UID: \"6fd65f7b-2f44-4b57-9675-358d64774fbc\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" Nov 03 20:22:35.760413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.760360 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fd65f7b-2f44-4b57-9675-358d64774fbc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk\" (UID: \"6fd65f7b-2f44-4b57-9675-358d64774fbc\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" Nov 03 20:22:35.824125 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.824092 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fd65f7b-2f44-4b57-9675-358d64774fbc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk\" (UID: \"6fd65f7b-2f44-4b57-9675-358d64774fbc\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" Nov 03 20:22:35.825268 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.824260 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmk59\" (UniqueName: \"kubernetes.io/projected/6fd65f7b-2f44-4b57-9675-358d64774fbc-kube-api-access-gmk59\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk\" (UID: \"6fd65f7b-2f44-4b57-9675-358d64774fbc\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" Nov 03 20:22:35.825268 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.824476 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fd65f7b-2f44-4b57-9675-358d64774fbc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk\" (UID: \"6fd65f7b-2f44-4b57-9675-358d64774fbc\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" Nov 03 20:22:35.825268 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.825202 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd65f7b-2f44-4b57-9675-358d64774fbc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk\" (UID: \"6fd65f7b-2f44-4b57-9675-358d64774fbc\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" Nov 03 20:22:35.935704 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:35.935633 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" Nov 03 20:22:36.319523 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:36.319485 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk"] Nov 03 20:22:37.095158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:37.095119 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk"] Nov 03 20:22:37.390196 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:37.390080 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" event={"ID":"6fd65f7b-2f44-4b57-9675-358d64774fbc","Type":"ContainerStarted","Data":"c96b579c764e3e8e8264ccd6fe73c84f7c5c4202b5ee9b4842fe8fa97d0bbd0c"} Nov 03 20:22:37.390518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:37.390469 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" event={"ID":"6fd65f7b-2f44-4b57-9675-358d64774fbc","Type":"ContainerStarted","Data":"b1fca81585b385fdb5d19839bea98701ede413e75aeff03de9153fb56675aff5"} Nov 03 20:22:37.392332 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:37.392309 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" Nov 03 20:22:37.465436 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:22:37.465400 5059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-98529d/agc-cred-f535493e5746: secret "agc-cred-f535493e5746" not found Nov 03 20:22:37.465726 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:22:37.465652 5059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk: secret "agc-cred-f535493e5746" not found Nov 03 20:22:37.465925 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:22:37.465912 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fd65f7b-2f44-4b57-9675-358d64774fbc-provider-credentials podName:6fd65f7b-2f44-4b57-9675-358d64774fbc nodeName:}" failed. No retries permitted until 2026-11-03 20:22:37.965879431 +0000 UTC m=+2752.223676683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6fd65f7b-2f44-4b57-9675-358d64774fbc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" (UID: "6fd65f7b-2f44-4b57-9675-358d64774fbc") : secret "agc-cred-f535493e5746" not found Nov 03 20:22:37.753738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:37.752300 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" podStartSLOduration=2.7522787539999998 podStartE2EDuration="2.752278754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:37.426086699 +0000 UTC m=+2751.683883982" watchObservedRunningTime="2026-11-03 20:22:37.752278754 +0000 UTC m=+2752.010076012" Nov 03 20:22:37.772686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:37.764644 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38cf8370-729gp"] Nov 03 20:22:37.842516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:37.842476 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38cf8370-729gp"] Nov 03 20:22:37.854965 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:37.842825 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cf8370-729gp" Nov 03 20:22:37.854965 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:37.848173 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:22:37.854965 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:37.848232 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:22:38.004155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:37.984853 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjhzz\" (UniqueName: \"kubernetes.io/projected/9064e69a-d47d-4bad-83b5-ad17fd2dd0c3-kube-api-access-pjhzz\") pod \"maas-api-revoke-sub-38cf8370-729gp\" (UID: \"9064e69a-d47d-4bad-83b5-ad17fd2dd0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cf8370-729gp" Nov 03 20:22:38.004155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:37.985008 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9064e69a-d47d-4bad-83b5-ad17fd2dd0c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38cf8370-729gp\" (UID: \"9064e69a-d47d-4bad-83b5-ad17fd2dd0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cf8370-729gp" Nov 03 20:22:38.004155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:37.985089 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9064e69a-d47d-4bad-83b5-ad17fd2dd0c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38cf8370-729gp\" (UID: \"9064e69a-d47d-4bad-83b5-ad17fd2dd0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cf8370-729gp" Nov 03 20:22:38.004155 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:22:37.992001 5059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-98529d/agc-cred-f535493e5746: secret "agc-cred-f535493e5746" not found Nov 03 20:22:38.004155 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:22:37.992030 5059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk: secret "agc-cred-f535493e5746" not found Nov 03 20:22:38.004155 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:22:37.992092 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fd65f7b-2f44-4b57-9675-358d64774fbc-provider-credentials podName:6fd65f7b-2f44-4b57-9675-358d64774fbc nodeName:}" failed. No retries permitted until 2026-11-03 20:22:38.992072184 +0000 UTC m=+2753.249869438 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6fd65f7b-2f44-4b57-9675-358d64774fbc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" (UID: "6fd65f7b-2f44-4b57-9675-358d64774fbc") : secret "agc-cred-f535493e5746" not found Nov 03 20:22:38.273673 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:38.266745 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjhzz\" (UniqueName: \"kubernetes.io/projected/9064e69a-d47d-4bad-83b5-ad17fd2dd0c3-kube-api-access-pjhzz\") pod \"maas-api-revoke-sub-38cf8370-729gp\" (UID: \"9064e69a-d47d-4bad-83b5-ad17fd2dd0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cf8370-729gp" Nov 03 20:22:38.273673 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:38.266878 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9064e69a-d47d-4bad-83b5-ad17fd2dd0c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38cf8370-729gp\" (UID: \"9064e69a-d47d-4bad-83b5-ad17fd2dd0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cf8370-729gp" Nov 03 20:22:38.273673 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:38.266942 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9064e69a-d47d-4bad-83b5-ad17fd2dd0c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38cf8370-729gp\" (UID: \"9064e69a-d47d-4bad-83b5-ad17fd2dd0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cf8370-729gp" Nov 03 20:22:38.274475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:38.274132 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9064e69a-d47d-4bad-83b5-ad17fd2dd0c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38cf8370-729gp\" (UID: \"9064e69a-d47d-4bad-83b5-ad17fd2dd0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cf8370-729gp" Nov 03 20:22:38.295540 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:38.295501 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9064e69a-d47d-4bad-83b5-ad17fd2dd0c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38cf8370-729gp\" (UID: \"9064e69a-d47d-4bad-83b5-ad17fd2dd0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cf8370-729gp" Nov 03 20:22:38.300642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:38.300621 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjhzz\" (UniqueName: \"kubernetes.io/projected/9064e69a-d47d-4bad-83b5-ad17fd2dd0c3-kube-api-access-pjhzz\") pod \"maas-api-revoke-sub-38cf8370-729gp\" (UID: \"9064e69a-d47d-4bad-83b5-ad17fd2dd0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cf8370-729gp" Nov 03 20:22:38.598466 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:38.598430 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cf8370-729gp" Nov 03 20:22:38.907696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:38.907657 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38cf8370-729gp"] Nov 03 20:22:38.910073 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:22:38.910047 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9064e69a_d47d_4bad_83b5_ad17fd2dd0c3.slice/crio-61ad90cec476e83ddf29cc413ea87bafda6b33c83f3e5feeaee1043036e5a343 WatchSource:0}: Error finding container 61ad90cec476e83ddf29cc413ea87bafda6b33c83f3e5feeaee1043036e5a343: Status 404 returned error can't find the container with id 61ad90cec476e83ddf29cc413ea87bafda6b33c83f3e5feeaee1043036e5a343 Nov 03 20:22:39.051939 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:22:39.051905 5059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-98529d/agc-cred-f535493e5746: secret "agc-cred-f535493e5746" not found Nov 03 20:22:39.051939 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:22:39.051935 5059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk: secret "agc-cred-f535493e5746" not found Nov 03 20:22:39.052152 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:22:39.052004 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fd65f7b-2f44-4b57-9675-358d64774fbc-provider-credentials podName:6fd65f7b-2f44-4b57-9675-358d64774fbc nodeName:}" failed. No retries permitted until 2026-11-03 20:22:41.051989 +0000 UTC m=+2755.309786236 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6fd65f7b-2f44-4b57-9675-358d64774fbc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" (UID: "6fd65f7b-2f44-4b57-9675-358d64774fbc") : secret "agc-cred-f535493e5746" not found Nov 03 20:22:39.424361 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:39.424281 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cf8370-729gp" event={"ID":"9064e69a-d47d-4bad-83b5-ad17fd2dd0c3","Type":"ContainerStarted","Data":"61ad90cec476e83ddf29cc413ea87bafda6b33c83f3e5feeaee1043036e5a343"} Nov 03 20:22:39.766991 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:39.766955 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce41531-n4559"] Nov 03 20:22:39.771887 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:39.771864 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce41531-n4559" Nov 03 20:22:39.781434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:39.781376 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce41531-n4559"] Nov 03 20:22:39.867342 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:39.867310 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5jq4\" (UniqueName: \"kubernetes.io/projected/11283cc0-04dc-42b5-8821-9540534b6781-kube-api-access-k5jq4\") pod \"maas-api-revoke-sub-cce41531-n4559\" (UID: \"11283cc0-04dc-42b5-8821-9540534b6781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce41531-n4559" Nov 03 20:22:39.867556 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:39.867350 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11283cc0-04dc-42b5-8821-9540534b6781-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cce41531-n4559\" (UID: \"11283cc0-04dc-42b5-8821-9540534b6781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce41531-n4559" Nov 03 20:22:39.867556 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:39.867455 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11283cc0-04dc-42b5-8821-9540534b6781-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cce41531-n4559\" (UID: \"11283cc0-04dc-42b5-8821-9540534b6781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce41531-n4559" Nov 03 20:22:39.968991 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:39.968961 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11283cc0-04dc-42b5-8821-9540534b6781-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cce41531-n4559\" (UID: \"11283cc0-04dc-42b5-8821-9540534b6781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce41531-n4559" Nov 03 20:22:39.969312 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:39.969295 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5jq4\" (UniqueName: \"kubernetes.io/projected/11283cc0-04dc-42b5-8821-9540534b6781-kube-api-access-k5jq4\") pod \"maas-api-revoke-sub-cce41531-n4559\" (UID: \"11283cc0-04dc-42b5-8821-9540534b6781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce41531-n4559" Nov 03 20:22:39.970027 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:39.969324 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11283cc0-04dc-42b5-8821-9540534b6781-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cce41531-n4559\" (UID: \"11283cc0-04dc-42b5-8821-9540534b6781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce41531-n4559" Nov 03 20:22:39.971075 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:39.971054 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11283cc0-04dc-42b5-8821-9540534b6781-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cce41531-n4559\" (UID: \"11283cc0-04dc-42b5-8821-9540534b6781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce41531-n4559" Nov 03 20:22:39.973704 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:39.973681 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11283cc0-04dc-42b5-8821-9540534b6781-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cce41531-n4559\" (UID: \"11283cc0-04dc-42b5-8821-9540534b6781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce41531-n4559" Nov 03 20:22:39.982047 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:39.982029 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5jq4\" (UniqueName: \"kubernetes.io/projected/11283cc0-04dc-42b5-8821-9540534b6781-kube-api-access-k5jq4\") pod \"maas-api-revoke-sub-cce41531-n4559\" (UID: \"11283cc0-04dc-42b5-8821-9540534b6781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce41531-n4559" Nov 03 20:22:40.090309 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:40.090282 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce41531-n4559" Nov 03 20:22:40.386390 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:40.386359 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce41531-n4559"] Nov 03 20:22:40.388622 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:22:40.388589 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11283cc0_04dc_42b5_8821_9540534b6781.slice/crio-f35da7b9890501260e0ec994f85b29d04ed1a6fc07aff90eefda74a74c8c59af WatchSource:0}: Error finding container f35da7b9890501260e0ec994f85b29d04ed1a6fc07aff90eefda74a74c8c59af: Status 404 returned error can't find the container with id f35da7b9890501260e0ec994f85b29d04ed1a6fc07aff90eefda74a74c8c59af Nov 03 20:22:40.433844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:40.433815 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce41531-n4559" event={"ID":"11283cc0-04dc-42b5-8821-9540534b6781","Type":"ContainerStarted","Data":"f35da7b9890501260e0ec994f85b29d04ed1a6fc07aff90eefda74a74c8c59af"} Nov 03 20:22:40.435698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:40.435673 5059 generic.go:356] "Generic (PLEG): container finished" podID="9064e69a-d47d-4bad-83b5-ad17fd2dd0c3" containerID="bc571c0f9d14acdd1fada8fd767a2f23f5170cfca0dc658da6d8f8ab02daf45d" exitCode=0 Nov 03 20:22:40.435844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:40.435717 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cf8370-729gp" event={"ID":"9064e69a-d47d-4bad-83b5-ad17fd2dd0c3","Type":"ContainerDied","Data":"bc571c0f9d14acdd1fada8fd767a2f23f5170cfca0dc658da6d8f8ab02daf45d"} Nov 03 20:22:40.623729 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:22:40.622501 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:22:41.123191 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:22:41.123158 5059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-98529d/agc-cred-f535493e5746: secret "agc-cred-f535493e5746" not found Nov 03 20:22:41.123191 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:22:41.123185 5059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk: secret "agc-cred-f535493e5746" not found Nov 03 20:22:41.123401 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:22:41.123242 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fd65f7b-2f44-4b57-9675-358d64774fbc-provider-credentials podName:6fd65f7b-2f44-4b57-9675-358d64774fbc nodeName:}" failed. No retries permitted until 2026-11-03 20:22:45.123228944 +0000 UTC m=+2759.381026178 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6fd65f7b-2f44-4b57-9675-358d64774fbc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" (UID: "6fd65f7b-2f44-4b57-9675-358d64774fbc") : secret "agc-cred-f535493e5746" not found Nov 03 20:22:41.446497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:41.446467 5059 generic.go:356] "Generic (PLEG): container finished" podID="11283cc0-04dc-42b5-8821-9540534b6781" containerID="fa697f01be324e223c752a12316d9a9c14ae9cae343d4fc9f472c399735cd9ad" exitCode=0 Nov 03 20:22:41.446953 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:41.446606 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce41531-n4559" event={"ID":"11283cc0-04dc-42b5-8821-9540534b6781","Type":"ContainerDied","Data":"fa697f01be324e223c752a12316d9a9c14ae9cae343d4fc9f472c399735cd9ad"} Nov 03 20:22:41.654033 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:41.653967 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cf8370-729gp" Nov 03 20:22:41.941299 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:41.941224 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9064e69a-d47d-4bad-83b5-ad17fd2dd0c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9064e69a-d47d-4bad-83b5-ad17fd2dd0c3-maas-api-openshift-ca\") pod \"9064e69a-d47d-4bad-83b5-ad17fd2dd0c3\" (UID: \"9064e69a-d47d-4bad-83b5-ad17fd2dd0c3\") " Nov 03 20:22:41.941472 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:41.941361 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9064e69a-d47d-4bad-83b5-ad17fd2dd0c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9064e69a-d47d-4bad-83b5-ad17fd2dd0c3-maas-api-xks-ca\") pod \"9064e69a-d47d-4bad-83b5-ad17fd2dd0c3\" (UID: \"9064e69a-d47d-4bad-83b5-ad17fd2dd0c3\") " Nov 03 20:22:41.941554 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:41.941481 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9064e69a-d47d-4bad-83b5-ad17fd2dd0c3-kube-api-access-pjhzz\" (UniqueName: \"kubernetes.io/projected/9064e69a-d47d-4bad-83b5-ad17fd2dd0c3-kube-api-access-pjhzz\") pod \"9064e69a-d47d-4bad-83b5-ad17fd2dd0c3\" (UID: \"9064e69a-d47d-4bad-83b5-ad17fd2dd0c3\") " Nov 03 20:22:41.941640 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:41.941612 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9064e69a-d47d-4bad-83b5-ad17fd2dd0c3-maas-api-openshift-ca" pod "9064e69a-d47d-4bad-83b5-ad17fd2dd0c3" (UID: "9064e69a-d47d-4bad-83b5-ad17fd2dd0c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:41.942387 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:41.942184 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9064e69a-d47d-4bad-83b5-ad17fd2dd0c3-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:41.946585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:41.946562 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9064e69a-d47d-4bad-83b5-ad17fd2dd0c3-maas-api-xks-ca" pod "9064e69a-d47d-4bad-83b5-ad17fd2dd0c3" (UID: "9064e69a-d47d-4bad-83b5-ad17fd2dd0c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:22:41.946761 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:41.946738 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9064e69a-d47d-4bad-83b5-ad17fd2dd0c3-kube-api-access-pjhzz" pod "9064e69a-d47d-4bad-83b5-ad17fd2dd0c3" (UID: "9064e69a-d47d-4bad-83b5-ad17fd2dd0c3"). InnerVolumeSpecName "kube-api-access-pjhzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:42.043481 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:42.043447 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9064e69a-d47d-4bad-83b5-ad17fd2dd0c3-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:42.043481 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:42.043480 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjhzz\" (UniqueName: \"kubernetes.io/projected/9064e69a-d47d-4bad-83b5-ad17fd2dd0c3-kube-api-access-pjhzz\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:42.458382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:42.458352 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cf8370-729gp" Nov 03 20:22:42.458875 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:42.458389 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cf8370-729gp" event={"ID":"9064e69a-d47d-4bad-83b5-ad17fd2dd0c3","Type":"ContainerDied","Data":"61ad90cec476e83ddf29cc413ea87bafda6b33c83f3e5feeaee1043036e5a343"} Nov 03 20:22:42.458875 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:42.458480 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61ad90cec476e83ddf29cc413ea87bafda6b33c83f3e5feeaee1043036e5a343" Nov 03 20:22:42.785325 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:42.785299 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce41531-n4559" Nov 03 20:22:42.858586 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:42.858554 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11283cc0-04dc-42b5-8821-9540534b6781-kube-api-access-k5jq4\" (UniqueName: \"kubernetes.io/projected/11283cc0-04dc-42b5-8821-9540534b6781-kube-api-access-k5jq4\") pod \"11283cc0-04dc-42b5-8821-9540534b6781\" (UID: \"11283cc0-04dc-42b5-8821-9540534b6781\") " Nov 03 20:22:42.859862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:42.858640 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11283cc0-04dc-42b5-8821-9540534b6781-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11283cc0-04dc-42b5-8821-9540534b6781-maas-api-xks-ca\") pod \"11283cc0-04dc-42b5-8821-9540534b6781\" (UID: \"11283cc0-04dc-42b5-8821-9540534b6781\") " Nov 03 20:22:42.859862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:42.858850 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11283cc0-04dc-42b5-8821-9540534b6781-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11283cc0-04dc-42b5-8821-9540534b6781-maas-api-openshift-ca\") pod \"11283cc0-04dc-42b5-8821-9540534b6781\" (UID: \"11283cc0-04dc-42b5-8821-9540534b6781\") " Nov 03 20:22:42.859862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:42.859224 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11283cc0-04dc-42b5-8821-9540534b6781-maas-api-openshift-ca" pod "11283cc0-04dc-42b5-8821-9540534b6781" (UID: "11283cc0-04dc-42b5-8821-9540534b6781"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:42.859862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:42.859545 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11283cc0-04dc-42b5-8821-9540534b6781-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:42.861345 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:42.861314 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11283cc0-04dc-42b5-8821-9540534b6781-kube-api-access-k5jq4" pod "11283cc0-04dc-42b5-8821-9540534b6781" (UID: "11283cc0-04dc-42b5-8821-9540534b6781"). InnerVolumeSpecName "kube-api-access-k5jq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:42.861446 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:42.861353 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11283cc0-04dc-42b5-8821-9540534b6781-maas-api-xks-ca" pod "11283cc0-04dc-42b5-8821-9540534b6781" (UID: "11283cc0-04dc-42b5-8821-9540534b6781"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:22:42.961884 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:42.961854 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5jq4\" (UniqueName: \"kubernetes.io/projected/11283cc0-04dc-42b5-8821-9540534b6781-kube-api-access-k5jq4\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:42.962038 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:42.961948 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11283cc0-04dc-42b5-8821-9540534b6781-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:43.403685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:43.401596 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" podUID="6fd65f7b-2f44-4b57-9675-358d64774fbc" containerName="payload-processing" containerID="cri-o://c96b579c764e3e8e8264ccd6fe73c84f7c5c4202b5ee9b4842fe8fa97d0bbd0c" gracePeriod=25 Nov 03 20:22:43.578688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:43.578191 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce41531-n4559" event={"ID":"11283cc0-04dc-42b5-8821-9540534b6781","Type":"ContainerDied","Data":"f35da7b9890501260e0ec994f85b29d04ed1a6fc07aff90eefda74a74c8c59af"} Nov 03 20:22:43.578688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:43.578261 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f35da7b9890501260e0ec994f85b29d04ed1a6fc07aff90eefda74a74c8c59af" Nov 03 20:22:43.578688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:43.578338 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce41531-n4559" Nov 03 20:22:43.776178 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:22:43.776139 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11283cc0_04dc_42b5_8821_9540534b6781.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd65f7b_2f44_4b57_9675_358d64774fbc.slice/crio-conmon-c96b579c764e3e8e8264ccd6fe73c84f7c5c4202b5ee9b4842fe8fa97d0bbd0c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd65f7b_2f44_4b57_9675_358d64774fbc.slice/crio-c96b579c764e3e8e8264ccd6fe73c84f7c5c4202b5ee9b4842fe8fa97d0bbd0c.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:22:43.885129 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:43.885075 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" Nov 03 20:22:44.038543 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:44.038453 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-366aef64-jxpvk"] Nov 03 20:22:44.043973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:44.043943 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-366aef64-jxpvk"] Nov 03 20:22:44.047602 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:44.047580 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fd65f7b-2f44-4b57-9675-358d64774fbc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fd65f7b-2f44-4b57-9675-358d64774fbc-provider-credentials\") pod \"6fd65f7b-2f44-4b57-9675-358d64774fbc\" (UID: \"6fd65f7b-2f44-4b57-9675-358d64774fbc\") " Nov 03 20:22:44.047746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:44.047622 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fd65f7b-2f44-4b57-9675-358d64774fbc-kube-api-access-gmk59\" (UniqueName: \"kubernetes.io/projected/6fd65f7b-2f44-4b57-9675-358d64774fbc-kube-api-access-gmk59\") pod \"6fd65f7b-2f44-4b57-9675-358d64774fbc\" (UID: \"6fd65f7b-2f44-4b57-9675-358d64774fbc\") " Nov 03 20:22:44.047861 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:44.047744 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fd65f7b-2f44-4b57-9675-358d64774fbc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd65f7b-2f44-4b57-9675-358d64774fbc-plugins-config-volume\") pod \"6fd65f7b-2f44-4b57-9675-358d64774fbc\" (UID: \"6fd65f7b-2f44-4b57-9675-358d64774fbc\") " Nov 03 20:22:44.047861 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:44.047844 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fd65f7b-2f44-4b57-9675-358d64774fbc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fd65f7b-2f44-4b57-9675-358d64774fbc-routing-overlay-volume\") pod \"6fd65f7b-2f44-4b57-9675-358d64774fbc\" (UID: \"6fd65f7b-2f44-4b57-9675-358d64774fbc\") " Nov 03 20:22:44.048871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:44.048197 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fd65f7b-2f44-4b57-9675-358d64774fbc-routing-overlay-volume" pod "6fd65f7b-2f44-4b57-9675-358d64774fbc" (UID: "6fd65f7b-2f44-4b57-9675-358d64774fbc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:44.048871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:44.048202 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fd65f7b-2f44-4b57-9675-358d64774fbc-plugins-config-volume" pod "6fd65f7b-2f44-4b57-9675-358d64774fbc" (UID: "6fd65f7b-2f44-4b57-9675-358d64774fbc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:44.048871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:44.048601 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd65f7b-2f44-4b57-9675-358d64774fbc-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:44.048871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:44.048618 5059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fd65f7b-2f44-4b57-9675-358d64774fbc-routing-overlay-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:44.049927 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:44.049905 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fd65f7b-2f44-4b57-9675-358d64774fbc-provider-credentials" pod "6fd65f7b-2f44-4b57-9675-358d64774fbc" (UID: "6fd65f7b-2f44-4b57-9675-358d64774fbc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:44.050632 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:44.050050 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fd65f7b-2f44-4b57-9675-358d64774fbc-kube-api-access-gmk59" pod "6fd65f7b-2f44-4b57-9675-358d64774fbc" (UID: "6fd65f7b-2f44-4b57-9675-358d64774fbc"). InnerVolumeSpecName "kube-api-access-gmk59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:44.150114 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:44.150079 5059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fd65f7b-2f44-4b57-9675-358d64774fbc-provider-credentials\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:44.150114 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:44.150109 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmk59\" (UniqueName: \"kubernetes.io/projected/6fd65f7b-2f44-4b57-9675-358d64774fbc-kube-api-access-gmk59\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:44.588325 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:44.588287 5059 generic.go:356] "Generic (PLEG): container finished" podID="6fd65f7b-2f44-4b57-9675-358d64774fbc" containerID="c96b579c764e3e8e8264ccd6fe73c84f7c5c4202b5ee9b4842fe8fa97d0bbd0c" exitCode=0 Nov 03 20:22:44.588823 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:44.588334 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" Nov 03 20:22:44.588823 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:44.588360 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" event={"ID":"6fd65f7b-2f44-4b57-9675-358d64774fbc","Type":"ContainerDied","Data":"c96b579c764e3e8e8264ccd6fe73c84f7c5c4202b5ee9b4842fe8fa97d0bbd0c"} Nov 03 20:22:44.588823 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:44.588413 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk" event={"ID":"6fd65f7b-2f44-4b57-9675-358d64774fbc","Type":"ContainerDied","Data":"b1fca81585b385fdb5d19839bea98701ede413e75aeff03de9153fb56675aff5"} Nov 03 20:22:44.588823 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:44.588447 5059 scope.go:122] "RemoveContainer" containerID="c96b579c764e3e8e8264ccd6fe73c84f7c5c4202b5ee9b4842fe8fa97d0bbd0c" Nov 03 20:22:44.600989 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:44.600971 5059 scope.go:122] "RemoveContainer" containerID="c96b579c764e3e8e8264ccd6fe73c84f7c5c4202b5ee9b4842fe8fa97d0bbd0c" Nov 03 20:22:44.601265 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:22:44.601244 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c96b579c764e3e8e8264ccd6fe73c84f7c5c4202b5ee9b4842fe8fa97d0bbd0c\": container with ID starting with c96b579c764e3e8e8264ccd6fe73c84f7c5c4202b5ee9b4842fe8fa97d0bbd0c not found: ID does not exist" containerID="c96b579c764e3e8e8264ccd6fe73c84f7c5c4202b5ee9b4842fe8fa97d0bbd0c" Nov 03 20:22:44.601314 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:44.601274 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c96b579c764e3e8e8264ccd6fe73c84f7c5c4202b5ee9b4842fe8fa97d0bbd0c"} err="failed to get container status \"c96b579c764e3e8e8264ccd6fe73c84f7c5c4202b5ee9b4842fe8fa97d0bbd0c\": rpc error: code = NotFound desc = could not find container \"c96b579c764e3e8e8264ccd6fe73c84f7c5c4202b5ee9b4842fe8fa97d0bbd0c\": container with ID starting with c96b579c764e3e8e8264ccd6fe73c84f7c5c4202b5ee9b4842fe8fa97d0bbd0c not found: ID does not exist" Nov 03 20:22:44.619549 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:44.619524 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk"] Nov 03 20:22:44.625009 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:44.624984 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5968rpmxk"] Nov 03 20:22:45.872835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:45.872798 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fd65f7b-2f44-4b57-9675-358d64774fbc" path="/var/lib/kubelet/pods/6fd65f7b-2f44-4b57-9675-358d64774fbc/volumes" Nov 03 20:22:45.874106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:45.874082 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c930c184-40d2-4b5a-ac89-a24e4e7a6590" path="/var/lib/kubelet/pods/c930c184-40d2-4b5a-ac89-a24e4e7a6590/volumes" Nov 03 20:22:48.298036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.297999 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d937bf0d-c87mk"] Nov 03 20:22:48.298860 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.298844 5059 cpu_manager.go:414] "containerMap: removing container" podUID="6fd65f7b-2f44-4b57-9675-358d64774fbc" containerName="payload-processing" Nov 03 20:22:48.298905 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.298861 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fd65f7b-2f44-4b57-9675-358d64774fbc" containerName="payload-processing" Nov 03 20:22:48.298905 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.298893 5059 cpu_manager.go:414] "containerMap: removing container" podUID="9064e69a-d47d-4bad-83b5-ad17fd2dd0c3" containerName="revoke-keys" Nov 03 20:22:48.298905 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.298898 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9064e69a-d47d-4bad-83b5-ad17fd2dd0c3" containerName="revoke-keys" Nov 03 20:22:48.299023 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.298906 5059 cpu_manager.go:414] "containerMap: removing container" podUID="11283cc0-04dc-42b5-8821-9540534b6781" containerName="revoke-keys" Nov 03 20:22:48.299023 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.298911 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11283cc0-04dc-42b5-8821-9540534b6781" containerName="revoke-keys" Nov 03 20:22:48.299023 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.299007 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9064e69a-d47d-4bad-83b5-ad17fd2dd0c3" containerName="revoke-keys" Nov 03 20:22:48.299023 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.299018 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="11283cc0-04dc-42b5-8821-9540534b6781" containerName="revoke-keys" Nov 03 20:22:48.299202 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.299032 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fd65f7b-2f44-4b57-9675-358d64774fbc" containerName="payload-processing" Nov 03 20:22:48.304121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.304098 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d937bf0d-c87mk" Nov 03 20:22:48.308784 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.308756 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:22:48.309705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.309646 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:22:48.312142 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.312049 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d937bf0d-c87mk"] Nov 03 20:22:48.409684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.402193 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c459f5-0958-4b2b-8527-c733819d0a56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d937bf0d-c87mk\" (UID: \"84c459f5-0958-4b2b-8527-c733819d0a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d937bf0d-c87mk" Nov 03 20:22:48.409684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.402345 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c459f5-0958-4b2b-8527-c733819d0a56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d937bf0d-c87mk\" (UID: \"84c459f5-0958-4b2b-8527-c733819d0a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d937bf0d-c87mk" Nov 03 20:22:48.409684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.402745 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvdtc\" (UniqueName: \"kubernetes.io/projected/84c459f5-0958-4b2b-8527-c733819d0a56-kube-api-access-hvdtc\") pod \"maas-api-revoke-sub-d937bf0d-c87mk\" (UID: \"84c459f5-0958-4b2b-8527-c733819d0a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d937bf0d-c87mk" Nov 03 20:22:48.504746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.504709 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvdtc\" (UniqueName: \"kubernetes.io/projected/84c459f5-0958-4b2b-8527-c733819d0a56-kube-api-access-hvdtc\") pod \"maas-api-revoke-sub-d937bf0d-c87mk\" (UID: \"84c459f5-0958-4b2b-8527-c733819d0a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d937bf0d-c87mk" Nov 03 20:22:48.504954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.504778 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c459f5-0958-4b2b-8527-c733819d0a56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d937bf0d-c87mk\" (UID: \"84c459f5-0958-4b2b-8527-c733819d0a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d937bf0d-c87mk" Nov 03 20:22:48.504954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.504865 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c459f5-0958-4b2b-8527-c733819d0a56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d937bf0d-c87mk\" (UID: \"84c459f5-0958-4b2b-8527-c733819d0a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d937bf0d-c87mk" Nov 03 20:22:48.506430 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.506402 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c459f5-0958-4b2b-8527-c733819d0a56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d937bf0d-c87mk\" (UID: \"84c459f5-0958-4b2b-8527-c733819d0a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d937bf0d-c87mk" Nov 03 20:22:48.507197 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.507168 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c459f5-0958-4b2b-8527-c733819d0a56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d937bf0d-c87mk\" (UID: \"84c459f5-0958-4b2b-8527-c733819d0a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d937bf0d-c87mk" Nov 03 20:22:48.517341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.517313 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvdtc\" (UniqueName: \"kubernetes.io/projected/84c459f5-0958-4b2b-8527-c733819d0a56-kube-api-access-hvdtc\") pod \"maas-api-revoke-sub-d937bf0d-c87mk\" (UID: \"84c459f5-0958-4b2b-8527-c733819d0a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d937bf0d-c87mk" Nov 03 20:22:48.623491 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.623462 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d937bf0d-c87mk" Nov 03 20:22:48.929491 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:48.929457 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d937bf0d-c87mk"] Nov 03 20:22:48.929722 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:22:48.929697 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c459f5_0958_4b2b_8527_c733819d0a56.slice/crio-46bfe68e9e014d2aa3cec95103d707e13b46d4545dc9b426ca1e8a0384899c11 WatchSource:0}: Error finding container 46bfe68e9e014d2aa3cec95103d707e13b46d4545dc9b426ca1e8a0384899c11: Status 404 returned error can't find the container with id 46bfe68e9e014d2aa3cec95103d707e13b46d4545dc9b426ca1e8a0384899c11 Nov 03 20:22:49.646408 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:49.646364 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d937bf0d-c87mk" event={"ID":"84c459f5-0958-4b2b-8527-c733819d0a56","Type":"ContainerStarted","Data":"50a334bf18a40ce26b6f3aa885ab25406b24910372b8a40e1242ebd2b8c1f4b1"} Nov 03 20:22:49.646819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:49.646440 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d937bf0d-c87mk" event={"ID":"84c459f5-0958-4b2b-8527-c733819d0a56","Type":"ContainerStarted","Data":"46bfe68e9e014d2aa3cec95103d707e13b46d4545dc9b426ca1e8a0384899c11"} Nov 03 20:22:49.666786 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:49.666739 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d937bf0d-c87mk" podStartSLOduration=1.666723172 podStartE2EDuration="1.666723172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:49.665579449 +0000 UTC m=+2763.923376706" watchObservedRunningTime="2026-11-03 20:22:49.666723172 +0000 UTC m=+2763.924520429" Nov 03 20:22:50.676144 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:50.676102 5059 generic.go:356] "Generic (PLEG): container finished" podID="84c459f5-0958-4b2b-8527-c733819d0a56" containerID="50a334bf18a40ce26b6f3aa885ab25406b24910372b8a40e1242ebd2b8c1f4b1" exitCode=0 Nov 03 20:22:50.676563 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:50.676187 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d937bf0d-c87mk" event={"ID":"84c459f5-0958-4b2b-8527-c733819d0a56","Type":"ContainerDied","Data":"50a334bf18a40ce26b6f3aa885ab25406b24910372b8a40e1242ebd2b8c1f4b1"} Nov 03 20:22:51.973228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:51.973206 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d937bf0d-c87mk" Nov 03 20:22:52.035082 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:52.035048 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c459f5-0958-4b2b-8527-c733819d0a56-kube-api-access-hvdtc\" (UniqueName: \"kubernetes.io/projected/84c459f5-0958-4b2b-8527-c733819d0a56-kube-api-access-hvdtc\") pod \"84c459f5-0958-4b2b-8527-c733819d0a56\" (UID: \"84c459f5-0958-4b2b-8527-c733819d0a56\") " Nov 03 20:22:52.035228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:52.035156 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84c459f5-0958-4b2b-8527-c733819d0a56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c459f5-0958-4b2b-8527-c733819d0a56-maas-api-xks-ca\") pod \"84c459f5-0958-4b2b-8527-c733819d0a56\" (UID: \"84c459f5-0958-4b2b-8527-c733819d0a56\") " Nov 03 20:22:52.035410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:52.035394 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84c459f5-0958-4b2b-8527-c733819d0a56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c459f5-0958-4b2b-8527-c733819d0a56-maas-api-openshift-ca\") pod \"84c459f5-0958-4b2b-8527-c733819d0a56\" (UID: \"84c459f5-0958-4b2b-8527-c733819d0a56\") " Nov 03 20:22:52.036881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:52.036852 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c459f5-0958-4b2b-8527-c733819d0a56-maas-api-openshift-ca" pod "84c459f5-0958-4b2b-8527-c733819d0a56" (UID: "84c459f5-0958-4b2b-8527-c733819d0a56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:52.039270 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:52.039238 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84c459f5-0958-4b2b-8527-c733819d0a56-maas-api-xks-ca" pod "84c459f5-0958-4b2b-8527-c733819d0a56" (UID: "84c459f5-0958-4b2b-8527-c733819d0a56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:22:52.039386 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:52.039342 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c459f5-0958-4b2b-8527-c733819d0a56-kube-api-access-hvdtc" pod "84c459f5-0958-4b2b-8527-c733819d0a56" (UID: "84c459f5-0958-4b2b-8527-c733819d0a56"). InnerVolumeSpecName "kube-api-access-hvdtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:52.137694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:52.137641 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvdtc\" (UniqueName: \"kubernetes.io/projected/84c459f5-0958-4b2b-8527-c733819d0a56-kube-api-access-hvdtc\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:52.137694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:52.137686 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c459f5-0958-4b2b-8527-c733819d0a56-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:52.137694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:52.137698 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c459f5-0958-4b2b-8527-c733819d0a56-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:52.844702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:52.834037 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d937bf0d-c87mk" event={"ID":"84c459f5-0958-4b2b-8527-c733819d0a56","Type":"ContainerDied","Data":"46bfe68e9e014d2aa3cec95103d707e13b46d4545dc9b426ca1e8a0384899c11"} Nov 03 20:22:52.844702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:52.834112 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46bfe68e9e014d2aa3cec95103d707e13b46d4545dc9b426ca1e8a0384899c11" Nov 03 20:22:52.844702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:52.834196 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d937bf0d-c87mk" Nov 03 20:22:54.451304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:54.451254 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2"] Nov 03 20:22:54.452388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:54.452361 5059 cpu_manager.go:414] "containerMap: removing container" podUID="84c459f5-0958-4b2b-8527-c733819d0a56" containerName="revoke-keys" Nov 03 20:22:54.453459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:54.452389 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c459f5-0958-4b2b-8527-c733819d0a56" containerName="revoke-keys" Nov 03 20:22:54.453459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:54.452629 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c459f5-0958-4b2b-8527-c733819d0a56" containerName="revoke-keys" Nov 03 20:22:54.456594 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:54.456576 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" Nov 03 20:22:54.463153 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:54.463129 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"routing-overlay\"" Nov 03 20:22:54.463295 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:54.463153 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"kube-root-ca.crt\"" Nov 03 20:22:54.463362 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:54.463347 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"openshift-service-ca.crt\"" Nov 03 20:22:54.464228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:54.464210 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0e4ce7\"" Nov 03 20:22:54.464314 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:54.464277 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"agc-cred-5f4fe13bd610\"" Nov 03 20:22:54.464402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:54.464382 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"payload-processing-external-model-e2e--b1878920-dockercfg-vbr6x\"" Nov 03 20:22:54.465201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:54.465181 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2"] Nov 03 20:22:54.650684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:54.641870 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0309a608-f6c4-45ca-991b-33ba73221a6c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2\" (UID: \"0309a608-f6c4-45ca-991b-33ba73221a6c\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" Nov 03 20:22:54.650684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:54.642148 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0309a608-f6c4-45ca-991b-33ba73221a6c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2\" (UID: \"0309a608-f6c4-45ca-991b-33ba73221a6c\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" Nov 03 20:22:54.650684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:54.642193 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtp2k\" (UniqueName: \"kubernetes.io/projected/0309a608-f6c4-45ca-991b-33ba73221a6c-kube-api-access-mtp2k\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2\" (UID: \"0309a608-f6c4-45ca-991b-33ba73221a6c\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" Nov 03 20:22:54.650684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:54.642242 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0309a608-f6c4-45ca-991b-33ba73221a6c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2\" (UID: \"0309a608-f6c4-45ca-991b-33ba73221a6c\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" Nov 03 20:22:54.784933 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:54.756596 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtp2k\" (UniqueName: \"kubernetes.io/projected/0309a608-f6c4-45ca-991b-33ba73221a6c-kube-api-access-mtp2k\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2\" (UID: \"0309a608-f6c4-45ca-991b-33ba73221a6c\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" Nov 03 20:22:54.784933 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:54.756692 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0309a608-f6c4-45ca-991b-33ba73221a6c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2\" (UID: \"0309a608-f6c4-45ca-991b-33ba73221a6c\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" Nov 03 20:22:54.784933 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:54.756945 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0309a608-f6c4-45ca-991b-33ba73221a6c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2\" (UID: \"0309a608-f6c4-45ca-991b-33ba73221a6c\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" Nov 03 20:22:54.784933 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:54.757274 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0309a608-f6c4-45ca-991b-33ba73221a6c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2\" (UID: \"0309a608-f6c4-45ca-991b-33ba73221a6c\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" Nov 03 20:22:54.784933 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:54.758903 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0309a608-f6c4-45ca-991b-33ba73221a6c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2\" (UID: \"0309a608-f6c4-45ca-991b-33ba73221a6c\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" Nov 03 20:22:54.842095 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:54.824104 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0309a608-f6c4-45ca-991b-33ba73221a6c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2\" (UID: \"0309a608-f6c4-45ca-991b-33ba73221a6c\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" Nov 03 20:22:54.860966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:54.842878 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0309a608-f6c4-45ca-991b-33ba73221a6c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2\" (UID: \"0309a608-f6c4-45ca-991b-33ba73221a6c\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" Nov 03 20:22:55.240832 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:55.223114 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtp2k\" (UniqueName: \"kubernetes.io/projected/0309a608-f6c4-45ca-991b-33ba73221a6c-kube-api-access-mtp2k\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2\" (UID: \"0309a608-f6c4-45ca-991b-33ba73221a6c\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" Nov 03 20:22:55.386996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:55.386958 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" Nov 03 20:22:55.545741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:55.542912 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:22:56.047037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:56.047005 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2"] Nov 03 20:22:56.047295 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:22:56.047273 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0309a608_f6c4_45ca_991b_33ba73221a6c.slice/crio-9e5ee2cf73166662f5459bdbc4c34f404a24c7301d0c98128d4de9fb5bd8e3e0 WatchSource:0}: Error finding container 9e5ee2cf73166662f5459bdbc4c34f404a24c7301d0c98128d4de9fb5bd8e3e0: Status 404 returned error can't find the container with id 9e5ee2cf73166662f5459bdbc4c34f404a24c7301d0c98128d4de9fb5bd8e3e0 Nov 03 20:22:56.098346 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:56.098313 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" event={"ID":"0309a608-f6c4-45ca-991b-33ba73221a6c","Type":"ContainerStarted","Data":"9e5ee2cf73166662f5459bdbc4c34f404a24c7301d0c98128d4de9fb5bd8e3e0"} Nov 03 20:22:56.890999 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:56.890962 5059 scope.go:122] "RemoveContainer" containerID="ea7d947176a43903416157e70e7189267dc836e12d224da3cf26ca6db31bddd6" Nov 03 20:22:56.909887 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:56.909859 5059 scope.go:122] "RemoveContainer" containerID="3441d537f563b84dc27ed266da6bc868378f8c7da2dc25adc1c8b82b015cbcd8" Nov 03 20:22:56.928965 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:56.928937 5059 scope.go:122] "RemoveContainer" containerID="6f00779d178bbb7df7d0a1c42362a3f0bd0b7d1a7b5cf5eac297d3a24b472712" Nov 03 20:22:56.943979 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:56.943951 5059 scope.go:122] "RemoveContainer" containerID="9342254650cedc9bcaea3549b02ad08e1cbc29ba8f7299a5e40b49daba6753ae" Nov 03 20:22:56.960814 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:56.960789 5059 scope.go:122] "RemoveContainer" containerID="275d675106b85b74d23e458c800730f3bbecf7283214b15237e01a1bf1419cbf" Nov 03 20:22:56.974991 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:56.974969 5059 scope.go:122] "RemoveContainer" containerID="a9ea8abd23c9fc68664ed01ad3f064efcecd25de6bfa9eb3e95b9cef8938bcaa" Nov 03 20:22:57.187742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:57.187139 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" event={"ID":"0309a608-f6c4-45ca-991b-33ba73221a6c","Type":"ContainerStarted","Data":"6e4d31c85e8cd1b2b8591eeb7383bfd43ae2ffe368b7b0f552663b7b668eb233"} Nov 03 20:22:57.191691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:57.188479 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" Nov 03 20:22:57.230686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:22:57.222370 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" podStartSLOduration=3.222350972 podStartE2EDuration="3.222350972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:57.220057683 +0000 UTC m=+2771.477854940" watchObservedRunningTime="2026-11-03 20:22:57.222350972 +0000 UTC m=+2771.480148229" Nov 03 20:23:09.207562 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:09.207532 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" Nov 03 20:23:10.503694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:10.476931 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h"] Nov 03 20:23:11.149222 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:23:11.149184 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:23:15.505763 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:15.505720 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h" podUID="5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56" containerName="payload-processing" containerID="cri-o://eba398fcd62354e2f9a8fd1d8c08ddf515bc6ab98059119a9ca256a880a3c73a" gracePeriod=25 Nov 03 20:23:15.805168 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:15.805144 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h" Nov 03 20:23:15.880063 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:15.880034 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-routing-overlay-volume\") pod \"5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56\" (UID: \"5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56\") " Nov 03 20:23:15.880219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:15.880104 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-kube-api-access-sdsts\" (UniqueName: \"kubernetes.io/projected/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-kube-api-access-sdsts\") pod \"5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56\" (UID: \"5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56\") " Nov 03 20:23:15.880304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:15.880246 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-plugins-config-volume\") pod \"5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56\" (UID: \"5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56\") " Nov 03 20:23:15.880403 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:15.880383 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-provider-credentials\") pod \"5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56\" (UID: \"5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56\") " Nov 03 20:23:15.880517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:15.880488 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-routing-overlay-volume" pod "5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56" (UID: "5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:15.880938 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:15.880920 5059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-routing-overlay-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:15.881992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:15.881965 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-plugins-config-volume" pod "5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56" (UID: "5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:15.884078 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:15.884042 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-provider-credentials" pod "5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56" (UID: "5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:15.884078 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:15.883943 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-kube-api-access-sdsts" pod "5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56" (UID: "5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56"). InnerVolumeSpecName "kube-api-access-sdsts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:15.982555 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:15.982521 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:15.982555 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:15.982556 5059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-provider-credentials\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:15.982776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:15.982571 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdsts\" (UniqueName: \"kubernetes.io/projected/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56-kube-api-access-sdsts\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:16.627972 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:16.627931 5059 generic.go:356] "Generic (PLEG): container finished" podID="5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56" containerID="eba398fcd62354e2f9a8fd1d8c08ddf515bc6ab98059119a9ca256a880a3c73a" exitCode=0 Nov 03 20:23:16.630118 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:16.627980 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h" Nov 03 20:23:16.630118 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:16.628012 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h" event={"ID":"5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56","Type":"ContainerDied","Data":"eba398fcd62354e2f9a8fd1d8c08ddf515bc6ab98059119a9ca256a880a3c73a"} Nov 03 20:23:16.630118 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:16.628070 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h" event={"ID":"5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56","Type":"ContainerDied","Data":"3a7a67940ebd337d1183d5db3111b377ce58c0c2c3b4762a7bfc99a9dc064db3"} Nov 03 20:23:16.630118 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:16.628093 5059 scope.go:122] "RemoveContainer" containerID="eba398fcd62354e2f9a8fd1d8c08ddf515bc6ab98059119a9ca256a880a3c73a" Nov 03 20:23:16.643245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:16.643219 5059 scope.go:122] "RemoveContainer" containerID="eba398fcd62354e2f9a8fd1d8c08ddf515bc6ab98059119a9ca256a880a3c73a" Nov 03 20:23:16.644336 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:23:16.643537 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eba398fcd62354e2f9a8fd1d8c08ddf515bc6ab98059119a9ca256a880a3c73a\": container with ID starting with eba398fcd62354e2f9a8fd1d8c08ddf515bc6ab98059119a9ca256a880a3c73a not found: ID does not exist" containerID="eba398fcd62354e2f9a8fd1d8c08ddf515bc6ab98059119a9ca256a880a3c73a" Nov 03 20:23:16.644336 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:16.643560 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eba398fcd62354e2f9a8fd1d8c08ddf515bc6ab98059119a9ca256a880a3c73a"} err="failed to get container status \"eba398fcd62354e2f9a8fd1d8c08ddf515bc6ab98059119a9ca256a880a3c73a\": rpc error: code = NotFound desc = could not find container \"eba398fcd62354e2f9a8fd1d8c08ddf515bc6ab98059119a9ca256a880a3c73a\": container with ID starting with eba398fcd62354e2f9a8fd1d8c08ddf515bc6ab98059119a9ca256a880a3c73a not found: ID does not exist" Nov 03 20:23:16.673371 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:16.673343 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h"] Nov 03 20:23:16.679017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:16.678994 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-549fjp58h"] Nov 03 20:23:17.860105 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:17.860069 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56" path="/var/lib/kubelet/pods/5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56/volumes" Nov 03 20:23:40.124031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:40.123980 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:23:40.855688 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:23:40.854835 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:23:43.363687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:43.362805 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-464r7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:23:43.363687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:43.362898 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" podUID="c8683a65-9898-4b4f-a692-f2da325a93b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:53.984838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:53.984795 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:23:53.985492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:53.984876 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" podUID="f0d3c590-f523-4b7e-9086-60df5fc585e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:56.465942 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:56.465902 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25401a7d-6m5ws"] Nov 03 20:23:56.497838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:56.470589 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25401a7d-6m5ws"] Nov 03 20:23:57.067358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:57.067316 5059 scope.go:122] "RemoveContainer" containerID="8965713e7b67fd786a3f2eb079b3bf700a4c843fc6da3de604aa79b0f7c40b3e" Nov 03 20:23:57.886700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:57.886633 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b96b45e9-ce7d-43cf-b2ac-4df96acca191" path="/var/lib/kubelet/pods/b96b45e9-ce7d-43cf-b2ac-4df96acca191/volumes" Nov 03 20:23:59.877087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:23:59.877041 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts" podUID="633c352c-f07e-4032-b962-23a148297e45" containerName="postgres" probeResult="failure" output="command timed out" Nov 03 20:24:05.844884 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:05.844850 5059 generic.go:356] "Generic (PLEG): container finished" podID="f0d3c590-f523-4b7e-9086-60df5fc585e8" containerID="efcfe340defa1e3223df36397fde5c79fbe4b5c7832e90583c68ba8293d3e314" exitCode=0 Nov 03 20:24:05.845721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:05.844929 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" event={"ID":"f0d3c590-f523-4b7e-9086-60df5fc585e8","Type":"ContainerDied","Data":"efcfe340defa1e3223df36397fde5c79fbe4b5c7832e90583c68ba8293d3e314"} Nov 03 20:24:05.845721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:05.845443 5059 scope.go:122] "RemoveContainer" containerID="efcfe340defa1e3223df36397fde5c79fbe4b5c7832e90583c68ba8293d3e314" Nov 03 20:24:05.850387 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:05.850352 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:24:05.851509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:05.851480 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:24:05.863037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:05.851530 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:24:06.881893 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:06.881854 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" event={"ID":"f0d3c590-f523-4b7e-9086-60df5fc585e8","Type":"ContainerStarted","Data":"1638b310e0522c5f2a20ee3d226436456869be0ba4a5f3e1f6c3bdebb4a48c87"} Nov 03 20:24:07.631695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:07.631282 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:24:07.647708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:07.647627 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body= Nov 03 20:24:07.647987 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:07.647730 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" podUID="f0d3c590-f523-4b7e-9086-60df5fc585e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" Nov 03 20:24:07.936375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:07.936299 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:24:08.619959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:08.619925 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body= Nov 03 20:24:08.620138 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:08.619981 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" podUID="f0d3c590-f523-4b7e-9086-60df5fc585e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" Nov 03 20:24:09.619512 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:09.619479 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body= Nov 03 20:24:09.619935 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:09.619531 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" podUID="f0d3c590-f523-4b7e-9086-60df5fc585e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" Nov 03 20:24:10.623554 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:10.623511 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body= Nov 03 20:24:10.624152 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:10.624121 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" podUID="f0d3c590-f523-4b7e-9086-60df5fc585e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" Nov 03 20:24:10.734764 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:24:10.734725 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:24:11.619687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:11.619630 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body= Nov 03 20:24:11.620748 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:11.619737 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" podUID="f0d3c590-f523-4b7e-9086-60df5fc585e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" Nov 03 20:24:12.619645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:12.619611 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body= Nov 03 20:24:12.620108 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:12.619697 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" podUID="f0d3c590-f523-4b7e-9086-60df5fc585e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" Nov 03 20:24:13.797631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:13.796528 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:24:13.799904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:13.798457 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:24:36.181581 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:36.181543 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85c49152-qxjf2"] Nov 03 20:24:36.184786 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:36.184763 5059 cpu_manager.go:414] "containerMap: removing container" podUID="5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56" containerName="payload-processing" Nov 03 20:24:36.184911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:36.184789 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56" containerName="payload-processing" Nov 03 20:24:36.184983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:36.184973 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5317e6f6-e1b2-4bea-a6a1-6e4439c6ce56" containerName="payload-processing" Nov 03 20:24:36.193133 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:36.192519 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c49152-qxjf2" Nov 03 20:24:36.201804 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:36.201778 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:24:36.201959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:36.201906 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:24:36.257742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:36.257707 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85c49152-qxjf2"] Nov 03 20:24:36.323068 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:36.323029 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85c49152-qxjf2\" (UID: \"34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c49152-qxjf2" Nov 03 20:24:36.323565 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:36.323532 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvvqw\" (UniqueName: \"kubernetes.io/projected/34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d-kube-api-access-xvvqw\") pod \"maas-api-revoke-sub-85c49152-qxjf2\" (UID: \"34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c49152-qxjf2" Nov 03 20:24:36.324070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:36.323742 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85c49152-qxjf2\" (UID: \"34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c49152-qxjf2" Nov 03 20:24:36.425524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:36.425492 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvvqw\" (UniqueName: \"kubernetes.io/projected/34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d-kube-api-access-xvvqw\") pod \"maas-api-revoke-sub-85c49152-qxjf2\" (UID: \"34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c49152-qxjf2" Nov 03 20:24:36.425643 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:36.425572 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85c49152-qxjf2\" (UID: \"34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c49152-qxjf2" Nov 03 20:24:36.425756 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:36.425746 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85c49152-qxjf2\" (UID: \"34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c49152-qxjf2" Nov 03 20:24:36.426954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:36.426930 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85c49152-qxjf2\" (UID: \"34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c49152-qxjf2" Nov 03 20:24:36.436710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:36.436644 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85c49152-qxjf2\" (UID: \"34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c49152-qxjf2" Nov 03 20:24:36.439045 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:36.439023 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvvqw\" (UniqueName: \"kubernetes.io/projected/34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d-kube-api-access-xvvqw\") pod \"maas-api-revoke-sub-85c49152-qxjf2\" (UID: \"34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c49152-qxjf2" Nov 03 20:24:36.514373 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:36.514345 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c49152-qxjf2" Nov 03 20:24:37.102643 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:37.102614 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85c49152-qxjf2"] Nov 03 20:24:37.104977 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:24:37.104947 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34dc7a0b_ebbe_49b3_95de_0a77c7f63e6d.slice/crio-6c8eb9893fdc49649994b43a6c03d7b8c44e4d908c818bdb8dee3bdb37bdc188 WatchSource:0}: Error finding container 6c8eb9893fdc49649994b43a6c03d7b8c44e4d908c818bdb8dee3bdb37bdc188: Status 404 returned error can't find the container with id 6c8eb9893fdc49649994b43a6c03d7b8c44e4d908c818bdb8dee3bdb37bdc188 Nov 03 20:24:37.390702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:37.390583 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c49152-qxjf2" event={"ID":"34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d","Type":"ContainerStarted","Data":"6c8eb9893fdc49649994b43a6c03d7b8c44e4d908c818bdb8dee3bdb37bdc188"} Nov 03 20:24:38.401195 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:38.401157 5059 generic.go:356] "Generic (PLEG): container finished" podID="34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d" containerID="e9d8391fb50e417bbc209099a26beaeb42cc1de9591126f570e39ea8262098a0" exitCode=0 Nov 03 20:24:38.401583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:38.401245 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c49152-qxjf2" event={"ID":"34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d","Type":"ContainerDied","Data":"e9d8391fb50e417bbc209099a26beaeb42cc1de9591126f570e39ea8262098a0"} Nov 03 20:24:39.710733 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:39.710707 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c49152-qxjf2" Nov 03 20:24:39.784883 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:39.784853 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d-maas-api-openshift-ca\") pod \"34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d\" (UID: \"34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d\") " Nov 03 20:24:39.785077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:39.785038 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d-kube-api-access-xvvqw\" (UniqueName: \"kubernetes.io/projected/34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d-kube-api-access-xvvqw\") pod \"34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d\" (UID: \"34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d\") " Nov 03 20:24:39.786130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:39.785152 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d-maas-api-xks-ca\") pod \"34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d\" (UID: \"34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d\") " Nov 03 20:24:39.786130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:39.785237 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d-maas-api-openshift-ca" pod "34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d" (UID: "34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:39.786130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:39.785958 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:39.793198 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:39.793167 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d-maas-api-xks-ca" pod "34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d" (UID: "34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:24:39.793319 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:39.793237 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d-kube-api-access-xvvqw" pod "34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d" (UID: "34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d"). InnerVolumeSpecName "kube-api-access-xvvqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:39.887364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:39.887334 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvvqw\" (UniqueName: \"kubernetes.io/projected/34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d-kube-api-access-xvvqw\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:39.887489 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:39.887367 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:40.419073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:40.419039 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c49152-qxjf2" Nov 03 20:24:40.420343 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:40.419068 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85c49152-qxjf2" event={"ID":"34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d","Type":"ContainerDied","Data":"6c8eb9893fdc49649994b43a6c03d7b8c44e4d908c818bdb8dee3bdb37bdc188"} Nov 03 20:24:40.420343 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:40.419126 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c8eb9893fdc49649994b43a6c03d7b8c44e4d908c818bdb8dee3bdb37bdc188" Nov 03 20:24:40.613721 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:24:40.613682 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:24:43.534686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:43.527751 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:43.645014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:43.644971 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:46.572063 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.572027 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c6990cd-vnf4z"] Nov 03 20:24:46.575162 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.573084 5059 cpu_manager.go:414] "containerMap: removing container" podUID="34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d" containerName="revoke-keys" Nov 03 20:24:46.575162 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.573105 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d" containerName="revoke-keys" Nov 03 20:24:46.575162 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.573315 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d" containerName="revoke-keys" Nov 03 20:24:46.584918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.584893 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c6990cd-vnf4z" Nov 03 20:24:46.588939 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.588914 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c6990cd-vnf4z"] Nov 03 20:24:46.591863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.591838 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:24:46.592164 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.592145 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:24:46.680470 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.680435 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ec89e5b-1640-4b12-9150-b40a3a353c8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c6990cd-vnf4z\" (UID: \"5ec89e5b-1640-4b12-9150-b40a3a353c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c6990cd-vnf4z" Nov 03 20:24:46.680620 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.680544 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ec89e5b-1640-4b12-9150-b40a3a353c8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c6990cd-vnf4z\" (UID: \"5ec89e5b-1640-4b12-9150-b40a3a353c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c6990cd-vnf4z" Nov 03 20:24:46.680773 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.680754 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m89p\" (UniqueName: \"kubernetes.io/projected/5ec89e5b-1640-4b12-9150-b40a3a353c8e-kube-api-access-8m89p\") pod \"maas-api-revoke-sub-5c6990cd-vnf4z\" (UID: \"5ec89e5b-1640-4b12-9150-b40a3a353c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c6990cd-vnf4z" Nov 03 20:24:46.712601 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.712572 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b89f5007-8qfqc"] Nov 03 20:24:46.717185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.717168 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89f5007-8qfqc" Nov 03 20:24:46.730319 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.728821 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b89f5007-8qfqc"] Nov 03 20:24:46.782627 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.782593 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b69fe94-91fc-44a0-a91b-b7470105d659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b89f5007-8qfqc\" (UID: \"5b69fe94-91fc-44a0-a91b-b7470105d659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89f5007-8qfqc" Nov 03 20:24:46.782874 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.782853 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ec89e5b-1640-4b12-9150-b40a3a353c8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c6990cd-vnf4z\" (UID: \"5ec89e5b-1640-4b12-9150-b40a3a353c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c6990cd-vnf4z" Nov 03 20:24:46.783035 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.782944 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ec89e5b-1640-4b12-9150-b40a3a353c8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c6990cd-vnf4z\" (UID: \"5ec89e5b-1640-4b12-9150-b40a3a353c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c6990cd-vnf4z" Nov 03 20:24:46.783227 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.783116 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m89p\" (UniqueName: \"kubernetes.io/projected/5ec89e5b-1640-4b12-9150-b40a3a353c8e-kube-api-access-8m89p\") pod \"maas-api-revoke-sub-5c6990cd-vnf4z\" (UID: \"5ec89e5b-1640-4b12-9150-b40a3a353c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c6990cd-vnf4z" Nov 03 20:24:46.783227 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.783161 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trfrk\" (UniqueName: \"kubernetes.io/projected/5b69fe94-91fc-44a0-a91b-b7470105d659-kube-api-access-trfrk\") pod \"maas-api-revoke-sub-b89f5007-8qfqc\" (UID: \"5b69fe94-91fc-44a0-a91b-b7470105d659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89f5007-8qfqc" Nov 03 20:24:46.783227 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.783196 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b69fe94-91fc-44a0-a91b-b7470105d659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b89f5007-8qfqc\" (UID: \"5b69fe94-91fc-44a0-a91b-b7470105d659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89f5007-8qfqc" Nov 03 20:24:46.790973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.790946 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ec89e5b-1640-4b12-9150-b40a3a353c8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c6990cd-vnf4z\" (UID: \"5ec89e5b-1640-4b12-9150-b40a3a353c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c6990cd-vnf4z" Nov 03 20:24:46.795389 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.795359 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ec89e5b-1640-4b12-9150-b40a3a353c8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c6990cd-vnf4z\" (UID: \"5ec89e5b-1640-4b12-9150-b40a3a353c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c6990cd-vnf4z" Nov 03 20:24:46.796304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.796285 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m89p\" (UniqueName: \"kubernetes.io/projected/5ec89e5b-1640-4b12-9150-b40a3a353c8e-kube-api-access-8m89p\") pod \"maas-api-revoke-sub-5c6990cd-vnf4z\" (UID: \"5ec89e5b-1640-4b12-9150-b40a3a353c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c6990cd-vnf4z" Nov 03 20:24:46.884995 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.884923 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trfrk\" (UniqueName: \"kubernetes.io/projected/5b69fe94-91fc-44a0-a91b-b7470105d659-kube-api-access-trfrk\") pod \"maas-api-revoke-sub-b89f5007-8qfqc\" (UID: \"5b69fe94-91fc-44a0-a91b-b7470105d659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89f5007-8qfqc" Nov 03 20:24:46.884995 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.884960 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b69fe94-91fc-44a0-a91b-b7470105d659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b89f5007-8qfqc\" (UID: \"5b69fe94-91fc-44a0-a91b-b7470105d659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89f5007-8qfqc" Nov 03 20:24:46.886848 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.885036 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b69fe94-91fc-44a0-a91b-b7470105d659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b89f5007-8qfqc\" (UID: \"5b69fe94-91fc-44a0-a91b-b7470105d659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89f5007-8qfqc" Nov 03 20:24:46.886848 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.885732 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b69fe94-91fc-44a0-a91b-b7470105d659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b89f5007-8qfqc\" (UID: \"5b69fe94-91fc-44a0-a91b-b7470105d659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89f5007-8qfqc" Nov 03 20:24:46.889267 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.887429 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b69fe94-91fc-44a0-a91b-b7470105d659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b89f5007-8qfqc\" (UID: \"5b69fe94-91fc-44a0-a91b-b7470105d659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89f5007-8qfqc" Nov 03 20:24:46.897256 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.897230 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trfrk\" (UniqueName: \"kubernetes.io/projected/5b69fe94-91fc-44a0-a91b-b7470105d659-kube-api-access-trfrk\") pod \"maas-api-revoke-sub-b89f5007-8qfqc\" (UID: \"5b69fe94-91fc-44a0-a91b-b7470105d659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89f5007-8qfqc" Nov 03 20:24:46.908688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:46.908636 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c6990cd-vnf4z" Nov 03 20:24:47.039983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:47.039939 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89f5007-8qfqc" Nov 03 20:24:47.331451 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:24:47.331418 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec89e5b_1640_4b12_9150_b40a3a353c8e.slice/crio-9ca8e5d1dc8e819dbd6902bb153ddd8e015ba92ee87f0f111abffc7dfa9892d7 WatchSource:0}: Error finding container 9ca8e5d1dc8e819dbd6902bb153ddd8e015ba92ee87f0f111abffc7dfa9892d7: Status 404 returned error can't find the container with id 9ca8e5d1dc8e819dbd6902bb153ddd8e015ba92ee87f0f111abffc7dfa9892d7 Nov 03 20:24:47.332593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:47.332569 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c6990cd-vnf4z"] Nov 03 20:24:47.350603 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:47.350576 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b89f5007-8qfqc"] Nov 03 20:24:47.351087 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:24:47.351065 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b69fe94_91fc_44a0_a91b_b7470105d659.slice/crio-0398bd49287ebe35def1f3ab9570ec0b3cd35430730f630ecf6c776ed020fee2 WatchSource:0}: Error finding container 0398bd49287ebe35def1f3ab9570ec0b3cd35430730f630ecf6c776ed020fee2: Status 404 returned error can't find the container with id 0398bd49287ebe35def1f3ab9570ec0b3cd35430730f630ecf6c776ed020fee2 Nov 03 20:24:47.619234 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:47.619195 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89f5007-8qfqc" event={"ID":"5b69fe94-91fc-44a0-a91b-b7470105d659","Type":"ContainerStarted","Data":"0398bd49287ebe35def1f3ab9570ec0b3cd35430730f630ecf6c776ed020fee2"} Nov 03 20:24:47.620547 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:47.620523 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c6990cd-vnf4z" event={"ID":"5ec89e5b-1640-4b12-9150-b40a3a353c8e","Type":"ContainerStarted","Data":"9ca8e5d1dc8e819dbd6902bb153ddd8e015ba92ee87f0f111abffc7dfa9892d7"} Nov 03 20:24:48.631283 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:48.631199 5059 generic.go:356] "Generic (PLEG): container finished" podID="5ec89e5b-1640-4b12-9150-b40a3a353c8e" containerID="b55d8d94b1bfd091549e5f64c5b4ba80d22392a32e05076a92b999b27dc44f7a" exitCode=0 Nov 03 20:24:48.631686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:48.631284 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c6990cd-vnf4z" event={"ID":"5ec89e5b-1640-4b12-9150-b40a3a353c8e","Type":"ContainerDied","Data":"b55d8d94b1bfd091549e5f64c5b4ba80d22392a32e05076a92b999b27dc44f7a"} Nov 03 20:24:48.633239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:48.633209 5059 generic.go:356] "Generic (PLEG): container finished" podID="5b69fe94-91fc-44a0-a91b-b7470105d659" containerID="63bc22494254a32bee2bb7f180343e815aa6a4b2d9c80eef741cbee37519d4fe" exitCode=0 Nov 03 20:24:48.633350 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:48.633273 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89f5007-8qfqc" event={"ID":"5b69fe94-91fc-44a0-a91b-b7470105d659","Type":"ContainerDied","Data":"63bc22494254a32bee2bb7f180343e815aa6a4b2d9c80eef741cbee37519d4fe"} Nov 03 20:24:49.966411 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:49.966013 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c6990cd-vnf4z" Nov 03 20:24:50.018701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.018634 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89f5007-8qfqc" Nov 03 20:24:50.084175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.084137 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ec89e5b-1640-4b12-9150-b40a3a353c8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ec89e5b-1640-4b12-9150-b40a3a353c8e-maas-api-xks-ca\") pod \"5ec89e5b-1640-4b12-9150-b40a3a353c8e\" (UID: \"5ec89e5b-1640-4b12-9150-b40a3a353c8e\") " Nov 03 20:24:50.084378 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.084209 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec89e5b-1640-4b12-9150-b40a3a353c8e-kube-api-access-8m89p\" (UniqueName: \"kubernetes.io/projected/5ec89e5b-1640-4b12-9150-b40a3a353c8e-kube-api-access-8m89p\") pod \"5ec89e5b-1640-4b12-9150-b40a3a353c8e\" (UID: \"5ec89e5b-1640-4b12-9150-b40a3a353c8e\") " Nov 03 20:24:50.084378 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.084331 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b69fe94-91fc-44a0-a91b-b7470105d659-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b69fe94-91fc-44a0-a91b-b7470105d659-maas-api-xks-ca\") pod \"5b69fe94-91fc-44a0-a91b-b7470105d659\" (UID: \"5b69fe94-91fc-44a0-a91b-b7470105d659\") " Nov 03 20:24:50.085477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.084543 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b69fe94-91fc-44a0-a91b-b7470105d659-kube-api-access-trfrk\" (UniqueName: \"kubernetes.io/projected/5b69fe94-91fc-44a0-a91b-b7470105d659-kube-api-access-trfrk\") pod \"5b69fe94-91fc-44a0-a91b-b7470105d659\" (UID: \"5b69fe94-91fc-44a0-a91b-b7470105d659\") " Nov 03 20:24:50.085477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.085172 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ec89e5b-1640-4b12-9150-b40a3a353c8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ec89e5b-1640-4b12-9150-b40a3a353c8e-maas-api-openshift-ca\") pod \"5ec89e5b-1640-4b12-9150-b40a3a353c8e\" (UID: \"5ec89e5b-1640-4b12-9150-b40a3a353c8e\") " Nov 03 20:24:50.085477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.085309 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b69fe94-91fc-44a0-a91b-b7470105d659-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b69fe94-91fc-44a0-a91b-b7470105d659-maas-api-openshift-ca\") pod \"5b69fe94-91fc-44a0-a91b-b7470105d659\" (UID: \"5b69fe94-91fc-44a0-a91b-b7470105d659\") " Nov 03 20:24:50.085713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.085639 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec89e5b-1640-4b12-9150-b40a3a353c8e-maas-api-openshift-ca" pod "5ec89e5b-1640-4b12-9150-b40a3a353c8e" (UID: "5ec89e5b-1640-4b12-9150-b40a3a353c8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:50.085792 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.085740 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b69fe94-91fc-44a0-a91b-b7470105d659-maas-api-openshift-ca" pod "5b69fe94-91fc-44a0-a91b-b7470105d659" (UID: "5b69fe94-91fc-44a0-a91b-b7470105d659"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:50.086745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.086718 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ec89e5b-1640-4b12-9150-b40a3a353c8e-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:50.086839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.086749 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b69fe94-91fc-44a0-a91b-b7470105d659-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:50.087011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.086984 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ec89e5b-1640-4b12-9150-b40a3a353c8e-maas-api-xks-ca" pod "5ec89e5b-1640-4b12-9150-b40a3a353c8e" (UID: "5ec89e5b-1640-4b12-9150-b40a3a353c8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:24:50.087837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.087723 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec89e5b-1640-4b12-9150-b40a3a353c8e-kube-api-access-8m89p" pod "5ec89e5b-1640-4b12-9150-b40a3a353c8e" (UID: "5ec89e5b-1640-4b12-9150-b40a3a353c8e"). InnerVolumeSpecName "kube-api-access-8m89p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:50.088780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.087959 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b69fe94-91fc-44a0-a91b-b7470105d659-kube-api-access-trfrk" pod "5b69fe94-91fc-44a0-a91b-b7470105d659" (UID: "5b69fe94-91fc-44a0-a91b-b7470105d659"). InnerVolumeSpecName "kube-api-access-trfrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:50.088780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.088159 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b69fe94-91fc-44a0-a91b-b7470105d659-maas-api-xks-ca" pod "5b69fe94-91fc-44a0-a91b-b7470105d659" (UID: "5b69fe94-91fc-44a0-a91b-b7470105d659"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:24:50.188089 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.188063 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b69fe94-91fc-44a0-a91b-b7470105d659-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:50.188214 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.188093 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trfrk\" (UniqueName: \"kubernetes.io/projected/5b69fe94-91fc-44a0-a91b-b7470105d659-kube-api-access-trfrk\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:50.188214 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.188110 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ec89e5b-1640-4b12-9150-b40a3a353c8e-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:50.188214 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.188122 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m89p\" (UniqueName: \"kubernetes.io/projected/5ec89e5b-1640-4b12-9150-b40a3a353c8e-kube-api-access-8m89p\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:50.654516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.654490 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c6990cd-vnf4z" Nov 03 20:24:50.654709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.654527 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c6990cd-vnf4z" event={"ID":"5ec89e5b-1640-4b12-9150-b40a3a353c8e","Type":"ContainerDied","Data":"9ca8e5d1dc8e819dbd6902bb153ddd8e015ba92ee87f0f111abffc7dfa9892d7"} Nov 03 20:24:50.654709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.654578 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ca8e5d1dc8e819dbd6902bb153ddd8e015ba92ee87f0f111abffc7dfa9892d7" Nov 03 20:24:50.656238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.656212 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89f5007-8qfqc" event={"ID":"5b69fe94-91fc-44a0-a91b-b7470105d659","Type":"ContainerDied","Data":"0398bd49287ebe35def1f3ab9570ec0b3cd35430730f630ecf6c776ed020fee2"} Nov 03 20:24:50.656238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.656247 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0398bd49287ebe35def1f3ab9570ec0b3cd35430730f630ecf6c776ed020fee2" Nov 03 20:24:50.656414 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:50.656219 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89f5007-8qfqc" Nov 03 20:24:53.064652 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:53.064609 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae219-5mpbm"] Nov 03 20:24:53.089725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:53.086686 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4ae219-5mpbm"] Nov 03 20:24:53.862549 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:53.862510 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea2b3d82-2b69-485e-a576-c8d805fca8b4" path="/var/lib/kubelet/pods/ea2b3d82-2b69-485e-a576-c8d805fca8b4/volumes" Nov 03 20:24:57.275806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:57.275654 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da7869e3-kttw5"] Nov 03 20:24:57.277238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:57.277205 5059 cpu_manager.go:414] "containerMap: removing container" podUID="5b69fe94-91fc-44a0-a91b-b7470105d659" containerName="revoke-keys" Nov 03 20:24:57.278020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:57.277349 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b69fe94-91fc-44a0-a91b-b7470105d659" containerName="revoke-keys" Nov 03 20:24:57.278020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:57.277428 5059 cpu_manager.go:414] "containerMap: removing container" podUID="5ec89e5b-1640-4b12-9150-b40a3a353c8e" containerName="revoke-keys" Nov 03 20:24:57.278020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:57.277436 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ec89e5b-1640-4b12-9150-b40a3a353c8e" containerName="revoke-keys" Nov 03 20:24:57.278020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:57.277656 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ec89e5b-1640-4b12-9150-b40a3a353c8e" containerName="revoke-keys" Nov 03 20:24:57.278020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:57.277706 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b69fe94-91fc-44a0-a91b-b7470105d659" containerName="revoke-keys" Nov 03 20:24:57.286069 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:57.286041 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da7869e3-kttw5"] Nov 03 20:24:57.287071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:57.286290 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7869e3-kttw5" Nov 03 20:24:57.295495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:57.295471 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:24:57.295799 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:57.295778 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:24:57.412370 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:57.412334 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c0c10d-2b49-481e-85d8-50d0a3b251a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da7869e3-kttw5\" (UID: \"44c0c10d-2b49-481e-85d8-50d0a3b251a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7869e3-kttw5" Nov 03 20:24:57.412532 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:57.412455 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c0c10d-2b49-481e-85d8-50d0a3b251a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da7869e3-kttw5\" (UID: \"44c0c10d-2b49-481e-85d8-50d0a3b251a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7869e3-kttw5" Nov 03 20:24:57.413806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:57.412580 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5kqn\" (UniqueName: \"kubernetes.io/projected/44c0c10d-2b49-481e-85d8-50d0a3b251a1-kube-api-access-d5kqn\") pod \"maas-api-revoke-sub-da7869e3-kttw5\" (UID: \"44c0c10d-2b49-481e-85d8-50d0a3b251a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7869e3-kttw5" Nov 03 20:24:57.514024 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:57.513988 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c0c10d-2b49-481e-85d8-50d0a3b251a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da7869e3-kttw5\" (UID: \"44c0c10d-2b49-481e-85d8-50d0a3b251a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7869e3-kttw5" Nov 03 20:24:57.514167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:57.514083 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c0c10d-2b49-481e-85d8-50d0a3b251a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da7869e3-kttw5\" (UID: \"44c0c10d-2b49-481e-85d8-50d0a3b251a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7869e3-kttw5" Nov 03 20:24:57.514242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:57.514207 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5kqn\" (UniqueName: \"kubernetes.io/projected/44c0c10d-2b49-481e-85d8-50d0a3b251a1-kube-api-access-d5kqn\") pod \"maas-api-revoke-sub-da7869e3-kttw5\" (UID: \"44c0c10d-2b49-481e-85d8-50d0a3b251a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7869e3-kttw5" Nov 03 20:24:57.515843 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:57.515823 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c0c10d-2b49-481e-85d8-50d0a3b251a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da7869e3-kttw5\" (UID: \"44c0c10d-2b49-481e-85d8-50d0a3b251a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7869e3-kttw5" Nov 03 20:24:57.516547 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:57.516517 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c0c10d-2b49-481e-85d8-50d0a3b251a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da7869e3-kttw5\" (UID: \"44c0c10d-2b49-481e-85d8-50d0a3b251a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7869e3-kttw5" Nov 03 20:24:57.526854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:57.526793 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5kqn\" (UniqueName: \"kubernetes.io/projected/44c0c10d-2b49-481e-85d8-50d0a3b251a1-kube-api-access-d5kqn\") pod \"maas-api-revoke-sub-da7869e3-kttw5\" (UID: \"44c0c10d-2b49-481e-85d8-50d0a3b251a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7869e3-kttw5" Nov 03 20:24:57.612437 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:57.612411 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7869e3-kttw5" Nov 03 20:24:58.013858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:58.013823 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da7869e3-kttw5"] Nov 03 20:24:58.014168 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:24:58.014142 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c0c10d_2b49_481e_85d8_50d0a3b251a1.slice/crio-50158d6b5c0a785f53048ef597abdc6592e893d28ff780f3124f2665d4bf8e3c WatchSource:0}: Error finding container 50158d6b5c0a785f53048ef597abdc6592e893d28ff780f3124f2665d4bf8e3c: Status 404 returned error can't find the container with id 50158d6b5c0a785f53048ef597abdc6592e893d28ff780f3124f2665d4bf8e3c Nov 03 20:24:58.668990 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:58.668611 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l"] Nov 03 20:24:59.139214 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:59.139173 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l"] Nov 03 20:24:59.159153 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:59.154466 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l" Nov 03 20:24:59.256041 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:59.255994 5059 scope.go:122] "RemoveContainer" containerID="94821d8bafaa395235edfb3518b58bbfa0a80434927536272c057fe1ccbaa4ad" Nov 03 20:24:59.258114 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:59.258077 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54ff6643-48a2-4fc3-9446-dd096b73b09a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l\" (UID: \"54ff6643-48a2-4fc3-9446-dd096b73b09a\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l" Nov 03 20:24:59.270827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:59.258301 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54ff6643-48a2-4fc3-9446-dd096b73b09a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l\" (UID: \"54ff6643-48a2-4fc3-9446-dd096b73b09a\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l" Nov 03 20:24:59.270827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:59.258488 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54ff6643-48a2-4fc3-9446-dd096b73b09a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l\" (UID: \"54ff6643-48a2-4fc3-9446-dd096b73b09a\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l" Nov 03 20:24:59.270827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:59.258570 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4n2h\" (UniqueName: \"kubernetes.io/projected/54ff6643-48a2-4fc3-9446-dd096b73b09a-kube-api-access-g4n2h\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l\" (UID: \"54ff6643-48a2-4fc3-9446-dd096b73b09a\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l" Nov 03 20:24:59.422918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:59.421621 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54ff6643-48a2-4fc3-9446-dd096b73b09a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l\" (UID: \"54ff6643-48a2-4fc3-9446-dd096b73b09a\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l" Nov 03 20:24:59.422918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:59.421714 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54ff6643-48a2-4fc3-9446-dd096b73b09a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l\" (UID: \"54ff6643-48a2-4fc3-9446-dd096b73b09a\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l" Nov 03 20:24:59.422918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:59.421927 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54ff6643-48a2-4fc3-9446-dd096b73b09a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l\" (UID: \"54ff6643-48a2-4fc3-9446-dd096b73b09a\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l" Nov 03 20:24:59.422918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:59.422021 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4n2h\" (UniqueName: \"kubernetes.io/projected/54ff6643-48a2-4fc3-9446-dd096b73b09a-kube-api-access-g4n2h\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l\" (UID: \"54ff6643-48a2-4fc3-9446-dd096b73b09a\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l" Nov 03 20:24:59.431597 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:59.431555 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54ff6643-48a2-4fc3-9446-dd096b73b09a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l\" (UID: \"54ff6643-48a2-4fc3-9446-dd096b73b09a\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l" Nov 03 20:24:59.460558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:59.460517 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7869e3-kttw5" event={"ID":"44c0c10d-2b49-481e-85d8-50d0a3b251a1","Type":"ContainerStarted","Data":"926702d6bfa8f7006addfc1ed714e4470d66af979102156b85f6cad5b34d1221"} Nov 03 20:24:59.461647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:59.460791 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7869e3-kttw5" event={"ID":"44c0c10d-2b49-481e-85d8-50d0a3b251a1","Type":"ContainerStarted","Data":"50158d6b5c0a785f53048ef597abdc6592e893d28ff780f3124f2665d4bf8e3c"} Nov 03 20:24:59.461647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:59.461389 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54ff6643-48a2-4fc3-9446-dd096b73b09a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l\" (UID: \"54ff6643-48a2-4fc3-9446-dd096b73b09a\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l" Nov 03 20:24:59.489404 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:59.489337 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7869e3-kttw5" podStartSLOduration=2.489306933 podStartE2EDuration="2.489306933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:59.486643135 +0000 UTC m=+2893.744440400" watchObservedRunningTime="2026-11-03 20:24:59.489306933 +0000 UTC m=+2893.747104203" Nov 03 20:24:59.559037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:59.539446 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4n2h\" (UniqueName: \"kubernetes.io/projected/54ff6643-48a2-4fc3-9446-dd096b73b09a-kube-api-access-g4n2h\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l\" (UID: \"54ff6643-48a2-4fc3-9446-dd096b73b09a\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l" Nov 03 20:24:59.561026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:59.560938 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54ff6643-48a2-4fc3-9446-dd096b73b09a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l\" (UID: \"54ff6643-48a2-4fc3-9446-dd096b73b09a\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l" Nov 03 20:24:59.759273 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:24:59.759197 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l" Nov 03 20:25:00.054736 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:25:00.054703 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ff6643_48a2_4fc3_9446_dd096b73b09a.slice/crio-928c55fa72997aa9b8de5c3d7019cd3deea31c22ce8cfa2f29d53d7bb631164e WatchSource:0}: Error finding container 928c55fa72997aa9b8de5c3d7019cd3deea31c22ce8cfa2f29d53d7bb631164e: Status 404 returned error can't find the container with id 928c55fa72997aa9b8de5c3d7019cd3deea31c22ce8cfa2f29d53d7bb631164e Nov 03 20:25:00.055627 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:00.055228 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l"] Nov 03 20:25:00.472636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:00.472598 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l" event={"ID":"54ff6643-48a2-4fc3-9446-dd096b73b09a","Type":"ContainerStarted","Data":"103312d7497946c4c859f1b107fdc31bd0618ca0fec73a597c66b6369bb5e019"} Nov 03 20:25:00.472825 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:00.472671 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l" Nov 03 20:25:00.472825 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:00.472720 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l" event={"ID":"54ff6643-48a2-4fc3-9446-dd096b73b09a","Type":"ContainerStarted","Data":"928c55fa72997aa9b8de5c3d7019cd3deea31c22ce8cfa2f29d53d7bb631164e"} Nov 03 20:25:00.474590 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:00.474563 5059 generic.go:356] "Generic (PLEG): container finished" podID="44c0c10d-2b49-481e-85d8-50d0a3b251a1" containerID="926702d6bfa8f7006addfc1ed714e4470d66af979102156b85f6cad5b34d1221" exitCode=0 Nov 03 20:25:00.474680 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:00.474606 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7869e3-kttw5" event={"ID":"44c0c10d-2b49-481e-85d8-50d0a3b251a1","Type":"ContainerDied","Data":"926702d6bfa8f7006addfc1ed714e4470d66af979102156b85f6cad5b34d1221"} Nov 03 20:25:00.502778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:00.502731 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l" podStartSLOduration=2.5027182850000003 podStartE2EDuration="2.502718285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:00.499892751 +0000 UTC m=+2894.757690008" watchObservedRunningTime="2026-11-03 20:25:00.502718285 +0000 UTC m=+2894.760515567" Nov 03 20:25:01.810549 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:01.810521 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7869e3-kttw5" Nov 03 20:25:01.922327 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:01.922287 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44c0c10d-2b49-481e-85d8-50d0a3b251a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c0c10d-2b49-481e-85d8-50d0a3b251a1-maas-api-xks-ca\") pod \"44c0c10d-2b49-481e-85d8-50d0a3b251a1\" (UID: \"44c0c10d-2b49-481e-85d8-50d0a3b251a1\") " Nov 03 20:25:01.922521 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:01.922426 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44c0c10d-2b49-481e-85d8-50d0a3b251a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c0c10d-2b49-481e-85d8-50d0a3b251a1-maas-api-openshift-ca\") pod \"44c0c10d-2b49-481e-85d8-50d0a3b251a1\" (UID: \"44c0c10d-2b49-481e-85d8-50d0a3b251a1\") " Nov 03 20:25:01.922521 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:01.922518 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c0c10d-2b49-481e-85d8-50d0a3b251a1-kube-api-access-d5kqn\" (UniqueName: \"kubernetes.io/projected/44c0c10d-2b49-481e-85d8-50d0a3b251a1-kube-api-access-d5kqn\") pod \"44c0c10d-2b49-481e-85d8-50d0a3b251a1\" (UID: \"44c0c10d-2b49-481e-85d8-50d0a3b251a1\") " Nov 03 20:25:01.922873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:01.922834 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44c0c10d-2b49-481e-85d8-50d0a3b251a1-maas-api-openshift-ca" pod "44c0c10d-2b49-481e-85d8-50d0a3b251a1" (UID: "44c0c10d-2b49-481e-85d8-50d0a3b251a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:01.923543 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:01.923507 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c0c10d-2b49-481e-85d8-50d0a3b251a1-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:01.924757 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:01.924730 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44c0c10d-2b49-481e-85d8-50d0a3b251a1-maas-api-xks-ca" pod "44c0c10d-2b49-481e-85d8-50d0a3b251a1" (UID: "44c0c10d-2b49-481e-85d8-50d0a3b251a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:01.924872 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:01.924852 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c0c10d-2b49-481e-85d8-50d0a3b251a1-kube-api-access-d5kqn" pod "44c0c10d-2b49-481e-85d8-50d0a3b251a1" (UID: "44c0c10d-2b49-481e-85d8-50d0a3b251a1"). InnerVolumeSpecName "kube-api-access-d5kqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:02.025136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:02.025072 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5kqn\" (UniqueName: \"kubernetes.io/projected/44c0c10d-2b49-481e-85d8-50d0a3b251a1-kube-api-access-d5kqn\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:02.025136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:02.025101 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c0c10d-2b49-481e-85d8-50d0a3b251a1-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:02.493473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:02.493440 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7869e3-kttw5" Nov 03 20:25:02.493649 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:02.493473 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7869e3-kttw5" event={"ID":"44c0c10d-2b49-481e-85d8-50d0a3b251a1","Type":"ContainerDied","Data":"50158d6b5c0a785f53048ef597abdc6592e893d28ff780f3124f2665d4bf8e3c"} Nov 03 20:25:02.493649 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:02.493520 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50158d6b5c0a785f53048ef597abdc6592e893d28ff780f3124f2665d4bf8e3c" Nov 03 20:25:03.029108 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:03.029073 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20c16c95-k4k62"] Nov 03 20:25:03.033931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:03.033905 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20c16c95-k4k62"] Nov 03 20:25:03.964123 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:03.963646 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c717fa3f-98a4-418a-a857-7425412e727f" path="/var/lib/kubelet/pods/c717fa3f-98a4-418a-a857-7425412e727f/volumes" Nov 03 20:25:11.306693 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:25:11.295080 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:25:11.326410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:11.309391 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l"] Nov 03 20:25:11.399690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:11.391892 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l" Nov 03 20:25:11.626949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:11.626916 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx"] Nov 03 20:25:12.471318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:12.471276 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c52bfc6c-d9n4v"] Nov 03 20:25:12.597953 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:12.597461 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c52bfc6c-d9n4v"] Nov 03 20:25:12.865400 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:12.864762 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:25:13.862036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:13.861999 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3548c76a-c800-4837-a85c-aa4ffcc97d61" path="/var/lib/kubelet/pods/3548c76a-c800-4837-a85c-aa4ffcc97d61/volumes" Nov 03 20:25:16.377175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:16.377136 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l" podUID="54ff6643-48a2-4fc3-9446-dd096b73b09a" containerName="payload-processing" containerID="cri-o://103312d7497946c4c859f1b107fdc31bd0618ca0fec73a597c66b6369bb5e019" gracePeriod=25 Nov 03 20:25:16.628208 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:16.627988 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx" podUID="2e50ede5-4e49-4912-9e00-9a0d37ba0614" containerName="payload-processing" containerID="cri-o://67fa604274d25af74fb4ee60ff533af52d0922d1d3635aceaa63a1b4afd51d87" gracePeriod=25 Nov 03 20:25:16.912730 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:16.907866 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l" Nov 03 20:25:17.316886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.189002 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54ff6643-48a2-4fc3-9446-dd096b73b09a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/54ff6643-48a2-4fc3-9446-dd096b73b09a-provider-credentials\") pod \"54ff6643-48a2-4fc3-9446-dd096b73b09a\" (UID: \"54ff6643-48a2-4fc3-9446-dd096b73b09a\") " Nov 03 20:25:17.316886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.189100 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54ff6643-48a2-4fc3-9446-dd096b73b09a-kube-api-access-g4n2h\" (UniqueName: \"kubernetes.io/projected/54ff6643-48a2-4fc3-9446-dd096b73b09a-kube-api-access-g4n2h\") pod \"54ff6643-48a2-4fc3-9446-dd096b73b09a\" (UID: \"54ff6643-48a2-4fc3-9446-dd096b73b09a\") " Nov 03 20:25:17.316886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.189315 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ff6643-48a2-4fc3-9446-dd096b73b09a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54ff6643-48a2-4fc3-9446-dd096b73b09a-plugins-config-volume\") pod \"54ff6643-48a2-4fc3-9446-dd096b73b09a\" (UID: \"54ff6643-48a2-4fc3-9446-dd096b73b09a\") " Nov 03 20:25:17.316886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.189430 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ff6643-48a2-4fc3-9446-dd096b73b09a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54ff6643-48a2-4fc3-9446-dd096b73b09a-routing-overlay-volume\") pod \"54ff6643-48a2-4fc3-9446-dd096b73b09a\" (UID: \"54ff6643-48a2-4fc3-9446-dd096b73b09a\") " Nov 03 20:25:17.316886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.208338 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ff6643-48a2-4fc3-9446-dd096b73b09a-plugins-config-volume" pod "54ff6643-48a2-4fc3-9446-dd096b73b09a" (UID: "54ff6643-48a2-4fc3-9446-dd096b73b09a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:17.316886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.285801 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ff6643-48a2-4fc3-9446-dd096b73b09a-routing-overlay-volume" pod "54ff6643-48a2-4fc3-9446-dd096b73b09a" (UID: "54ff6643-48a2-4fc3-9446-dd096b73b09a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:17.316886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.301516 5059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54ff6643-48a2-4fc3-9446-dd096b73b09a-routing-overlay-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:17.316886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.301556 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54ff6643-48a2-4fc3-9446-dd096b73b09a-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:17.470925 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.469900 5059 generic.go:356] "Generic (PLEG): container finished" podID="54ff6643-48a2-4fc3-9446-dd096b73b09a" containerID="103312d7497946c4c859f1b107fdc31bd0618ca0fec73a597c66b6369bb5e019" exitCode=0 Nov 03 20:25:17.470925 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.470249 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l" event={"ID":"54ff6643-48a2-4fc3-9446-dd096b73b09a","Type":"ContainerDied","Data":"103312d7497946c4c859f1b107fdc31bd0618ca0fec73a597c66b6369bb5e019"} Nov 03 20:25:17.470925 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.470367 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l" event={"ID":"54ff6643-48a2-4fc3-9446-dd096b73b09a","Type":"ContainerDied","Data":"928c55fa72997aa9b8de5c3d7019cd3deea31c22ce8cfa2f29d53d7bb631164e"} Nov 03 20:25:17.470925 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.470414 5059 scope.go:122] "RemoveContainer" containerID="103312d7497946c4c859f1b107fdc31bd0618ca0fec73a597c66b6369bb5e019" Nov 03 20:25:17.472519 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.471907 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l" Nov 03 20:25:17.481752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.481704 5059 generic.go:356] "Generic (PLEG): container finished" podID="2e50ede5-4e49-4912-9e00-9a0d37ba0614" containerID="67fa604274d25af74fb4ee60ff533af52d0922d1d3635aceaa63a1b4afd51d87" exitCode=0 Nov 03 20:25:17.482834 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.482806 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx" event={"ID":"2e50ede5-4e49-4912-9e00-9a0d37ba0614","Type":"ContainerDied","Data":"67fa604274d25af74fb4ee60ff533af52d0922d1d3635aceaa63a1b4afd51d87"} Nov 03 20:25:17.483475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.483427 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ff6643-48a2-4fc3-9446-dd096b73b09a-kube-api-access-g4n2h" pod "54ff6643-48a2-4fc3-9446-dd096b73b09a" (UID: "54ff6643-48a2-4fc3-9446-dd096b73b09a"). InnerVolumeSpecName "kube-api-access-g4n2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:17.506263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.503170 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ff6643-48a2-4fc3-9446-dd096b73b09a-provider-credentials" pod "54ff6643-48a2-4fc3-9446-dd096b73b09a" (UID: "54ff6643-48a2-4fc3-9446-dd096b73b09a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:17.523488 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.523441 5059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54ff6643-48a2-4fc3-9446-dd096b73b09a-provider-credentials\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:17.523488 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.523472 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4n2h\" (UniqueName: \"kubernetes.io/projected/54ff6643-48a2-4fc3-9446-dd096b73b09a-kube-api-access-g4n2h\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:17.530657 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.530624 5059 scope.go:122] "RemoveContainer" containerID="103312d7497946c4c859f1b107fdc31bd0618ca0fec73a597c66b6369bb5e019" Nov 03 20:25:17.531759 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:25:17.531724 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"103312d7497946c4c859f1b107fdc31bd0618ca0fec73a597c66b6369bb5e019\": container with ID starting with 103312d7497946c4c859f1b107fdc31bd0618ca0fec73a597c66b6369bb5e019 not found: ID does not exist" containerID="103312d7497946c4c859f1b107fdc31bd0618ca0fec73a597c66b6369bb5e019" Nov 03 20:25:17.531856 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.531765 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"103312d7497946c4c859f1b107fdc31bd0618ca0fec73a597c66b6369bb5e019"} err="failed to get container status \"103312d7497946c4c859f1b107fdc31bd0618ca0fec73a597c66b6369bb5e019\": rpc error: code = NotFound desc = could not find container \"103312d7497946c4c859f1b107fdc31bd0618ca0fec73a597c66b6369bb5e019\": container with ID starting with 103312d7497946c4c859f1b107fdc31bd0618ca0fec73a597c66b6369bb5e019 not found: ID does not exist" Nov 03 20:25:17.649452 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.649426 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx" Nov 03 20:25:17.692383 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.692350 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w"] Nov 03 20:25:17.693860 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.693834 5059 cpu_manager.go:414] "containerMap: removing container" podUID="44c0c10d-2b49-481e-85d8-50d0a3b251a1" containerName="revoke-keys" Nov 03 20:25:17.693860 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.693856 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c0c10d-2b49-481e-85d8-50d0a3b251a1" containerName="revoke-keys" Nov 03 20:25:17.694074 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.693888 5059 cpu_manager.go:414] "containerMap: removing container" podUID="54ff6643-48a2-4fc3-9446-dd096b73b09a" containerName="payload-processing" Nov 03 20:25:17.694074 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.693894 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ff6643-48a2-4fc3-9446-dd096b73b09a" containerName="payload-processing" Nov 03 20:25:17.694074 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.693919 5059 cpu_manager.go:414] "containerMap: removing container" podUID="2e50ede5-4e49-4912-9e00-9a0d37ba0614" containerName="payload-processing" Nov 03 20:25:17.694074 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.693923 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e50ede5-4e49-4912-9e00-9a0d37ba0614" containerName="payload-processing" Nov 03 20:25:17.694074 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.694049 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="44c0c10d-2b49-481e-85d8-50d0a3b251a1" containerName="revoke-keys" Nov 03 20:25:17.694074 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.694063 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ff6643-48a2-4fc3-9446-dd096b73b09a" containerName="payload-processing" Nov 03 20:25:17.694074 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.694074 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e50ede5-4e49-4912-9e00-9a0d37ba0614" containerName="payload-processing" Nov 03 20:25:17.698027 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.698004 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.702439 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.702413 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d2aa3b-openshift-default-dockercfg-g8c6q\"" Nov 03 20:25:17.706747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.706705 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w"] Nov 03 20:25:17.803107 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.803071 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l"] Nov 03 20:25:17.808167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.808140 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc8hcx9l"] Nov 03 20:25:17.828139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.828113 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e50ede5-4e49-4912-9e00-9a0d37ba0614-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e50ede5-4e49-4912-9e00-9a0d37ba0614-provider-credentials\") pod \"2e50ede5-4e49-4912-9e00-9a0d37ba0614\" (UID: \"2e50ede5-4e49-4912-9e00-9a0d37ba0614\") " Nov 03 20:25:17.828240 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.828187 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e50ede5-4e49-4912-9e00-9a0d37ba0614-kube-api-access-gkt9c\" (UniqueName: \"kubernetes.io/projected/2e50ede5-4e49-4912-9e00-9a0d37ba0614-kube-api-access-gkt9c\") pod \"2e50ede5-4e49-4912-9e00-9a0d37ba0614\" (UID: \"2e50ede5-4e49-4912-9e00-9a0d37ba0614\") " Nov 03 20:25:17.828356 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.828340 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e50ede5-4e49-4912-9e00-9a0d37ba0614-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e50ede5-4e49-4912-9e00-9a0d37ba0614-plugins-config-volume\") pod \"2e50ede5-4e49-4912-9e00-9a0d37ba0614\" (UID: \"2e50ede5-4e49-4912-9e00-9a0d37ba0614\") " Nov 03 20:25:17.828532 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.828511 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e50ede5-4e49-4912-9e00-9a0d37ba0614-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e50ede5-4e49-4912-9e00-9a0d37ba0614-routing-overlay-volume\") pod \"2e50ede5-4e49-4912-9e00-9a0d37ba0614\" (UID: \"2e50ede5-4e49-4912-9e00-9a0d37ba0614\") " Nov 03 20:25:17.828796 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.828767 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e50ede5-4e49-4912-9e00-9a0d37ba0614-plugins-config-volume" pod "2e50ede5-4e49-4912-9e00-9a0d37ba0614" (UID: "2e50ede5-4e49-4912-9e00-9a0d37ba0614"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:17.829631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.828886 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-podinfo\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.829631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.828866 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e50ede5-4e49-4912-9e00-9a0d37ba0614-routing-overlay-volume" pod "2e50ede5-4e49-4912-9e00-9a0d37ba0614" (UID: "2e50ede5-4e49-4912-9e00-9a0d37ba0614"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:17.829631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.829029 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-workload-socket\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.829631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.829053 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czj55\" (UniqueName: \"kubernetes.io/projected/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-kube-api-access-czj55\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.829631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.829122 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-envoy\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.829631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.829243 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-data\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.829631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.829299 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istiod-ca-cert\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.829631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.829319 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-token\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.829631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.829341 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-credential-socket\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.829631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.829500 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-workload-certs\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.832259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.829699 5059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e50ede5-4e49-4912-9e00-9a0d37ba0614-routing-overlay-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:17.832259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.829727 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e50ede5-4e49-4912-9e00-9a0d37ba0614-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:17.832259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.830495 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e50ede5-4e49-4912-9e00-9a0d37ba0614-provider-credentials" pod "2e50ede5-4e49-4912-9e00-9a0d37ba0614" (UID: "2e50ede5-4e49-4912-9e00-9a0d37ba0614"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:17.832259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.830946 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e50ede5-4e49-4912-9e00-9a0d37ba0614-kube-api-access-gkt9c" pod "2e50ede5-4e49-4912-9e00-9a0d37ba0614" (UID: "2e50ede5-4e49-4912-9e00-9a0d37ba0614"). InnerVolumeSpecName "kube-api-access-gkt9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:17.861643 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.861581 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54ff6643-48a2-4fc3-9446-dd096b73b09a" path="/var/lib/kubelet/pods/54ff6643-48a2-4fc3-9446-dd096b73b09a/volumes" Nov 03 20:25:17.931628 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.931594 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-workload-socket\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.931749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.931651 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czj55\" (UniqueName: \"kubernetes.io/projected/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-kube-api-access-czj55\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.931749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.931731 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-envoy\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.931850 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.931838 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-data\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.931897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.931883 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istiod-ca-cert\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.931994 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.931905 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-token\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.931994 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.931937 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-credential-socket\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.932122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.932045 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-workload-certs\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.932122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.932089 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-workload-socket\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.933798 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.932355 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-podinfo\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.933798 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.932490 5059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e50ede5-4e49-4912-9e00-9a0d37ba0614-provider-credentials\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:17.933798 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.932512 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkt9c\" (UniqueName: \"kubernetes.io/projected/2e50ede5-4e49-4912-9e00-9a0d37ba0614-kube-api-access-gkt9c\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:17.933798 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.932771 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-data\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.934185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.934139 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-credential-socket\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.934304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.934216 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-workload-certs\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.934578 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.934553 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istiod-ca-cert\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.935549 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.935522 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-envoy\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.935804 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.935786 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-podinfo\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.944166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.944139 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-token\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:17.945080 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:17.944430 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czj55\" (UniqueName: \"kubernetes.io/projected/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-kube-api-access-czj55\") pod \"e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:18.027420 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:18.027004 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:18.360224 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:25:18.360182 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1731953b_1850_40d1_9ba4_cb08dc4ebc5d.slice/crio-a86c61ebb787ebdf61871133a5429b258ffa023ca9d00cf03137a978d0bb7408 WatchSource:0}: Error finding container a86c61ebb787ebdf61871133a5429b258ffa023ca9d00cf03137a978d0bb7408: Status 404 returned error can't find the container with id a86c61ebb787ebdf61871133a5429b258ffa023ca9d00cf03137a978d0bb7408 Nov 03 20:25:18.361357 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:18.361328 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w"] Nov 03 20:25:18.363363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:18.363316 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:25:18.363464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:18.363393 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:25:18.363464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:18.363423 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:25:18.499781 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:18.499732 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" event={"ID":"1731953b-1850-40d1-9ba4-cb08dc4ebc5d","Type":"ContainerStarted","Data":"073b6545188a48162ae3968a113b770a3a9d7b5615b65392e4809e1551e74cc2"} Nov 03 20:25:18.500190 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:18.499794 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" event={"ID":"1731953b-1850-40d1-9ba4-cb08dc4ebc5d","Type":"ContainerStarted","Data":"a86c61ebb787ebdf61871133a5429b258ffa023ca9d00cf03137a978d0bb7408"} Nov 03 20:25:18.501424 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:18.501395 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx" Nov 03 20:25:18.501591 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:18.501424 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx" event={"ID":"2e50ede5-4e49-4912-9e00-9a0d37ba0614","Type":"ContainerDied","Data":"f6e76b1f4eb653d18779761d5d74f3247407dc1680b4f02f5dc339e76aacda58"} Nov 03 20:25:18.501591 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:18.501490 5059 scope.go:122] "RemoveContainer" containerID="67fa604274d25af74fb4ee60ff533af52d0922d1d3635aceaa63a1b4afd51d87" Nov 03 20:25:18.531345 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:18.531296 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" podStartSLOduration=2.531281964 podStartE2EDuration="2.531281964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:18.523905681 +0000 UTC m=+2912.781702938" watchObservedRunningTime="2026-11-03 20:25:18.531281964 +0000 UTC m=+2912.789079220" Nov 03 20:25:18.553756 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:18.553708 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx"] Nov 03 20:25:18.558560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:18.558538 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-9bb4pd7nx"] Nov 03 20:25:19.028143 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:19.028105 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:19.032397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:19.032356 5059 patch_prober.go:29] interesting pod/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Nov 03 20:25:19.034350 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:19.032429 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" podUID="1731953b-1850-40d1-9ba4-cb08dc4ebc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Nov 03 20:25:19.859244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:19.859208 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e50ede5-4e49-4912-9e00-9a0d37ba0614" path="/var/lib/kubelet/pods/2e50ede5-4e49-4912-9e00-9a0d37ba0614/volumes" Nov 03 20:25:20.034069 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:20.034036 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:20.588682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:20.587215 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:20.595686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:20.590943 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:25:34.025363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:34.025325 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3c252b2-tln54"] Nov 03 20:25:34.030207 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:34.030181 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3c252b2-tln54"] Nov 03 20:25:35.860780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:35.860745 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="197e1bd8-4f7a-4366-b0b9-88bcc9cf515a" path="/var/lib/kubelet/pods/197e1bd8-4f7a-4366-b0b9-88bcc9cf515a/volumes" Nov 03 20:25:40.651892 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:25:40.651853 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:25:58.058229 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:58.058185 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bec3b13-hb84q"] Nov 03 20:25:58.160740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:58.064547 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bec3b13-hb84q"] Nov 03 20:25:59.667062 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:59.667023 5059 scope.go:122] "RemoveContainer" containerID="4047cf868565f81ce559fe77432c44c45df4e1fead0fa2f927393c42875bfb67" Nov 03 20:25:59.861188 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:25:59.861154 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f29306d-863d-484a-80b7-e05e6effbd51" path="/var/lib/kubelet/pods/2f29306d-863d-484a-80b7-e05e6effbd51/volumes" Nov 03 20:26:10.838720 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:26:10.832622 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:26:12.905937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:26:12.905878 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded" Nov 03 20:26:13.717193 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:26:13.442803 5059 patch_prober.go:29] interesting pod/router-default-f7b87d695-cwlfq container/router namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.9:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:26:14.703781 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:26:14.703734 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:14.775095 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:26:14.733171 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:14.820261 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:26:14.820223 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:14.856728 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:26:14.856684 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:15.402604 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:26:15.402559 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:15.453122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:26:15.453038 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:15.484115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:26:15.453884 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-f7b87d695-cwlfq" podUID="603868b0-8628-4227-8471-1894e2e1de50" containerName="router" probeResult="failure" output="Get \"http://10.132.0.9:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:15.484115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:26:15.457594 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:26:15.484115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:26:15.457640 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" podUID="f0d3c590-f523-4b7e-9086-60df5fc585e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:15.615031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:26:15.488477 5059 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-qchj6 container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.132.0.28:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:26:15.615031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:26:15.488521 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qchj6" podUID="ad491ce3-2f92-45a7-9b4f-278309989e64" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.28:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:15.615031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:26:15.588060 5059 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-qchj6 container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.28:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:26:15.615031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:26:15.588122 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qchj6" podUID="ad491ce3-2f92-45a7-9b4f-278309989e64" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.28:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:16.062045 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:26:15.837815 5059 patch_prober.go:29] interesting pod/jobset-controller-manager-5d9f95b69-9vwq7 container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.132.0.44:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:26:16.062045 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:26:15.837869 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" podUID="1072628f-f513-4b4e-8954-afee5a1aa733" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.44:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:16.824022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:26:16.823967 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" podUID="4ea6e69e-3556-4ffe-bb9e-796e8795d747" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.1.58:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:23.840532 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:26:21.651831 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.163050529s: [/var/lib/containers/storage/overlay/ab20ddba4ce467229f22d33db5fafd5d048d7904e2d5f46fdda07463291e410d/diff /var/log/pods/openshift-ingress_e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g_5ea38154-8a1f-4e1f-8fa0-50481ba4fec0/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:26:27.567091 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:26:24.068066 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.23025771s: [/var/lib/containers/storage/overlay/cf20c89a59abe7a607a4c4fc3989a0700c8b52ef6b4b7ec8f4ce59b58a807569/diff /var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5_541bc63e-66f1-42ca-a946-aedc34beb62f/main/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:27:30.173557 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:16.246787 5059 trace.go:236] Trace[1146549800]: "Calculate volume metrics of workload-socket for pod openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" (03-Nov-2026 20:26:52.249) (total time: 23995ms): Nov 03 20:27:30.173557 ip-10-0-132-83 kubenswrapper[5059]: Trace[1146549800]: [23.995147114s] [23.995147114s] END Nov 03 20:27:30.173557 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:16.246814 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.084526912s: [/var/lib/containers/storage/overlay/43201a78b713ec15496bb789b9294c6a0013d5c89dba58b4ae0edd1a0971be62/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2m4mq_c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/kube-rbac-proxy-ovn-metrics/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:27:30.173557 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:16.247868 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 22.530547892s: [/var/lib/containers/storage/overlay/69d000787c06a185865f58066c28dd8c5a083b56b5987b22d50bb9ddd8b80af3/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.804270 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:58.143294 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.948732602s: [/var/lib/containers/storage/overlay/5ca4279d7298c4d616461615afd646ad3eb5ab2d9fc04b7395ee9566a88db63d/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.804270 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:58.143394 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.94884808s: [/var/lib/containers/storage/overlay/7a542b6487238caa6f2b3f7a1a6c7b90dd3b718c7b8d47eb8857c76cccb9f1d1/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.804270 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:58.143432 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.948885724s: [/var/lib/containers/storage/overlay/ff02ed8e0192f2b5f8a09e55aa451c0ec876698c921658cf8618aac18a84ca84/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.804270 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:58.143611 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.949060013s: [/var/lib/containers/storage/overlay/1ab252080a6da0c26d9aed23639cf17237b7e42ba6e207939b96152dca047393/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.804270 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:58.143894 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.949340841s: [/var/lib/containers/storage/overlay/f0f2ca521d042f46a869ff8db4935ca1a58cfef7b1a4ed9b5ed6a4c77999f76c/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.804270 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:58.143939 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.949384656s: [/var/lib/containers/storage/overlay/e3f26c564d54b842c20f87ef8b782f5560957d6752e7297cc52d36a8b3e16d04/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.804270 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:58.143975 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.949419087s: [/var/lib/containers/storage/overlay/f0554608437114cfd30a852e952ef989245f496f382a495e89e41089979bea9e/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.805278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:58.144014 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.949448682s: [/var/lib/containers/storage/overlay/a05ea4c91cdd1eeaa152497f63911dfe58c1907944efac9945b0167bf064ce0d/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.805278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:58.190914 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.591408298s: [/var/lib/containers/storage/overlay/0bcd3ad108cb6e59ca34fbe5d0bdabe1fa2c72489d6b64f202cd98113a84a542/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.805278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.109411 5059 trace.go:236] Trace[340458136]: "Calculate volume metrics of env-overrides for pod openshift-ovn-kubernetes/ovnkube-node-2m4mq" (03-Nov-2026 20:27:56.742) (total time: 2366ms): Nov 03 20:32:50.805278 ip-10-0-132-83 kubenswrapper[5059]: Trace[340458136]: [2.366636681s] [2.366636681s] END Nov 03 20:32:50.805278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.126642 5059 ???:1] "http: TLS handshake error from 10.0.133.249:52205: EOF" Nov 03 20:32:50.805278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.149843 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"agc-cred-5f4fe13bd610\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.805278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.149890 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.805278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.149927 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.805913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.149969 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rwt79\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.805913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.150010 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b2f85\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.805913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.150039 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-0e1546-dockercfg-hwbh2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.805913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.150072 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.805913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.150106 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tx8mk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.805913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.150132 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.806494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.150165 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.806494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.150207 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jzp4r\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.806494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.150261 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.806494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.150303 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.806494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.150333 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a0d4ec-openshift-default-dockercfg-gsslf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.806494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.150365 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.807429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.150410 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.807429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.150442 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.807429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.150467 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.807429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.150494 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.807429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.150544 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-p6m7s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.807429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.150576 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.807986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.290480 5059 trace.go:236] Trace[1106108966]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p" (03-Nov-2026 20:27:56.728) (total time: 1518ms): Nov 03 20:32:50.807986 ip-10-0-132-83 kubenswrapper[5059]: Trace[1106108966]: [1.518147332s] [1.518147332s] END Nov 03 20:32:50.807986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.309325 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.807986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.309386 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-2b5703-dockercfg-b84j2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.807986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.309434 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-81a5ae-openshift-default-dockercfg-pvl2f\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.807986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.309480 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vlmgx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.807986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.309540 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"cacerts\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.808551 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.309578 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.808551 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.309613 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-b22jw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.808551 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.309675 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-8b7351-openshift-default-dockercfg-lx6hb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.808551 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.309713 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ndfpg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.808551 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.309748 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.808551 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.309801 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.809087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.309844 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a0d4ec\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.809087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.309881 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.809087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.309914 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sgdb8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.809087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.309966 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.809087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.310002 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.809087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.310037 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-vx824\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.809572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.310087 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"payload-processing-external-model-e2e--b1878920-dockercfg-vbr6x\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.809572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.310147 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.809572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.310191 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"routing-overlay\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.809572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.310227 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.809572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.310261 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-88a18e-openshift-default-dockercfg-bs4qg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.868230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.310295 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.868230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.310354 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.868230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.310389 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.868230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.310440 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.868230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.310488 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.868230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.310522 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.868742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.310583 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-942930-openshift-default-dockercfg-vwmg4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.868742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.310617 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-43c674-openshift-default-dockercfg-slz8b\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.868742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.310653 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.868742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.310716 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.868742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.310798 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.868742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.310833 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rs44g\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.869024 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.310865 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.869024 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.310898 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.869024 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.310947 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.869024 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.310981 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"default-dockercfg-8tzmp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.869024 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.311021 5059 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.869024 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.311067 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.869289 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.311100 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.869289 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.311143 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.869289 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.311184 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.869289 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.311230 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.869289 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.311282 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.869289 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.311315 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0e4ce7\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.870192 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.311349 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.870192 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.311387 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.870192 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.311427 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.870192 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.311525 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.870192 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.311574 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ldllm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.870192 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.311605 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dgjzc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.870952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.311643 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xv4gr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.870952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.311723 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lhxcg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.870952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.311759 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-33e267-openshift-default-dockercfg-lkhx5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.870952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.311791 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.870952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.311837 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a0d4ec-dockercfg-d47xx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.871255 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.311869 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.871255 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.311913 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.871255 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.311961 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.871255 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.311996 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lrbp4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.871255 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312035 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e7b70d-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.871255 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312070 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.872252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312117 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.872252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312149 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.872252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312179 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.872252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312220 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.872252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312252 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.872252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312284 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-fsb74\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.872496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312318 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.872496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312358 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.872496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312390 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.872496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312425 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.872496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312458 5059 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.872496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312499 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.872742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312537 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.872742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312574 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.872742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312639 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-2b5703-openshift-default-dockercfg-jmjbm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.872742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312698 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.872742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312729 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.872742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312760 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.872961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312806 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.872961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312843 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0e1546\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.872961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312874 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.872961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312910 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-32e482-openshift-default-dockercfg-5pv2v\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.872961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312939 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.873190 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.312983 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-bcfde8-openshift-default-dockercfg-6xbwb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.873190 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.313022 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fw8qp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.873190 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.313075 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.873190 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.313123 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-32e482-dockercfg-4mbnz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.873190 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.313158 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-3fbb1a-dockercfg-b6ghv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.873379 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.313194 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0e4ce7-openshift-default-dockercfg-nppzh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.873379 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.313240 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2r65v\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.873379 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.313276 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-h4l58\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.873379 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.313311 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"signing-key\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.873379 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.313435 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.873379 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.313471 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-32e482\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.873605 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.313504 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-3fbb1a\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.873605 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.313536 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.873605 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.313567 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.873605 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.313598 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.873605 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.313628 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.873605 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.313953 5059 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.875353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.313988 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-9f46e5\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.875353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314021 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.875353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314051 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.875353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314084 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-46a6d7-openshift-default-dockercfg-5jblx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.875353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314114 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.875353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314151 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.875583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314189 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.875583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314219 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.875583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314250 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.875583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314280 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e7b70d-openshift-default-dockercfg-jjqkc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.875583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314313 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.875583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314343 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-nh6bx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.875842 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314376 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.875842 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314420 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-k5vf6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.875842 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314451 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.875842 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314482 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.875842 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314511 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-847xh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.875842 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314542 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314575 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314607 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"default-dockercfg-5r8nn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314642 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"simulator-w3-9f46e5-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314689 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xr7jf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314721 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"default-dockercfg-zrxnc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876264 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314751 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-lw7w7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876264 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314782 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876264 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314825 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876264 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314858 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876264 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.314997 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876264 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315027 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315059 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315091 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5qm8x\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315122 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315153 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315203 5059 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315234 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876730 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315278 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-b503a8-openshift-default-dockercfg-tchtj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876730 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315309 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-eb69cb-openshift-default-dockercfg-w96q5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876730 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315342 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qhpjg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876730 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315373 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-98529d-openshift-default-dockercfg-l472v\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876730 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315404 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3fbb1a-openshift-default-dockercfg-247ct\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315434 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"downloads-dockercfg-wftlv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315470 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xbkg2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315500 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4p6h4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315529 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315563 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.876923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315594 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.877134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315629 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.877134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315693 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.877134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315780 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-660d95-openshift-default-dockercfg-df2mh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.877134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315811 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.877134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315841 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.877134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315871 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.877338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315903 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.877338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315936 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.877338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315965 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-98529d\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.877338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.315996 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.877338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.316026 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.877539 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.316059 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-bfc240-openshift-default-dockercfg-dxqxq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.877539 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.316090 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d2aa3b-openshift-default-dockercfg-g8c6q\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.877539 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.316145 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-942q2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.877539 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.316185 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"default-dockercfg-lxnx2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.877539 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.316242 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.877539 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.316277 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.878413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.316312 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.878413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.316348 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-9f46e5-dockercfg-6g47t\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.878413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.316382 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.878413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.316417 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-dq8cl\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.878413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.316536 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"distinct-w2-2b5703-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.878614 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.316575 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"default-dockercfg-4rsdq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.878614 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.316612 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d3d0df-openshift-default-dockercfg-m4lvp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.878614 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.316648 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-65hnm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.878614 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.316700 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vf7n8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.878614 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.316735 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.879637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.316775 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0e1546-openshift-default-dockercfg-jcbdc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.879637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.316813 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"simulator-w4-3fbb1a-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.879637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.316849 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-4c2a28-dockercfg-psjnh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.879637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.316885 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.879637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.316951 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.879637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.316986 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.879894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.317026 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.879894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.317064 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-4c2a28-openshift-default-dockercfg-vhjbn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.879894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.317121 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-rb4sg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.879894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.317158 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.879894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.317193 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"simulator-w1-32e482-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.880172 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.317227 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.880172 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:27:59.150602 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bkd9f\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.880172 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:00.729738 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-f154ac-openshift-default-dockercfg-xndgl\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.880172 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:00.744409 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-4c2a28\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.880172 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:00.744489 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.880384 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:00.759279 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-9f46e5-openshift-default-dockercfg-4djl8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.880384 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:00.865154 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qg9jj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.880384 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:00.865358 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.880384 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:00.865422 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-w84zh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.880384 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:00.865557 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bfvcm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.880384 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:00.900501 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 44.699632762s: [/var/lib/containers/storage/overlay/78131cdf25b8ea22b8b4e2ecfba14b69aed6fec9e7be52667fc8d84c2d1c1f4f/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.880613 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:00.900755 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m32.946936866s: [/var/lib/containers/storage/overlay/79c575ea534c7b0bb095fa21c0a8cd9709d073b5c95f1b6f855db1b7fc78a8ef/diff /var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-nj52h_941b16dd-0550-4434-86f9-26e38efb68ee/manager/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.880613 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:00.901212 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 44.693120863s: [/var/lib/containers/storage/overlay/8194d450ef80314a478a977ccfb3a412ac089da2624c6da21e565bb18c732c4c/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.880613 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:28:00.902052 5059 controller.go:251] "Failed to update lease" err="Put \"https://acb016a5e504c4ca985d18876c4d6627-9b035d8de54f8ea2.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-83.us-east-2.compute.internal?timeout=10s\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:50.880613 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:00.938193 5059 trace.go:236] Trace[271734497]: "Calculate volume metrics of ovnkube-script-lib for pod openshift-ovn-kubernetes/ovnkube-node-2m4mq" (03-Nov-2026 20:27:59.441) (total time: 1497ms): Nov 03 20:32:50.880613 ip-10-0-132-83 kubenswrapper[5059]: Trace[271734497]: [1.497107602s] [1.497107602s] END Nov 03 20:32:50.880613 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.425172 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 20:32:50.880613 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.425271 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 20:32:50.880613 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.425389 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 20:32:50.880613 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.425490 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-0e1546-dockercfg-hwbh2\"" Nov 03 20:32:50.880963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.425576 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"kube-root-ca.crt\"" Nov 03 20:32:50.880963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.425684 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-vx824\"" Nov 03 20:32:50.880963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.425778 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 20:32:50.880963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.425866 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:32:50.880963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.425961 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 20:32:50.880963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.426050 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-fsb74\"" Nov 03 20:32:50.880963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.426143 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 20:32:50.880963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.426255 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 20:32:50.880963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.426339 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 20:32:50.880963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.426431 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 03 20:32:50.880963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.426530 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-98529d-openshift-default-dockercfg-l472v\"" Nov 03 20:32:50.880963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.426596 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 20:32:50.882057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.426694 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"kube-root-ca.crt\"" Nov 03 20:32:50.882057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.426765 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 20:32:50.882057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.426849 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 20:32:50.882057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.426912 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 20:32:50.882057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.427006 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 20:32:50.882057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.429755 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 20:32:50.882057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.429833 5059 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:32:50.882057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.429918 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vlmgx\"" Nov 03 20:32:50.882057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.429979 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 20:32:50.882057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.430070 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 20:32:50.882057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.430165 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-p6m7s\"" Nov 03 20:32:50.882057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.430229 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"agc-cred-5f4fe13bd610\"" Nov 03 20:32:50.882600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.430327 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a0d4ec-openshift-default-dockercfg-gsslf\"" Nov 03 20:32:50.882600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.430410 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b2f85\"" Nov 03 20:32:50.882600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.430476 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 20:32:50.882600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.430535 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 20:32:50.882600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.430611 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jzp4r\"" Nov 03 20:32:50.882600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.430707 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 20:32:50.882600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.430793 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rwt79\"" Nov 03 20:32:50.882600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.430873 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tx8mk\"" Nov 03 20:32:50.882600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.430945 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 20:32:50.882600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.433981 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 20:32:50.882600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.434056 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 20:32:50.882600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.434239 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"simulator-w4-3fbb1a-kserve-self-signed-certs\"" Nov 03 20:32:50.883242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.434301 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-eb69cb-openshift-default-dockercfg-w96q5\"" Nov 03 20:32:50.883242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.434374 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 20:32:50.883242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.434463 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 20:32:50.883242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.434544 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 20:32:50.883242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.434613 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 20:32:50.883242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.434744 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-h4l58\"" Nov 03 20:32:50.883242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.434863 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dgjzc\"" Nov 03 20:32:50.883242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.434986 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-b22jw\"" Nov 03 20:32:50.883242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.435084 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d3d0df-openshift-default-dockercfg-m4lvp\"" Nov 03 20:32:50.883242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.435171 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-2b5703-openshift-default-dockercfg-jmjbm\"" Nov 03 20:32:50.883242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.435300 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 20:32:50.883913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.435357 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"openshift-service-ca.crt\"" Nov 03 20:32:50.883913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.435434 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lhxcg\"" Nov 03 20:32:50.883913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.447407 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-4c2a28-openshift-default-dockercfg-vhjbn\"" Nov 03 20:32:50.883913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.448121 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 20:32:50.883913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.448221 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 20:32:50.883913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.448342 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 20:32:50.883913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.448445 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 20:32:50.883913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.448515 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 20:32:50.883913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.448609 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-9f46e5\"" Nov 03 20:32:50.883913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.448730 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 20:32:50.883913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.448804 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 20:32:50.883913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.448958 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"openshift-service-ca.crt\"" Nov 03 20:32:50.885938 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.449026 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 20:32:50.885938 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:02.030465 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:50.885938 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:02.030730 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 45.807217102s: [/var/lib/containers/storage/overlay/f339f937f6b2246ad7339663ba783e2ca19158b7afa66ca30c7cc59cade7ae73/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.885938 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:02.249970 5059 trace.go:236] Trace[223615938]: "Calculate volume metrics of kube-api-access-hdbbr for pod openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" (03-Nov-2026 20:27:17.391) (total time: 44856ms): Nov 03 20:32:50.885938 ip-10-0-132-83 kubenswrapper[5059]: Trace[223615938]: [44.856138867s] [44.856138867s] END Nov 03 20:32:50.885938 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.678756 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5qm8x\": awaiting required bookmark event for initial events stream, no events received for 43.928995216s" Nov 03 20:32:50.885938 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.678831 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5qm8x\": awaiting required bookmark event for initial events stream, no events received for 43.929120124s" Nov 03 20:32:50.885938 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.678871 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"default-dockercfg-zrxnc\": awaiting required bookmark event for initial events stream, no events received for 43.92872085s" Nov 03 20:32:50.886278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.678884 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"default-dockercfg-zrxnc\": awaiting required bookmark event for initial events stream, no events received for 43.928755325s" Nov 03 20:32:50.886278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.678897 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-k5vf6\": awaiting required bookmark event for initial events stream, no events received for 43.928520406s" Nov 03 20:32:50.886278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.678923 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-k5vf6\": awaiting required bookmark event for initial events stream, no events received for 43.928546988s" Nov 03 20:32:50.886278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.678935 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"e2e-shared-b-w0-81a5ae-openshift-default-dockercfg-pvl2f\": awaiting required bookmark event for initial events stream, no events received for 43.928129006s" Nov 03 20:32:50.886278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.678946 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"e2e-shared-b-w0-81a5ae-openshift-default-dockercfg-pvl2f\": awaiting required bookmark event for initial events stream, no events received for 43.928139774s" Nov 03 20:32:50.886278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.678958 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-image-registry\"/\"registry-dockercfg-vf7n8\": awaiting required bookmark event for initial events stream, no events received for 43.927747988s" Nov 03 20:32:50.886278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.678969 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-image-registry\"/\"registry-dockercfg-vf7n8\": awaiting required bookmark event for initial events stream, no events received for 43.927759691s" Nov 03 20:32:50.886845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.678980 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-multus\"/\"metrics-daemon-secret\": awaiting required bookmark event for initial events stream, no events received for 43.926537451s" Nov 03 20:32:50.886845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.678990 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-multus\"/\"metrics-daemon-secret\": awaiting required bookmark event for initial events stream, no events received for 43.926547464s" Nov 03 20:32:50.886845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679001 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-monitoring\"/\"kube-root-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 43.926279209s" Nov 03 20:32:50.886845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679010 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-monitoring\"/\"kube-root-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 43.926288603s" Nov 03 20:32:50.886845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679020 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-rb4sg\": awaiting required bookmark event for initial events stream, no events received for 43.925892155s" Nov 03 20:32:50.886845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679031 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-rb4sg\": awaiting required bookmark event for initial events stream, no events received for 43.925902911s" Nov 03 20:32:50.886845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679042 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\": awaiting required bookmark event for initial events stream, no events received for 43.925536319s" Nov 03 20:32:50.886845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679054 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\": awaiting required bookmark event for initial events stream, no events received for 43.925548079s" Nov 03 20:32:50.889032 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679065 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-dns\"/\"node-resolver-dockercfg-ndfpg\": awaiting required bookmark event for initial events stream, no events received for 43.92529414s" Nov 03 20:32:50.889032 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679074 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-dns\"/\"node-resolver-dockercfg-ndfpg\": awaiting required bookmark event for initial events stream, no events received for 43.925303179s" Nov 03 20:32:50.889032 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679085 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"e2e-worker-w1-32e482-openshift-default-dockercfg-5pv2v\": awaiting required bookmark event for initial events stream, no events received for 43.924899666s" Nov 03 20:32:50.889032 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679100 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"e2e-worker-w1-32e482-openshift-default-dockercfg-5pv2v\": awaiting required bookmark event for initial events stream, no events received for 43.924914435s" Nov 03 20:32:50.889032 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679112 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-console\"/\"downloads-dockercfg-wftlv\": awaiting required bookmark event for initial events stream, no events received for 43.924520699s" Nov 03 20:32:50.889032 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679123 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-console\"/\"downloads-dockercfg-wftlv\": awaiting required bookmark event for initial events stream, no events received for 43.924531689s" Nov 03 20:32:50.889032 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679134 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-network-operator\"/\"iptables-alerter-script\": awaiting required bookmark event for initial events stream, no events received for 43.924157455s" Nov 03 20:32:50.889032 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679144 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-network-operator\"/\"iptables-alerter-script\": awaiting required bookmark event for initial events stream, no events received for 43.924166845s" Nov 03 20:32:50.889498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679156 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"e2e-worker-w4-3fbb1a-openshift-default-dockercfg-247ct\": awaiting required bookmark event for initial events stream, no events received for 43.923768143s" Nov 03 20:32:50.889498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679168 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"e2e-worker-w4-3fbb1a-openshift-default-dockercfg-247ct\": awaiting required bookmark event for initial events stream, no events received for 43.923781548s" Nov 03 20:32:50.889498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679183 5059 reflector.go:1159] "Warning: event bookmark expired" err="k8s.io/client-go/informers/factory.go:161: awaiting required bookmark event for initial events stream, no events received for 43.923382343s" Nov 03 20:32:50.889498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679194 5059 reflector.go:1159] "Warning: event bookmark expired" err="k8s.io/client-go/informers/factory.go:161: awaiting required bookmark event for initial events stream, no events received for 43.923393212s" Nov 03 20:32:50.889498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679205 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 43.923026771s" Nov 03 20:32:50.889498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679214 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 43.923036428s" Nov 03 20:32:50.889498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679229 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"openshift-service-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 43.922621385s" Nov 03 20:32:50.890057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679240 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"openshift-service-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 43.922633678s" Nov 03 20:32:50.890057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679252 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-monitoring\"/\"metrics-client-ca\": awaiting required bookmark event for initial events stream, no events received for 43.922226305s" Nov 03 20:32:50.890057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679264 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-monitoring\"/\"metrics-client-ca\": awaiting required bookmark event for initial events stream, no events received for 43.922239025s" Nov 03 20:32:50.890057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679275 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-insights\"/\"openshift-insights-serving-cert\": awaiting required bookmark event for initial events stream, no events received for 43.921870768s" Nov 03 20:32:50.890057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679285 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-insights\"/\"openshift-insights-serving-cert\": awaiting required bookmark event for initial events stream, no events received for 43.921880561s" Nov 03 20:32:50.890057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679297 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"e2e-extmod-a-8b7351-openshift-default-dockercfg-lx6hb\": awaiting required bookmark event for initial events stream, no events received for 43.921489416s" Nov 03 20:32:50.890057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679306 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"e2e-extmod-a-8b7351-openshift-default-dockercfg-lx6hb\": awaiting required bookmark event for initial events stream, no events received for 43.921499223s" Nov 03 20:32:50.890057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679317 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-console\"/\"openshift-service-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 43.921129447s" Nov 03 20:32:50.890402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679327 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-console\"/\"openshift-service-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 43.921139397s" Nov 03 20:32:50.890402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679338 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-insights\"/\"kube-rbac-proxy\": awaiting required bookmark event for initial events stream, no events received for 43.920753023s" Nov 03 20:32:50.890402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679347 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-insights\"/\"kube-rbac-proxy\": awaiting required bookmark event for initial events stream, no events received for 43.920762771s" Nov 03 20:32:50.890402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679357 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-multus\"/\"whereabouts-flatfile-config\": awaiting required bookmark event for initial events stream, no events received for 43.919859102s" Nov 03 20:32:50.890402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679366 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-multus\"/\"whereabouts-flatfile-config\": awaiting required bookmark event for initial events stream, no events received for 43.919867528s" Nov 03 20:32:50.890402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679375 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-dns\"/\"dns-default\": awaiting required bookmark event for initial events stream, no events received for 43.919286796s" Nov 03 20:32:50.890402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679385 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-dns\"/\"dns-default\": awaiting required bookmark event for initial events stream, no events received for 43.919295662s" Nov 03 20:32:50.890402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679395 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"llm\"/\"default-dockercfg-8tzmp\": awaiting required bookmark event for initial events stream, no events received for 43.918795107s" Nov 03 20:32:50.890742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679404 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"llm\"/\"default-dockercfg-8tzmp\": awaiting required bookmark event for initial events stream, no events received for 43.918804294s" Nov 03 20:32:50.890742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679415 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"e2e-extmod-a-33e267-openshift-default-dockercfg-lkhx5\": awaiting required bookmark event for initial events stream, no events received for 43.918266381s" Nov 03 20:32:50.890742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679424 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"e2e-extmod-a-33e267-openshift-default-dockercfg-lkhx5\": awaiting required bookmark event for initial events stream, no events received for 43.918276846s" Nov 03 20:32:50.890742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679437 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-942q2\": awaiting required bookmark event for initial events stream, no events received for 43.917836223s" Nov 03 20:32:50.890742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679447 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-942q2\": awaiting required bookmark event for initial events stream, no events received for 43.917845925s" Nov 03 20:32:50.890742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679458 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-98529d\": awaiting required bookmark event for initial events stream, no events received for 43.917361122s" Nov 03 20:32:50.890742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679467 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-98529d\": awaiting required bookmark event for initial events stream, no events received for 43.917371333s" Nov 03 20:32:50.891015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679479 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 43.916851917s" Nov 03 20:32:50.891015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679490 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 43.916861644s" Nov 03 20:32:50.891015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679501 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sgdb8\": awaiting required bookmark event for initial events stream, no events received for 43.916597346s" Nov 03 20:32:50.891015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679513 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sgdb8\": awaiting required bookmark event for initial events stream, no events received for 43.916609168s" Nov 03 20:32:50.891015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679524 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"kube-system\"/\"kube-root-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 43.91611834s" Nov 03 20:32:50.891015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679533 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"kube-system\"/\"kube-root-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 43.916127122s" Nov 03 20:32:50.891015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679544 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"cacerts\": awaiting required bookmark event for initial events stream, no events received for 43.915571699s" Nov 03 20:32:50.891015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679553 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"cacerts\": awaiting required bookmark event for initial events stream, no events received for 43.91558161s" Nov 03 20:32:50.891318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679563 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-3fbb1a-dockercfg-b6ghv\": awaiting required bookmark event for initial events stream, no events received for 43.915092971s" Nov 03 20:32:50.891318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679573 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-3fbb1a-dockercfg-b6ghv\": awaiting required bookmark event for initial events stream, no events received for 43.915103363s" Nov 03 20:32:50.891318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679585 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 43.914600556s" Nov 03 20:32:50.891318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.679595 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 43.914610283s" Nov 03 20:32:50.891318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.874106 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-jobset-operator\"/\"jobset-manager-config\": awaiting required bookmark event for initial events stream, no events received for 44.10868814s" Nov 03 20:32:50.891318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.874139 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-jobset-operator\"/\"jobset-manager-config\": awaiting required bookmark event for initial events stream, no events received for 44.108732555s" Nov 03 20:32:50.891318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.874161 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ovn-kubernetes\"/\"env-overrides\": awaiting required bookmark event for initial events stream, no events received for 44.108213594s" Nov 03 20:32:50.891318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.874180 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ovn-kubernetes\"/\"env-overrides\": awaiting required bookmark event for initial events stream, no events received for 44.108240459s" Nov 03 20:32:50.891618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.874192 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"kuadrant-system\"/\"openshift-service-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 44.107785279s" Nov 03 20:32:50.891618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.874201 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"kuadrant-system\"/\"openshift-service-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 44.107794628s" Nov 03 20:32:50.891618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.874214 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"e2e-models-e2e-worker-w1-32e482\"/\"kube-root-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 44.107322024s" Nov 03 20:32:50.891618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.874222 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"e2e-models-e2e-worker-w1-32e482\"/\"kube-root-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 44.107331292s" Nov 03 20:32:50.891618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.874237 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"istiod-tls\": awaiting required bookmark event for initial events stream, no events received for 44.106964046s" Nov 03 20:32:50.891618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.874251 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"istiod-tls\": awaiting required bookmark event for initial events stream, no events received for 44.106979215s" Nov 03 20:32:50.891618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.874262 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"router-dockercfg-xr7jf\": awaiting required bookmark event for initial events stream, no events received for 44.106550679s" Nov 03 20:32:50.891618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.874271 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"router-dockercfg-xr7jf\": awaiting required bookmark event for initial events stream, no events received for 44.106560217s" Nov 03 20:32:50.892278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.874281 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"kube-root-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 44.106097662s" Nov 03 20:32:50.892278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.874289 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"kube-root-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 44.106106509s" Nov 03 20:32:50.892278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.874300 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"default-dockercfg-lxnx2\": awaiting required bookmark event for initial events stream, no events received for 44.10537742s" Nov 03 20:32:50.892278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.874315 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"default-dockercfg-lxnx2\": awaiting required bookmark event for initial events stream, no events received for 44.105392098s" Nov 03 20:32:50.892278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.874328 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 44.105045225s" Nov 03 20:32:50.892278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:44.874338 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 44.105055343s" Nov 03 20:32:50.892278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:47.470920 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xv4gr\": awaiting required bookmark event for initial events stream, no events received for 46.722533867s" Nov 03 20:32:50.892537 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:47.470957 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xv4gr\": awaiting required bookmark event for initial events stream, no events received for 46.722586769s" Nov 03 20:32:50.892537 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:47.470969 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-network-operator\"/\"kube-root-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 41.51684153s" Nov 03 20:32:50.892537 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:47.470977 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-network-operator\"/\"kube-root-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 46.72237576s" Nov 03 20:32:50.892537 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:47.470990 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fw8qp\": awaiting required bookmark event for initial events stream, no events received for 41.701152884s" Nov 03 20:32:50.892537 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:47.470997 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fw8qp\": awaiting required bookmark event for initial events stream, no events received for 46.722019304s" Nov 03 20:32:50.892537 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:47.479175 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-2b5703-dockercfg-b84j2\": awaiting required bookmark event for initial events stream, no events received for 41.955676242s" Nov 03 20:32:50.892537 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:47.479200 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-2b5703-dockercfg-b84j2\": awaiting required bookmark event for initial events stream, no events received for 46.72988748s" Nov 03 20:32:50.892915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:46.226335 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m29.993815348s: [/var/lib/containers/storage/overlay/961637059afbf81633d4eba2dfb534ab9a290d37b78ad8d5526994d2ad6c0348/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.892915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:46.239067 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m30.00655723s: [/var/lib/containers/storage/overlay/e18fb873dfe538f424b3c44816bc601922d0b829179e2579487f6c834acdb0af/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.892915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:46.239105 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m30.00659852s: [/var/lib/containers/storage/overlay/8e00d3fbc10cb977be544c47c232e9871e548417e5eb5a3e5c48dc1cbde46a01/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.892915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:46.239136 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m30.006625754s: [/var/lib/containers/storage/overlay/036ad7274ad1232ed107d37abc90a18d84e2c58366eba450be383b1c7539919b/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.892915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:46.239166 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m30.00665728s: [/var/lib/containers/storage/overlay/d91a2eef7704086128be09e4b79681dd76ac562bcbb98517cf669c2b35c77693/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.892915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:46.239193 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m30.006684373s: [/var/lib/containers/storage/overlay/90917c44b902559da175c71d7fd1c2bf17a72658a6a077025bc4f6e26f5f0b71/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.893245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:46.239222 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m30.006705938s: [/var/lib/containers/storage/overlay/e426a31bf0b7a61b98bd48d03f31dcee6f91725a89c8e93541fbe6c276f22bbd/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.893245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:46.239250 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m30.006730333s: [/var/lib/containers/storage/overlay/0edfa7bdd1c0bb6e171a437f6bce6c93a31b9489e465414ee1c1d9d1a8585a8a/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.893245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:46.239400 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m30.006876722s: [/var/lib/containers/storage/overlay/6108362e423e122429ec08608da59a2f40f00273b2584dca92bd8a7570448b0c/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.893245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:46.239429 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m30.006902997s: [/var/lib/containers/storage/overlay/cc33b7715971fc9b11e0f8ee7fff767603d70576b6ba7c1fde7231b29d5e89b6/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.893245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:46.239456 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m30.006929043s: [/var/lib/containers/storage/overlay/d9cdf270537c2f1686a905fce82a567a5543e02a3c3f34d533083daf08be4559/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.893245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:46.239756 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m30.007223408s: [/var/lib/containers/storage/overlay/9b844bd54d6bb3ae9b0c6fa4199967987d71c6c3891e99f9455c5977ec052ffc/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.893622 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:46.239785 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m30.007251496s: [/var/lib/containers/storage/overlay/6bc0f7827722b17cd75299041f34b5e43ac68089229c28450eb493c786a5bc0b/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.893622 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:47.479987 5059 trace.go:236] Trace[1181159522]: "Calculate volume metrics of local-certs for pod openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" (03-Nov-2026 20:28:44.678) (total time: 2801ms): Nov 03 20:32:50.893622 ip-10-0-132-83 kubenswrapper[5059]: Trace[1181159522]: [2.80171563s] [2.80171563s] END Nov 03 20:32:50.893622 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:47.483813 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m30.270846148s: [/var/lib/containers/storage/overlay/2a43d0fb23e55b728f436ffbb8ff2f5b9316bfa0129b85eab729dcbb72fc0152/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:50.893622 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:47.484747 5059 trace.go:236] Trace[1070326365]: "Calculate volume metrics of istio-data for pod openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" (03-Nov-2026 20:28:46.171) (total time: 1313ms): Nov 03 20:32:50.893622 ip-10-0-132-83 kubenswrapper[5059]: Trace[1070326365]: [1.313102132s] [1.313102132s] END Nov 03 20:32:50.893622 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:47.485059 5059 trace.go:236] Trace[1908385608]: "Calculate volume metrics of home for pod e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" (03-Nov-2026 20:28:46.215) (total time: 1269ms): Nov 03 20:32:50.893622 ip-10-0-132-83 kubenswrapper[5059]: Trace[1908385608]: [1.269771006s] [1.269771006s] END Nov 03 20:32:50.893622 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:47.505645 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-9f46e5-dockercfg-6g47t\": awaiting required bookmark event for initial events stream, no events received for 46.758413443s" Nov 03 20:32:50.893622 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:47.505683 5059 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-9f46e5-dockercfg-6g47t\": awaiting required bookmark event for initial events stream, no events received for 46.758459596s" Nov 03 20:32:50.894121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:47.581121 5059 ???:1] "http: TLS handshake error from 10.0.133.249:56344: EOF" Nov 03 20:32:50.894121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:48.467775 5059 ???:1] "http: TLS handshake error from 10.0.133.249:40673: EOF" Nov 03 20:32:50.894121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.359735 5059 trace.go:236] Trace[2046600140]: "Reflector WatchList" name:object-"openshift-ingress"/"istiod-tls" (03-Nov-2026 20:28:00.767) (total time: 48587ms): Nov 03 20:32:50.894121 ip-10-0-132-83 kubenswrapper[5059]: Trace[2046600140]: ---"Objects streamed" count:0 48587ms (20:28:49.354) Nov 03 20:32:50.894121 ip-10-0-132-83 kubenswrapper[5059]: Trace[2046600140]: [48.58757648s] [48.58757648s] END Nov 03 20:32:50.894121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.359774 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 20:32:50.894121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.382473 5059 trace.go:236] Trace[862315324]: "Reflector WatchList" name:object-"e2e-models-e2e-worker-w1-32e482"/"kube-root-ca.crt" (03-Nov-2026 20:28:00.766) (total time: 48615ms): Nov 03 20:32:50.894121 ip-10-0-132-83 kubenswrapper[5059]: Trace[862315324]: ---"Objects streamed" count:1 48615ms (20:28:49.382) Nov 03 20:32:50.894121 ip-10-0-132-83 kubenswrapper[5059]: Trace[862315324]: [48.615545233s] [48.615545233s] END Nov 03 20:32:50.894121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.382506 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"kube-root-ca.crt\"" Nov 03 20:32:50.894121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.487349 5059 trace.go:236] Trace[1222662858]: "Reflector WatchList" name:object-"openshift-ingress"/"payload-processing-e2e-worker-w3-9f46e5-dockercfg-6g47t" (03-Nov-2026 20:28:00.747) (total time: 48740ms): Nov 03 20:32:50.894121 ip-10-0-132-83 kubenswrapper[5059]: Trace[1222662858]: ---"Objects streamed" count:1 48740ms (20:28:49.487) Nov 03 20:32:50.894121 ip-10-0-132-83 kubenswrapper[5059]: Trace[1222662858]: [48.740092764s] [48.740092764s] END Nov 03 20:32:50.894121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.487375 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-9f46e5-dockercfg-6g47t\"" Nov 03 20:32:50.894121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.516839 5059 trace.go:236] Trace[1364960426]: "Reflector WatchList" name:object-"openshift-ingress"/"openshift-ai-inference-openshift-ai-inference-dockercfg-rb4sg" (03-Nov-2026 20:28:00.753) (total time: 48763ms): Nov 03 20:32:50.894121 ip-10-0-132-83 kubenswrapper[5059]: Trace[1364960426]: ---"Objects streamed" count:1 48763ms (20:28:49.516) Nov 03 20:32:50.894121 ip-10-0-132-83 kubenswrapper[5059]: Trace[1364960426]: [48.763682213s] [48.763682213s] END Nov 03 20:32:50.894828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.516864 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-rb4sg\"" Nov 03 20:32:50.894828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.516927 5059 trace.go:236] Trace[1666537367]: "Reflector WatchList" name:object-"openshift-cluster-csi-drivers"/"aws-ebs-csi-driver-node-sa-dockercfg-xv4gr" (03-Nov-2026 20:28:00.748) (total time: 48768ms): Nov 03 20:32:50.894828 ip-10-0-132-83 kubenswrapper[5059]: Trace[1666537367]: ---"Objects streamed" count:1 48768ms (20:28:49.516) Nov 03 20:32:50.894828 ip-10-0-132-83 kubenswrapper[5059]: Trace[1666537367]: [48.768548915s] [48.768548915s] END Nov 03 20:32:50.894828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.516936 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xv4gr\"" Nov 03 20:32:50.894828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.517210 5059 trace.go:236] Trace[1508632251]: "Reflector WatchList" name:object-"kube-system"/"kube-root-ca.crt" (03-Nov-2026 20:28:00.763) (total time: 48753ms): Nov 03 20:32:50.894828 ip-10-0-132-83 kubenswrapper[5059]: Trace[1508632251]: ---"Objects streamed" count:1 48753ms (20:28:49.517) Nov 03 20:32:50.894828 ip-10-0-132-83 kubenswrapper[5059]: Trace[1508632251]: [48.753792886s] [48.753792886s] END Nov 03 20:32:50.894828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.517222 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 20:32:50.894828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.517315 5059 trace.go:236] Trace[665186741]: "Reflector WatchList" name:object-"odh-ai-gateway-infra"/"maas-api-dockercfg-942q2" (03-Nov-2026 20:28:00.761) (total time: 48755ms): Nov 03 20:32:50.894828 ip-10-0-132-83 kubenswrapper[5059]: Trace[665186741]: ---"Objects streamed" count:1 48755ms (20:28:49.517) Nov 03 20:32:50.894828 ip-10-0-132-83 kubenswrapper[5059]: Trace[665186741]: [48.755710084s] [48.755710084s] END Nov 03 20:32:50.894828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.517323 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-942q2\"" Nov 03 20:32:50.894828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.517381 5059 trace.go:236] Trace[200411439]: "Reflector WatchList" name:object-"openshift-network-operator"/"kube-root-ca.crt" (03-Nov-2026 20:28:00.748) (total time: 48768ms): Nov 03 20:32:50.894828 ip-10-0-132-83 kubenswrapper[5059]: Trace[200411439]: ---"Objects streamed" count:1 48768ms (20:28:49.517) Nov 03 20:32:50.894828 ip-10-0-132-83 kubenswrapper[5059]: Trace[200411439]: [48.768772854s] [48.768772854s] END Nov 03 20:32:50.895509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.517390 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 20:32:50.895509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.517468 5059 trace.go:236] Trace[1428336296]: "Reflector WatchList" name:object-"openshift-console-operator"/"console-operator-dockercfg-fw8qp" (03-Nov-2026 20:28:00.748) (total time: 48768ms): Nov 03 20:32:50.895509 ip-10-0-132-83 kubenswrapper[5059]: Trace[1428336296]: ---"Objects streamed" count:1 48768ms (20:28:49.517) Nov 03 20:32:50.895509 ip-10-0-132-83 kubenswrapper[5059]: Trace[1428336296]: [48.76847971s] [48.76847971s] END Nov 03 20:32:50.895509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.517476 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fw8qp\"" Nov 03 20:32:50.895509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.517554 5059 trace.go:236] Trace[1454749930]: "Reflector WatchList" name:object-"openshift-ingress"/"payload-processing-e2e-worker-w2-2b5703-dockercfg-b84j2" (03-Nov-2026 20:28:00.749) (total time: 48768ms): Nov 03 20:32:50.895509 ip-10-0-132-83 kubenswrapper[5059]: Trace[1454749930]: ---"Objects streamed" count:1 48768ms (20:28:49.517) Nov 03 20:32:50.895509 ip-10-0-132-83 kubenswrapper[5059]: Trace[1454749930]: [48.768237184s] [48.768237184s] END Nov 03 20:32:50.895509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.517562 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-2b5703-dockercfg-b84j2\"" Nov 03 20:32:50.895509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.517633 5059 trace.go:236] Trace[1551302611]: "Reflector WatchList" name:object-"openshift-network-diagnostics"/"network-check-target-dockercfg-5qm8x" (03-Nov-2026 20:28:00.749) (total time: 48767ms): Nov 03 20:32:50.895509 ip-10-0-132-83 kubenswrapper[5059]: Trace[1551302611]: ---"Objects streamed" count:1 48767ms (20:28:49.517) Nov 03 20:32:50.895509 ip-10-0-132-83 kubenswrapper[5059]: Trace[1551302611]: [48.767916119s] [48.767916119s] END Nov 03 20:32:50.895509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.517640 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5qm8x\"" Nov 03 20:32:50.895509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.517726 5059 trace.go:236] Trace[1001302096]: "Reflector WatchList" name:object-"openshift-ingress"/"payload-processing-plugins-e2e-extmod-a-98529d" (03-Nov-2026 20:28:00.762) (total time: 48755ms): Nov 03 20:32:50.895509 ip-10-0-132-83 kubenswrapper[5059]: Trace[1001302096]: ---"Objects streamed" count:1 48755ms (20:28:49.517) Nov 03 20:32:50.909877 ip-10-0-132-83 kubenswrapper[5059]: Trace[1001302096]: [48.755621655s] [48.755621655s] END Nov 03 20:32:50.909877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.517735 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-98529d\"" Nov 03 20:32:50.909877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.517798 5059 trace.go:236] Trace[1647926668]: "Reflector WatchList" name:object-"e2e-models-e2e-worker-w3-9f46e5"/"default-dockercfg-zrxnc" (03-Nov-2026 20:28:00.750) (total time: 48767ms): Nov 03 20:32:50.909877 ip-10-0-132-83 kubenswrapper[5059]: Trace[1647926668]: ---"Objects streamed" count:1 48767ms (20:28:49.517) Nov 03 20:32:50.909877 ip-10-0-132-83 kubenswrapper[5059]: Trace[1647926668]: [48.767662055s] [48.767662055s] END Nov 03 20:32:50.909877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.517805 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"default-dockercfg-zrxnc\"" Nov 03 20:32:50.909877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.517850 5059 trace.go:236] Trace[789103406]: "Reflector WatchList" name:object-"openshift-ovn-kubernetes"/"openshift-service-ca.crt" (03-Nov-2026 20:28:00.762) (total time: 48755ms): Nov 03 20:32:50.909877 ip-10-0-132-83 kubenswrapper[5059]: Trace[789103406]: ---"Objects streamed" count:1 48755ms (20:28:49.517) Nov 03 20:32:50.909877 ip-10-0-132-83 kubenswrapper[5059]: Trace[789103406]: [48.755215549s] [48.755215549s] END Nov 03 20:32:50.909877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.517857 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 20:32:50.909877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.517920 5059 trace.go:236] Trace[243303770]: "Reflector WatchList" name:object-"kuadrant-system"/"authorino-authorino-dockercfg-sgdb8" (03-Nov-2026 20:28:00.762) (total time: 48755ms): Nov 03 20:32:50.909877 ip-10-0-132-83 kubenswrapper[5059]: Trace[243303770]: ---"Objects streamed" count:1 48755ms (20:28:49.517) Nov 03 20:32:50.909877 ip-10-0-132-83 kubenswrapper[5059]: Trace[243303770]: [48.75500903s] [48.75500903s] END Nov 03 20:32:50.909877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.517927 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sgdb8\"" Nov 03 20:32:50.909877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.517983 5059 trace.go:236] Trace[1059478501]: "Reflector WatchList" name:object-"openshift-jobset-operator"/"jobset-controller-manager-dockercfg-k5vf6" (03-Nov-2026 20:28:00.750) (total time: 48767ms): Nov 03 20:32:50.909877 ip-10-0-132-83 kubenswrapper[5059]: Trace[1059478501]: ---"Objects streamed" count:1 48767ms (20:28:49.517) Nov 03 20:32:50.953253 ip-10-0-132-83 kubenswrapper[5059]: Trace[1059478501]: [48.767600053s] [48.767600053s] END Nov 03 20:32:50.953253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.517990 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-k5vf6\"" Nov 03 20:32:50.953253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.518071 5059 trace.go:236] Trace[2098238708]: "Reflector WatchList" name:object-"openshift-image-registry"/"registry-dockercfg-vf7n8" (03-Nov-2026 20:28:00.751) (total time: 48766ms): Nov 03 20:32:50.953253 ip-10-0-132-83 kubenswrapper[5059]: Trace[2098238708]: ---"Objects streamed" count:1 48766ms (20:28:49.518) Nov 03 20:32:50.953253 ip-10-0-132-83 kubenswrapper[5059]: Trace[2098238708]: [48.76685299s] [48.76685299s] END Nov 03 20:32:50.953253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.518077 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vf7n8\"" Nov 03 20:32:50.953253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.518112 5059 trace.go:236] Trace[606593736]: "Reflector WatchList" name:object-"openshift-ingress"/"cacerts" (03-Nov-2026 20:28:00.763) (total time: 48754ms): Nov 03 20:32:50.953253 ip-10-0-132-83 kubenswrapper[5059]: Trace[606593736]: ---"Objects streamed" count:0 48754ms (20:28:49.518) Nov 03 20:32:50.953253 ip-10-0-132-83 kubenswrapper[5059]: Trace[606593736]: [48.754133984s] [48.754133984s] END Nov 03 20:32:50.953253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.518120 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 20:32:50.953253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.518193 5059 trace.go:236] Trace[1493814033]: "Reflector WatchList" name:object-"openshift-ingress"/"e2e-shared-b-w0-81a5ae-openshift-default-dockercfg-pvl2f" (03-Nov-2026 20:28:00.750) (total time: 48767ms): Nov 03 20:32:50.953253 ip-10-0-132-83 kubenswrapper[5059]: Trace[1493814033]: ---"Objects streamed" count:1 48767ms (20:28:49.518) Nov 03 20:32:50.953253 ip-10-0-132-83 kubenswrapper[5059]: Trace[1493814033]: [48.767378158s] [48.767378158s] END Nov 03 20:32:50.953253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.518202 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-81a5ae-openshift-default-dockercfg-pvl2f\"" Nov 03 20:32:50.953253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.518268 5059 trace.go:236] Trace[2007772894]: "Reflector WatchList" name:object-"openshift-ingress"/"payload-processing-e2e-worker-w4-3fbb1a-dockercfg-b6ghv" (03-Nov-2026 20:28:00.764) (total time: 48753ms): Nov 03 20:32:50.953253 ip-10-0-132-83 kubenswrapper[5059]: Trace[2007772894]: ---"Objects streamed" count:1 48753ms (20:28:49.518) Nov 03 20:32:50.953253 ip-10-0-132-83 kubenswrapper[5059]: Trace[2007772894]: [48.753790951s] [48.753790951s] END Nov 03 20:32:50.979500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.518275 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-3fbb1a-dockercfg-b6ghv\"" Nov 03 20:32:50.979500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.518342 5059 trace.go:236] Trace[1652970742]: "Reflector WatchList" name:object-"openshift-monitoring"/"openshift-service-ca.crt" (03-Nov-2026 20:28:00.764) (total time: 48753ms): Nov 03 20:32:50.979500 ip-10-0-132-83 kubenswrapper[5059]: Trace[1652970742]: ---"Objects streamed" count:1 48753ms (20:28:49.518) Nov 03 20:32:50.979500 ip-10-0-132-83 kubenswrapper[5059]: Trace[1652970742]: [48.753348583s] [48.753348583s] END Nov 03 20:32:50.979500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.518349 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 20:32:50.979500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.522405 5059 trace.go:236] Trace[1991612815]: "Reflector WatchList" name:k8s.io/client-go/informers/factory.go:161 (03-Nov-2026 20:28:00.755) (total time: 48766ms): Nov 03 20:32:50.979500 ip-10-0-132-83 kubenswrapper[5059]: Trace[1991612815]: ---"Objects streamed" count:229 48766ms (20:28:49.522) Nov 03 20:32:50.979500 ip-10-0-132-83 kubenswrapper[5059]: Trace[1991612815]: [48.766583936s] [48.766583936s] END Nov 03 20:32:50.979500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.522431 5059 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:32:50.979500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.522575 5059 trace.go:236] Trace[634090545]: "Reflector WatchList" name:object-"openshift-ovn-kubernetes"/"ovnkube-script-lib" (03-Nov-2026 20:28:00.753) (total time: 48769ms): Nov 03 20:32:50.979500 ip-10-0-132-83 kubenswrapper[5059]: Trace[634090545]: ---"Objects streamed" count:1 48769ms (20:28:49.522) Nov 03 20:32:50.979500 ip-10-0-132-83 kubenswrapper[5059]: Trace[634090545]: [48.769056431s] [48.769056431s] END Nov 03 20:32:50.979500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.522585 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 20:32:50.979500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.522684 5059 trace.go:236] Trace[1238111891]: "Reflector WatchList" name:object-"openshift-ingress"/"router-dockercfg-xr7jf" (03-Nov-2026 20:28:00.767) (total time: 48754ms): Nov 03 20:32:50.979500 ip-10-0-132-83 kubenswrapper[5059]: Trace[1238111891]: ---"Objects streamed" count:1 48754ms (20:28:49.522) Nov 03 20:32:50.979500 ip-10-0-132-83 kubenswrapper[5059]: Trace[1238111891]: [48.754947252s] [48.754947252s] END Nov 03 20:32:50.980106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.522693 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xr7jf\"" Nov 03 20:32:50.980106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.522780 5059 trace.go:236] Trace[1134160340]: "Reflector WatchList" name:object-"openshift-dns"/"node-resolver-dockercfg-ndfpg" (03-Nov-2026 20:28:00.753) (total time: 48769ms): Nov 03 20:32:50.980106 ip-10-0-132-83 kubenswrapper[5059]: Trace[1134160340]: ---"Objects streamed" count:1 48768ms (20:28:49.522) Nov 03 20:32:50.980106 ip-10-0-132-83 kubenswrapper[5059]: Trace[1134160340]: [48.769001207s] [48.769001207s] END Nov 03 20:32:50.980106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.522788 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ndfpg\"" Nov 03 20:32:50.980106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.522853 5059 trace.go:236] Trace[1278041031]: "Reflector WatchList" name:object-"e2e-models-e2e-worker-w3-9f46e5"/"kube-root-ca.crt" (03-Nov-2026 20:28:00.768) (total time: 48754ms): Nov 03 20:32:50.980106 ip-10-0-132-83 kubenswrapper[5059]: Trace[1278041031]: ---"Objects streamed" count:1 48754ms (20:28:49.522) Nov 03 20:32:50.980106 ip-10-0-132-83 kubenswrapper[5059]: Trace[1278041031]: [48.75466089s] [48.75466089s] END Nov 03 20:32:50.980106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.522862 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"kube-root-ca.crt\"" Nov 03 20:32:50.980106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.522929 5059 trace.go:236] Trace[167630360]: "Reflector WatchList" name:object-"openshift-ingress"/"e2e-worker-w1-32e482-openshift-default-dockercfg-5pv2v" (03-Nov-2026 20:28:00.754) (total time: 48768ms): Nov 03 20:32:50.980106 ip-10-0-132-83 kubenswrapper[5059]: Trace[167630360]: ---"Objects streamed" count:1 48768ms (20:28:49.522) Nov 03 20:32:50.980106 ip-10-0-132-83 kubenswrapper[5059]: Trace[167630360]: [48.768735937s] [48.768735937s] END Nov 03 20:32:50.980106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.522936 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-32e482-openshift-default-dockercfg-5pv2v\"" Nov 03 20:32:50.980106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.523012 5059 trace.go:236] Trace[1490724544]: "Reflector WatchList" name:object-"e2e-models-e2e-worker-w4-3fbb1a"/"default-dockercfg-lxnx2" (03-Nov-2026 20:28:00.768) (total time: 48754ms): Nov 03 20:32:50.980106 ip-10-0-132-83 kubenswrapper[5059]: Trace[1490724544]: ---"Objects streamed" count:1 48754ms (20:28:49.522) Nov 03 20:32:50.980106 ip-10-0-132-83 kubenswrapper[5059]: Trace[1490724544]: [48.754082622s] [48.754082622s] END Nov 03 20:32:50.980743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.523020 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"default-dockercfg-lxnx2\"" Nov 03 20:32:50.980743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.523128 5059 trace.go:236] Trace[2080142498]: "Reflector WatchList" name:object-"openshift-console"/"downloads-dockercfg-wftlv" (03-Nov-2026 20:28:00.754) (total time: 48768ms): Nov 03 20:32:50.980743 ip-10-0-132-83 kubenswrapper[5059]: Trace[2080142498]: ---"Objects streamed" count:1 48768ms (20:28:49.523) Nov 03 20:32:50.980743 ip-10-0-132-83 kubenswrapper[5059]: Trace[2080142498]: [48.768527628s] [48.768527628s] END Nov 03 20:32:50.980743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.523140 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-wftlv\"" Nov 03 20:32:50.980743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.523191 5059 trace.go:236] Trace[1846933739]: "Reflector WatchList" name:object-"openshift-cluster-node-tuning-operator"/"openshift-service-ca.crt" (03-Nov-2026 20:28:00.769) (total time: 48753ms): Nov 03 20:32:50.980743 ip-10-0-132-83 kubenswrapper[5059]: Trace[1846933739]: ---"Objects streamed" count:1 48753ms (20:28:49.523) Nov 03 20:32:50.980743 ip-10-0-132-83 kubenswrapper[5059]: Trace[1846933739]: [48.753898843s] [48.753898843s] END Nov 03 20:32:50.980743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.523199 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:32:50.980743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.523268 5059 trace.go:236] Trace[581540939]: "Reflector WatchList" name:object-"openshift-network-operator"/"iptables-alerter-script" (03-Nov-2026 20:28:00.754) (total time: 48768ms): Nov 03 20:32:50.980743 ip-10-0-132-83 kubenswrapper[5059]: Trace[581540939]: ---"Objects streamed" count:1 48768ms (20:28:49.523) Nov 03 20:32:50.980743 ip-10-0-132-83 kubenswrapper[5059]: Trace[581540939]: [48.768283933s] [48.768283933s] END Nov 03 20:32:50.980743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.523275 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 20:32:50.980743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.523348 5059 trace.go:236] Trace[867183694]: "Reflector WatchList" name:object-"openshift-ingress"/"e2e-worker-w4-3fbb1a-openshift-default-dockercfg-247ct" (03-Nov-2026 20:28:00.755) (total time: 48767ms): Nov 03 20:32:50.980743 ip-10-0-132-83 kubenswrapper[5059]: Trace[867183694]: ---"Objects streamed" count:1 48767ms (20:28:49.523) Nov 03 20:32:50.980743 ip-10-0-132-83 kubenswrapper[5059]: Trace[867183694]: [48.767951849s] [48.767951849s] END Nov 03 20:32:50.981340 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.523356 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3fbb1a-openshift-default-dockercfg-247ct\"" Nov 03 20:32:50.981340 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.525827 5059 trace.go:236] Trace[1565838045]: "Reflector WatchList" name:object-"openshift-multus"/"metrics-daemon-secret" (03-Nov-2026 20:28:00.752) (total time: 48773ms): Nov 03 20:32:50.981340 ip-10-0-132-83 kubenswrapper[5059]: Trace[1565838045]: ---"Objects streamed" count:1 48773ms (20:28:49.525) Nov 03 20:32:50.981340 ip-10-0-132-83 kubenswrapper[5059]: Trace[1565838045]: [48.773373168s] [48.773373168s] END Nov 03 20:32:50.981340 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.525845 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 20:32:50.981340 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.525897 5059 trace.go:236] Trace[1183851316]: "Reflector WatchList" name:object-"openshift-jobset-operator"/"jobset-manager-config" (03-Nov-2026 20:28:00.765) (total time: 48760ms): Nov 03 20:32:50.981340 ip-10-0-132-83 kubenswrapper[5059]: Trace[1183851316]: ---"Objects streamed" count:1 48760ms (20:28:49.525) Nov 03 20:32:50.981340 ip-10-0-132-83 kubenswrapper[5059]: Trace[1183851316]: [48.760484148s] [48.760484148s] END Nov 03 20:32:50.981340 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.525907 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 20:32:50.981340 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.525988 5059 trace.go:236] Trace[381491751]: "Reflector WatchList" name:object-"openshift-ovn-kubernetes"/"env-overrides" (03-Nov-2026 20:28:00.765) (total time: 48760ms): Nov 03 20:32:50.981340 ip-10-0-132-83 kubenswrapper[5059]: Trace[381491751]: ---"Objects streamed" count:0 48760ms (20:28:49.525) Nov 03 20:32:50.981340 ip-10-0-132-83 kubenswrapper[5059]: Trace[381491751]: [48.760041224s] [48.760041224s] END Nov 03 20:32:50.981340 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.525998 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 20:32:50.981340 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526079 5059 trace.go:236] Trace[674299910]: "Reflector WatchList" name:object-"openshift-console"/"openshift-service-ca.crt" (03-Nov-2026 20:28:00.758) (total time: 48767ms): Nov 03 20:32:50.981340 ip-10-0-132-83 kubenswrapper[5059]: Trace[674299910]: ---"Objects streamed" count:1 48767ms (20:28:49.526) Nov 03 20:32:50.981340 ip-10-0-132-83 kubenswrapper[5059]: Trace[674299910]: [48.767881974s] [48.767881974s] END Nov 03 20:32:50.981896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526088 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 20:32:50.981896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526134 5059 trace.go:236] Trace[800872188]: "Reflector WatchList" name:object-"openshift-kube-storage-version-migrator-operator"/"openshift-service-ca.crt" (03-Nov-2026 20:28:00.756) (total time: 48769ms): Nov 03 20:32:50.981896 ip-10-0-132-83 kubenswrapper[5059]: Trace[800872188]: ---"Objects streamed" count:1 48769ms (20:28:49.526) Nov 03 20:32:50.981896 ip-10-0-132-83 kubenswrapper[5059]: Trace[800872188]: [48.769947914s] [48.769947914s] END Nov 03 20:32:50.981896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526141 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:32:50.981896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526192 5059 trace.go:236] Trace[961319657]: "Reflector WatchList" name:object-"e2e-models-e2e-worker-w4-3fbb1a"/"openshift-service-ca.crt" (03-Nov-2026 20:28:00.756) (total time: 48769ms): Nov 03 20:32:50.981896 ip-10-0-132-83 kubenswrapper[5059]: Trace[961319657]: ---"Objects streamed" count:1 48769ms (20:28:49.526) Nov 03 20:32:50.981896 ip-10-0-132-83 kubenswrapper[5059]: Trace[961319657]: [48.769576978s] [48.769576978s] END Nov 03 20:32:50.981896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526200 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"openshift-service-ca.crt\"" Nov 03 20:32:50.981896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526328 5059 trace.go:236] Trace[1642116954]: "Reflector WatchList" name:object-"openshift-monitoring"/"metrics-client-ca" (03-Nov-2026 20:28:00.757) (total time: 48769ms): Nov 03 20:32:50.981896 ip-10-0-132-83 kubenswrapper[5059]: Trace[1642116954]: ---"Objects streamed" count:1 48769ms (20:28:49.526) Nov 03 20:32:50.981896 ip-10-0-132-83 kubenswrapper[5059]: Trace[1642116954]: [48.769294747s] [48.769294747s] END Nov 03 20:32:50.981896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526336 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 20:32:50.981896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526405 5059 trace.go:236] Trace[1530396628]: "Reflector WatchList" name:object-"openshift-insights"/"openshift-insights-serving-cert" (03-Nov-2026 20:28:00.757) (total time: 48768ms): Nov 03 20:32:50.981896 ip-10-0-132-83 kubenswrapper[5059]: Trace[1530396628]: ---"Objects streamed" count:1 48768ms (20:28:49.526) Nov 03 20:32:50.981896 ip-10-0-132-83 kubenswrapper[5059]: Trace[1530396628]: [48.768993244s] [48.768993244s] END Nov 03 20:32:50.982463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526413 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 20:32:50.982463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526488 5059 trace.go:236] Trace[1973674022]: "Reflector WatchList" name:object-"openshift-ingress"/"e2e-extmod-a-8b7351-openshift-default-dockercfg-lx6hb" (03-Nov-2026 20:28:00.757) (total time: 48768ms): Nov 03 20:32:50.982463 ip-10-0-132-83 kubenswrapper[5059]: Trace[1973674022]: ---"Objects streamed" count:1 48768ms (20:28:49.526) Nov 03 20:32:50.982463 ip-10-0-132-83 kubenswrapper[5059]: Trace[1973674022]: [48.768672651s] [48.768672651s] END Nov 03 20:32:50.982463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526495 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-8b7351-openshift-default-dockercfg-lx6hb\"" Nov 03 20:32:50.982463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526560 5059 trace.go:236] Trace[1802593019]: "Reflector WatchList" name:object-"kuadrant-system"/"openshift-service-ca.crt" (03-Nov-2026 20:28:00.766) (total time: 48760ms): Nov 03 20:32:50.982463 ip-10-0-132-83 kubenswrapper[5059]: Trace[1802593019]: ---"Objects streamed" count:1 48760ms (20:28:49.526) Nov 03 20:32:50.982463 ip-10-0-132-83 kubenswrapper[5059]: Trace[1802593019]: [48.760146423s] [48.760146423s] END Nov 03 20:32:50.982463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526567 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 20:32:50.982463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526618 5059 trace.go:236] Trace[1050453549]: "Reflector WatchList" name:object-"openshift-dns"/"dns-default" (03-Nov-2026 20:28:00.760) (total time: 48766ms): Nov 03 20:32:50.982463 ip-10-0-132-83 kubenswrapper[5059]: Trace[1050453549]: ---"Objects streamed" count:1 48766ms (20:28:49.526) Nov 03 20:32:50.982463 ip-10-0-132-83 kubenswrapper[5059]: Trace[1050453549]: [48.766522352s] [48.766522352s] END Nov 03 20:32:50.982463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526625 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 20:32:50.982463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526702 5059 trace.go:236] Trace[1875039152]: "Reflector WatchList" name:object-"openshift-insights"/"kube-rbac-proxy" (03-Nov-2026 20:28:00.758) (total time: 48768ms): Nov 03 20:32:50.982463 ip-10-0-132-83 kubenswrapper[5059]: Trace[1875039152]: ---"Objects streamed" count:1 48768ms (20:28:49.526) Nov 03 20:32:50.982463 ip-10-0-132-83 kubenswrapper[5059]: Trace[1875039152]: [48.768109779s] [48.768109779s] END Nov 03 20:32:51.012288 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526710 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 20:32:51.012288 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526753 5059 trace.go:236] Trace[2114166848]: "Reflector WatchList" name:object-"openshift-multus"/"whereabouts-flatfile-config" (03-Nov-2026 20:28:00.759) (total time: 48767ms): Nov 03 20:32:51.012288 ip-10-0-132-83 kubenswrapper[5059]: Trace[2114166848]: ---"Objects streamed" count:1 48767ms (20:28:49.526) Nov 03 20:32:51.012288 ip-10-0-132-83 kubenswrapper[5059]: Trace[2114166848]: [48.767249065s] [48.767249065s] END Nov 03 20:32:51.012288 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526760 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 20:32:51.012288 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526844 5059 trace.go:236] Trace[1620441158]: "Reflector WatchList" name:object-"llm"/"default-dockercfg-8tzmp" (03-Nov-2026 20:28:00.760) (total time: 48766ms): Nov 03 20:32:51.012288 ip-10-0-132-83 kubenswrapper[5059]: Trace[1620441158]: ---"Objects streamed" count:1 48766ms (20:28:49.526) Nov 03 20:32:51.012288 ip-10-0-132-83 kubenswrapper[5059]: Trace[1620441158]: [48.766235684s] [48.766235684s] END Nov 03 20:32:51.012288 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526853 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8tzmp\"" Nov 03 20:32:51.012288 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526922 5059 trace.go:236] Trace[1024557729]: "Reflector WatchList" name:object-"openshift-ingress"/"e2e-extmod-a-33e267-openshift-default-dockercfg-lkhx5" (03-Nov-2026 20:28:00.761) (total time: 48765ms): Nov 03 20:32:51.012288 ip-10-0-132-83 kubenswrapper[5059]: Trace[1024557729]: ---"Objects streamed" count:1 48765ms (20:28:49.526) Nov 03 20:32:51.012288 ip-10-0-132-83 kubenswrapper[5059]: Trace[1024557729]: [48.765765271s] [48.765765271s] END Nov 03 20:32:51.012288 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526930 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-33e267-openshift-default-dockercfg-lkhx5\"" Nov 03 20:32:51.012288 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526991 5059 trace.go:236] Trace[1931050592]: "Reflector WatchList" name:object-"openshift-monitoring"/"kube-root-ca.crt" (03-Nov-2026 20:28:00.752) (total time: 48774ms): Nov 03 20:32:51.012288 ip-10-0-132-83 kubenswrapper[5059]: Trace[1931050592]: ---"Objects streamed" count:1 48774ms (20:28:49.526) Nov 03 20:32:51.012288 ip-10-0-132-83 kubenswrapper[5059]: Trace[1931050592]: [48.774261571s] [48.774261571s] END Nov 03 20:32:51.012646 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.526998 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 20:32:51.012646 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:49.755975 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts" podUID="633c352c-f07e-4032-b962-23a148297e45" containerName="postgres" probeResult="failure" output="command timed out" Nov 03 20:32:51.012646 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:28:49.792996 5059 webhook.go:269] Failed to make webhook authorizer request: webhook call failed: context canceled Nov 03 20:32:51.012646 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:28:49.870900 5059 webhook.go:269] Failed to make webhook authorizer request: webhook call failed: context canceled Nov 03 20:32:51.012646 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:28:49.870925 5059 server.go:358] "Authorization error" err="webhook call failed: context canceled" user="system:serviceaccount:openshift-monitoring:metrics-server" verb="get" resource="nodes" subresource="metrics" Nov 03 20:32:51.012646 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.033194 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m59.069064387s: [/var/lib/containers/storage/overlay/6b837f63593a17f863e6b609d3c6f7d939a72c0b3cc0a9bf79876776dbd4f0d9/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn_25ceab3d-6344-4512-a471-d91aec0cb48c/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.012646 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:28:49.793023 5059 server.go:358] "Authorization error" err="webhook call failed: context canceled" user="system:serviceaccount:openshift-monitoring:metrics-server" verb="get" resource="nodes" subresource="metrics" Nov 03 20:32:51.012646 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:28:50.080516 5059 webhook.go:269] Failed to make webhook authorizer request: webhook call failed: context canceled Nov 03 20:32:51.012646 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:28:50.080541 5059 server.go:358] "Authorization error" err="webhook call failed: context canceled" user="system:serviceaccount:openshift-monitoring:metrics-server" verb="get" resource="nodes" subresource="metrics" Nov 03 20:32:51.012646 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:28:50.085055 5059 webhook.go:269] Failed to make webhook authorizer request: webhook call failed: context canceled Nov 03 20:32:51.012907 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:28:50.085084 5059 server.go:358] "Authorization error" err="webhook call failed: context canceled" user="system:serviceaccount:openshift-monitoring:metrics-server" verb="get" resource="nodes" subresource="metrics" Nov 03 20:32:51.012907 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:28:50.085242 5059 webhook.go:269] Failed to make webhook authorizer request: webhook call failed: context canceled Nov 03 20:32:51.012907 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:28:50.085252 5059 server.go:358] "Authorization error" err="webhook call failed: context canceled" user="system:serviceaccount:openshift-monitoring:metrics-server" verb="get" resource="nodes" subresource="metrics" Nov 03 20:32:51.012907 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:28:50.085388 5059 webhook.go:269] Failed to make webhook authorizer request: webhook call failed: context canceled Nov 03 20:32:51.012907 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:28:50.085398 5059 server.go:358] "Authorization error" err="webhook call failed: context canceled" user="system:serviceaccount:openshift-monitoring:metrics-server" verb="get" resource="nodes" subresource="metrics" Nov 03 20:32:51.012907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.085702 5059 trace.go:236] Trace[2055850903]: "Calculate volume metrics of openshift-service-ca for pod odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" (03-Nov-2026 20:26:20.523) (total time: 149562ms): Nov 03 20:32:51.012907 ip-10-0-132-83 kubenswrapper[5059]: Trace[2055850903]: [2m29.562322378s] [2m29.562322378s] END Nov 03 20:32:51.012907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.085766 5059 trace.go:236] Trace[1731494344]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" (03-Nov-2026 20:26:18.739) (total time: 151346ms): Nov 03 20:32:51.012907 ip-10-0-132-83 kubenswrapper[5059]: Trace[1731494344]: [2m31.346337269s] [2m31.346337269s] END Nov 03 20:32:51.012907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:01.575216 5059 trace.go:236] Trace[1579359128]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" (03-Nov-2026 20:26:18.751) (total time: 1751ms): Nov 03 20:32:51.012907 ip-10-0-132-83 kubenswrapper[5059]: Trace[1579359128]: [1.751599843s] [1.751599843s] END Nov 03 20:32:51.012907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.589871 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"openshift-service-ca.crt\"" Nov 03 20:32:51.013226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.589995 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 20:32:51.013226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.590079 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-46a6d7-openshift-default-dockercfg-5jblx\"" Nov 03 20:32:51.013226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.592607 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 20:32:51.013226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.592801 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-4c2a28-dockercfg-psjnh\"" Nov 03 20:32:51.013226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.592932 5059 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:32:51.013226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.593031 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 20:32:51.013226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.593199 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-88a18e-openshift-default-dockercfg-bs4qg\"" Nov 03 20:32:51.013226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.593302 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e7b70d-openshift-default-dockercfg-jjqkc\"" Nov 03 20:32:51.013226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.593431 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"default-dockercfg-5r8nn\"" Nov 03 20:32:51.013226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.593577 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d2aa3b-openshift-default-dockercfg-g8c6q\"" Nov 03 20:32:51.013226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.593642 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 20:32:51.029300 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.141377 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.946853935s: [/var/lib/containers/storage/overlay/fbcc3fd0d14d87f947f11a6270e50c6a4a9b838bf13ea2df20555628ebca92ca/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.029300 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:28:50.141727 5059 webhook.go:269] Failed to make webhook authorizer request: webhook call failed: context canceled Nov 03 20:32:51.029300 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:28:50.594960 5059 server.go:358] "Authorization error" err="webhook call failed: context canceled" user="system:serviceaccount:openshift-monitoring:metrics-server" verb="get" resource="nodes" subresource="metrics" Nov 03 20:32:51.029300 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:28:50.141854 5059 webhook.go:269] Failed to make webhook authorizer request: webhook call failed: context canceled Nov 03 20:32:51.029300 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:28:50.595007 5059 server.go:358] "Authorization error" err="webhook call failed: context canceled" user="system:serviceaccount:openshift-monitoring:metrics-server" verb="get" resource="nodes" subresource="metrics" Nov 03 20:32:51.029300 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.145096 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.950566217s: [/var/lib/containers/storage/overlay/067ca349ee1450e78700331d6f208c849004833b993038177db3236b89e0c254/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.029300 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.145132 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.896557249s: [/var/lib/containers/storage/overlay/1c18da9d60946c03b1cb4c419644a5ccb427409980e444d0b82c88043ec78a6a/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.029300 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.159962 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.965429493s: [/var/lib/containers/storage/overlay/f4223c7ada52388a0e1115563a3e2c2098950a9bffbaa80181178892ef6cf8d3/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.029879 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.160013 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.908351983s: [/var/lib/containers/storage/overlay/3c1933e8657ed24a888e3078150a15152408aef3de6d06715661d3b955bdeb03/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.029879 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.160039 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.908389051s: [/var/lib/containers/storage/overlay/f76791e3b40bd77e4177e755af9593b8264a6f9584df82df5f3ad48f8536a824/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.029879 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.160060 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.90840928s: [/var/lib/containers/storage/overlay/5c5f82c4ddbea13aa5f7e0f0d8a4dea32beeac342c1caa0870a75fc8b4f3994f/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.029879 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.160524 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.908867995s: [/var/lib/containers/storage/overlay/2ab4d09781f5bb38cf39dad12e66699114c6e92eb3b29dd65c026a0af1b404cf/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.029879 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.160547 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.90889074s: [/var/lib/containers/storage/overlay/9a287a1ec7ba2ac17e0f0d11b194f49d00d362cedf55c894346e179195a8e98e/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.029879 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.160572 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.908913091s: [/var/lib/containers/storage/overlay/acb75c12d3b37b911ca96f7741e934e88cdad5615e7b123974ba4d09049057a8/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.030328 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.160595 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.90893639s: [/var/lib/containers/storage/overlay/6e9994b975a1d209a1e7ad1c69fd10d9cc3dcfdb7ae65b13f21dcbd140058659/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.030328 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.160616 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.908943786s: [/var/lib/containers/storage/overlay/2acad7ba6294ceb7ed21eec32f82e3dd0f18687ff5816a03411e1545b134088c/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.030328 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.162363 5059 trace.go:236] Trace[700202064]: "Calculate volume metrics of kube-api-access-jzmbv for pod openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" (03-Nov-2026 20:27:17.308) (total time: 92853ms): Nov 03 20:32:51.030328 ip-10-0-132-83 kubenswrapper[5059]: Trace[700202064]: [1m32.853954915s] [1m32.853954915s] END Nov 03 20:32:51.030328 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.162900 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.911207104s: [/var/lib/containers/storage/overlay/696bd9c68ee459239e11ef2c5be2c47c25002ec59a7388640a6ee78d147b05aa/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.030328 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.162983 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.911292131s: [/var/lib/containers/storage/overlay/2cf0cbdd877424c5f8c0110195c9f03a8f8dac9e852d359293da71d0737916a0/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.030328 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.163010 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.911315192s: [/var/lib/containers/storage/overlay/fff3e4757f69e495c651e4c9e0afd850cf031c4c2b428196eb46a31c2ff022ea/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.030625 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.163032 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.91133533s: [/var/lib/containers/storage/overlay/0e72203bbe7cb0cd3f9fc6eab23667a1b76b6bb41836316e63fd885d237bb420/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.030625 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.163164 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.911464423s: [/var/lib/containers/storage/overlay/e3a08dd38e0673fee7ee93b1f7ca301b5aa65f8f1089002352132e3cd010d249/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.030625 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.163853 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.90925421s: [/var/lib/containers/storage/overlay/0c52062b38c59af46beb68c242b3a38f795b07b0c8b1ad76126162c8c3c3e1d5/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.030625 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.163907 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.909306995s: [/var/lib/containers/storage/overlay/56a1a4fe0beea2934f0067c34f07d9b9de48a79ddeaebc10dcc1918e249e4fe9/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.030625 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.163934 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.909333579s: [/var/lib/containers/storage/overlay/974994766dd4ff7caa644bd6347d79d35dd36116080142add2f681ab8815f5ab/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.030625 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.163954 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.909352304s: [/var/lib/containers/storage/overlay/5e9d0591d7c92f06d2c04115fb8e8d565a3e0b02ab2c918f95ea53689fea97eb/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.030876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.163979 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.909366823s: [/var/lib/containers/storage/overlay/d49f130f651bf65ace5eace09a0ebe2fa45d704fd81e259422bdddca3c58a0d6/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.030876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.164114 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.909498027s: [/var/lib/containers/storage/overlay/89ea4b8b93aee8b87c44a82fd9fd20add9e924a331496366f375f3833e6e457c/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.030876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.164138 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.909519908s: [/var/lib/containers/storage/overlay/1a222e0eecb04b88b5dabdbc6d32ec15f7b6d7e865f4b269e1fef7a61f1e0f0a/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.030876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.164163 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.909543933s: [/var/lib/containers/storage/overlay/e8120ead559c0bf33201f22ae411f9741ede65e14e3388fa66493a090c2ce6e6/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.030876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.164183 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.909562257s: [/var/lib/containers/storage/overlay/326b71f661ef9f0085c5b27b89c59064eec03ee6148d513a218dcd440991fe6d/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.030876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.164236 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.909613003s: [/var/lib/containers/storage/overlay/6982a733f632f82f58560039947e668a41a46bac9f9f4ddef9335e74f577da9d/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.031375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.164262 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.909633514s: [/var/lib/containers/storage/overlay/3e151306ed961ad6bfc0a65d6d7858de0210f8149234a8750218b7bce9f28c39/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.031375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.164285 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.909645623s: [/var/lib/containers/storage/overlay/c3f9bac49400fba6152bdce7d3ee4ea3aa0bf276d345e56e6566efd24f4aa5ba/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.031375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.164310 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.909670949s: [/var/lib/containers/storage/overlay/07aeb6439b6aab304aeaf1b043ccaeee044b85ae3facc335f860427356ab4c41/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.031375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.164332 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.909691327s: [/var/lib/containers/storage/overlay/ef5bbc2ac888611f17ab3446f15465cb87e7e0732a42e0f69870ac3291beae0b/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.031375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.164352 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.909710936s: [/var/lib/containers/storage/overlay/5f7322f6b59f9ec6bdaf95083ffd18198d220b0ba84771ed863a48299964c622/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.031375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.164407 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.909762702s: [/var/lib/containers/storage/overlay/807b5d91ec044b6e3690fd687ada9bd4e01ef9aefd7c5be109d86f8020447460/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.031719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.164432 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.909788128s: [/var/lib/containers/storage/overlay/e508a4fba182975f628a2965bdd2bc06db2f762a03cb80182d6faab7219e0ccd/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.031719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.164452 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.909807451s: [/var/lib/containers/storage/overlay/364e21f717075de023707ce0eadcab5ff3e172710140545d125c62e36740887e/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.031719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.164486 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.909839219s: [/var/lib/containers/storage/overlay/42ec34392fa2e3d7cbe31b8fa86900cece879ef5fb5d2e80b32bb490c7a8fbb4/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.031719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.164516 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.909868876s: [/var/lib/containers/storage/overlay/15a96bafbdc9757f9d817aa97e7845340b628642f6a672a3e31c5f490b8f0fb1/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.031719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.164575 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.909926279s: [/var/lib/containers/storage/overlay/5b44229870991839ead05bbd6d43ad3c92fae6398297149bd793ebda3f403234/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.031719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.164607 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.909957104s: [/var/lib/containers/storage/overlay/927ae99c7bc3b06e217e45d2cf6db349f238086d590a4a2b7e234cbb66369af6/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.031970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.165335 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.970824075s: [/var/lib/containers/storage/overlay/2b818e04a987bed97dc81c53c55626df56032b4ea5b65abebfe974b29ef334b6/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.031970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.165794 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.971283426s: [/var/lib/containers/storage/overlay/7f28e0f7f0452864f37340b84f720939062a84b92404c9b09cbac54186e86f29/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.031970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.585070 5059 trace.go:236] Trace[1257022950]: "Reflector WatchList" name:object-"odh-ai-gateway-infra"/"default-dockercfg-w84zh" (03-Nov-2026 20:28:01.759) (total time: 48825ms): Nov 03 20:32:51.031970 ip-10-0-132-83 kubenswrapper[5059]: Trace[1257022950]: ---"Objects streamed" count:1 48825ms (20:28:50.585) Nov 03 20:32:51.031970 ip-10-0-132-83 kubenswrapper[5059]: Trace[1257022950]: [48.825804812s] [48.825804812s] END Nov 03 20:32:51.031970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.595403 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-w84zh\"" Nov 03 20:32:51.031970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.585311 5059 trace.go:236] Trace[1401362006]: "Reflector WatchList" name:k8s.io/client-go/informers/factory.go:161 (03-Nov-2026 20:28:00.769) (total time: 49815ms): Nov 03 20:32:51.031970 ip-10-0-132-83 kubenswrapper[5059]: Trace[1401362006]: ---"Objects streamed" count:1 49815ms (20:28:50.585) Nov 03 20:32:51.031970 ip-10-0-132-83 kubenswrapper[5059]: Trace[1401362006]: [49.815698527s] [49.815698527s] END Nov 03 20:32:51.031970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.595454 5059 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:32:51.031970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.585405 5059 trace.go:236] Trace[669917110]: "Reflector WatchList" name:object-"e2e-models-e2e-worker-w2-2b5703"/"openshift-service-ca.crt" (03-Nov-2026 20:28:01.771) (total time: 48813ms): Nov 03 20:32:51.031970 ip-10-0-132-83 kubenswrapper[5059]: Trace[669917110]: ---"Objects streamed" count:1 48813ms (20:28:50.585) Nov 03 20:32:51.031970 ip-10-0-132-83 kubenswrapper[5059]: Trace[669917110]: [48.813935461s] [48.813935461s] END Nov 03 20:32:51.031970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.595473 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"openshift-service-ca.crt\"" Nov 03 20:32:51.032505 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.585497 5059 trace.go:236] Trace[1250867842]: "Reflector WatchList" name:object-"openshift-service-ca-operator"/"service-ca-operator-dockercfg-bfvcm" (03-Nov-2026 20:28:01.841) (total time: 48743ms): Nov 03 20:32:51.032505 ip-10-0-132-83 kubenswrapper[5059]: Trace[1250867842]: ---"Objects streamed" count:1 48743ms (20:28:50.585) Nov 03 20:32:51.032505 ip-10-0-132-83 kubenswrapper[5059]: Trace[1250867842]: [48.743505803s] [48.743505803s] END Nov 03 20:32:51.032505 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.595520 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bfvcm\"" Nov 03 20:32:51.032505 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.585590 5059 trace.go:236] Trace[1227898653]: "Reflector WatchList" name:object-"openshift-ingress"/"e2e-shared-a-w1-f154ac-openshift-default-dockercfg-xndgl" (03-Nov-2026 20:28:01.854) (total time: 48730ms): Nov 03 20:32:51.032505 ip-10-0-132-83 kubenswrapper[5059]: Trace[1227898653]: ---"Objects streamed" count:1 48730ms (20:28:50.585) Nov 03 20:32:51.032505 ip-10-0-132-83 kubenswrapper[5059]: Trace[1227898653]: [48.730814999s] [48.730814999s] END Nov 03 20:32:51.032505 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.595537 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-f154ac-openshift-default-dockercfg-xndgl\"" Nov 03 20:32:51.032505 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.585652 5059 trace.go:236] Trace[210701696]: "Reflector WatchList" name:object-"openshift-ingress"/"payload-processing-plugins-e2e-extmod-b-4c2a28" (03-Nov-2026 20:28:01.925) (total time: 48660ms): Nov 03 20:32:51.032505 ip-10-0-132-83 kubenswrapper[5059]: Trace[210701696]: ---"Objects streamed" count:1 48660ms (20:28:50.585) Nov 03 20:32:51.032505 ip-10-0-132-83 kubenswrapper[5059]: Trace[210701696]: [48.66060268s] [48.66060268s] END Nov 03 20:32:51.032505 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.595556 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-4c2a28\"" Nov 03 20:32:51.032505 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.585769 5059 trace.go:236] Trace[13400278]: "Reflector WatchList" name:object-"openshift-ingress"/"e2e-worker-w3-9f46e5-openshift-default-dockercfg-4djl8" (03-Nov-2026 20:28:01.953) (total time: 48632ms): Nov 03 20:32:51.032505 ip-10-0-132-83 kubenswrapper[5059]: Trace[13400278]: ---"Objects streamed" count:1 48632ms (20:28:50.585) Nov 03 20:32:51.032505 ip-10-0-132-83 kubenswrapper[5059]: Trace[13400278]: [48.632486766s] [48.632486766s] END Nov 03 20:32:51.033203 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.595600 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-9f46e5-openshift-default-dockercfg-4djl8\"" Nov 03 20:32:51.033203 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.591817 5059 trace.go:236] Trace[760369036]: "Reflector WatchList" name:object-"openshift-network-operator"/"openshift-service-ca.crt" (03-Nov-2026 20:28:00.989) (total time: 49602ms): Nov 03 20:32:51.033203 ip-10-0-132-83 kubenswrapper[5059]: Trace[760369036]: ---"Objects streamed" count:1 49602ms (20:28:50.591) Nov 03 20:32:51.033203 ip-10-0-132-83 kubenswrapper[5059]: Trace[760369036]: [49.602718632s] [49.602718632s] END Nov 03 20:32:51.033203 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.595619 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:32:51.033203 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.604782 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ldllm\"" Nov 03 20:32:51.033203 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.604937 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 20:32:51.033203 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.605076 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lrbp4\"" Nov 03 20:32:51.033203 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.605166 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"payload-processing-external-model-e2e--b1878920-dockercfg-vbr6x\"" Nov 03 20:32:51.033203 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.605242 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 20:32:51.033203 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.605391 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0e4ce7-openshift-default-dockercfg-nppzh\"" Nov 03 20:32:51.033203 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.605500 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-dq8cl\"" Nov 03 20:32:51.033203 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.605572 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:32:51.033705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.605620 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0e4ce7\"" Nov 03 20:32:51.033705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.605761 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"simulator-w3-9f46e5-kserve-self-signed-certs\"" Nov 03 20:32:51.033705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.605900 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 20:32:51.033705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.605981 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 20:32:51.033705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.606123 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-nh6bx\"" Nov 03 20:32:51.033705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.606206 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 20:32:51.033705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.606299 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 20:32:51.033705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.606363 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 20:32:51.033705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.606510 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4p6h4\"" Nov 03 20:32:51.033705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.606623 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-32e482-dockercfg-4mbnz\"" Nov 03 20:32:51.033705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.606702 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 20:32:51.034147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.606881 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0e1546-openshift-default-dockercfg-jcbdc\"" Nov 03 20:32:51.034147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.606979 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:32:51.034147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.607081 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 20:32:51.034147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.607943 5059 trace.go:236] Trace[1859614372]: "Reflector WatchList" name:object-"openshift-monitoring"/"node-exporter-accelerators-collector-config" (03-Nov-2026 20:28:02.250) (total time: 48357ms): Nov 03 20:32:51.034147 ip-10-0-132-83 kubenswrapper[5059]: Trace[1859614372]: ---"Objects streamed" count:1 48357ms (20:28:50.607) Nov 03 20:32:51.034147 ip-10-0-132-83 kubenswrapper[5059]: Trace[1859614372]: [48.357655784s] [48.357655784s] END Nov 03 20:32:51.034147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.607961 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 20:32:51.034147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.608087 5059 trace.go:236] Trace[23708277]: "Reflector WatchList" name:object-"openshift-multus"/"multus-node-identity-dockercfg-qhpjg" (03-Nov-2026 20:28:02.250) (total time: 48357ms): Nov 03 20:32:51.034147 ip-10-0-132-83 kubenswrapper[5059]: Trace[23708277]: ---"Objects streamed" count:1 48357ms (20:28:50.608) Nov 03 20:32:51.034147 ip-10-0-132-83 kubenswrapper[5059]: Trace[23708277]: [48.357681834s] [48.357681834s] END Nov 03 20:32:51.034147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.608095 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qhpjg\"" Nov 03 20:32:51.034147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.608197 5059 trace.go:236] Trace[277254172]: "Reflector WatchList" name:object-"openshift-ingress"/"e2e-shared-b-w2-660d95-openshift-default-dockercfg-df2mh" (03-Nov-2026 20:28:02.249) (total time: 48358ms): Nov 03 20:32:51.034147 ip-10-0-132-83 kubenswrapper[5059]: Trace[277254172]: ---"Objects streamed" count:1 48358ms (20:28:50.608) Nov 03 20:32:51.034147 ip-10-0-132-83 kubenswrapper[5059]: Trace[277254172]: [48.358372694s] [48.358372694s] END Nov 03 20:32:51.034147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.608205 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-660d95-openshift-default-dockercfg-df2mh\"" Nov 03 20:32:51.035075 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.608291 5059 trace.go:236] Trace[508420449]: "Reflector WatchList" name:object-"openshift-dns"/"dns-dockercfg-qg9jj" (03-Nov-2026 20:28:02.278) (total time: 48329ms): Nov 03 20:32:51.035075 ip-10-0-132-83 kubenswrapper[5059]: Trace[508420449]: ---"Objects streamed" count:1 48329ms (20:28:50.608) Nov 03 20:32:51.035075 ip-10-0-132-83 kubenswrapper[5059]: Trace[508420449]: [48.329957582s] [48.329957582s] END Nov 03 20:32:51.035075 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.608299 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qg9jj\"" Nov 03 20:32:51.035075 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.608365 5059 trace.go:236] Trace[286098721]: "Reflector WatchList" name:object-"openshift-ingress"/"openshift-gw-ca-root-cert" (03-Nov-2026 20:28:03.008) (total time: 47600ms): Nov 03 20:32:51.035075 ip-10-0-132-83 kubenswrapper[5059]: Trace[286098721]: ---"Objects streamed" count:1 47600ms (20:28:50.608) Nov 03 20:32:51.035075 ip-10-0-132-83 kubenswrapper[5059]: Trace[286098721]: [47.600291093s] [47.600291093s] END Nov 03 20:32:51.035075 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.608373 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 20:32:51.035075 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.609097 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m59.028879601s: [/var/lib/containers/storage/overlay/daf6579fec241cf0ecce296260dfaa6f1c89bdaaf862dc139e0bb6d7b465719f/diff /var/log/pods/e2e-models-e2e-worker-w1-32e482_simulator-w1-32e482-kserve-776bf6595d-mgx55_16c463fa-8ddf-43c8-b00f-3f5f9ca33c02/main/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.035075 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.609128 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m57.247096817s: [/var/lib/containers/storage/overlay/15e71d53803ae942e3049eeff235ba243399e5cdb82a33d54954a3a1b2a1f3be/diff /var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-h9f7b_8d0db609-b271-4c98-a303-fb3ee71f152c/cluster-monitoring-operator/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.035477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.609149 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m57.452908387s: [/var/lib/containers/storage/overlay/160601fcbeefc18ee81d0c7f6eb1441d505559030088ac0f615f82da9ad436a4/diff /var/log/pods/openshift-ingress_e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l_05fceb9c-4a04-470f-a94a-055121319f82/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.035477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.609170 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m57.381811651s: [/var/lib/containers/storage/overlay/233ddc6796fc9d42177ba3d89ce263361d459ea54af88fb400264c7f01dee413/diff /var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-c2m84_4a35329c-82fb-479f-abca-bff9ed082558/manager/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.035477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.609809 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m57.478715575s: [/var/lib/containers/storage/overlay/b2bcbf7f5b1d1d97e274441018168e42a9e8779416c966ffc0c40bc5575414a0/diff /var/log/pods/openshift-ingress_e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn_08b3a993-1a64-4fe5-9577-b0f11a36779e/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.035477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.609848 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m57.46415776s: [/var/lib/containers/storage/overlay/962bdb1ea9f0fbdf7d9766a1032960fd9a27c344be31f462484a7ef5facfd38c/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-vznnm_171c7288-3157-4fe0-80ee-94cf5b2ca2dd/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.035677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.609871 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m57.452931562s: [/var/lib/containers/storage/overlay/1d7c6bc86a06ce9ae161b91fcbef9d4526c125d63fcd9993f06aa5a3d6b77636/diff /var/log/pods/openshift-ingress_e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd_f88ae6f3-f1d6-4aa3-ad33-589490696167/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.035677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.609892 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m57.453414017s: [/var/lib/containers/storage/overlay/ff68119ff6c1ec3b8d8999fce0f9c89e61ff03e3d3dac3235815761191644c7b/diff /var/log/pods/openshift-ingress_e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx_2ed21c5c-ef42-4fb0-bf41-34e84b9cf193/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.035677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.609913 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2m23.577366711s: [/var/lib/containers/storage/overlay/428935c09ddf323d498310f18396171954216064977823c19d83c4df7e2d6b7f/diff /var/log/pods/e2e-models-e2e-worker-w2-2b5703_distinct-w2-2b5703-kserve-676bd58bd6-qhvz4_1a57d851-12b3-4535-b12c-b7d2c579d0fb/main/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.035677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.609935 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m37.401053609s: [/var/lib/containers/storage/overlay/3474501ae08013e59580c990423bd84e932c126eeb6012f33f01041d02df7c01/diff /var/log/pods/openshift-monitoring_node-exporter-8s8ss_cc202826-dc4c-4f05-b4bc-63fc108188c2/node-exporter/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.035863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.610397 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m57.382238554s: [/var/lib/containers/storage/overlay/b8c70cc7c1cd703826834e8d483928dd61ec449b277430ef89f32568e47f3d3f/diff /var/log/pods/openshift-ingress_e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf_8adff4a5-5376-43ec-9b53-888705ae8236/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.035863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.610471 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.228923958s: [/var/lib/containers/storage/overlay/7ee751a49452db10e44d99c3d16e1705262f035def3da602cdb5f0ec0d850e03/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.035863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.610510 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.228961539s: [/var/lib/containers/storage/overlay/f2c7f00aa454c8cc582a4c36e607079857c142026b24edfe46a8cd2d4529c10f/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.035863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.610546 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.22899539s: [/var/lib/containers/storage/overlay/3664b495f860b8b12a14f83f6f7268d45d1d6788d533b83f182667924c5c86e2/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.035863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.610580 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.229027358s: [/var/lib/containers/storage/overlay/7234d3b3796fcc180309f9d1e93dcbb1cda70514c5cbe8e2f97a929556ed2eee/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.035863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.610647 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.229092771s: [/var/lib/containers/storage/overlay/ca9a0fb1928ab3bba4b110e4d32cb6c316afa494aa3d037d2c76980cff6cf009/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.036196 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.610707 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.229151521s: [/var/lib/containers/storage/overlay/94455cc20d0b1129824f7cb5c5a094f97f0a608b14545bdc84604cc0fdbe3db9/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.036196 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.610740 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.229183296s: [/var/lib/containers/storage/overlay/9a840ab2681e15e51c6020aa6a5097916390aa67e7392c8ee7473775da518ee8/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.036196 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.610811 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.229250147s: [/var/lib/containers/storage/overlay/07af3ff7c020d195ba5fa404ed7a2979a52805c7facc79393490756afd6b625d/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.036196 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.610844 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.229282528s: [/var/lib/containers/storage/overlay/0a728ec770e6ffa3c0d8df9535746dbcb124a23ff2fea94ab2fc314ec32ae537/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.036196 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.610877 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.229297926s: [/var/lib/containers/storage/overlay/d1b34dee7bb386b4b08b6201e76895f6f53c61c71d5047b48d1b40a81fdc836c/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.036196 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.610943 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.229348714s: [/var/lib/containers/storage/overlay/65a7f2743a6baf66f9e9795694b13722c929a35c96e6ad652f638f9899e25507/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.036809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.610992 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.229397661s: [/var/lib/containers/storage/overlay/91fa9618cfa590d0973a3622557914d662d549de7b2bf74b03586221a48abc66/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.036809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.611024 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.229427057s: [/var/lib/containers/storage/overlay/9b5b445cad281296a69a13715d23a258a00871cd7a404857597c880c7c344ef2/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.036809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.611057 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.229457794s: [/var/lib/containers/storage/overlay/05b98974003a4181cc525d06dd6951339b1182ba0e8da0ec7cc1521a072638d0/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.036809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.611128 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.229526589s: [/var/lib/containers/storage/overlay/83f2877cc6d21618287aabda5fe219114908303ccc1e352fde2d18088555b36f/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.036809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.611163 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.229559757s: [/var/lib/containers/storage/overlay/0ff717d5140c034ea683dfff3cefbceae3d1da30d032121ac374b7822fbfed17/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.036809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.611195 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.229590671s: [/var/lib/containers/storage/overlay/990167ec6f6877b6979123d6c2939438f468234423aa7808f21a809920da5fc8/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.037076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.611227 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.229619734s: [/var/lib/containers/storage/overlay/30867ac3c4283948de88e14428fc7c9ac0901d779c82fdb4b0222913e208a1d6/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.037076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.611642 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.229057332s: [/var/lib/containers/storage/overlay/10510a6cfdd969bf4f3ffee52a0d729e68c4fb0387c5264c3cbe862a8fb57585/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.037076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.611958 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.229352904s: [/var/lib/containers/storage/overlay/bc363165f9f21ea338e9ed2bfdedf1f3550f2bcb4ee5f479d635338ae290bcb4/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.037076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.611987 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m57.453986131s: [/var/lib/containers/storage/overlay/3ae1666a70fe1ee29ea3032c4280f1c66a3efccb208906869f521f4554b08316/diff /var/log/pods/openshift-ingress_e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r_ade5b9f4-d822-4418-a31b-f0c0a21ddb0c/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.037076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.612008 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m59.046124528s: [/var/lib/containers/storage/overlay/057778d1fef160a784176c21333cc7ddb765ca129ee233239cde8b2204224102/diff /var/log/pods/e2e-models-e2e-worker-w4-3fbb1a_simulator-w4-3fbb1a-kserve-7868585bbb-4892w_20690fd2-7022-4657-8c8c-bd738493ffc0/main/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.037076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.612874 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 20:32:51.037506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.612964 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 20:32:51.037506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.612989 5059 trace.go:236] Trace[698753972]: "Reflector WatchList" name:object-"openshift-cluster-node-tuning-operator"/"tuned-dockercfg-bkd9f" (03-Nov-2026 20:28:01.655) (total time: 48957ms): Nov 03 20:32:51.037506 ip-10-0-132-83 kubenswrapper[5059]: Trace[698753972]: ---"Objects streamed" count:1 48957ms (20:28:50.612) Nov 03 20:32:51.037506 ip-10-0-132-83 kubenswrapper[5059]: Trace[698753972]: [48.957090893s] [48.957090893s] END Nov 03 20:32:51.037506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.613031 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bkd9f\"" Nov 03 20:32:51.037506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.613089 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-b503a8-openshift-default-dockercfg-tchtj\"" Nov 03 20:32:51.037506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.613195 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 20:32:51.037506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.613261 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0e1546\"" Nov 03 20:32:51.037506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.613569 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:32:51.037506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.613699 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 20:32:51.037506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.613750 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xbkg2\"" Nov 03 20:32:51.037506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.613845 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a0d4ec-dockercfg-d47xx\"" Nov 03 20:32:51.037506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.613825 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 20:32:51.038618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.613955 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rs44g\"" Nov 03 20:32:51.038618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.614037 5059 trace.go:236] Trace[235073075]: "Reflector WatchList" name:object-"openshift-console-operator"/"openshift-service-ca.crt" (03-Nov-2026 20:28:01.575) (total time: 49038ms): Nov 03 20:32:51.038618 ip-10-0-132-83 kubenswrapper[5059]: Trace[235073075]: ---"Objects streamed" count:1 49038ms (20:28:50.614) Nov 03 20:32:51.038618 ip-10-0-132-83 kubenswrapper[5059]: Trace[235073075]: [49.038999162s] [49.038999162s] END Nov 03 20:32:51.038618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.614048 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:32:51.038618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.614205 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 20:32:51.038618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.614286 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 20:32:51.038618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.614374 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:32:51.038618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.614443 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m56.658980476s: [/var/lib/containers/storage/overlay/e730290aa13437c09e45b8f08838fe214bb40ce68dfc579337e5198f7056d30e/diff /var/log/pods/openshift-ingress_e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt_d912f1fd-b76b-4bb6-a5d9-ccc8b4536840/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.038618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.614464 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:32:51.039110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.614485 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m56.654129215s: [/var/lib/containers/storage/overlay/4d1d763bdcd2be1ec8c6a0b81723c1837cc2c1780f6677d73989b2c1133b7e69/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44_26375297-81f4-427f-94c3-d56f7a6a11fb/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.039110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.614512 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m35.062607663s: [/var/lib/containers/storage/overlay/13a0492b9fae19a6960dac59694e25302a8507425c7946f0778b7c1e26af65ef/diff /var/log/pods/openshift-ingress_e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d_40b88c7d-08e9-4203-9ebd-2c9f69b884dc/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.039110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.614530 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 20:32:51.039110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.614534 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m35.005209437s: [/var/lib/containers/storage/overlay/77be1a78d40f6b27ca2dc53b367cb858613fc53efb5d6d6ca3f4ac6646a0e853/diff /var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-tf75l_cde430c2-2518-49a0-9851-42356d3f2fe9/main/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.039110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.614601 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m56.654263999s: [/var/lib/containers/storage/overlay/0f417ef24c6af7f731bfdb4a657261136f1640a2fce34a1f97bdd82370d8fd37/diff /var/log/pods/openshift-ingress_payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8_705dea2b-ad85-4b1c-a3c6-a9f705ac9039/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.039585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.614630 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m56.652865735s: [/var/lib/containers/storage/overlay/6fe7ca5b5fed409629d0a8bb564f4f835c60bfdb5e674c50db47461f210e2eb7/diff /var/log/pods/openshift-ingress_e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5_ab8320f7-2354-433a-a8a7-4d91cc5d37d2/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.039585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.614643 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-bfc240-openshift-default-dockercfg-dxqxq\"" Nov 03 20:32:51.039585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.614700 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m56.648947606s: [/var/lib/containers/storage/overlay/4a676e520264ef3cc5f5220c041fab55a3740033f9d9f3f0c382bc8105f63576/diff /var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c557ztrnm_37e90ab0-e9c0-420c-a0f4-c7923aa00297/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.039585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.614745 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 20:32:51.039585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.614795 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m59.640788355s: [/var/lib/containers/storage/overlay/8cd4679b81ead040c961217792e0816971c04f0c13ae609dea6a2489efcc2ff8/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7_2c363281-0226-420f-a963-5922d5ca609e/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.039807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.614822 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m56.653054657s: [/var/lib/containers/storage/overlay/e18a17f3c6667965b57f8915cf1b24098b5fbfa576766528a9c24e90c047017a/diff /var/log/pods/openshift-ingress_e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg_518169ac-2f8d-4645-ae47-210b67ca080a/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.039807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.614836 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 20:32:51.039807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.614921 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-942930-openshift-default-dockercfg-vwmg4\"" Nov 03 20:32:51.039807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.615000 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 20:32:51.039807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.615068 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 20:32:51.039807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.615160 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2r65v\"" Nov 03 20:32:51.039807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.615249 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a0d4ec\"" Nov 03 20:32:51.039807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.616844 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"routing-overlay\"" Nov 03 20:32:51.039807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.617009 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 20:32:51.039807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.617149 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-3fbb1a\"" Nov 03 20:32:51.040176 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.623237 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m57.379998288s: [/var/lib/containers/storage/overlay/8ea54aa04516a03f1047de79fbfbd116b50ffbdc25a62c78219aeec1b7a7cc9f/diff /var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4_70d45aa9-bfa8-43c6-8674-6f8f14a3097e/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.040176 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.623269 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m57.212150586s: [/var/lib/containers/storage/overlay/548b56c6b19ef7cd87b984ab1c6b190ae06edeb55cfce7bb12e0610e4c31569f/diff /var/log/pods/openshift-ingress_e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95_c2d80995-5f1c-46db-8d71-a130cb92e279/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.040176 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.623293 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2m31.566869738s: [/var/lib/containers/storage/overlay/4909810f6862308c7b8e948b9acdcc7c0d05e34ac5ad3eb68a2814d8552abed2/diff /var/log/pods/openshift-ingress_e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w_1731953b-1850-40d1-9ba4-cb08dc4ebc5d/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.040176 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.623339 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m57.397436536s: [/var/lib/containers/storage/overlay/70930d38077b10a20b0ce4df30e17d1da1c20ec8bcdce6a5603b005d7a575af2/diff /var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-846mt_04a982f9-143c-406c-99d5-9ef73d5c571a/networking-console-plugin/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.040176 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.623515 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m29.640597757s: [/var/lib/containers/storage/overlay/b0d1e39d17ccc7a902eeab6ffca28bdff0fd8d7dc91746f24dd421fffbc289b0/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.040372 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.628274 5059 ???:1] "http: TLS handshake error from 10.0.133.249:60830: write tcp 10.0.132.83:10250->10.0.133.249:60830: write: broken pipe" Nov 03 20:32:51.040372 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.630847 5059 trace.go:236] Trace[1087853246]: "Reflector WatchList" name:object-"openshift-ingress"/"default-ingress-cert" (03-Nov-2026 20:28:02.207) (total time: 48423ms): Nov 03 20:32:51.040372 ip-10-0-132-83 kubenswrapper[5059]: Trace[1087853246]: ---"Objects streamed" count:1 48423ms (20:28:50.630) Nov 03 20:32:51.040372 ip-10-0-132-83 kubenswrapper[5059]: Trace[1087853246]: [48.423467361s] [48.423467361s] END Nov 03 20:32:51.040372 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.630864 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 20:32:51.040372 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.631018 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e7b70d-serving-cert\"" Nov 03 20:32:51.040372 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.631084 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:32:51.040372 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.631214 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-lw7w7\"" Nov 03 20:32:51.040372 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.631328 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 20:32:51.040372 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.631422 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-43c674-openshift-default-dockercfg-slz8b\"" Nov 03 20:32:51.040372 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.638268 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 20:32:51.040372 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:28:50.641259 5059 log.go:32] "Status from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" Nov 03 20:32:51.041014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.705952 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.118650004s: [/var/lib/containers/storage/overlay/d2e80b4794268fec4a961ae72b980628d6cfeab77a767fc85c67ad64630354e0/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.041014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.706001 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.118686912s: [/var/lib/containers/storage/overlay/85c1fb9d4ff88cba050b8812344293d7e1bf6bc385014c69d5d4be0f8fd22664/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.041014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.706037 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 53.961121843s: [/var/lib/containers/storage/overlay/9b58e66893e846684294552f052e4a90fdd8746f4b3637827c3ccb0e52c95e78/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.041014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.706070 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 53.961152034s: [/var/lib/containers/storage/overlay/8c48fd187f175613a5793fa62d578c4b3a935ca8ba2d18903e3169b48d6de89d/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.041014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.706103 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 53.961183654s: [/var/lib/containers/storage/overlay/dca21d2cdbf4ac9467b499e27e9aa76419bff93f79dd7e293c13870cfd22c2f6/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.041014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.706142 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 53.961220807s: [/var/lib/containers/storage/overlay/d839857b75b0b46e2db3536ada903cde1fd0faabe7319f8b0b4b8f5d82ec5583/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.041014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.706180 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 53.961258221s: [/var/lib/containers/storage/overlay/560bd35915a3e4ed3a92ebec5a37ee6c8c2b5d602e52db698c006c8f6d871c49/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.041431 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.706217 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 53.941898191s: [/var/lib/containers/storage/overlay/1cf8faf81515e3252d55ad5fa3a6f01aade4cbcfc250d0b89f7f4abe3f998eb5/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.041431 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.706368 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 56.944613383s: [/var/lib/containers/storage/overlay/ba408307c7920c24e1b289a6ddecb3cf2b8cf03ea2ca16d87c7f5ff7b4e0d72f/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.041431 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.706534 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 57.061900331s: [/var/lib/containers/storage/overlay/ecf8c759fcf916418871e8a8350ee9535c279c4497742cc2efb82eb693743add/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.041431 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.706557 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m57.549849099s: [/var/lib/containers/storage/overlay/faed3101d2fbf6ec02b09ee279cb2ef2e02bff6c99401bdbf2f7da9c4678b55c/diff /var/log/pods/openshift-ingress_e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk_8b648c6b-1f7d-42ca-9c3f-a435494bad03/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.041431 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.706581 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.512033893s: [/var/lib/containers/storage/overlay/de6ca20b9377c28cabe4038e64abbe6a95d1dd277c5ca5ddfa3af5b8c5e76c54/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-gb8r8_e64f9d9f-c868-470e-bd78-4ce8baabd97a/csi-liveness-probe/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.041925 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.706601 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.51205067s: [/var/lib/containers/storage/overlay/421c143b7c80827d2cc5a062879acfb554cf4fb285709dba5f301636b77ba0e8/diff /var/log/pods/kube-system_global-pull-secret-syncer-wh9tj_116af81a-d3a0-4558-93c4-8bf061b6815f/global-pull-secret-syncer/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.041925 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.706620 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.510042277s: [/var/lib/containers/storage/overlay/3590b0870ebbb7484f75a61269deee6b5456d7e78326def3787ba771f3783e4d/diff /var/log/pods/openshift-multus_multus-xlfb8_c6025fec-2ce4-4638-9a74-1f58c7ba7e8b/kube-multus/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.041925 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.706640 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.509161345s: [/var/lib/containers/storage/overlay/35affdd196fb11150949ff43876d496060e5efae610c103523fe68d5ceac4faa/diff /var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-x8zts_633c352c-f07e-4032-b962-23a148297e45/postgres/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.041925 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.706672 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.507497128s: [/var/lib/containers/storage/overlay/aa85e2b57d88bff2e55c442f79cb18559d789aa60ffc4f6bae25b4627a4c09d3/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt_c92b1be0-0890-4db6-ac75-968b6e5a3434/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.042101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.706693 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.507529198s: [/var/lib/containers/storage/overlay/0610b8954c6e6648affb0dd0874ae4c15a85f66db8f91e1f59381d694d28fbe6/diff /var/log/pods/openshift-insights_insights-operator-7c4986d449-wdls7_40834bff-2db0-4690-9cb9-7b4aacdab3ab/insights-operator/1.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.042101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.706714 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.505862147s: [/var/lib/containers/storage/overlay/76011fc3cc8cd31792e1a47889733536bd5e73bb3d979dccf9abc68fee1315e0/diff /var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-lf9vc_677a5153-5fe3-4624-bf06-3c8dfc6bc72d/discovery/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.042101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.706734 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.502501611s: [/var/lib/containers/storage/overlay/25fd98fb3d4fc0e795fcacdfc638e10cf768ef8545540e30d90b8efdb375bec1/diff /var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-qchj6_ad491ce3-2f92-45a7-9b4f-278309989e64/prometheus-operator-admission-webhook/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.042101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.706758 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.498648047s: [/var/lib/containers/storage/overlay/97db272a4e9d602f960815093200513ae8212138a7aa60545a5d09e979bd3e88/diff /var/log/pods/openshift-ingress_e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8_4c609469-d7d7-4530-8d74-e34aa56dda8f/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.042293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.706778 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.49401352s: [/var/lib/containers/storage/overlay/83bd2190b8aa145be528c41fb67adba5f21d9d3f828fd3c580d14625b15ec8c9/diff /var/log/pods/kuadrant-system_authorino-66b59d468d-5bfd2_c7fce352-1850-4048-be2e-fa458fd4d2f3/authorino/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.042293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.706798 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.491857899s: [/var/lib/containers/storage/overlay/2def3de689f3697117c5b2b888ee4f0f0714c497d59d71bf5ba44e80b2884cdd/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-g6xz7_0f32ecf7-06f7-4098-9793-acf761a30c09/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.042293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.706818 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m35.00925482s: [/var/lib/containers/storage/overlay/644b60c0f8b58a334a1a7fc4a6319153055998acb8143ecbe17f7375356566c1/diff /var/log/pods/openshift-insights_insights-runtime-extractor-hmnx4_377ffcf5-b097-495d-b986-e61c58ae8ec1/extractor/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.042293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.706840 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.480900112s: [/var/lib/containers/storage/overlay/772b2fa301202b16036e5a2f97de9948e4d62690b1312c6009f39270bee18938/diff /var/log/pods/openshift-ingress_e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f_14a06bdf-e15d-406c-bb76-47ae5dd06b2d/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.042524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.706861 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.533393143s: [/var/lib/containers/storage/overlay/5cc649e7a330cae75c81c80a41964c2c3c6ce9688dad07d03ada6fd24f317291/diff /var/log/pods/openshift-ingress_e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42_a594e899-889e-4f7f-a8ba-b7e7a389ba8a/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.042524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.707576 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 4.502453449s: [/var/lib/containers/storage/overlay/063b6ab855c8ff15074e39c756449c29ce35cfb5d4ffc77409f59ad34e12ea31/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.042524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.707604 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.475103253s: [/var/lib/containers/storage/overlay/db08eff9b22a4500cec0d1680cd6484dadd3fdd3de35cc7f22a12a0590ccdc68/diff /var/log/pods/openshift-dns_node-resolver-4d4ll_94d082d5-1c33-4394-9d93-67645f9b60cd/dns-node-resolver/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.042524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.707653 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.475130597s: [/var/lib/containers/storage/overlay/b6cd9e4f4ab5200d0481e1c96d9431f1e12949004fd0870f5eab0a08be25c4ee/diff /var/log/pods/openshift-ingress-canary_ingress-canary-fgdh8_18ffe451-b8eb-41ad-a19d-0390f3e1f8e2/serve-healthcheck-canary/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.042524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.707696 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.475166284s: [/var/lib/containers/storage/overlay/f26445c31409eb167bf27cc2cbab2017408dbd457f03729e67cd2f30f1ea53f0/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2m4mq_c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/northd/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.049013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.707739 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.475203495s: [/var/lib/containers/storage/overlay/ccf806ac6ec23ad74bcc74db02ca5f266c08af5a5088b87059866bfd5ad3a508/diff /var/log/pods/openshift-multus_multus-additional-cni-plugins-cxdrc_b6f0d931-3a92-424f-a5fc-d9ebef985dfd/kube-multus-additional-cni-plugins/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.049013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.707765 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.475227857s: [/var/lib/containers/storage/overlay/29312326d1c5d67a009d3f2a0ead3df745c17fd18ca61679cf5a8d648501ee1c/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2m4mq_c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/sbdb/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.049013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.707843 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.773641165s: [/var/lib/containers/storage/overlay/69d000787c06a185865f58066c28dd8c5a083b56b5987b22d50bb9ddd8b80af3/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.049013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.710206 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.672810846s: [/var/lib/containers/storage/overlay/4e5492bb5eff4ed1d55fe1a44966920b4e9fe42aa08f0f1abe51ffbc8a7e7223/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.049013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.710256 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.672864721s: [/var/lib/containers/storage/overlay/b3721a6a9f08643666ccb64e9abff935757945728992436081afdf324727651a/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.049244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.710418 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.477879164s: [/var/lib/containers/storage/overlay/c37ca7a1d4a292f83c6190b9174a510fc5046ebdb1f5f1453a9ab145c3e6857a/diff /var/log/pods/openshift-service-ca_service-ca-5fffc97bdf-brvgz_7b8992fc-fba6-4135-90f8-fb259e765dd5/service-ca-controller/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.049244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.710441 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.477901736s: [/var/lib/containers/storage/overlay/5630a22ba1b338988a20699748117abbe435074b2a4d9dcff245e43eb9767391/diff /var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal_ea3063cb5f7f736e0060bab60ff36a5e/haproxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.049244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.710460 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.477916341s: [/var/lib/containers/storage/overlay/d3141a050734ea30444cd519454c37c2cee0e793a05fad7726c15fd6b8e5ce74/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2m4mq_c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/ovnkube-controller/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.049244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.710484 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.475266001s: [/var/lib/containers/storage/overlay/eea3e87288386c0194020c711ea81acb99383e2d30a4286346e27933ee057561/diff /var/log/pods/openshift-multus_network-metrics-daemon-qvxjq_caf7d6c9-0bb2-4da0-be2d-caea7734bbcb/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.049415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.710547 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.475324753s: [/var/lib/containers/storage/overlay/46b70b8d6a52bbe639d1b79f0e47e999bf5c479db6750a707367f79d848146ce/diff /var/log/pods/openshift-dns_dns-default-nhf67_419841be-7a31-4cdb-b0b7-9d90a3c376c7/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.049415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.710567 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.474436009s: [/var/lib/containers/storage/overlay/9df3bda628332b9dc87ebcd7ad96757f67125cc8d0ed149f1b9cf8ae1a6abd67/diff /var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-86qxx_40b2621f-854e-44e0-9b01-c877e333f644/cluster-samples-operator/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.049415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.710589 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.47445445s: [/var/lib/containers/storage/overlay/99057c763637840ee3d311f80feb2ce08c68c4bd388fa3db08885cc03330a746/diff /var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-86qxx_40b2621f-854e-44e0-9b01-c877e333f644/cluster-samples-operator-watch/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.049415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.710608 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.474463321s: [/var/lib/containers/storage/overlay/06a0ba1cbe573083e0a7d84264e2f1b125b9836c39424aaa1ca550774ca8462b/diff /var/log/pods/e2e-models-e2e-worker-w3-9f46e5_simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22_f14445d5-52ed-487d-972c-4499966065fa/main/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.049415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.710643 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.205012064s: [/var/lib/containers/storage/overlay/18543d407168eb4576353392a1afd991f157ee565b131d04360e902c8407fdd2/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.049626 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711095 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.459439654s: [/var/lib/containers/storage/overlay/7d118d46ddd9118020120bf4abeda717a9745f884905c6c04f3bec41139d79e2/diff /var/log/pods/openshift-cluster-node-tuning-operator_tuned-mkm7n_53b14f9a-431e-4dbb-a725-4c8470262ca7/tuned/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.049626 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711120 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.459464963s: [/var/lib/containers/storage/overlay/b85378590d3a8b51c3720c14d24215dbda1e35f20453ad77550072c510cf3714/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2m4mq_c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/ovn-controller/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.049626 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711142 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.459466851s: [/var/lib/containers/storage/overlay/5b3629ccac63294c67978800bc554ad8685d0a4172b11f35016df65821206978/diff /var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-6k9n9_f485893c-6f14-449b-b837-8ac181315aeb/migrator/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.049626 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711197 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.459485095s: [/var/lib/containers/storage/overlay/18bc1e0ac5f089f3489772fb39b4cc72a12ceea81b05d60570e379b8672727f2/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-gb8r8_e64f9d9f-c868-470e-bd78-4ce8baabd97a/csi-driver/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.049834 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711222 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.516718961s: [/var/lib/containers/storage/overlay/7fcc2c309088f71f15e3de261c00b834cb912549820428227aa73c7f19b19398/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2m4mq_c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/kube-rbac-proxy-node/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.049834 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711242 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.459543328s: [/var/lib/containers/storage/overlay/cbb879871a44605c610147751343b85dedca5ecbe37123acd39e2fd41eedaa35/diff /var/log/pods/kube-system_kas-connection-checker-685895c7c9-6hx8t_f9d935b2-aa51-474c-836d-a74f91efab07/connection-checker/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.049834 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711262 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.459560808s: [/var/lib/containers/storage/overlay/ee7f02d9be1fdeb93909fdbcfaf9d4038d7c16d9e26b087b8d5ace145804b85a/diff /var/log/pods/openshift-network-diagnostics_network-check-target-b4x9q_c95249a6-899f-4faf-82dc-6e553453ff80/network-check-target-container/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.049834 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711285 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.51677817s: [/var/lib/containers/storage/overlay/395fbf4d5d4ec8a134ec8da50084515d06500fbb092d7ed16790f32d60204633/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2m4mq_c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/ovn-acl-logging/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.050005 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711306 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.459602904s: [/var/lib/containers/storage/overlay/af2813e3cc345e41fada3e12c6cc0bafde54c03ea986315db35b5d8e8606a959/diff /var/log/pods/kube-system_konnectivity-agent-q4zkg_4fdbd17a-1e15-4442-b715-aaec1087da11/konnectivity-agent/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.050005 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711361 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.456746224s: [/var/lib/containers/storage/overlay/67643f6bfc43ab27acce59d3f46ca4439902175e226d9e26fa2b0544a356d3a7/diff /var/log/pods/kube-system_kas-connection-checker-685895c7c9-jt54m_e2155512-2116-44f6-be5e-77cef72f9e14/connection-checker/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.050005 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711385 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.456727153s: [/var/lib/containers/storage/overlay/4b8288f2e07855b92c0673979bbc56f16b95f104d7f3d331523a2b249fd005be/diff /var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal_3e0047dbb7ee8d4e56f23e3bd51214c3/kube-rbac-proxy-crio/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.050005 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711406 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.456475724s: [/var/lib/containers/storage/overlay/eaf2894f871c7923b917e5b3aeb7e3f05b762c5775c3e7f9dc2c9ff74861f681/diff /var/log/pods/openshift-service-ca-operator_service-ca-operator-544d749c8c-xzpgl_c99b0c19-004e-450c-852a-e38338635b61/service-ca-operator/1.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.050183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711426 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.456491677s: [/var/lib/containers/storage/overlay/49a371edd202683666dd0b270130376ae59bf99c2a0df270b51976ef77c62c30/diff /var/log/pods/openshift-insights_insights-runtime-extractor-hmnx4_377ffcf5-b097-495d-b986-e61c58ae8ec1/exporter/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.050183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711446 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.787732717s: [/var/lib/containers/storage/overlay/42df31a418b95b1237dc0b8536ace06c25a4eac010478e46aeba8d03b09ef4b2/diff /var/log/pods/openshift-ingress_e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj_c9721811-d27d-4e13-a53d-4ae81e16fb96/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.050183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711467 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.459764322s: [/var/lib/containers/storage/overlay/d68139bb7d74a6179bfbf30966bab9231b936b5fed801b743293927edf9117a4/diff /var/log/pods/openshift-image-registry_node-ca-kzr2l_2398ba99-35d3-4461-b66e-f44ac013b2d2/node-ca/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.050183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711523 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.455969904s: [/var/lib/containers/storage/overlay/aae27b6ee5fc90d09200b2fca469bcc5b7e2572a8f74d6ee3fb9fc3e123281c7/diff /var/log/pods/openshift-image-registry_image-registry-6c57db574-fmm6v_3265bd58-d654-467c-9b2a-14cedc27b754/registry/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.050358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711547 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m37.583506906s: [/var/lib/containers/storage/overlay/06e27953833e922dd29fdfb0190b6691a452f9aecad198b52de016a37c8e197e/diff /var/log/pods/openshift-monitoring_node-exporter-8s8ss_cc202826-dc4c-4f05-b4bc-63fc108188c2/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.050358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711567 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.456638581s: [/var/lib/containers/storage/overlay/00030eca2f61227baeb170bb925aa2236cdf3480d33657979fdb973e7e9e694d/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2_cfcb49c5-4381-411e-9d64-d9dcd6181239/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.050358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711587 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.452993639s: [/var/lib/containers/storage/overlay/0a7d2b6c184bbc9ea7aca802195d9a55372400e1cc4f3cf22c0893256c58b9e4/diff /var/log/pods/openshift-ingress_e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6_eac98813-7b4c-4c4a-8bf9-444bba2fa644/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.050358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711606 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.371673888s: [/var/lib/containers/storage/overlay/84167313719d1991fbbadc24e0c05628d697e9c890ed2e2ddafa01fcce5c18e9/diff /var/log/pods/ai-tenant-e2e-extmod-b-0e4ce7_payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2_0309a608-f6c4-45ca-991b-33ba73221a6c/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.050531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711631 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.461376815s: [/var/lib/containers/storage/overlay/b287206ebf5abf02de46d1dfecb037e038a60773df298de9394db3ac0ca2306b/diff /var/log/pods/openshift-ingress_e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj_d3de18b3-c3d1-4ec3-961f-d95ca3656f4b/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.050531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711717 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.330106867s: [/var/lib/containers/storage/overlay/89ca02667dcab424b5177bc62aab53cbd5ac6a1d58eee3167b0960fcfc2cafb7/diff /var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/2.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.050531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711741 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.460054978s: [/var/lib/containers/storage/overlay/67b680590f5eefdd8ad88492c01e8616609706fb8ff8b7571351b1b139c2be6d/diff /var/log/pods/openshift-network-diagnostics_network-check-source-6f78d59f99-tvb96_0a985b50-c4c0-4a39-abed-29f53e2be747/check-endpoints/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.050531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711761 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.531575609s: [/var/lib/containers/storage/overlay/108b016c7af9497826514874c593f020e6d61efe256b2502992c3661bb6d96ff/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9_890f130b-cdeb-43da-acd4-1654673271cc/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.050722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711782 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.539789767s: [/var/lib/containers/storage/overlay/8d9a1b5f5af6fb294a385b5c40459d1d5a37a514e98a1e388be257b935b02e6a/diff /var/log/pods/openshift-ingress_e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x_7909b42c-1572-4fe2-816a-447020594fe4/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.050722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711804 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.748238406s: [/var/lib/containers/storage/overlay/61b109d2e47522fb73054e0ea35fe258491cc3975ae8ebfcce6b66801ef946db/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2_4ea6e69e-3556-4ffe-bb9e-796e8795d747/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.050722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711823 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.531188533s: [/var/lib/containers/storage/overlay/129473a8ae2b6e6cb76eb14fb406bdbf019ed61e93c41aac613910bbd58651eb/diff /var/log/pods/openshift-ingress_e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h_fc57130b-c950-49f2-8a98-22c47bd98288/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.050722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711882 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.649184132s: [/var/lib/containers/storage/overlay/65ccba7b8af5c70659046888199f0ac866f2edde7cc1d777af004dd86605e1b9/diff /var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-86qxx_40b2621f-854e-44e0-9b01-c877e333f644/d8b3f14031a53eecbdb2db39f15186180f1b7059e2868a65f8b7a4fe42d2380a.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.050889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711906 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.468271059s: [/var/lib/containers/storage/overlay/0cf3a6fbf2a9bc19ba19b0638dd47eabc50ce49bea682a54d2c9f0fc3e9857bb/diff /var/log/pods/openshift-ingress_e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl_b962b15a-12b8-4b36-bc1d-ce5ed903f1d3/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.050889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711925 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.649225099s: [/var/lib/containers/storage/overlay/a8e2771f9883e0cdaf40d4061a2f08e80e068684c3ef6c19bc9998b87ae3d8c3/diff /var/log/pods/openshift-insights_insights-runtime-extractor-hmnx4_377ffcf5-b097-495d-b986-e61c58ae8ec1/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.050889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711945 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m58.972869311s: [/var/lib/containers/storage/overlay/599ba7e3c05d74cc5b782e3f79128931abfe2e885836c5b495046f23b84a4e4a/diff /var/log/pods/openshift-ingress_e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd_9d60eb56-9852-4c25-ae50-8a932e752793/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.050889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711965 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.517455327s: [/var/lib/containers/storage/overlay/8d5a8bfb065b77d3516c9b12dc7440a9b213e9e14b732e71883de279b65cd3ce/diff /var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-pbgwk_815b15b6-2c37-4ca3-9b38-e7e1328d0aa8/migrator/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.051157 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.711984 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m30.532077378s: [/var/lib/containers/storage/overlay/0b869a1572abc120d48a24674560fbd62371975dce75976b1de34bf21a0f7324/diff /var/log/pods/openshift-console_downloads-6dd789c8b7-c88f8_6a7937ca-c705-4091-967e-2e640c622a26/download-server/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.051157 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.712043 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.517531126s: [/var/lib/containers/storage/overlay/3fe862ed3e308dc9fd551945d6f248a08da522305664f95336d9bb32796ea201/diff /var/log/pods/kube-system_kas-connection-checker-685895c7c9-bd8f8_0fc518fc-6302-4a1a-9ff9-f1e036a2199b/connection-checker/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.051157 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.712066 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.463660131s: [/var/lib/containers/storage/overlay/1f29dc544a15045e1dc5d30c93ff89b9d8dc147583a95169dc0867bf987d73b4/diff /var/log/pods/openshift-ingress_e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5_f0d3c590-f523-4b7e-9086-60df5fc585e8/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.051157 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.712085 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.329500281s: [/var/lib/containers/storage/overlay/8620f98241dd1d45a676340e68150fcebbdb656ee3ea2ca830fb8502bdcc7302/diff /var/log/pods/openshift-ingress_e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w_cef4417c-3d04-4d65-a5bc-6ec1af6f09bb/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.051348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.712108 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.66585347s: [/var/lib/containers/storage/overlay/6c0972a025a1d080f263d3e1efa9ee45ba0cf370ca3a4b39a456b1e8d775f4c7/diff /var/log/pods/openshift-ingress_e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk_edcfb7ab-b14f-4d5c-b752-5b43a6eee163/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.051348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.712129 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 53.943715295s: [/var/lib/containers/storage/overlay/ab6eeb0d8131876a3c4186f525187a9889b17f6b72c96426f7192132193bfaaa/diff /var/log/pods/openshift-dns_dns-default-nhf67_419841be-7a31-4cdb-b0b7-9d90a3c376c7/dns/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.051348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.712150 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 57.069942675s: [/var/lib/containers/storage/overlay/b44c3e3c39b2f4af23975cd957b5394aaa9ef8ef0b64adfdc2aa404a92d61503/diff /var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-wh95x_492c7b4e-84c0-49a7-b12f-153a021f971b/kube-storage-version-migrator-operator/1.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.051348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.712207 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.649507548s: [/var/lib/containers/storage/overlay/c1ac6b2ddb5ce33b84d769d5200a7b867b1dbeead0b830359507083657717be0/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq_0d960e4e-9539-4a53-9df7-1c520ab7d708/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.051523 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.712228 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 4.507109563s: [/var/lib/containers/storage/overlay/9edd96ae054cc1dc8e9dfc2a28e2169a5c4bcc43353e82ed8bf524e2c8467dbc/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2m4mq_c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/nbdb/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.051523 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.712248 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m29.713106636s: [/var/lib/containers/storage/overlay/525f162f804c3962fa3a18a74a23555a9451e6576687225a267ea143486f4ee8/diff /var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp_e7cf1267-7d06-459b-9df7-fc7c477d36e1/main/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.051523 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.712269 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.778063747s: [/var/lib/containers/storage/overlay/15de5c78c58cb7031b136a1c35a90045dd6aebec927308d49a43d91a20ef351d/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-gb8r8_e64f9d9f-c868-470e-bd78-4ce8baabd97a/csi-node-driver-registrar/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.051523 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.712289 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.778081437s: [/var/lib/containers/storage/overlay/4a77bbd76b8d6f139795f66a89d4209b4d85e3189b0d33316e94acef23c21ee8/diff /var/log/pods/openshift-network-operator_iptables-alerter-7zx7h_55f124eb-2ebf-4675-a08a-bad3757061b3/iptables-alerter/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.051892 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.712309 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.665869821s: [/var/lib/containers/storage/overlay/3e134a1fe674754c9e9a38acd15a24432bc657789a1cd146ef7749623da3c543/diff /var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-8464c976fd-9b8p5_6785e24f-6fb3-43ab-a469-84d30d94b2f0/manager/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.051892 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.712362 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.67497677s: [/var/lib/containers/storage/overlay/43201a78b713ec15496bb789b9294c6a0013d5c89dba58b4ae0edd1a0971be62/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2m4mq_c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/kube-rbac-proxy-ovn-metrics/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.051892 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.712386 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.674998327s: [/var/lib/containers/storage/overlay/285095d558a08566a705a3940ab4af0b45e7f143d3fb6d23370fc566c03184ba/diff /var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-fsbsp_1bcd4690-4753-4246-abb4-8668a0a56231/volume-data-source-validator/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.051892 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.712405 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.67501352s: [/var/lib/containers/storage/overlay/32e345b5d629dba2b5bd17e9c25f60a6c11de452ab6a193c6b6e03707507be02/diff /var/log/pods/openshift-multus_network-metrics-daemon-qvxjq_caf7d6c9-0bb2-4da0-be2d-caea7734bbcb/network-metrics-daemon/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.052077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.712425 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.66600927s: [/var/lib/containers/storage/overlay/dd8d2e028364b69cdce847e6e49ad546ee219955d9ccb8ac2291b09be3224a75/diff /var/log/pods/openshift-ingress_e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc_474b2e1d-832b-4edd-8765-31e0bc77cc2a/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.052077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.716293 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.334660154s: [/var/lib/containers/storage/overlay/98c6bfc9ab89d11a4ca2043d5cabcbe5116c9dff2217abac8fcdf7162650651c/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl_9c3d72b6-73c4-402f-8308-ee5d87a9dd83/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.052077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.716372 5059 trace.go:236] Trace[1619212384]: "iptables ChainExists" (03-Nov-2026 20:27:15.760) (total time: 94956ms): Nov 03 20:32:51.052077 ip-10-0-132-83 kubenswrapper[5059]: Trace[1619212384]: [1m34.956031018s] [1m34.956031018s] END Nov 03 20:32:51.052077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.730261 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.390769734s: [/var/lib/containers/storage/overlay/f254a34c2d7b07cdb8243cfef02ae48f7764efa44a95f01755e0ef12b15fb3dc/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj_06b78cd0-94df-40f9-a1ed-87c1751675d3/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.052077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.730293 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2m6.991032675s: [/var/lib/containers/storage/overlay/cc64e95826aceb44df14fcfcfbeb1abcee57869143f35bd62776a6422cfda66c/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-464r7_c8683a65-9898-4b4f-a692-f2da325a93b4/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.052331 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.738679 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.47606917s: [/var/lib/containers/storage/overlay/d209d2a4eaaa3bcefdb4f01ea0aa4775a1ed9635fee488a5dab9a0c0f5cea560/diff /var/log/pods/openshift-ingress_e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r_34880975-a897-4ad4-af59-18617a93cdad/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.052331 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.748172 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.486865318s: [/var/lib/containers/storage/overlay/a13dfa739c11295b8b17047cfd67cfcb736f6e317fc4fd8510561e20df9c398b/diff /var/log/pods/openshift-ingress_e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff_3b9d0847-df69-4b6a-b346-2cd206e42339/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.052331 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:28:50.778530 5059 webhook.go:269] Failed to make webhook authorizer request: webhook call failed: context canceled Nov 03 20:32:51.052331 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:28:50.778554 5059 server.go:358] "Authorization error" err="webhook call failed: context canceled" user="system:serviceaccount:openshift-monitoring:metrics-server" verb="get" resource="nodes" subresource="metrics" Nov 03 20:32:51.052331 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.867605 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 20:32:51.052331 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.868046 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 20:32:51.052331 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.868176 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 20:32:51.052331 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.868240 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 20:32:51.052331 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.868356 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 20:32:51.052714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.868454 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:32:51.052714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.868541 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 20:32:51.052714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.868600 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 20:32:51.052714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.868755 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-bcfde8-openshift-default-dockercfg-6xbwb\"" Nov 03 20:32:51.052714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.868862 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"default-dockercfg-4rsdq\"" Nov 03 20:32:51.052714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.868927 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 20:32:51.052714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.869067 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-65hnm\"" Nov 03 20:32:51.052714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.869148 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"kube-root-ca.crt\"" Nov 03 20:32:51.052714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.869239 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 20:32:51.052714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.869375 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"simulator-w1-32e482-kserve-self-signed-certs\"" Nov 03 20:32:51.052714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.869448 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-32e482\"" Nov 03 20:32:51.053150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.869533 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 20:32:51.053150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.872036 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 20:32:51.053150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.872182 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-847xh\"" Nov 03 20:32:51.053150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.872262 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 20:32:51.053150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.872367 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"distinct-w2-2b5703-kserve-self-signed-certs\"" Nov 03 20:32:51.053150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.872453 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 20:32:51.053150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.872514 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 20:32:51.053150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.872676 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 20:32:51.053150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.873235 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.939023544s: [/var/lib/containers/storage/overlay/7e28be4717606b4b52525a911bdb8c86f57f2856bebf297a1322355bafc3e847/diff /var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-d7wsq_5ac27a9c-459f-4310-aef7-8dc643b46c6c/main/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.053506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.894373 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 4.741188007s: [/var/lib/containers/storage/overlay/9f34caa2489d14df64cdc8294e5fd26b65b76f5be247c8fbf4f6e0e4303813eb/diff /var/log/pods/openshift-ingress_e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn_8153e018-7f9a-4645-b102-06ff3913f682/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.053506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.894559 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m26.826078375s: [/var/lib/containers/storage/overlay/cf20c89a59abe7a607a4c4fc3989a0700c8b52ef6b4b7ec8f4ce59b58a807569/diff /var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5_541bc63e-66f1-42ca-a946-aedc34beb62f/main/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.053506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.915198 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m29.262299266s: [/var/lib/containers/storage/overlay/ab20ddba4ce467229f22d33db5fafd5d048d7904e2d5f46fdda07463291e410d/diff /var/log/pods/openshift-ingress_e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g_5ea38154-8a1f-4e1f-8fa0-50481ba4fec0/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.053506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.915429 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.567480974s: [/var/lib/containers/storage/overlay/94e23a93257da876c38398186f8a6015998dba78499a0f10e1ac126481bbe14d/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-7krsk_4946e3aa-a961-415a-b71d-95164d5fbca3/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.053506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.916507 5059 trace.go:236] Trace[2062041910]: "Calculate volume metrics of tmp for pod openshift-console/downloads-6dd789c8b7-c88f8" (03-Nov-2026 20:27:16.181) (total time: 94734ms): Nov 03 20:32:51.053506 ip-10-0-132-83 kubenswrapper[5059]: Trace[2062041910]: [1m34.734332274s] [1m34.734332274s] END Nov 03 20:32:51.053783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.916592 5059 trace.go:236] Trace[1648236278]: "iptables ChainExists" (03-Nov-2026 20:27:16.232) (total time: 94684ms): Nov 03 20:32:51.053783 ip-10-0-132-83 kubenswrapper[5059]: Trace[1648236278]: [1m34.684553911s] [1m34.684553911s] END Nov 03 20:32:51.053783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.916973 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.679889227s: [/var/lib/containers/storage/overlay/2e395e81281e0705e1f09a9f9b1a516397cfddbdec2ec5dad6e0af28960240c3/diff /var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-slpzw_270c522c-ef72-46a7-9e65-3be2ebdbcfa6/manager/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.053783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:50.922500 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.988288003s: [/var/lib/containers/storage/overlay/78d5a25fd67befb86fd5ea2280d7175455ecdb53478522ad87a939cb0ef5c7c1/diff /var/log/pods/cert-manager-operator_cert-manager-operator-controller-manager-7d87bcd5fd-mm85p_532de14d-f2dd-4c66-ba6f-8b103f7c9375/cert-manager-operator/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.053783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:51.185081 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m34.122360389s: [/var/lib/containers/storage/overlay/8b2b6e7cad8ba0d31b01bad21f7c4e39df2debf268786e2befafef641bd5c6db/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p_ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.053783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:28:51.234279 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m35.00713275s: [/var/lib/containers/storage/overlay/f11eef46c0d6bd5328d511b7c3d8a2d7d49dbcbba4f1248321afe89be929c8f7/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.054036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:30:54.087621 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3m41.242522828s: [/var/lib/containers/storage/overlay/de16d7c33f60c35b85e789b6226417a11d8d0fe2e80e094daa63d5abb746ba38/diff /var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-l5g9c_64153a99-d5f7-4f92-a0c9-f79b39eba97c/discovery/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.054036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:30:54.144385 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3m33.145224756s: [/var/lib/containers/storage/overlay/acca74e545ee51a535769353a41c16a0a8524a86505ba1c9e4ec7337e4336414/diff /var/log/pods/openshift-ingress_payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps_2faa14fa-ec86-4633-a968-c3f761d13443/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.054036 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:30:55.008324 5059 log.go:32] "ListImages with filter from image service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" filter="" Nov 03 20:32:51.054036 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:30:55.008458 5059 kuberuntime_image.go:108] "Failed to list images" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" Nov 03 20:32:51.054036 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:30:55.008498 5059 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" Nov 03 20:32:51.054036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:30:55.204491 5059 trace.go:236] Trace[1273033396]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" (03-Nov-2026 20:28:50.722) (total time: 124482ms): Nov 03 20:32:51.054036 ip-10-0-132-83 kubenswrapper[5059]: Trace[1273033396]: [2m4.482111172s] [2m4.482111172s] END Nov 03 20:32:51.054337 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:30:55.219331 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 4m21.339807747s: [/var/lib/containers/storage/overlay/9b282c02d3b21d264d07e011c358a2c9a318e4f879a24e9b38ec0cd89449d657/diff /var/log/pods/openshift-jobset-operator_jobset-controller-manager-5d9f95b69-9vwq7_1072628f-f513-4b4e-8954-afee5a1aa733/manager/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.054337 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:30:55.232533 5059 log.go:32] "Version from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" Nov 03 20:32:51.054337 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:30:55.254256 5059 csi_client.go:182] "Error calling CSI NodeGetInfo()" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded while waiting for connections to become ready" Nov 03 20:32:51.054337 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:30:55.254304 5059 csi_node_updater.go:178] "Failed to update CSIDriver" err="failed to get NodeGetInfo from driver \"ebs.csi.aws.com\": rpc error: code = DeadlineExceeded desc = context deadline exceeded while waiting for connections to become ready" driver="ebs.csi.aws.com" Nov 03 20:32:51.054337 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:30:55.509973 5059 trace.go:236] Trace[1681876567]: "Calculate volume metrics of tmp-dir for pod openshift-dns/node-resolver-4d4ll" (03-Nov-2026 20:30:54.095) (total time: 1414ms): Nov 03 20:32:51.054337 ip-10-0-132-83 kubenswrapper[5059]: Trace[1681876567]: [1.414723503s] [1.414723503s] END Nov 03 20:32:51.054337 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:30:55.120606 5059 log.go:32] "ListContainers with filter from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" filter="" Nov 03 20:32:51.054337 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:30:55.534050 5059 kuberuntime_container.go:554] "ListContainers failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" Nov 03 20:32:51.088468 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:30:55.534265 5059 logging.go:55] [core] [Channel #6667 SubChannel #6668]grpc: addrConn.createTransport failed to connect to {Addr: "/var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock", ServerName: "localhost", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read unix @->/csi/csi.sock: use of closed network connection" Nov 03 20:32:51.088468 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:30:55.302692 5059 log.go:32] "ListContainers with filter from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" filter="" Nov 03 20:32:51.088468 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:30:55.534887 5059 kuberuntime_container.go:554] "ListContainers failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" Nov 03 20:32:51.088468 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:30:55.536318 5059 generic.go:254] "GenericPLEG: Unable to retrieve pods" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" Nov 03 20:32:51.088468 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:30:55.537229 5059 trace.go:236] Trace[107255895]: "Calculate volume metrics of snapshots for pod openshift-insights/insights-operator-7c4986d449-wdls7" (03-Nov-2026 20:30:54.238) (total time: 1298ms): Nov 03 20:32:51.088468 ip-10-0-132-83 kubenswrapper[5059]: Trace[107255895]: [1.298814556s] [1.298814556s] END Nov 03 20:32:51.088468 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:30:55.538022 5059 trace.go:236] Trace[1023194250]: "Calculate volume metrics of istio-envoy for pod openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" (03-Nov-2026 20:30:54.305) (total time: 1232ms): Nov 03 20:32:51.088468 ip-10-0-132-83 kubenswrapper[5059]: Trace[1023194250]: [1.232386866s] [1.232386866s] END Nov 03 20:32:51.088468 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:30:55.990798 5059 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeNotReady" Nov 03 20:32:51.089334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:30:55.990856 5059 setters.go:546] "Node became not ready" node="ip-10-0-132-83.us-east-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T20:30:55Z","lastTransitionTime":"2026-11-03T20:30:55Z","reason":"KubeletNotReady","message":"[container runtime is down, PLEG is not healthy: pleg was last seen active 4m39.47088269s ago; threshold is 3m0s]"} Nov 03 20:32:51.089334 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:30:55.119854 5059 log.go:32] "ListContainers with filter from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" filter="" Nov 03 20:32:51.089334 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:30:57.046846 5059 container_log_manager.go:154] "Failed to rotate container logs" err="failed to list containers: rpc error: code = DeadlineExceeded desc = context deadline exceeded" Nov 03 20:32:51.089334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.326477 5059 trace.go:236] Trace[1942787115]: "Calculate volume metrics of agent-certs for pod kube-system/konnectivity-agent-q4zkg" (03-Nov-2026 20:32:23.224) (total time: 22101ms): Nov 03 20:32:51.089334 ip-10-0-132-83 kubenswrapper[5059]: Trace[1942787115]: [22.101850721s] [22.101850721s] END Nov 03 20:32:51.089334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.327721 5059 trace.go:236] Trace[92360070]: "Calculate volume metrics of istio-podinfo for pod openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" (03-Nov-2026 20:32:18.878) (total time: 26449ms): Nov 03 20:32:51.089334 ip-10-0-132-83 kubenswrapper[5059]: Trace[92360070]: [26.449354998s] [26.449354998s] END Nov 03 20:32:51.089334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.327976 5059 trace.go:236] Trace[2075830383]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" (03-Nov-2026 20:32:18.896) (total time: 26431ms): Nov 03 20:32:51.089334 ip-10-0-132-83 kubenswrapper[5059]: Trace[2075830383]: [26.431469935s] [26.431469935s] END Nov 03 20:32:51.089334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.328198 5059 trace.go:236] Trace[1710905328]: "Calculate volume metrics of kube-api-access-mkxhk for pod openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" (03-Nov-2026 20:32:19.856) (total time: 25471ms): Nov 03 20:32:51.089334 ip-10-0-132-83 kubenswrapper[5059]: Trace[1710905328]: [25.47167336s] [25.47167336s] END Nov 03 20:32:51.089906 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.368600 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jzp4r\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:51.089906 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.368685 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tx8mk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:51.089906 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.368725 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:51.089906 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.368797 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:51.089906 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.369065 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a0d4ec-openshift-default-dockercfg-gsslf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:51.089906 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.369109 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:51.090158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.369143 5059 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:51.090158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.369180 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-p6m7s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:51.090158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.369214 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b2f85\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:51.090158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.369245 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:51.090158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.369278 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:51.090158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.369313 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:51.090427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.369346 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:51.090427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.369378 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rwt79\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:51.090427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.369411 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vlmgx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:51.090427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.369440 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:51.090427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.369473 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"agc-cred-5f4fe13bd610\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:51.090427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.369506 5059 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:51.090838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.399198 5059 trace.go:236] Trace[906877599]: "Calculate volume metrics of kube-api-access-hpnqh for pod openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn" (03-Nov-2026 20:32:23.224) (total time: 22174ms): Nov 03 20:32:51.090838 ip-10-0-132-83 kubenswrapper[5059]: Trace[906877599]: [22.174996648s] [22.174996648s] END Nov 03 20:32:51.090838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.405499 5059 trace.go:236] Trace[1110960472]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2" (03-Nov-2026 20:32:43.770) (total time: 1634ms): Nov 03 20:32:51.090838 ip-10-0-132-83 kubenswrapper[5059]: Trace[1110960472]: [1.634961075s] [1.634961075s] END Nov 03 20:32:51.090838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.405543 5059 trace.go:236] Trace[804510847]: "Calculate volume metrics of tls-certs for pod llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" (03-Nov-2026 20:32:43.760) (total time: 1645ms): Nov 03 20:32:51.090838 ip-10-0-132-83 kubenswrapper[5059]: Trace[804510847]: [1.645226629s] [1.645226629s] END Nov 03 20:32:51.090838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.408080 5059 trace.go:236] Trace[132037120]: "Calculate volume metrics of workload-socket for pod openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" (03-Nov-2026 20:32:43.907) (total time: 1500ms): Nov 03 20:32:51.090838 ip-10-0-132-83 kubenswrapper[5059]: Trace[132037120]: [1.500433533s] [1.500433533s] END Nov 03 20:32:51.090838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.439613 5059 trace.go:236] Trace[633714210]: "iptables ChainExists" (03-Nov-2026 20:30:55.058) (total time: 110380ms): Nov 03 20:32:51.090838 ip-10-0-132-83 kubenswrapper[5059]: Trace[633714210]: [1m50.380775745s] [1m50.380775745s] END Nov 03 20:32:51.090838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.445960 5059 trace.go:236] Trace[1864162557]: "Calculate volume metrics of istio-data for pod openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" (03-Nov-2026 20:32:43.226) (total time: 2219ms): Nov 03 20:32:51.090838 ip-10-0-132-83 kubenswrapper[5059]: Trace[1864162557]: [2.219334181s] [2.219334181s] END Nov 03 20:32:51.090838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.486310 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5m29.254000037s: [/var/lib/containers/storage/overlay/630999a4dc3864050ae8c3e3659c5a44defa3de8c0ab983ed793c303bfb15434/diff ]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.091366 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.486387 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.38435569s: [/var/lib/containers/storage/overlay/990167ec6f6877b6979123d6c2939438f468234423aa7808f21a809920da5fc8/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.091366 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.486429 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.384398115s: [/var/lib/containers/storage/overlay/30867ac3c4283948de88e14428fc7c9ac0901d779c82fdb4b0222913e208a1d6/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.091366 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.486465 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.384432026s: [/var/lib/containers/storage/overlay/bc363165f9f21ea338e9ed2bfdedf1f3550f2bcb4ee5f479d635338ae290bcb4/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.091366 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.588371 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.486373585s: [/var/lib/containers/storage/overlay/7ee751a49452db10e44d99c3d16e1705262f035def3da602cdb5f0ec0d850e03/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.091366 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.588513 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.486519826s: [/var/lib/containers/storage/overlay/f2c7f00aa454c8cc582a4c36e607079857c142026b24edfe46a8cd2d4529c10f/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.091366 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.588783 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.486782502s: [/var/lib/containers/storage/overlay/7234d3b3796fcc180309f9d1e93dcbb1cda70514c5cbe8e2f97a929556ed2eee/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.091607 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.594606 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.492602892s: [/var/lib/containers/storage/overlay/ca9a0fb1928ab3bba4b110e4d32cb6c316afa494aa3d037d2c76980cff6cf009/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.091607 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.594764 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.492761777s: [/var/lib/containers/storage/overlay/94455cc20d0b1129824f7cb5c5a094f97f0a608b14545bdc84604cc0fdbe3db9/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.091607 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.596602 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.494597203s: [/var/lib/containers/storage/overlay/9a840ab2681e15e51c6020aa6a5097916390aa67e7392c8ee7473775da518ee8/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.091607 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.596751 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.494736335s: [/var/lib/containers/storage/overlay/07af3ff7c020d195ba5fa404ed7a2979a52805c7facc79393490756afd6b625d/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.091607 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.597761 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.495743711s: [/var/lib/containers/storage/overlay/0a728ec770e6ffa3c0d8df9535746dbcb124a23ff2fea94ab2fc314ec32ae537/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.091607 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.605177 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.503179341s: [/var/lib/containers/storage/overlay/3664b495f860b8b12a14f83f6f7268d45d1d6788d533b83f182667924c5c86e2/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.091998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.605219 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.577207468s: [/var/lib/containers/storage/overlay/97db272a4e9d602f960815093200513ae8212138a7aa60545a5d09e979bd3e88/diff /var/log/pods/openshift-ingress_e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8_4c609469-d7d7-4530-8d74-e34aa56dda8f/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.091998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.605261 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.502765477s: [/var/lib/containers/storage/overlay/18543d407168eb4576353392a1afd991f157ee565b131d04360e902c8407fdd2/diff ]; will not log again for this container unless duration exceeds 4s Nov 03 20:32:51.091998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.743555 5059 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:51.091998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.743627 5059 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:51.091998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.777911 5059 ???:1] "http: TLS handshake error from 10.0.133.249:45296: EOF" Nov 03 20:32:51.091998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.781472 5059 ???:1] "http: TLS handshake error from 10.0.133.249:38263: EOF" Nov 03 20:32:51.091998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.781553 5059 ???:1] "http: TLS handshake error from 10.0.133.249:36972: EOF" Nov 03 20:32:51.091998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.782225 5059 ???:1] "http: TLS handshake error from 10.0.133.249:49190: EOF" Nov 03 20:32:51.091998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.783144 5059 ???:1] "http: TLS handshake error from 10.0.133.249:39591: EOF" Nov 03 20:32:51.091998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.783909 5059 ???:1] "http: TLS handshake error from 10.0.133.249:39768: EOF" Nov 03 20:32:51.091998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.784622 5059 ???:1] "http: TLS handshake error from 10.0.133.249:41793: EOF" Nov 03 20:32:51.092518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.786144 5059 ???:1] "http: TLS handshake error from 10.0.133.249:57364: EOF" Nov 03 20:32:51.092518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.786955 5059 ???:1] "http: TLS handshake error from 10.0.133.249:40195: EOF" Nov 03 20:32:51.092518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.787708 5059 ???:1] "http: TLS handshake error from 10.0.133.249:34563: EOF" Nov 03 20:32:51.092518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.788412 5059 ???:1] "http: TLS handshake error from 10.0.133.249:35677: EOF" Nov 03 20:32:51.092518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.857049 5059 ???:1] "http: TLS handshake error from 10.0.133.249:33483: EOF" Nov 03 20:32:51.092518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.857090 5059 ???:1] "http: TLS handshake error from 10.0.133.249:45071: EOF" Nov 03 20:32:51.092518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.857138 5059 ???:1] "http: TLS handshake error from 10.0.133.249:37988: EOF" Nov 03 20:32:51.092518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.857158 5059 ???:1] "http: TLS handshake error from 10.0.133.249:49282: EOF" Nov 03 20:32:51.092518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.857176 5059 ???:1] "http: TLS handshake error from 10.0.133.249:37506: EOF" Nov 03 20:32:51.092518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.857206 5059 ???:1] "http: TLS handshake error from 10.0.133.249:56452: EOF" Nov 03 20:32:51.092518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.857230 5059 ???:1] "http: TLS handshake error from 10.0.133.249:41067: EOF" Nov 03 20:32:51.092518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.857273 5059 ???:1] "http: TLS handshake error from 10.0.133.249:45354: EOF" Nov 03 20:32:51.092518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.857301 5059 ???:1] "http: TLS handshake error from 10.0.133.249:59759: EOF" Nov 03 20:32:51.092518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.857316 5059 ???:1] "http: TLS handshake error from 10.0.133.249:33677: EOF" Nov 03 20:32:51.092518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.857360 5059 ???:1] "http: TLS handshake error from 10.0.133.249:52488: EOF" Nov 03 20:32:51.092518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.857391 5059 ???:1] "http: TLS handshake error from 10.0.133.249:36679: EOF" Nov 03 20:32:51.092518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.857414 5059 ???:1] "http: TLS handshake error from 10.0.133.249:33758: EOF" Nov 03 20:32:51.092518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.789216 5059 container_manager_linux.go:682] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 55342 into the system container \"/system.slice\": " Nov 03 20:32:51.092518 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:45.801729 5059 kubelet.go:3135] "Container runtime sanity check failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" Nov 03 20:32:51.093432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.822646 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.718872129s: [/var/lib/containers/storage/overlay/f11eef46c0d6bd5328d511b7c3d8a2d7d49dbcbba4f1248321afe89be929c8f7/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.093432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.828482 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.722787113s: [/var/lib/containers/storage/overlay/b0d1e39d17ccc7a902eeab6ffca28bdff0fd8d7dc91746f24dd421fffbc289b0/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.093432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.837148 5059 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:51.093432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.862899 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.760875706s: [/var/lib/containers/storage/overlay/65a7f2743a6baf66f9e9795694b13722c929a35c96e6ad652f638f9899e25507/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.093432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.863238 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.761213782s: [/var/lib/containers/storage/overlay/9b5b445cad281296a69a13715d23a258a00871cd7a404857597c880c7c344ef2/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.093432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.837614 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.735594242s: [/var/lib/containers/storage/overlay/d1b34dee7bb386b4b08b6201e76895f6f53c61c71d5047b48d1b40a81fdc836c/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.093714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.863391 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.761366816s: [/var/lib/containers/storage/overlay/05b98974003a4181cc525d06dd6951339b1182ba0e8da0ec7cc1521a072638d0/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.093714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.863418 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.858567659s: [/var/lib/containers/storage/overlay/8b2b6e7cad8ba0d31b01bad21f7c4e39df2debf268786e2befafef641bd5c6db/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p_ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.093714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.864515 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.919762279s: [/var/lib/containers/storage/overlay/6b837f63593a17f863e6b609d3c6f7d939a72c0b3cc0a9bf79876776dbd4f0d9/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn_25ceab3d-6344-4512-a471-d91aec0cb48c/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.093714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.864561 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.839743882s: [/var/lib/containers/storage/overlay/76011fc3cc8cd31792e1a47889733536bd5e73bb3d979dccf9abc68fee1315e0/diff /var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-lf9vc_677a5153-5fe3-4624-bf06-3c8dfc6bc72d/discovery/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.093714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.864581 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.819724003s: [/var/lib/containers/storage/overlay/772b2fa301202b16036e5a2f97de9948e4d62690b1312c6009f39270bee18938/diff /var/log/pods/openshift-ingress_e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f_14a06bdf-e15d-406c-bb76-47ae5dd06b2d/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.094254 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.864601 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.788556795s: [/var/lib/containers/storage/overlay/5cc649e7a330cae75c81c80a41964c2c3c6ce9688dad07d03ada6fd24f317291/diff /var/log/pods/openshift-ingress_e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42_a594e899-889e-4f7f-a8ba-b7e7a389ba8a/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.094254 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.864619 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.790142438s: [/var/lib/containers/storage/overlay/8ea54aa04516a03f1047de79fbfbd116b50ffbdc25a62c78219aeec1b7a7cc9f/diff /var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4_70d45aa9-bfa8-43c6-8674-6f8f14a3097e/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.094254 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.864634 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.764857428s: [/var/lib/containers/storage/overlay/06a0ba1cbe573083e0a7d84264e2f1b125b9836c39424aaa1ca550774ca8462b/diff /var/log/pods/e2e-models-e2e-worker-w3-9f46e5_simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22_f14445d5-52ed-487d-972c-4499966065fa/main/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.094254 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.864651 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.850956906s: [/var/lib/containers/storage/overlay/aa85e2b57d88bff2e55c442f79cb18559d789aa60ffc4f6bae25b4627a4c09d3/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt_c92b1be0-0890-4db6-ac75-968b6e5a3434/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.094462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.864680 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.762737711s: [/var/lib/containers/storage/overlay/b2bcbf7f5b1d1d97e274441018168e42a9e8779416c966ffc0c40bc5575414a0/diff /var/log/pods/openshift-ingress_e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn_08b3a993-1a64-4fe5-9577-b0f11a36779e/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.094462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.864697 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.76273128s: [/var/lib/containers/storage/overlay/962bdb1ea9f0fbdf7d9766a1032960fd9a27c344be31f462484a7ef5facfd38c/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-vznnm_171c7288-3157-4fe0-80ee-94cf5b2ca2dd/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.094462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.864714 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.766028606s: [/var/lib/containers/storage/overlay/4909810f6862308c7b8e948b9acdcc7c0d05e34ac5ad3eb68a2814d8552abed2/diff /var/log/pods/openshift-ingress_e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w_1731953b-1850-40d1-9ba4-cb08dc4ebc5d/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.094462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.864731 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.820193018s: [/var/lib/containers/storage/overlay/2def3de689f3697117c5b2b888ee4f0f0714c497d59d71bf5ba44e80b2884cdd/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-g6xz7_0f32ecf7-06f7-4098-9793-acf761a30c09/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.094636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.864746 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.782976313s: [/var/lib/containers/storage/overlay/c37ca7a1d4a292f83c6190b9174a510fc5046ebdb1f5f1453a9ab145c3e6857a/diff /var/log/pods/openshift-service-ca_service-ca-5fffc97bdf-brvgz_7b8992fc-fba6-4135-90f8-fb259e765dd5/service-ca-controller/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.094636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.866864 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.764836115s: [/var/lib/containers/storage/overlay/83f2877cc6d21618287aabda5fe219114908303ccc1e352fde2d18088555b36f/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.094636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.871252 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.76927933s: [/var/lib/containers/storage/overlay/1d7c6bc86a06ce9ae161b91fcbef9d4526c125d63fcd9993f06aa5a3d6b77636/diff /var/log/pods/openshift-ingress_e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd_f88ae6f3-f1d6-4aa3-ad33-589490696167/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.094636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.871284 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.864284276s: [/var/lib/containers/storage/overlay/35affdd196fb11150949ff43876d496060e5efae610c103523fe68d5ceac4faa/diff /var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-x8zts_633c352c-f07e-4032-b962-23a148297e45/postgres/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.094636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.871307 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.828007738s: [/var/lib/containers/storage/overlay/79c575ea534c7b0bb095fa21c0a8cd9709d073b5c95f1b6f855db1b7fc78a8ef/diff /var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-nj52h_941b16dd-0550-4434-86f9-26e38efb68ee/manager/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.094862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.871328 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.769346918s: [/var/lib/containers/storage/overlay/ff68119ff6c1ec3b8d8999fce0f9c89e61ff03e3d3dac3235815761191644c7b/diff /var/log/pods/openshift-ingress_e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx_2ed21c5c-ef42-4fb0-bf41-34e84b9cf193/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.094862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.871350 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.841651667s: [/var/lib/containers/storage/overlay/83bd2190b8aa145be528c41fb67adba5f21d9d3f828fd3c580d14625b15ec8c9/diff /var/log/pods/kuadrant-system_authorino-66b59d468d-5bfd2_c7fce352-1850-4048-be2e-fa458fd4d2f3/authorino/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.094862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.871370 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.790587043s: [/var/lib/containers/storage/overlay/548b56c6b19ef7cd87b984ab1c6b190ae06edeb55cfce7bb12e0610e4c31569f/diff /var/log/pods/openshift-ingress_e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95_c2d80995-5f1c-46db-8d71-a130cb92e279/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.094862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.871404 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 50.017007256s: [/var/lib/containers/storage/overlay/326b71f661ef9f0085c5b27b89c59064eec03ee6148d513a218dcd440991fe6d/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.094862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.871888 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.449455605s: [/var/lib/containers/storage/overlay/78131cdf25b8ea22b8b4e2ecfba14b69aed6fec9e7be52667fc8d84c2d1c1f4f/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.095085 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.871929 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.404692566s: [/var/lib/containers/storage/overlay/5ca4279d7298c4d616461615afd646ad3eb5ab2d9fc04b7395ee9566a88db63d/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.095085 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.871964 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.404726173s: [/var/lib/containers/storage/overlay/7a542b6487238caa6f2b3f7a1a6c7b90dd3b718c7b8d47eb8857c76cccb9f1d1/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.095085 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.871999 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.404759098s: [/var/lib/containers/storage/overlay/ff02ed8e0192f2b5f8a09e55aa451c0ec876698c921658cf8618aac18a84ca84/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.095085 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872034 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.404793173s: [/var/lib/containers/storage/overlay/1ab252080a6da0c26d9aed23639cf17237b7e42ba6e207939b96152dca047393/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.095085 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872071 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.404828173s: [/var/lib/containers/storage/overlay/f0f2ca521d042f46a869ff8db4935ca1a58cfef7b1a4ed9b5ed6a4c77999f76c/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.095085 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872104 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.404860295s: [/var/lib/containers/storage/overlay/e3f26c564d54b842c20f87ef8b782f5560957d6752e7297cc52d36a8b3e16d04/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.095085 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872139 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.404892871s: [/var/lib/containers/storage/overlay/f0554608437114cfd30a852e952ef989245f496f382a495e89e41089979bea9e/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.095736 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872173 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.404924722s: [/var/lib/containers/storage/overlay/a05ea4c91cdd1eeaa152497f63911dfe58c1907944efac9945b0167bf064ce0d/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.095736 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872207 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.404958192s: [/var/lib/containers/storage/overlay/f339f937f6b2246ad7339663ba783e2ca19158b7afa66ca30c7cc59cade7ae73/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.095736 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872245 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.404992088s: [/var/lib/containers/storage/overlay/fbcc3fd0d14d87f947f11a6270e50c6a4a9b838bf13ea2df20555628ebca92ca/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.095736 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872277 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405024359s: [/var/lib/containers/storage/overlay/067ca349ee1450e78700331d6f208c849004833b993038177db3236b89e0c254/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.095736 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872312 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405057287s: [/var/lib/containers/storage/overlay/1c18da9d60946c03b1cb4c419644a5ccb427409980e444d0b82c88043ec78a6a/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.095736 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872346 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405089445s: [/var/lib/containers/storage/overlay/f4223c7ada52388a0e1115563a3e2c2098950a9bffbaa80181178892ef6cf8d3/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.095736 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872379 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405120897s: [/var/lib/containers/storage/overlay/3c1933e8657ed24a888e3078150a15152408aef3de6d06715661d3b955bdeb03/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.096371 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872412 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405152179s: [/var/lib/containers/storage/overlay/f76791e3b40bd77e4177e755af9593b8264a6f9584df82df5f3ad48f8536a824/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.096371 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872445 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405183452s: [/var/lib/containers/storage/overlay/5c5f82c4ddbea13aa5f7e0f0d8a4dea32beeac342c1caa0870a75fc8b4f3994f/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.096371 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872477 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405214522s: [/var/lib/containers/storage/overlay/2ab4d09781f5bb38cf39dad12e66699114c6e92eb3b29dd65c026a0af1b404cf/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.096371 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872510 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405246205s: [/var/lib/containers/storage/overlay/9a287a1ec7ba2ac17e0f0d11b194f49d00d362cedf55c894346e179195a8e98e/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.096371 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872542 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405276651s: [/var/lib/containers/storage/overlay/acb75c12d3b37b911ca96f7741e934e88cdad5615e7b123974ba4d09049057a8/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.096371 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872575 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405307664s: [/var/lib/containers/storage/overlay/6e9994b975a1d209a1e7ad1c69fd10d9cc3dcfdb7ae65b13f21dcbd140058659/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.096371 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872607 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405338892s: [/var/lib/containers/storage/overlay/2acad7ba6294ceb7ed21eec32f82e3dd0f18687ff5816a03411e1545b134088c/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.097110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872643 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405372427s: [/var/lib/containers/storage/overlay/696bd9c68ee459239e11ef2c5be2c47c25002ec59a7388640a6ee78d147b05aa/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.097110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872692 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405419656s: [/var/lib/containers/storage/overlay/2cf0cbdd877424c5f8c0110195c9f03a8f8dac9e852d359293da71d0737916a0/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.097110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872725 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405451836s: [/var/lib/containers/storage/overlay/fff3e4757f69e495c651e4c9e0afd850cf031c4c2b428196eb46a31c2ff022ea/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.097110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872758 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405483365s: [/var/lib/containers/storage/overlay/0e72203bbe7cb0cd3f9fc6eab23667a1b76b6bb41836316e63fd885d237bb420/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.097110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872792 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405516024s: [/var/lib/containers/storage/overlay/e3a08dd38e0673fee7ee93b1f7ca301b5aa65f8f1089002352132e3cd010d249/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.097110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872824 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405546665s: [/var/lib/containers/storage/overlay/0c52062b38c59af46beb68c242b3a38f795b07b0c8b1ad76126162c8c3c3e1d5/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.097110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872858 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405578556s: [/var/lib/containers/storage/overlay/56a1a4fe0beea2934f0067c34f07d9b9de48a79ddeaebc10dcc1918e249e4fe9/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.097793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872892 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405610996s: [/var/lib/containers/storage/overlay/974994766dd4ff7caa644bd6347d79d35dd36116080142add2f681ab8815f5ab/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.097793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872926 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405643376s: [/var/lib/containers/storage/overlay/5e9d0591d7c92f06d2c04115fb8e8d565a3e0b02ab2c918f95ea53689fea97eb/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.097793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872962 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405677637s: [/var/lib/containers/storage/overlay/d49f130f651bf65ace5eace09a0ebe2fa45d704fd81e259422bdddca3c58a0d6/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.097793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.872997 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405710026s: [/var/lib/containers/storage/overlay/89ea4b8b93aee8b87c44a82fd9fd20add9e924a331496366f375f3833e6e457c/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.097793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873031 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405743013s: [/var/lib/containers/storage/overlay/1a222e0eecb04b88b5dabdbc6d32ec15f7b6d7e865f4b269e1fef7a61f1e0f0a/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.097793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873067 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405775654s: [/var/lib/containers/storage/overlay/e8120ead559c0bf33201f22ae411f9741ede65e14e3388fa66493a090c2ce6e6/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.097793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873100 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405808371s: [/var/lib/containers/storage/overlay/6982a733f632f82f58560039947e668a41a46bac9f9f4ddef9335e74f577da9d/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.098092 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873134 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405841737s: [/var/lib/containers/storage/overlay/3e151306ed961ad6bfc0a65d6d7858de0210f8149234a8750218b7bce9f28c39/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.098092 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873172 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405878073s: [/var/lib/containers/storage/overlay/c3f9bac49400fba6152bdce7d3ee4ea3aa0bf276d345e56e6566efd24f4aa5ba/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.098092 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873205 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405908862s: [/var/lib/containers/storage/overlay/07aeb6439b6aab304aeaf1b043ccaeee044b85ae3facc335f860427356ab4c41/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.098092 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873238 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.405940577s: [/var/lib/containers/storage/overlay/ef5bbc2ac888611f17ab3446f15465cb87e7e0732a42e0f69870ac3291beae0b/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.098092 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873273 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.40597323s: [/var/lib/containers/storage/overlay/5f7322f6b59f9ec6bdaf95083ffd18198d220b0ba84771ed863a48299964c622/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.098092 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873310 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.406009276s: [/var/lib/containers/storage/overlay/807b5d91ec044b6e3690fd687ada9bd4e01ef9aefd7c5be109d86f8020447460/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.098092 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873343 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.406040994s: [/var/lib/containers/storage/overlay/e508a4fba182975f628a2965bdd2bc06db2f762a03cb80182d6faab7219e0ccd/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.098402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873377 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.406072767s: [/var/lib/containers/storage/overlay/364e21f717075de023707ce0eadcab5ff3e172710140545d125c62e36740887e/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.098402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873410 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.406105504s: [/var/lib/containers/storage/overlay/42ec34392fa2e3d7cbe31b8fa86900cece879ef5fb5d2e80b32bb490c7a8fbb4/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.098402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873445 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.406138517s: [/var/lib/containers/storage/overlay/15a96bafbdc9757f9d817aa97e7845340b628642f6a672a3e31c5f490b8f0fb1/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.098402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873478 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.406170401s: [/var/lib/containers/storage/overlay/5b44229870991839ead05bbd6d43ad3c92fae6398297149bd793ebda3f403234/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.098402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873510 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.406201085s: [/var/lib/containers/storage/overlay/927ae99c7bc3b06e217e45d2cf6db349f238086d590a4a2b7e234cbb66369af6/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.098402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873544 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.406232732s: [/var/lib/containers/storage/overlay/2b818e04a987bed97dc81c53c55626df56032b4ea5b65abebfe974b29ef334b6/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.098402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873578 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.406265028s: [/var/lib/containers/storage/overlay/7f28e0f7f0452864f37340b84f720939062a84b92404c9b09cbac54186e86f29/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.098707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873613 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.406298024s: [/var/lib/containers/storage/overlay/d2e80b4794268fec4a961ae72b980628d6cfeab77a767fc85c67ad64630354e0/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.098707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873646 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.406330256s: [/var/lib/containers/storage/overlay/85c1fb9d4ff88cba050b8812344293d7e1bf6bc385014c69d5d4be0f8fd22664/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.098707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873701 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.406383188s: [/var/lib/containers/storage/overlay/9b58e66893e846684294552f052e4a90fdd8746f4b3637827c3ccb0e52c95e78/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.098707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873735 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.697944276s: [/var/lib/containers/storage/overlay/dca21d2cdbf4ac9467b499e27e9aa76419bff93f79dd7e293c13870cfd22c2f6/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.098707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873786 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.697986903s: [/var/lib/containers/storage/overlay/d839857b75b0b46e2db3536ada903cde1fd0faabe7319f8b0b4b8f5d82ec5583/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.098707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873821 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.698020523s: [/var/lib/containers/storage/overlay/560bd35915a3e4ed3a92ebec5a37ee6c8c2b5d602e52db698c006c8f6d871c49/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.098707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873858 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.698056169s: [/var/lib/containers/storage/overlay/1cf8faf81515e3252d55ad5fa3a6f01aade4cbcfc250d0b89f7f4abe3f998eb5/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.099219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873894 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.698089939s: [/var/lib/containers/storage/overlay/ba408307c7920c24e1b289a6ddecb3cf2b8cf03ea2ca16d87c7f5ff7b4e0d72f/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.099219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.873928 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.698122549s: [/var/lib/containers/storage/overlay/ecf8c759fcf916418871e8a8350ee9535c279c4497742cc2efb82eb693743add/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.099219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.877841 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.702017545s: [/var/lib/containers/storage/overlay/063b6ab855c8ff15074e39c756449c29ce35cfb5d4ffc77409f59ad34e12ea31/diff ]; will not log again for this container unless duration exceeds 4s Nov 03 20:32:51.099219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.878640 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.702810375s: [/var/lib/containers/storage/overlay/69d000787c06a185865f58066c28dd8c5a083b56b5987b22d50bb9ddd8b80af3/diff ]; will not log again for this container unless duration exceeds 4s Nov 03 20:32:51.099219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.878691 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.702860609s: [/var/lib/containers/storage/overlay/4e5492bb5eff4ed1d55fe1a44966920b4e9fe42aa08f0f1abe51ffbc8a7e7223/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.099219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.878725 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.702894241s: [/var/lib/containers/storage/overlay/b3721a6a9f08643666ccb64e9abff935757945728992436081afdf324727651a/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.099219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.879742 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.653041068s: [/var/lib/containers/storage/overlay/e18fb873dfe538f424b3c44816bc601922d0b829179e2579487f6c834acdb0af/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.099502 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.879787 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.653083497s: [/var/lib/containers/storage/overlay/8e00d3fbc10cb977be544c47c232e9871e548417e5eb5a3e5c48dc1cbde46a01/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.099502 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.879820 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.653114669s: [/var/lib/containers/storage/overlay/036ad7274ad1232ed107d37abc90a18d84e2c58366eba450be383b1c7539919b/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.099502 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.879854 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.653147621s: [/var/lib/containers/storage/overlay/d91a2eef7704086128be09e4b79681dd76ac562bcbb98517cf669c2b35c77693/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.099502 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.879887 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.653179073s: [/var/lib/containers/storage/overlay/90917c44b902559da175c71d7fd1c2bf17a72658a6a077025bc4f6e26f5f0b71/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.099502 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.879920 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.653210506s: [/var/lib/containers/storage/overlay/e426a31bf0b7a61b98bd48d03f31dcee6f91725a89c8e93541fbe6c276f22bbd/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.099502 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.879957 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.653244049s: [/var/lib/containers/storage/overlay/0edfa7bdd1c0bb6e171a437f6bce6c93a31b9489e465414ee1c1d9d1a8585a8a/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.099502 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.879989 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.653275997s: [/var/lib/containers/storage/overlay/6108362e423e122429ec08608da59a2f40f00273b2584dca92bd8a7570448b0c/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.099853 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.880023 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.653307727s: [/var/lib/containers/storage/overlay/cc33b7715971fc9b11e0f8ee7fff767603d70576b6ba7c1fde7231b29d5e89b6/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.099853 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.880056 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.653338781s: [/var/lib/containers/storage/overlay/d9cdf270537c2f1686a905fce82a567a5543e02a3c3f34d533083daf08be4559/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.099853 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.880088 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.653369173s: [/var/lib/containers/storage/overlay/9b844bd54d6bb3ae9b0c6fa4199967987d71c6c3891e99f9455c5977ec052ffc/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.099853 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.880120 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.653400062s: [/var/lib/containers/storage/overlay/6bc0f7827722b17cd75299041f34b5e43ac68089229c28450eb493c786a5bc0b/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.099853 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.880152 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.653429096s: [/var/lib/containers/storage/overlay/2a43d0fb23e55b728f436ffbb8ff2f5b9316bfa0129b85eab729dcbb72fc0152/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.099853 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.880186 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.653460752s: [/var/lib/containers/storage/overlay/10510a6cfdd969bf4f3ffee52a0d729e68c4fb0387c5264c3cbe862a8fb57585/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.099853 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.880368 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 29.224923386s: [/var/lib/containers/storage/overlay/8c48fd187f175613a5793fa62d578c4b3a935ca8ba2d18903e3169b48d6de89d/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.100133 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.880400 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 29.313216008s: [/var/lib/containers/storage/overlay/961637059afbf81633d4eba2dfb534ab9a290d37b78ad8d5526994d2ad6c0348/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.100133 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.880685 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778625856s: [/var/lib/containers/storage/overlay/3ae1666a70fe1ee29ea3032c4280f1c66a3efccb208906869f521f4554b08316/diff /var/log/pods/openshift-ingress_e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r_ade5b9f4-d822-4418-a31b-f0c0a21ddb0c/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.100133 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.880709 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778724952s: [/var/lib/containers/storage/overlay/3474501ae08013e59580c990423bd84e932c126eeb6012f33f01041d02df7c01/diff /var/log/pods/openshift-monitoring_node-exporter-8s8ss_cc202826-dc4c-4f05-b4bc-63fc108188c2/node-exporter/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.100133 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.880732 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778227151s: [/var/lib/containers/storage/overlay/7d118d46ddd9118020120bf4abeda717a9745f884905c6c04f3bec41139d79e2/diff /var/log/pods/openshift-cluster-node-tuning-operator_tuned-mkm7n_53b14f9a-431e-4dbb-a725-4c8470262ca7/tuned/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.100133 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.880752 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778245778s: [/var/lib/containers/storage/overlay/b85378590d3a8b51c3720c14d24215dbda1e35f20453ad77550072c510cf3714/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2m4mq_c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/ovn-controller/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.100333 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.880773 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778264085s: [/var/lib/containers/storage/overlay/5b3629ccac63294c67978800bc554ad8685d0a4172b11f35016df65821206978/diff /var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-6k9n9_f485893c-6f14-449b-b837-8ac181315aeb/migrator/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.100333 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.880794 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778284178s: [/var/lib/containers/storage/overlay/18bc1e0ac5f089f3489772fb39b4cc72a12ceea81b05d60570e379b8672727f2/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-gb8r8_e64f9d9f-c868-470e-bd78-4ce8baabd97a/csi-driver/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.100333 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.880813 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.77830122s: [/var/lib/containers/storage/overlay/7fcc2c309088f71f15e3de261c00b834cb912549820428227aa73c7f19b19398/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2m4mq_c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/kube-rbac-proxy-node/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.100333 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.880833 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778319523s: [/var/lib/containers/storage/overlay/cbb879871a44605c610147751343b85dedca5ecbe37123acd39e2fd41eedaa35/diff /var/log/pods/kube-system_kas-connection-checker-685895c7c9-6hx8t_f9d935b2-aa51-474c-836d-a74f91efab07/connection-checker/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.100498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.880852 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778337128s: [/var/lib/containers/storage/overlay/ee7f02d9be1fdeb93909fdbcfaf9d4038d7c16d9e26b087b8d5ace145804b85a/diff /var/log/pods/openshift-network-diagnostics_network-check-target-b4x9q_c95249a6-899f-4faf-82dc-6e553453ff80/network-check-target-container/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.100498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.880872 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.77882797s: [/var/lib/containers/storage/overlay/057778d1fef160a784176c21333cc7ddb765ca129ee233239cde8b2204224102/diff /var/log/pods/e2e-models-e2e-worker-w4-3fbb1a_simulator-w4-3fbb1a-kserve-7868585bbb-4892w_20690fd2-7022-4657-8c8c-bd738493ffc0/main/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.100498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.880893 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778373781s: [/var/lib/containers/storage/overlay/af2813e3cc345e41fada3e12c6cc0bafde54c03ea986315db35b5d8e8606a959/diff /var/log/pods/kube-system_konnectivity-agent-q4zkg_4fdbd17a-1e15-4442-b715-aaec1087da11/konnectivity-agent/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.100498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.880913 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778392591s: [/var/lib/containers/storage/overlay/67643f6bfc43ab27acce59d3f46ca4439902175e226d9e26fa2b0544a356d3a7/diff /var/log/pods/kube-system_kas-connection-checker-685895c7c9-jt54m_e2155512-2116-44f6-be5e-77cef72f9e14/connection-checker/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.100674 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.880935 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778413427s: [/var/lib/containers/storage/overlay/4b8288f2e07855b92c0673979bbc56f16b95f104d7f3d331523a2b249fd005be/diff /var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-132-83.us-east-2.compute.internal_3e0047dbb7ee8d4e56f23e3bd51214c3/kube-rbac-proxy-crio/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.100674 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.880957 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778432977s: [/var/lib/containers/storage/overlay/eaf2894f871c7923b917e5b3aeb7e3f05b762c5775c3e7f9dc2c9ff74861f681/diff /var/log/pods/openshift-service-ca-operator_service-ca-operator-544d749c8c-xzpgl_c99b0c19-004e-450c-852a-e38338635b61/service-ca-operator/1.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.100674 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.880976 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.77844352s: [/var/lib/containers/storage/overlay/49a371edd202683666dd0b270130376ae59bf99c2a0df270b51976ef77c62c30/diff /var/log/pods/openshift-insights_insights-runtime-extractor-hmnx4_377ffcf5-b097-495d-b986-e61c58ae8ec1/exporter/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.100674 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.880996 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778950086s: [/var/lib/containers/storage/overlay/e730290aa13437c09e45b8f08838fe214bb40ce68dfc579337e5198f7056d30e/diff /var/log/pods/openshift-ingress_e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt_d912f1fd-b76b-4bb6-a5d9-ccc8b4536840/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.100846 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881016 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778964516s: [/var/lib/containers/storage/overlay/77be1a78d40f6b27ca2dc53b367cb858613fc53efb5d6d6ca3f4ac6646a0e853/diff /var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-tf75l_cde430c2-2518-49a0-9851-42356d3f2fe9/main/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.100846 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881038 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778981041s: [/var/lib/containers/storage/overlay/6fe7ca5b5fed409629d0a8bb564f4f835c60bfdb5e674c50db47461f210e2eb7/diff /var/log/pods/openshift-ingress_e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5_ab8320f7-2354-433a-a8a7-4d91cc5d37d2/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.100846 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881066 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778518549s: [/var/lib/containers/storage/overlay/06e27953833e922dd29fdfb0190b6691a452f9aecad198b52de016a37c8e197e/diff /var/log/pods/openshift-monitoring_node-exporter-8s8ss_cc202826-dc4c-4f05-b4bc-63fc108188c2/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.100846 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881088 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.779034311s: [/var/lib/containers/storage/overlay/0f417ef24c6af7f731bfdb4a657261136f1640a2fce34a1f97bdd82370d8fd37/diff /var/log/pods/openshift-ingress_payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8_705dea2b-ad85-4b1c-a3c6-a9f705ac9039/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.101013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881110 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.779051263s: [/var/lib/containers/storage/overlay/8cd4679b81ead040c961217792e0816971c04f0c13ae609dea6a2489efcc2ff8/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7_2c363281-0226-420f-a963-5922d5ca609e/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.101013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881131 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778578208s: [/var/lib/containers/storage/overlay/84167313719d1991fbbadc24e0c05628d697e9c890ed2e2ddafa01fcce5c18e9/diff /var/log/pods/ai-tenant-e2e-extmod-b-0e4ce7_payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2_0309a608-f6c4-45ca-991b-33ba73221a6c/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.101013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881155 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.779104948s: [/var/lib/containers/storage/overlay/13a0492b9fae19a6960dac59694e25302a8507425c7946f0778b7c1e26af65ef/diff /var/log/pods/openshift-ingress_e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d_40b88c7d-08e9-4203-9ebd-2c9f69b884dc/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.101013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881176 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.777490084s: [/var/lib/containers/storage/overlay/89ca02667dcab424b5177bc62aab53cbd5ac6a1d58eee3167b0960fcfc2cafb7/diff /var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/2.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.101177 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881198 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.779149218s: [/var/lib/containers/storage/overlay/4d1d763bdcd2be1ec8c6a0b81723c1837cc2c1780f6677d73989b2c1133b7e69/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44_26375297-81f4-427f-94c3-d56f7a6a11fb/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.101177 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881219 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.779235986s: [/var/lib/containers/storage/overlay/428935c09ddf323d498310f18396171954216064977823c19d83c4df7e2d6b7f/diff /var/log/pods/e2e-models-e2e-worker-w2-2b5703_distinct-w2-2b5703-kserve-676bd58bd6-qhvz4_1a57d851-12b3-4535-b12c-b7d2c579d0fb/main/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.101177 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881238 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.779177163s: [/var/lib/containers/storage/overlay/e18a17f3c6667965b57f8915cf1b24098b5fbfa576766528a9c24e90c047017a/diff /var/log/pods/openshift-ingress_e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg_518169ac-2f8d-4645-ae47-210b67ca080a/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.101177 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881260 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.779273262s: [/var/lib/containers/storage/overlay/b8c70cc7c1cd703826834e8d483928dd61ec449b277430ef89f32568e47f3d3f/diff /var/log/pods/openshift-ingress_e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf_8adff4a5-5376-43ec-9b53-888705ae8236/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.101347 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881278 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.77876132s: [/var/lib/containers/storage/overlay/395fbf4d5d4ec8a134ec8da50084515d06500fbb092d7ed16790f32d60204633/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2m4mq_c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/ovn-acl-logging/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.101347 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881298 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.777594924s: [/var/lib/containers/storage/overlay/65ccba7b8af5c70659046888199f0ac866f2edde7cc1d777af004dd86605e1b9/diff /var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-86qxx_40b2621f-854e-44e0-9b01-c877e333f644/d8b3f14031a53eecbdb2db39f15186180f1b7059e2868a65f8b7a4fe42d2380a.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.101347 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881318 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778772376s: [/var/lib/containers/storage/overlay/aae27b6ee5fc90d09200b2fca469bcc5b7e2572a8f74d6ee3fb9fc3e123281c7/diff /var/log/pods/openshift-image-registry_image-registry-6c57db574-fmm6v_3265bd58-d654-467c-9b2a-14cedc27b754/registry/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.101347 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881339 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.777631847s: [/var/lib/containers/storage/overlay/a8e2771f9883e0cdaf40d4061a2f08e80e068684c3ef6c19bc9998b87ae3d8c3/diff /var/log/pods/openshift-insights_insights-runtime-extractor-hmnx4_377ffcf5-b097-495d-b986-e61c58ae8ec1/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.101515 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881359 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778808624s: [/var/lib/containers/storage/overlay/0a7d2b6c184bbc9ea7aca802195d9a55372400e1cc4f3cf22c0893256c58b9e4/diff /var/log/pods/openshift-ingress_e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6_eac98813-7b4c-4c4a-8bf9-444bba2fa644/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.101515 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881379 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.777668005s: [/var/lib/containers/storage/overlay/8d5a8bfb065b77d3516c9b12dc7440a9b213e9e14b732e71883de279b65cd3ce/diff /var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-pbgwk_815b15b6-2c37-4ca3-9b38-e7e1328d0aa8/migrator/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.101515 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881399 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.777686367s: [/var/lib/containers/storage/overlay/0b869a1572abc120d48a24674560fbd62371975dce75976b1de34bf21a0f7324/diff /var/log/pods/openshift-console_downloads-6dd789c8b7-c88f8_6a7937ca-c705-4091-967e-2e640c622a26/download-server/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.101515 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881419 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.777705219s: [/var/lib/containers/storage/overlay/3fe862ed3e308dc9fd551945d6f248a08da522305664f95336d9bb32796ea201/diff /var/log/pods/kube-system_kas-connection-checker-685895c7c9-bd8f8_0fc518fc-6302-4a1a-9ff9-f1e036a2199b/connection-checker/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.101691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881440 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.777757267s: [/var/lib/containers/storage/overlay/b287206ebf5abf02de46d1dfecb037e038a60773df298de9394db3ac0ca2306b/diff /var/log/pods/openshift-ingress_e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj_d3de18b3-c3d1-4ec3-961f-d95ca3656f4b/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.101691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881462 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.777760108s: [/var/lib/containers/storage/overlay/129473a8ae2b6e6cb76eb14fb406bdbf019ed61e93c41aac613910bbd58651eb/diff /var/log/pods/openshift-ingress_e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h_fc57130b-c950-49f2-8a98-22c47bd98288/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.101691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881482 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778932586s: [/var/lib/containers/storage/overlay/00030eca2f61227baeb170bb925aa2236cdf3480d33657979fdb973e7e9e694d/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2_cfcb49c5-4381-411e-9d64-d9dcd6181239/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.101691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881503 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.777774192s: [/var/lib/containers/storage/overlay/ab6eeb0d8131876a3c4186f525187a9889b17f6b72c96426f7192132193bfaaa/diff /var/log/pods/openshift-dns_dns-default-nhf67_419841be-7a31-4cdb-b0b7-9d90a3c376c7/dns/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.101864 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881523 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.777793516s: [/var/lib/containers/storage/overlay/b44c3e3c39b2f4af23975cd957b5394aaa9ef8ef0b64adfdc2aa404a92d61503/diff /var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-wh95x_492c7b4e-84c0-49a7-b12f-153a021f971b/kube-storage-version-migrator-operator/1.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.101864 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881544 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.777857118s: [/var/lib/containers/storage/overlay/67b680590f5eefdd8ad88492c01e8616609706fb8ff8b7571351b1b139c2be6d/diff /var/log/pods/openshift-network-diagnostics_network-check-source-6f78d59f99-tvb96_0a985b50-c4c0-4a39-abed-29f53e2be747/check-endpoints/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.101864 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881563 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.777828536s: [/var/lib/containers/storage/overlay/9edd96ae054cc1dc8e9dfc2a28e2169a5c4bcc43353e82ed8bf524e2c8467dbc/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2m4mq_c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/nbdb/0.log]; will not log again for this container unless duration exceeds 4s Nov 03 20:32:51.101864 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881582 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.77789393s: [/var/lib/containers/storage/overlay/108b016c7af9497826514874c593f020e6d61efe256b2502992c3661bb6d96ff/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9_890f130b-cdeb-43da-acd4-1654673271cc/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.102030 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881602 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.777864161s: [/var/lib/containers/storage/overlay/15de5c78c58cb7031b136a1c35a90045dd6aebec927308d49a43d91a20ef351d/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-gb8r8_e64f9d9f-c868-470e-bd78-4ce8baabd97a/csi-node-driver-registrar/0.log]; will not log again for this container unless duration exceeds 4s Nov 03 20:32:51.102030 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881621 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.777881421s: [/var/lib/containers/storage/overlay/4a77bbd76b8d6f139795f66a89d4209b4d85e3189b0d33316e94acef23c21ee8/diff /var/log/pods/openshift-network-operator_iptables-alerter-7zx7h_55f124eb-2ebf-4675-a08a-bad3757061b3/iptables-alerter/0.log]; will not log again for this container unless duration exceeds 4s Nov 03 20:32:51.102030 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881640 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.777941902s: [/var/lib/containers/storage/overlay/8d9a1b5f5af6fb294a385b5c40459d1d5a37a514e98a1e388be257b935b02e6a/diff /var/log/pods/openshift-ingress_e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x_7909b42c-1572-4fe2-816a-447020594fe4/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.102030 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881775 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778029121s: [/var/lib/containers/storage/overlay/43201a78b713ec15496bb789b9294c6a0013d5c89dba58b4ae0edd1a0971be62/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2m4mq_c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/kube-rbac-proxy-ovn-metrics/0.log]; will not log again for this container unless duration exceeds 4s Nov 03 20:32:51.102205 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881803 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778058385s: [/var/lib/containers/storage/overlay/285095d558a08566a705a3940ab4af0b45e7f143d3fb6d23370fc566c03184ba/diff /var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-fsbsp_1bcd4690-4753-4246-abb4-8668a0a56231/volume-data-source-validator/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.102205 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881822 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778076338s: [/var/lib/containers/storage/overlay/32e345b5d629dba2b5bd17e9c25f60a6c11de452ab6a193c6b6e03707507be02/diff /var/log/pods/openshift-multus_network-metrics-daemon-qvxjq_caf7d6c9-0bb2-4da0-be2d-caea7734bbcb/network-metrics-daemon/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.102205 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881848 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.779789066s: [/var/lib/containers/storage/overlay/4a676e520264ef3cc5f5220c041fab55a3740033f9d9f3f0c382bc8105f63576/diff /var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c557ztrnm_37e90ab0-e9c0-420c-a0f4-c7923aa00297/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.102205 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881869 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778168595s: [/var/lib/containers/storage/overlay/61b109d2e47522fb73054e0ea35fe258491cc3975ae8ebfcce6b66801ef946db/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2_4ea6e69e-3556-4ffe-bb9e-796e8795d747/maas-api/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.102370 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881889 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.779344704s: [/var/lib/containers/storage/overlay/d68139bb7d74a6179bfbf30966bab9231b936b5fed801b743293927edf9117a4/diff /var/log/pods/openshift-image-registry_node-ca-kzr2l_2398ba99-35d3-4461-b66e-f44ac013b2d2/node-ca/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.102370 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881908 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778202798s: [/var/lib/containers/storage/overlay/0cf3a6fbf2a9bc19ba19b0638dd47eabc50ce49bea682a54d2c9f0fc3e9857bb/diff /var/log/pods/openshift-ingress_e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl_b962b15a-12b8-4b36-bc1d-ce5ed903f1d3/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.102370 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881932 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.77822335s: [/var/lib/containers/storage/overlay/599ba7e3c05d74cc5b782e3f79128931abfe2e885836c5b495046f23b84a4e4a/diff /var/log/pods/openshift-ingress_e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd_9d60eb56-9852-4c25-ae50-8a932e752793/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.102370 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881952 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778191647s: [/var/lib/containers/storage/overlay/cc64e95826aceb44df14fcfcfbeb1abcee57869143f35bd62776a6422cfda66c/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-464r7_c8683a65-9898-4b4f-a692-f2da325a93b4/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.102532 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881973 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778208679s: [/var/lib/containers/storage/overlay/a13dfa739c11295b8b17047cfd67cfcb736f6e317fc4fd8510561e20df9c398b/diff /var/log/pods/openshift-ingress_e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff_3b9d0847-df69-4b6a-b346-2cd206e42339/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.102532 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.881993 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778252051s: [/var/lib/containers/storage/overlay/3e134a1fe674754c9e9a38acd15a24432bc657789a1cd146ef7749623da3c543/diff /var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-8464c976fd-9b8p5_6785e24f-6fb3-43ab-a469-84d30d94b2f0/manager/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.102532 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.882014 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778281501s: [/var/lib/containers/storage/overlay/c1ac6b2ddb5ce33b84d769d5200a7b867b1dbeead0b830359507083657717be0/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq_0d960e4e-9539-4a53-9df7-1c520ab7d708/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.102532 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.882033 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.338746559s: [/var/lib/containers/storage/overlay/15e71d53803ae942e3049eeff235ba243399e5cdb82a33d54954a3a1b2a1f3be/diff /var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-h9f7b_8d0db609-b271-4c98-a303-fb3ee71f152c/cluster-monitoring-operator/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.102802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.882053 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778290623s: [/var/lib/containers/storage/overlay/d209d2a4eaaa3bcefdb4f01ea0aa4775a1ed9635fee488a5dab9a0c0f5cea560/diff /var/log/pods/openshift-ingress_e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r_34880975-a897-4ad4-af59-18617a93cdad/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.102802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.882073 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.77831451s: [/var/lib/containers/storage/overlay/f254a34c2d7b07cdb8243cfef02ae48f7764efa44a95f01755e0ef12b15fb3dc/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj_06b78cd0-94df-40f9-a1ed-87c1751675d3/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.102802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.882095 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778378954s: [/var/lib/containers/storage/overlay/1f29dc544a15045e1dc5d30c93ff89b9d8dc147583a95169dc0867bf987d73b4/diff /var/log/pods/openshift-ingress_e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5_f0d3c590-f523-4b7e-9086-60df5fc585e8/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.102802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.882116 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.706307866s: [/var/lib/containers/storage/overlay/faed3101d2fbf6ec02b09ee279cb2ef2e02bff6c99401bdbf2f7da9c4678b55c/diff /var/log/pods/openshift-ingress_e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk_8b648c6b-1f7d-42ca-9c3f-a435494bad03/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.102993 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.882135 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.706326175s: [/var/lib/containers/storage/overlay/de6ca20b9377c28cabe4038e64abbe6a95d1dd277c5ca5ddfa3af5b8c5e76c54/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-gb8r8_e64f9d9f-c868-470e-bd78-4ce8baabd97a/csi-liveness-probe/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.102993 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.882155 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.70634419s: [/var/lib/containers/storage/overlay/421c143b7c80827d2cc5a062879acfb554cf4fb285709dba5f301636b77ba0e8/diff /var/log/pods/kube-system_global-pull-secret-syncer-wh9tj_116af81a-d3a0-4558-93c4-8bf061b6815f/global-pull-secret-syncer/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.102993 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.882182 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.778446019s: [/var/lib/containers/storage/overlay/525f162f804c3962fa3a18a74a23555a9451e6576687225a267ea143486f4ee8/diff /var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp_e7cf1267-7d06-459b-9df7-fc7c477d36e1/main/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.102993 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.882201 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.706387972s: [/var/lib/containers/storage/overlay/0610b8954c6e6648affb0dd0874ae4c15a85f66db8f91e1f59381d694d28fbe6/diff /var/log/pods/openshift-insights_insights-operator-7c4986d449-wdls7_40834bff-2db0-4690-9cb9-7b4aacdab3ab/insights-operator/1.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.103142 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.882222 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.706407014s: [/var/lib/containers/storage/overlay/25fd98fb3d4fc0e795fcacdfc638e10cf768ef8545540e30d90b8efdb375bec1/diff /var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-qchj6_ad491ce3-2f92-45a7-9b4f-278309989e64/prometheus-operator-admission-webhook/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.103142 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.882243 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.706426272s: [/var/lib/containers/storage/overlay/644b60c0f8b58a334a1a7fc4a6319153055998acb8143ecbe17f7375356566c1/diff /var/log/pods/openshift-insights_insights-runtime-extractor-hmnx4_377ffcf5-b097-495d-b986-e61c58ae8ec1/extractor/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.103142 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.882261 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.706441003s: [/var/lib/containers/storage/overlay/db08eff9b22a4500cec0d1680cd6484dadd3fdd3de35cc7f22a12a0590ccdc68/diff /var/log/pods/openshift-dns_node-resolver-4d4ll_94d082d5-1c33-4394-9d93-67645f9b60cd/dns-node-resolver/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.103142 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.882281 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.706459289s: [/var/lib/containers/storage/overlay/b6cd9e4f4ab5200d0481e1c96d9431f1e12949004fd0870f5eab0a08be25c4ee/diff /var/log/pods/openshift-ingress-canary_ingress-canary-fgdh8_18ffe451-b8eb-41ad-a19d-0390f3e1f8e2/serve-healthcheck-canary/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.103291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.882301 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.706478617s: [/var/lib/containers/storage/overlay/f26445c31409eb167bf27cc2cbab2017408dbd457f03729e67cd2f30f1ea53f0/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2m4mq_c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/northd/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.103291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.882320 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.7064963s: [/var/lib/containers/storage/overlay/ccf806ac6ec23ad74bcc74db02ca5f266c08af5a5088b87059866bfd5ad3a508/diff /var/log/pods/openshift-multus_multus-additional-cni-plugins-cxdrc_b6f0d931-3a92-424f-a5fc-d9ebef985dfd/kube-multus-additional-cni-plugins/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.103291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.882340 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.706513417s: [/var/lib/containers/storage/overlay/29312326d1c5d67a009d3f2a0ead3df745c17fd18ca61679cf5a8d648501ee1c/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2m4mq_c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/sbdb/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.103291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.882359 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.706526608s: [/var/lib/containers/storage/overlay/5630a22ba1b338988a20699748117abbe435074b2a4d9dcff245e43eb9767391/diff /var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal_ea3063cb5f7f736e0060bab60ff36a5e/haproxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.103443 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.882378 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.706543505s: [/var/lib/containers/storage/overlay/d3141a050734ea30444cd519454c37c2cee0e793a05fad7726c15fd6b8e5ce74/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2m4mq_c6f8e09a-ee3b-4c5e-acd1-bce32adc0a42/ovnkube-controller/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.103443 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.882397 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.706561919s: [/var/lib/containers/storage/overlay/eea3e87288386c0194020c711ea81acb99383e2d30a4286346e27933ee057561/diff /var/log/pods/openshift-multus_network-metrics-daemon-qvxjq_caf7d6c9-0bb2-4da0-be2d-caea7734bbcb/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.103443 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.882418 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.706580751s: [/var/lib/containers/storage/overlay/46b70b8d6a52bbe639d1b79f0e47e999bf5c479db6750a707367f79d848146ce/diff /var/log/pods/openshift-dns_dns-default-nhf67_419841be-7a31-4cdb-b0b7-9d90a3c376c7/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.103443 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.882436 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.706598249s: [/var/lib/containers/storage/overlay/9df3bda628332b9dc87ebcd7ad96757f67125cc8d0ed149f1b9cf8ae1a6abd67/diff /var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-86qxx_40b2621f-854e-44e0-9b01-c877e333f644/cluster-samples-operator/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.103599 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.882454 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 29.917829592s: [/var/lib/containers/storage/overlay/99057c763637840ee3d311f80feb2ce08c68c4bd388fa3db08885cc03330a746/diff /var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-86qxx_40b2621f-854e-44e0-9b01-c877e333f644/cluster-samples-operator-watch/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.103599 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.882473 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 28.706660919s: [/var/lib/containers/storage/overlay/3590b0870ebbb7484f75a61269deee6b5456d7e78326def3787ba771f3783e4d/diff /var/log/pods/openshift-multus_multus-xlfb8_c6025fec-2ce4-4638-9a74-1f58c7ba7e8b/kube-multus/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.103599 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:45.885390 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@41-55151-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf485893c_6f14_449b_b837_8ac181315aeb.slice/crio-5d874b39931c2408a3ab4748545ce2ab475106ee717ad14248ee3ca358ef09d0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8992fc_fba6_4135_90f8_fb259e765dd5.slice/crio-a9bfcc2a7fbcc7ee6fea838b485c6e2a9dbb15881176ff436fbfec03c0df0eaf.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:32:51.103599 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.863640 5059 ???:1] "http: TLS handshake error from 10.0.133.249:53309: EOF" Nov 03 20:32:51.103599 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.885768 5059 ???:1] "http: TLS handshake error from 10.0.133.249:38815: EOF" Nov 03 20:32:51.103869 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.885940 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.783909134s: [/var/lib/containers/storage/overlay/0ff717d5140c034ea683dfff3cefbceae3d1da30d032121ac374b7822fbfed17/diff ]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.103869 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:45.893964 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@41-55151-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1731953b_1850_40d1_9ba4_cb08dc4ebc5d.slice/crio-073b6545188a48162ae3968a113b770a3a9d7b5615b65392e4809e1551e74cc2.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8992fc_fba6_4135_90f8_fb259e765dd5.slice/crio-a9bfcc2a7fbcc7ee6fea838b485c6e2a9dbb15881176ff436fbfec03c0df0eaf.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf485893c_6f14_449b_b837_8ac181315aeb.slice/crio-5d874b39931c2408a3ab4748545ce2ab475106ee717ad14248ee3ca358ef09d0.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:32:51.103869 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:45.894871 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.791149967s: [/var/lib/containers/storage/overlay/8620f98241dd1d45a676340e68150fcebbdb656ee3ea2ca830fb8502bdcc7302/diff /var/log/pods/openshift-ingress_e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w_cef4417c-3d04-4d65-a5bc-6ec1af6f09bb/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.103996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.437707 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m51.331991653s: [/var/lib/containers/storage/overlay/ab20ddba4ce467229f22d33db5fafd5d048d7904e2d5f46fdda07463291e410d/diff /var/log/pods/openshift-ingress_e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g_5ea38154-8a1f-4e1f-8fa0-50481ba4fec0/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s Nov 03 20:32:51.103996 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:46.440084 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@41-55151-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8992fc_fba6_4135_90f8_fb259e765dd5.slice/crio-a9bfcc2a7fbcc7ee6fea838b485c6e2a9dbb15881176ff436fbfec03c0df0eaf.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf485893c_6f14_449b_b837_8ac181315aeb.slice/crio-5d874b39931c2408a3ab4748545ce2ab475106ee717ad14248ee3ca358ef09d0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1731953b_1850_40d1_9ba4_cb08dc4ebc5d.slice/crio-073b6545188a48162ae3968a113b770a3a9d7b5615b65392e4809e1551e74cc2.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:32:51.103996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.442409 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.896360358s: [/var/lib/containers/storage/overlay/98c6bfc9ab89d11a4ca2043d5cabcbe5116c9dff2217abac8fcdf7162650651c/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl_9c3d72b6-73c4-402f-8308-ee5d87a9dd83/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.104167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.442613 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m51.338882925s: [/var/lib/containers/storage/overlay/6c0972a025a1d080f263d3e1efa9ee45ba0cf370ca3a4b39a456b1e8d775f4c7/diff /var/log/pods/openshift-ingress_e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk_edcfb7ab-b14f-4d5c-b752-5b43a6eee163/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.104167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.442769 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m51.337058832s: [/var/lib/containers/storage/overlay/cf20c89a59abe7a607a4c4fc3989a0700c8b52ef6b4b7ec8f4ce59b58a807569/diff /var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5_541bc63e-66f1-42ca-a946-aedc34beb62f/main/0.log]; will not log again for this container unless duration exceeds 4s Nov 03 20:32:51.104167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.442859 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m51.337147371s: [/var/lib/containers/storage/overlay/78d5a25fd67befb86fd5ea2280d7175455ecdb53478522ad87a939cb0ef5c7c1/diff /var/log/pods/cert-manager-operator_cert-manager-operator-controller-manager-7d87bcd5fd-mm85p_532de14d-f2dd-4c66-ba6f-8b103f7c9375/cert-manager-operator/0.log]; will not log again for this container unless duration exceeds 4s Nov 03 20:32:51.104167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.442987 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m51.337280073s: [/var/lib/containers/storage/overlay/9f34caa2489d14df64cdc8294e5fd26b65b76f5be247c8fbf4f6e0e4303813eb/diff /var/log/pods/openshift-ingress_e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn_8153e018-7f9a-4645-b102-06ff3913f682/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s Nov 03 20:32:51.104326 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.443189 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m51.337479355s: [/var/lib/containers/storage/overlay/94e23a93257da876c38398186f8a6015998dba78499a0f10e1ac126481bbe14d/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-7krsk_4946e3aa-a961-415a-b71d-95164d5fbca3/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.104326 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.452904 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.912817613s: [/var/lib/containers/storage/overlay/daf6579fec241cf0ecce296260dfaa6f1c89bdaaf862dc139e0bb6d7b465719f/diff /var/log/pods/e2e-models-e2e-worker-w1-32e482_simulator-w1-32e482-kserve-776bf6595d-mgx55_16c463fa-8ddf-43c8-b00f-3f5f9ca33c02/main/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.104326 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.453108 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.909817798s: [/var/lib/containers/storage/overlay/160601fcbeefc18ee81d0c7f6eb1441d505559030088ac0f615f82da9ad436a4/diff /var/log/pods/openshift-ingress_e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l_05fceb9c-4a04-470f-a94a-055121319f82/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.104326 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.453284 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m51.349519s: [/var/lib/containers/storage/overlay/dd8d2e028364b69cdce847e6e49ad546ee219955d9ccb8ac2291b09be3224a75/diff /var/log/pods/openshift-ingress_e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc_474b2e1d-832b-4edd-8765-31e0bc77cc2a/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.104495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.453404 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m50.910118102s: [/var/lib/containers/storage/overlay/233ddc6796fc9d42177ba3d89ce263361d459ea54af88fb400264c7f01dee413/diff /var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-c2m84_4a35329c-82fb-479f-abca-bff9ed082558/manager/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.104495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.453755 5059 trace.go:236] Trace[1425592708]: "Calculate volume metrics of tmp for pod openshift-console/downloads-6dd789c8b7-c88f8" (03-Nov-2026 20:30:55.108) (total time: 111344ms): Nov 03 20:32:51.104495 ip-10-0-132-83 kubenswrapper[5059]: Trace[1425592708]: [1m51.344803285s] [1m51.344803285s] END Nov 03 20:32:51.104495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.455280 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m51.352732713s: [/var/lib/containers/storage/overlay/42df31a418b95b1237dc0b8536ace06c25a4eac010478e46aeba8d03b09ef4b2/diff /var/log/pods/openshift-ingress_e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj_c9721811-d27d-4e13-a53d-4ae81e16fb96/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.104495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.456087 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 47.555178451s: [/var/lib/containers/storage/overlay/de16d7c33f60c35b85e789b6226417a11d8d0fe2e80e094daa63d5abb746ba38/diff /var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-l5g9c_64153a99-d5f7-4f92-a0c9-f79b39eba97c/discovery/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.104495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.467509 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m51.361802028s: [/var/lib/containers/storage/overlay/7e28be4717606b4b52525a911bdb8c86f57f2856bebf297a1322355bafc3e847/diff /var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-d7wsq_5ac27a9c-459f-4310-aef7-8dc643b46c6c/main/0.log]; will not log again for this container unless duration exceeds 4s Nov 03 20:32:51.104751 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:46.490766 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@41-55151-0.service\": RecentStats: unable to find data in memory cache]" Nov 03 20:32:51.104751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.512232 5059 patch_prober.go:29] interesting pod/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Nov 03 20:32:51.104751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.512280 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" podUID="1731953b-1850-40d1-9ba4-cb08dc4ebc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Nov 03 20:32:51.104751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.521408 5059 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-hmc5z container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.20:8443/healthz\": dial tcp 10.132.0.20:8443: connect: connection refused" start-of-body= Nov 03 20:32:51.104751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.521468 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" podUID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.20:8443/healthz\": dial tcp 10.132.0.20:8443: connect: connection refused" Nov 03 20:32:51.104751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.523426 5059 patch_prober.go:29] interesting pod/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Nov 03 20:32:51.105012 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.523475 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" podUID="1731953b-1850-40d1-9ba4-cb08dc4ebc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Nov 03 20:32:51.105012 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.531740 5059 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-hmc5z container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.20:8443/healthz\": dial tcp 10.132.0.20:8443: connect: connection refused" start-of-body= Nov 03 20:32:51.105012 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.531788 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" podUID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.20:8443/healthz\": dial tcp 10.132.0.20:8443: connect: connection refused" Nov 03 20:32:51.105012 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.563267 5059 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-hmc5z container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused" start-of-body= Nov 03 20:32:51.105012 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.563326 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" podUID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused" Nov 03 20:32:51.105012 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.595283 5059 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-hmc5z container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused" start-of-body= Nov 03 20:32:51.105268 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.595848 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" podUID="270c522c-ef72-46a7-9e65-3be2ebdbcfa6" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.35:8081/readyz\": dial tcp 10.132.0.35:8081: connect: connection refused" Nov 03 20:32:51.105268 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.601918 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" podUID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused" Nov 03 20:32:51.105268 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:46.626050 5059 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="6m34.776s" Nov 03 20:32:51.105268 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:46.626086 5059 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime is down, PLEG is not healthy: pleg was last seen active 6m30.106192555s ago; threshold is 3m0s]" Nov 03 20:32:51.105268 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.633235 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" podUID="270c522c-ef72-46a7-9e65-3be2ebdbcfa6" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.35:8081/healthz\": dial tcp 10.132.0.35:8081: connect: connection refused" Nov 03 20:32:51.105268 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.670331 5059 patch_prober.go:29] interesting pod/jobset-controller-manager-5d9f95b69-9vwq7 container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.132.0.44:8081/readyz\": dial tcp 10.132.0.44:8081: connect: connection refused" start-of-body= Nov 03 20:32:51.105508 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.670404 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" podUID="1072628f-f513-4b4e-8954-afee5a1aa733" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.44:8081/readyz\": dial tcp 10.132.0.44:8081: connect: connection refused" Nov 03 20:32:51.105508 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.677954 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" podUID="270c522c-ef72-46a7-9e65-3be2ebdbcfa6" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.35:8081/healthz\": dial tcp 10.132.0.35:8081: connect: connection refused" Nov 03 20:32:51.105508 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.672548 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" podUID="270c522c-ef72-46a7-9e65-3be2ebdbcfa6" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.35:8081/readyz\": dial tcp 10.132.0.35:8081: connect: connection refused" Nov 03 20:32:51.105508 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.688807 5059 patch_prober.go:29] interesting pod/jobset-controller-manager-5d9f95b69-9vwq7 container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.132.0.44:8081/readyz\": dial tcp 10.132.0.44:8081: connect: connection refused" start-of-body= Nov 03 20:32:51.105508 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.688857 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" podUID="1072628f-f513-4b4e-8954-afee5a1aa733" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.44:8081/readyz\": dial tcp 10.132.0.44:8081: connect: connection refused" Nov 03 20:32:51.105508 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.721773 5059 patch_prober.go:29] interesting pod/jobset-controller-manager-5d9f95b69-9vwq7 container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.132.0.44:8081/healthz\": dial tcp 10.132.0.44:8081: connect: connection refused" start-of-body= Nov 03 20:32:51.105732 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.721846 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" podUID="1072628f-f513-4b4e-8954-afee5a1aa733" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.44:8081/healthz\": dial tcp 10.132.0.44:8081: connect: connection refused" Nov 03 20:32:51.105732 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.725458 5059 patch_prober.go:29] interesting pod/jobset-controller-manager-5d9f95b69-9vwq7 container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.132.0.44:8081/healthz\": dial tcp 10.132.0.44:8081: connect: connection refused" start-of-body= Nov 03 20:32:51.105732 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.725512 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" podUID="1072628f-f513-4b4e-8954-afee5a1aa733" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.44:8081/healthz\": dial tcp 10.132.0.44:8081: connect: connection refused" Nov 03 20:32:51.105732 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:46.727702 5059 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime is down, PLEG is not healthy: pleg was last seen active 6m30.207792178s ago; threshold is 3m0s]" Nov 03 20:32:51.105732 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.881090 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5m29.818377074s: [/var/lib/containers/storage/overlay/638ecbffe5310c1de800b8f00d6b4f107f4ec383be032894ad1bb999bd9aa5f3/diff /var/log/pods/openshift-ingress_router-default-f7b87d695-cwlfq_603868b0-8628-4227-8471-1894e2e1de50/router/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 20:32:51.105732 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:46.928715 5059 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime is down, PLEG is not healthy: pleg was last seen active 6m30.408807607s ago; threshold is 3m0s]" Nov 03 20:32:51.105952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:46.988929 5059 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m51.883233775s: [/var/lib/containers/storage/overlay/70930d38077b10a20b0ce4df30e17d1da1c20ec8bcdce6a5603b005d7a575af2/diff /var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-846mt_04a982f9-143c-406c-99d5-9ef73d5c571a/networking-console-plugin/0.log]; will not log again for this container unless duration exceeds 3s Nov 03 20:32:51.105952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.051906 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" podUID="270c522c-ef72-46a7-9e65-3be2ebdbcfa6" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.35:8081/healthz\": dial tcp 10.132.0.35:8081: connect: connection refused" Nov 03 20:32:51.105952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.052886 5059 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:32:51.105952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.060596 5059 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:32:51.105952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.060835 5059 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:32:51.105952 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:47.329644 5059 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime is down, PLEG is not healthy: pleg was last seen active 6m30.80974183s ago; threshold is 3m0s]" Nov 03 20:32:51.105952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.390641 5059 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:32:51.105952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.500330 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:51.106236 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.500565 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt" podUID="c92b1be0-0890-4db6-ac75-968b6e5a3434" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.73:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.106236 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.500724 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2" podUID="cfcb49c5-4381-411e-9d64-d9dcd6181239" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.74:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.106236 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.500849 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p" podUID="ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.75:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.106236 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.501945 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq" podUID="0d960e4e-9539-4a53-9df7-1c520ab7d708" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.72:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.106236 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.505606 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:51.106411 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.506009 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded" Nov 03 20:32:51.106411 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.506321 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" podUID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Nov 03 20:32:51.106411 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.524750 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 20:32:51.106411 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.525394 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:32:51.106411 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.526697 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" podUID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:51.106411 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.527008 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:32:51.106630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.529280 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7" podUID="2c363281-0226-420f-a963-5922d5ca609e" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.255:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.106630 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:47.541882 5059 logging.go:55] [core] [Channel #6672 SubChannel #6676]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.73:50052", ServerName: "10.132.0.73:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:35698->10.132.0.73:50052: use of closed network connection" Nov 03 20:32:51.106630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.543467 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj" podUID="06b78cd0-94df-40f9-a1ed-87c1751675d3" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.234:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.106630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.544003 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8" podUID="705dea2b-ad85-4b1c-a3c6-a9f705ac9039" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.235:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.106630 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:47.544876 5059 logging.go:55] [core] [Channel #6671 SubChannel #6675]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.75:50052", ServerName: "10.132.0.75:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:59820->10.132.0.75:50052: use of closed network connection" Nov 03 20:32:51.106847 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:47.544960 5059 logging.go:55] [core] [Channel #6670 SubChannel #6674]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.74:50052", ServerName: "10.132.0.74:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:60344->10.132.0.74:50052: use of closed network connection" Nov 03 20:32:51.106847 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:47.549252 5059 logging.go:55] [core] [Channel #6673 SubChannel #6677]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.72:50052", ServerName: "10.132.0.72:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:39928->10.132.0.72:50052: use of closed network connection" Nov 03 20:32:51.106847 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.570488 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84" podUID="4a35329c-82fb-479f-abca-bff9ed082558" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:51.106847 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.571720 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:51.106847 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.572056 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded" Nov 03 20:32:51.108018 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.572425 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" podUID="0309a608-f6c4-45ca-991b-33ba73221a6c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.64:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.108018 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.573602 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn" podUID="25ceab3d-6344-4512-a471-d91aec0cb48c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.71:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.108018 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.573786 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Nov 03 20:32:51.108018 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.576105 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps" podUID="2faa14fa-ec86-4633-a968-c3f761d13443" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.242:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.108018 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.576894 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9" podUID="890f130b-cdeb-43da-acd4-1654673271cc" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.70:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.108229 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.580700 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:51.108229 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.581725 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:51.108229 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.587164 5059 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-c88f8 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:32:51.108229 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.587209 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-c88f8" podUID="6a7937ca-c705-4091-967e-2e640c622a26" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:51.108229 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:47.588044 5059 logging.go:55] [core] [Channel #6684 SubChannel #6685]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.234:50052", ServerName: "10.132.0.234:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:51190->10.132.0.234:50052: use of closed network connection" Nov 03 20:32:51.108439 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:47.590639 5059 logging.go:55] [core] [Channel #6683 SubChannel #6687]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.235:50052", ServerName: "10.132.0.235:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:42746->10.132.0.235:50052: use of closed network connection" Nov 03 20:32:51.108439 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:47.594244 5059 logging.go:55] [core] [Channel #6680 SubChannel #6681]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.255:50052", ServerName: "10.132.0.255:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:51606->10.132.0.255:50052: use of closed network connection" Nov 03 20:32:51.108439 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.594290 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" podUID="6785e24f-6fb3-43ab-a469-84d30d94b2f0" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.34:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:51.108439 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:47.612827 5059 logging.go:55] [core] [Channel #6693 SubChannel #6697]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.64:50052", ServerName: "10.132.1.64:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:50284->10.132.1.64:50052: use of closed network connection" Nov 03 20:32:51.108603 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.612855 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nj52h" podUID="941b16dd-0550-4434-86f9-26e38efb68ee" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.37:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:51.108603 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:47.614891 5059 logging.go:55] [core] [Channel #6694 SubChannel #6698]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.71:50052", ServerName: "10.132.0.71:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:57660->10.132.0.71:50052: use of closed network connection" Nov 03 20:32:51.108603 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.617550 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44" podUID="26375297-81f4-427f-94c3-d56f7a6a11fb" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.244:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.108603 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:47.617871 5059 logging.go:55] [core] [Channel #6692 SubChannel #6696]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.70:50052", ServerName: "10.132.0.70:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:47288->10.132.0.70:50052: use of closed network connection" Nov 03 20:32:51.108603 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:47.618721 5059 logging.go:55] [core] [Channel #6691 SubChannel #6695]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.242:50052", ServerName: "10.132.0.242:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:35832->10.132.0.242:50052: use of closed network connection" Nov 03 20:32:51.108834 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.622812 5059 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-c88f8 container/download-server namespace/openshift-console: Liveness probe status=failure output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:32:51.108834 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.622897 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nj52h" podUID="941b16dd-0550-4434-86f9-26e38efb68ee" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.37:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:51.108834 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.622930 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console/downloads-6dd789c8b7-c88f8" podUID="6a7937ca-c705-4091-967e-2e640c622a26" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:51.108834 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.624776 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:51.108834 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:47.658896 5059 logging.go:55] [core] [Channel #6701 SubChannel #6702]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.244:50052", ServerName: "10.132.0.244:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:34638->10.132.0.244:50052: use of closed network connection" Nov 03 20:32:51.109036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.776626 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl" podUID="9c3d72b6-73c4-402f-8308-ee5d87a9dd83" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.69:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.109036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.786850 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84" podUID="4a35329c-82fb-479f-abca-bff9ed082558" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:51.109036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.807806 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" podUID="6785e24f-6fb3-43ab-a469-84d30d94b2f0" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.34:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:51.109036 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:47.817847 5059 logging.go:55] [core] [Channel #6706 SubChannel #6707]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.69:50052", ServerName: "10.132.0.69:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:56328->10.132.0.69:50052: use of closed network connection" Nov 03 20:32:51.109036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.829387 5059 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-hmc5z container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.20:8443/healthz\": dial tcp 10.132.0.20:8443: connect: connection refused" start-of-body= Nov 03 20:32:51.109250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:47.829422 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" podUID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.20:8443/healthz\": dial tcp 10.132.0.20:8443: connect: connection refused" Nov 03 20:32:51.109250 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:48.130269 5059 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime is down, PLEG is not healthy: pleg was last seen active 6m31.610362966s ago; threshold is 3m0s]" Nov 03 20:32:51.109250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:48.501793 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt" podUID="c92b1be0-0890-4db6-ac75-968b6e5a3434" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.73:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.109250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:48.501811 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p" podUID="ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.75:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.109250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:48.501809 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2" podUID="cfcb49c5-4381-411e-9d64-d9dcd6181239" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.74:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.109250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:48.502816 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq" podUID="0d960e4e-9539-4a53-9df7-1c520ab7d708" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.72:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.109552 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:48.530390 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7" podUID="2c363281-0226-420f-a963-5922d5ca609e" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.255:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.109552 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:48.542925 5059 logging.go:55] [core] [Channel #6711 SubChannel #6719]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.75:50052", ServerName: "10.132.0.75:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:59842->10.132.0.75:50052: use of closed network connection" Nov 03 20:32:51.109552 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:48.543228 5059 logging.go:55] [core] [Channel #6709 SubChannel #6714]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.73:50052", ServerName: "10.132.0.73:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:35712->10.132.0.73:50052: use of closed network connection" Nov 03 20:32:51.109552 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:48.543780 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj" podUID="06b78cd0-94df-40f9-a1ed-87c1751675d3" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.234:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.109754 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:48.544397 5059 logging.go:55] [core] [Channel #6712 SubChannel #6713]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.72:50052", ServerName: "10.132.0.72:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:39952->10.132.0.72:50052: use of closed network connection" Nov 03 20:32:51.109754 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:48.544430 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8" podUID="705dea2b-ad85-4b1c-a3c6-a9f705ac9039" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.235:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.109754 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:48.546556 5059 logging.go:55] [core] [Channel #6710 SubChannel #6717]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.74:50052", ServerName: "10.132.0.74:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:60370->10.132.0.74:50052: use of closed network connection" Nov 03 20:32:51.109754 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:48.571889 5059 logging.go:55] [core] [Channel #6721 SubChannel #6722]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.255:50052", ServerName: "10.132.0.255:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:51616->10.132.0.255:50052: use of closed network connection" Nov 03 20:32:51.109927 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:48.572807 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" podUID="0309a608-f6c4-45ca-991b-33ba73221a6c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.64:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.109927 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:48.574369 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn" podUID="25ceab3d-6344-4512-a471-d91aec0cb48c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.71:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.109927 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:48.576970 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps" podUID="2faa14fa-ec86-4633-a968-c3f761d13443" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.242:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.109927 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:48.578817 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9" podUID="890f130b-cdeb-43da-acd4-1654673271cc" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.70:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.109927 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:48.584892 5059 logging.go:55] [core] [Channel #6725 SubChannel #6727]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.235:50052", ServerName: "10.132.0.235:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:42790->10.132.0.235:50052: use of closed network connection" Nov 03 20:32:51.110134 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:48.586317 5059 logging.go:55] [core] [Channel #6724 SubChannel #6726]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.234:50052", ServerName: "10.132.0.234:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:51222->10.132.0.234:50052: use of closed network connection" Nov 03 20:32:51.110134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:48.612841 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84" podUID="4a35329c-82fb-479f-abca-bff9ed082558" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:51.110134 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:48.613876 5059 logging.go:55] [core] [Channel #6730 SubChannel #6732]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.64:50052", ServerName: "10.132.1.64:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:50312->10.132.1.64:50052: use of closed network connection" Nov 03 20:32:51.110134 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:48.614855 5059 logging.go:55] [core] [Channel #6731 SubChannel #6733]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.71:50052", ServerName: "10.132.0.71:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:57686->10.132.0.71:50052: use of closed network connection" Nov 03 20:32:51.110134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:48.617954 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44" podUID="26375297-81f4-427f-94c3-d56f7a6a11fb" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.244:50052\" within 1s: context deadline exceeded" Nov 03 20:32:51.110360 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:48.618193 5059 logging.go:55] [core] [Channel #6736 SubChannel #6738]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.242:50052", ServerName: "10.132.0.242:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:35842->10.132.0.242:50052: use of closed network connection" Nov 03 20:32:51.110360 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:48.620907 5059 logging.go:55] [core] [Channel #6737 SubChannel #6740]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.70:50052", ServerName: "10.132.0.70:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:47330->10.132.0.70:50052: use of closed network connection" Nov 03 20:32:51.110360 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:48.630087 5059 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-c88f8 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:32:51.110360 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:48.630134 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-c88f8" podUID="6a7937ca-c705-4091-967e-2e640c622a26" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:51.110360 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:48.637872 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" podUID="6785e24f-6fb3-43ab-a469-84d30d94b2f0" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.34:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:51.110567 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:48.659818 5059 logging.go:55] [core] [Channel #6742 SubChannel #6743]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.244:50052", ServerName: "10.132.0.244:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:34648->10.132.0.244:50052: use of closed network connection" Nov 03 20:32:51.110567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:48.663839 5059 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-c88f8 container/download-server namespace/openshift-console: Liveness probe status=failure output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:32:51.110567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:48.663887 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console/downloads-6dd789c8b7-c88f8" podUID="6a7937ca-c705-4091-967e-2e640c622a26" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:51.110567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:48.727406 5059 trace.go:236] Trace[1894338313]: "Calculate volume metrics of ca-trust-extracted for pod openshift-image-registry/image-registry-6c57db574-fmm6v" (03-Nov-2026 20:27:17.403) (total time: 331323ms): Nov 03 20:32:51.110567 ip-10-0-132-83 kubenswrapper[5059]: Trace[1894338313]: [5m31.323965093s] [5m31.323965093s] END Nov 03 20:32:51.110567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:48.817790 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl" podUID="9c3d72b6-73c4-402f-8308-ee5d87a9dd83" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 03 20:32:51.110935 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:48.827813 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84" podUID="4a35329c-82fb-479f-abca-bff9ed082558" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:51.110935 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:48.849820 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" podUID="6785e24f-6fb3-43ab-a469-84d30d94b2f0" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.34:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:51.110935 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:49.520135 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" podUID="4ea6e69e-3556-4ffe-bb9e-796e8795d747" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.1.58:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:51.110935 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:49.559837 5059 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-gb8r8 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:32:51.110935 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:49.559916 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" podUID="e64f9d9f-c868-470e-bd78-4ce8baabd97a" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:51.111155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:49.561799 5059 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-gb8r8 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:32:51.111155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:49.562126 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb8r8" podUID="e64f9d9f-c868-470e-bd78-4ce8baabd97a" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:51.111155 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:49.753723 5059 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime is down, PLEG is not healthy: pleg was last seen active 6m33.233811413s ago; threshold is 3m0s]" Nov 03 20:32:51.111155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:49.893862 5059 trace.go:236] Trace[515795347]: "iptables ChainExists" (03-Nov-2026 20:32:45.861) (total time: 4032ms): Nov 03 20:32:51.111155 ip-10-0-132-83 kubenswrapper[5059]: Trace[515795347]: [4.032599088s] [4.032599088s] END Nov 03 20:32:51.111155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:49.893943 5059 trace.go:236] Trace[1823242417]: "iptables ChainExists" (03-Nov-2026 20:30:55.540) (total time: 114353ms): Nov 03 20:32:51.111155 ip-10-0-132-83 kubenswrapper[5059]: Trace[1823242417]: [1m54.353805633s] [1m54.353805633s] END Nov 03 20:32:51.111155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:50.467876 5059 generic.go:356] "Generic (PLEG): container finished" podID="1072628f-f513-4b4e-8954-afee5a1aa733" containerID="cadce9a446527c8c3f13706fda86af3620c7a7c77a7b43d35d1a7f7d6579e0e8" exitCode=1 Nov 03 20:32:51.111155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:50.590929 5059 patch_prober.go:29] interesting pod/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Nov 03 20:32:51.111486 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:50.591002 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" podUID="1731953b-1850-40d1-9ba4-cb08dc4ebc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Nov 03 20:32:51.111486 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:50.599421 5059 generic.go:356] "Generic (PLEG): container finished" podID="270c522c-ef72-46a7-9e65-3be2ebdbcfa6" containerID="a4f29644d81f5639b20431cb580b7160e30c9a7f0f42d00f2d940270329ea0f4" exitCode=1 Nov 03 20:32:51.111486 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:50.711155 5059 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-hmc5z container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused" start-of-body= Nov 03 20:32:51.111486 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:50.711220 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" podUID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused" Nov 03 20:32:51.111486 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:50.760886 5059 trace.go:236] Trace[2142455482]: "Calculate volume metrics of data for pod odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts" (03-Nov-2026 20:27:16.202) (total time: 334558ms): Nov 03 20:32:51.111486 ip-10-0-132-83 kubenswrapper[5059]: Trace[2142455482]: [5m34.558447095s] [5m34.558447095s] END Nov 03 20:32:51.111486 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:50.887350 5059 kubelet.go:1648] "Container garbage collection failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" Nov 03 20:32:51.219700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:51.218258 5059 trace.go:236] Trace[739657709]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" (03-Nov-2026 20:27:56.724) (total time: 294493ms): Nov 03 20:32:51.219700 ip-10-0-132-83 kubenswrapper[5059]: Trace[739657709]: [4m54.49372011s] [4m54.49372011s] END Nov 03 20:32:51.262429 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:51.260154 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:32:51.320676 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:51.320589 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:32:51.335202 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:51.335087 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 20:32:51.338969 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:51.338878 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-fsbsp_1bcd4690-4753-4246-abb4-8668a0a56231/volume-data-source-validator/0.log" Nov 03 20:32:51.366930 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:51.366537 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" podUID="270c522c-ef72-46a7-9e65-3be2ebdbcfa6" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.35:8081/readyz\": dial tcp 10.132.0.35:8081: connect: connection refused" Nov 03 20:32:51.618514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:51.618234 5059 generic.go:356] "Generic (PLEG): container finished" podID="4a35329c-82fb-479f-abca-bff9ed082558" containerID="667c8d1cf790b30fc65fdbeb2860e35758fd0e7eb1773d3f331c5e009252f137" exitCode=1 Nov 03 20:32:51.624267 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:51.624238 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/1.log" Nov 03 20:32:51.624405 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:51.624283 5059 generic.go:356] "Generic (PLEG): container finished" podID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" containerID="b7138eb33c5d6fa9eeb22093ad7d51deb0e28bca02ed9a1262e5df9e2a221d24" exitCode=0 Nov 03 20:32:51.627459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:51.627429 5059 generic.go:356] "Generic (PLEG): container finished" podID="6785e24f-6fb3-43ab-a469-84d30d94b2f0" containerID="24306abe36f01f202cf4225862e6256f61b8116c2b16568f30c02b610183da1c" exitCode=1 Nov 03 20:32:51.629803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:51.629776 5059 generic.go:356] "Generic (PLEG): container finished" podID="1731953b-1850-40d1-9ba4-cb08dc4ebc5d" containerID="073b6545188a48162ae3968a113b770a3a9d7b5615b65392e4809e1551e74cc2" exitCode=0 Nov 03 20:32:51.631733 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:51.631713 5059 generic.go:356] "Generic (PLEG): container finished" podID="532de14d-f2dd-4c66-ba6f-8b103f7c9375" containerID="111ab0145bc09ea2159df08b45a75e0a16c06fb447f17cd20b906e1b1a7c091f" exitCode=0 Nov 03 20:32:51.633507 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:51.633485 5059 generic.go:356] "Generic (PLEG): container finished" podID="f485893c-6f14-449b-b837-8ac181315aeb" containerID="5d874b39931c2408a3ab4748545ce2ab475106ee717ad14248ee3ca358ef09d0" exitCode=0 Nov 03 20:32:51.635236 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:51.635214 5059 generic.go:356] "Generic (PLEG): container finished" podID="7b8992fc-fba6-4135-90f8-fb259e765dd5" containerID="a9bfcc2a7fbcc7ee6fea838b485c6e2a9dbb15881176ff436fbfec03c0df0eaf" exitCode=0 Nov 03 20:32:51.637240 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:51.637219 5059 generic.go:356] "Generic (PLEG): container finished" podID="492c7b4e-84c0-49a7-b12f-153a021f971b" containerID="8b68d8a54774f3a932e8c394d04b88681ee2fd3757c1b50ef500cf35cb4119f9" exitCode=0 Nov 03 20:32:51.643279 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:51.643179 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-fsbsp_1bcd4690-4753-4246-abb4-8668a0a56231/volume-data-source-validator/0.log" Nov 03 20:32:51.643279 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:51.643211 5059 generic.go:356] "Generic (PLEG): container finished" podID="1bcd4690-4753-4246-abb4-8668a0a56231" containerID="56aa53c1d031f7ab6c282efd0400e4a7abf4371a2d9df0ad33beb6d199334684" exitCode=1 Nov 03 20:32:53.108392 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.108359 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 20:32:53.112764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.112734 5059 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" Nov 03 20:32:53.112977 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.112960 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 20:32:53.113122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.113108 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" Nov 03 20:32:53.113774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.113747 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" event={"ID":"1072628f-f513-4b4e-8954-afee5a1aa733","Type":"ContainerDied","Data":"cadce9a446527c8c3f13706fda86af3620c7a7c77a7b43d35d1a7f7d6579e0e8"} Nov 03 20:32:53.113946 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.113928 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 20:32:53.129682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.114069 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" Nov 03 20:32:53.130883 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.130862 5059 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 20:32:53.131597 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.131576 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" event={"ID":"270c522c-ef72-46a7-9e65-3be2ebdbcfa6","Type":"ContainerDied","Data":"a4f29644d81f5639b20431cb580b7160e30c9a7f0f42d00f2d940270329ea0f4"} Nov 03 20:32:53.131751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.131732 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84" event={"ID":"4a35329c-82fb-479f-abca-bff9ed082558","Type":"ContainerDied","Data":"667c8d1cf790b30fc65fdbeb2860e35758fd0e7eb1773d3f331c5e009252f137"} Nov 03 20:32:53.150252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.141901 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" event={"ID":"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac","Type":"ContainerDied","Data":"b7138eb33c5d6fa9eeb22093ad7d51deb0e28bca02ed9a1262e5df9e2a221d24"} Nov 03 20:32:53.150252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.141948 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" event={"ID":"6785e24f-6fb3-43ab-a469-84d30d94b2f0","Type":"ContainerDied","Data":"24306abe36f01f202cf4225862e6256f61b8116c2b16568f30c02b610183da1c"} Nov 03 20:32:53.150252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.141976 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" event={"ID":"1731953b-1850-40d1-9ba4-cb08dc4ebc5d","Type":"ContainerDied","Data":"073b6545188a48162ae3968a113b770a3a9d7b5615b65392e4809e1551e74cc2"} Nov 03 20:32:53.150252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.142002 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mm85p" event={"ID":"532de14d-f2dd-4c66-ba6f-8b103f7c9375","Type":"ContainerDied","Data":"111ab0145bc09ea2159df08b45a75e0a16c06fb447f17cd20b906e1b1a7c091f"} Nov 03 20:32:53.150252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.142023 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6k9n9" event={"ID":"f485893c-6f14-449b-b837-8ac181315aeb","Type":"ContainerDied","Data":"5d874b39931c2408a3ab4748545ce2ab475106ee717ad14248ee3ca358ef09d0"} Nov 03 20:32:53.150252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.142044 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" event={"ID":"7b8992fc-fba6-4135-90f8-fb259e765dd5","Type":"ContainerDied","Data":"a9bfcc2a7fbcc7ee6fea838b485c6e2a9dbb15881176ff436fbfec03c0df0eaf"} Nov 03 20:32:53.150591 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.142068 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" event={"ID":"492c7b4e-84c0-49a7-b12f-153a021f971b","Type":"ContainerDied","Data":"8b68d8a54774f3a932e8c394d04b88681ee2fd3757c1b50ef500cf35cb4119f9"} Nov 03 20:32:53.150591 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.142093 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fsbsp" event={"ID":"1bcd4690-4753-4246-abb4-8668a0a56231","Type":"ContainerDied","Data":"56aa53c1d031f7ab6c282efd0400e4a7abf4371a2d9df0ad33beb6d199334684"} Nov 03 20:32:53.150591 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.142339 5059 scope.go:122] "RemoveContainer" containerID="7bbcf23cd573d5c9e806dd6604b68ae67663631e74f89f722cb793dfee52aa69" Nov 03 20:32:53.150591 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.141713 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0e4ce7-openshift-default-dockercfg-nppzh\"" Nov 03 20:32:53.150591 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.141846 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-b503a8-openshift-default-dockercfg-tchtj\"" Nov 03 20:32:53.152739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.151004 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-dq8cl\"" Nov 03 20:32:53.156685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.153227 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jzp4r\"" Nov 03 20:32:53.156685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.154512 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8tzmp\"" Nov 03 20:32:53.156685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.154790 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xr7jf\"" Nov 03 20:32:53.156685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.154860 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-81a5ae-openshift-default-dockercfg-pvl2f\"" Nov 03 20:32:53.156685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.154900 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"default-dockercfg-zrxnc\"" Nov 03 20:32:53.156685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.154944 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-88a18e-openshift-default-dockercfg-bs4qg\"" Nov 03 20:32:53.156685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.154983 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ndfpg\"" Nov 03 20:32:53.174373 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.169347 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"payload-processing-external-model-e2e--b1878920-dockercfg-vbr6x\"" Nov 03 20:32:53.179301 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.179236 5059 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" secret="" err="secret \"payload-processing-external-model-e2e--b1878920-dockercfg-vbr6x\" not found" Nov 03 20:32:53.444600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.444509 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bkd9f\"" Nov 03 20:32:53.445250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.445226 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0e1546-openshift-default-dockercfg-jcbdc\"" Nov 03 20:32:53.446025 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.446006 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2r65v\"" Nov 03 20:32:53.446874 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.446850 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qg9jj\"" Nov 03 20:32:53.447550 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.447534 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a0d4ec-dockercfg-d47xx\"" Nov 03 20:32:53.448440 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.448423 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"default-dockercfg-5r8nn\"" Nov 03 20:32:53.454177 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.454136 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d3d0df-openshift-default-dockercfg-m4lvp\"" Nov 03 20:32:53.454439 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.454417 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-33e267-openshift-default-dockercfg-lkhx5\"" Nov 03 20:32:53.455811 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.455649 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ldllm\"" Nov 03 20:32:53.455996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.455966 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vlmgx\"" Nov 03 20:32:53.456065 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.455995 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fw8qp\"" Nov 03 20:32:53.456460 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.456439 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vf7n8\"" Nov 03 20:32:53.456597 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.456573 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-65hnm\"" Nov 03 20:32:53.456902 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.456880 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lrbp4\"" Nov 03 20:32:53.457304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.457284 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-9f46e5-openshift-default-dockercfg-4djl8\"" Nov 03 20:32:53.495688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.458013 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-vx824\"" Nov 03 20:32:53.540481 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.540448 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3fbb1a-openshift-default-dockercfg-247ct\"" Nov 03 20:32:53.540831 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.465296 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bfvcm\"" Nov 03 20:32:53.541637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.541603 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-nh6bx\"" Nov 03 20:32:53.543575 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.466249 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sgdb8\"" Nov 03 20:32:53.544441 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.466539 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-k5vf6\"" Nov 03 20:32:53.545806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.466818 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lhxcg\"" Nov 03 20:32:53.545985 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.522251 5059 scope.go:122] "RemoveContainer" containerID="cadce9a446527c8c3f13706fda86af3620c7a7c77a7b43d35d1a7f7d6579e0e8" Nov 03 20:32:53.546193 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.546177 5059 scope.go:122] "RemoveContainer" containerID="667c8d1cf790b30fc65fdbeb2860e35758fd0e7eb1773d3f331c5e009252f137" Nov 03 20:32:53.550861 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.550831 5059 scope.go:122] "RemoveContainer" containerID="a9bfcc2a7fbcc7ee6fea838b485c6e2a9dbb15881176ff436fbfec03c0df0eaf" Nov 03 20:32:53.550938 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.550916 5059 scope.go:122] "RemoveContainer" containerID="a4f29644d81f5639b20431cb580b7160e30c9a7f0f42d00f2d940270329ea0f4" Nov 03 20:32:53.551028 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.551013 5059 scope.go:122] "RemoveContainer" containerID="b7138eb33c5d6fa9eeb22093ad7d51deb0e28bca02ed9a1262e5df9e2a221d24" Nov 03 20:32:53.551104 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.538321 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-f154ac-openshift-default-dockercfg-xndgl\"" Nov 03 20:32:53.551153 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.536644 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a0d4ec-openshift-default-dockercfg-gsslf\"" Nov 03 20:32:53.551153 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.537064 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-w84zh\"" Nov 03 20:32:53.551226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.537465 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-eb69cb-openshift-default-dockercfg-w96q5\"" Nov 03 20:32:53.551265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.538541 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qhpjg\"" Nov 03 20:32:53.551312 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.538887 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-h4l58\"" Nov 03 20:32:53.551351 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.539184 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rwt79\"" Nov 03 20:32:53.551406 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.539769 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-b22jw\"" Nov 03 20:32:53.551447 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.541638 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-43c674-openshift-default-dockercfg-slz8b\"" Nov 03 20:32:53.551496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.465537 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-lw7w7\"" Nov 03 20:32:53.551544 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.466101 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-p6m7s\"" Nov 03 20:32:53.551597 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.541960 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-4c2a28-dockercfg-psjnh\"" Nov 03 20:32:53.551649 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.522631 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-8b7351-openshift-default-dockercfg-lx6hb\"" Nov 03 20:32:53.555974 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.555953 5059 scope.go:122] "RemoveContainer" containerID="56aa53c1d031f7ab6c282efd0400e4a7abf4371a2d9df0ad33beb6d199334684" Nov 03 20:32:53.556695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.556676 5059 scope.go:122] "RemoveContainer" containerID="8b68d8a54774f3a932e8c394d04b88681ee2fd3757c1b50ef500cf35cb4119f9" Nov 03 20:32:53.589959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.558380 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 20:32:53.642973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.642942 5059 scope.go:122] "RemoveContainer" containerID="418374b5472981e5cdbb3a5be895780bfb3ad1fc0b28a5e01266b5d997136ed3" Nov 03 20:32:53.647164 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.647118 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-3fbb1a-dockercfg-b6ghv\"" Nov 03 20:32:53.647578 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.647555 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-2b5703-dockercfg-b84j2\"" Nov 03 20:32:53.648280 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.648259 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-dkxw8\"" Nov 03 20:32:53.651467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.651450 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"default-dockercfg-lxnx2\"" Nov 03 20:32:53.651855 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.651821 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-wftlv\"" Nov 03 20:32:53.676650 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.676612 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 20:32:53.676892 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.652558 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 20:32:53.677096 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.652631 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-bfc240-openshift-default-dockercfg-dxqxq\"" Nov 03 20:32:53.677266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.661647 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-32e482-dockercfg-4mbnz\"" Nov 03 20:32:53.692628 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.652115 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-2b5703-openshift-default-dockercfg-jmjbm\"" Nov 03 20:32:53.697717 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.694705 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-942q2\"" Nov 03 20:32:53.697717 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.695194 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dgjzc\"" Nov 03 20:32:53.697717 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.695307 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tx8mk\"" Nov 03 20:32:53.697717 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.695431 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-847xh\"" Nov 03 20:32:53.697717 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.695525 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rs44g\"" Nov 03 20:32:53.717755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.717735 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 20:32:53.718107 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.718076 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xv4gr\"" Nov 03 20:32:53.737394 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.718438 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 20:32:53.737524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.718599 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-660d95-openshift-default-dockercfg-df2mh\"" Nov 03 20:32:53.737623 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.718765 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 20:32:53.737725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.718843 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d2aa3b-openshift-default-dockercfg-g8c6q\"" Nov 03 20:32:53.737838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.718994 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-46a6d7-openshift-default-dockercfg-5jblx\"" Nov 03 20:32:53.819045 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.819016 5059 scope.go:122] "RemoveContainer" containerID="24306abe36f01f202cf4225862e6256f61b8116c2b16568f30c02b610183da1c" Nov 03 20:32:53.819299 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.719298 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-fsb74\"" Nov 03 20:32:53.819841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.756822 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-98529d-openshift-default-dockercfg-l472v\"" Nov 03 20:32:53.843601 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.843572 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 20:32:53.843894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.843872 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 20:32:53.844284 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.844264 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 20:32:53.849381 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.719140 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xbkg2\"" Nov 03 20:32:53.850918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.850897 5059 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" secret="" err="secret \"e2e-inf-a-d2aa3b-openshift-default-dockercfg-g8c6q\" not found" Nov 03 20:32:53.851051 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.851040 5059 scope.go:122] "RemoveContainer" containerID="073b6545188a48162ae3968a113b770a3a9d7b5615b65392e4809e1551e74cc2" Nov 03 20:32:53.853604 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.853586 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 20:32:53.944381 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.944287 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"routing-overlay\"" Nov 03 20:32:53.946003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.945112 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 20:32:53.946003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.945808 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 20:32:53.946003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.945933 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 20:32:53.946204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.946020 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"simulator-w1-32e482-kserve-self-signed-certs\"" Nov 03 20:32:53.946204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.946064 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"kube-root-ca.crt\"" Nov 03 20:32:53.947610 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.947146 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-rb4sg\"" Nov 03 20:32:53.949766 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.949700 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 20:32:53.949856 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.949794 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 20:32:53.952147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.952129 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 20:32:53.952238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.952226 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 20:32:53.952869 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.952850 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 20:32:53.953047 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.953030 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 20:32:53.956882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.954330 5059 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:32:53.968687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.963020 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:32:53.996897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.996856 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:32:54.044807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.044468 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:32:54.046809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.045991 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 20:32:54.046809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.046076 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 20:32:54.046809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.046146 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 20:32:54.046809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.046204 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 20:32:54.046809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.046296 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 20:32:54.046809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.046389 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 20:32:54.046809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.046477 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 20:32:54.060008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.970678 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 20:32:54.060008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.014259 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 20:32:54.060008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.014346 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.060008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.014735 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-942930-openshift-default-dockercfg-vwmg4\"" Nov 03 20:32:54.060008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.015324 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.060008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.015373 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 20:32:54.060008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.048863 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4p6h4\"" Nov 03 20:32:54.060008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.048996 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-9f46e5\"" Nov 03 20:32:54.060008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.049066 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.060008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.049156 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.060008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.049220 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.060008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.049308 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.060579 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.049401 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 20:32:54.060579 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.049462 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 20:32:54.060579 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.049548 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.060579 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.049599 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0e4ce7\"" Nov 03 20:32:54.068828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.067847 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.068828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.067956 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 20:32:54.068828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.068098 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 20:32:54.068828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.068180 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.068828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.969500 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 20:32:54.068828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.969579 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b2f85\"" Nov 03 20:32:54.068828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.970487 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 20:32:54.068828 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:53.984378 5059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0e4ce7/agc-cred-5f4fe13bd610: secret "agc-cred-5f4fe13bd610" not found Nov 03 20:32:54.068828 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:54.068298 5059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2: secret "agc-cred-5f4fe13bd610" not found Nov 03 20:32:54.069287 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:54.068367 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0309a608-f6c4-45ca-991b-33ba73221a6c-provider-credentials podName:0309a608-f6c4-45ca-991b-33ba73221a6c nodeName:}" failed. No retries permitted until 2026-11-03 20:32:54.568337417 +0000 UTC m=+3368.826134666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0309a608-f6c4-45ca-991b-33ba73221a6c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" (UID: "0309a608-f6c4-45ca-991b-33ba73221a6c") : secret "agc-cred-5f4fe13bd610" not found Nov 03 20:32:54.069287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.042492 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-bcfde8-openshift-default-dockercfg-6xbwb\"" Nov 03 20:32:54.069287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.068378 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 20:32:54.069287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.997499 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 20:32:54.069287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.998193 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 20:32:54.069287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.068617 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 20:32:54.069287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.039139 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a0d4ec\"" Nov 03 20:32:54.069287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.039183 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 20:32:54.069287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.039260 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.069711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.041994 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-0e1546-dockercfg-hwbh2\"" Nov 03 20:32:54.069711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.015478 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-9f46e5-dockercfg-6g47t\"" Nov 03 20:32:54.069711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.042334 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.069711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.042399 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 20:32:54.069711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.015558 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"default-dockercfg-4rsdq\"" Nov 03 20:32:54.069711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.042601 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 20:32:54.069711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.021167 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-4c2a28\"" Nov 03 20:32:54.069711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.021256 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-w5t25\"" Nov 03 20:32:54.069711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.021311 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 20:32:54.081096 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.042191 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.081096 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.042275 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.081687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.015441 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e7b70d-openshift-default-dockercfg-jjqkc\"" Nov 03 20:32:54.081687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.081426 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5qm8x\"" Nov 03 20:32:54.082141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.021347 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 20:32:54.082270 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.021420 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.082352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.038943 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 20:32:54.082428 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.039045 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.124873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.082679 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 20:32:54.125436 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.969358 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"agc-cred-5f4fe13bd610\"" Nov 03 20:32:54.125571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.042083 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-32e482-openshift-default-dockercfg-5pv2v\"" Nov 03 20:32:54.125690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.015520 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-4c2a28-openshift-default-dockercfg-vhjbn\"" Nov 03 20:32:54.125872 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.083194 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.125978 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.083260 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.126079 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.085453 5059 scope.go:122] "RemoveContainer" containerID="5d874b39931c2408a3ab4748545ce2ab475106ee717ad14248ee3ca358ef09d0" Nov 03 20:32:54.126201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.088677 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.150900 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.129018 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.150900 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.129166 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 03 20:32:54.150900 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.129637 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.271636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.269063 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.271636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.269936 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.271636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.270006 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.271636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.270089 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.271636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.270153 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.273814 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.273222 5059 scope.go:122] "RemoveContainer" containerID="111ab0145bc09ea2159df08b45a75e0a16c06fb447f17cd20b906e1b1a7c091f" Nov 03 20:32:54.283729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.281464 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.283729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.281797 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.283729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.281895 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.315855 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.309135 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.315855 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.312610 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.336924 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.969399 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 20:32:54.337610 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:54.336956 5059 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-plugins-e2e-extmod-b-0e4ce7: configmap "payload-processing-external-model-plugins-e2e-extmod-b-0e4ce7" not found Nov 03 20:32:54.337744 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:54.337687 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0309a608-f6c4-45ca-991b-33ba73221a6c-plugins-config-volume podName:0309a608-f6c4-45ca-991b-33ba73221a6c nodeName:}" failed. No retries permitted until 2026-11-03 20:32:54.837648011 +0000 UTC m=+3369.095445263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0309a608-f6c4-45ca-991b-33ba73221a6c-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" (UID: "0309a608-f6c4-45ca-991b-33ba73221a6c") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-0e4ce7" not found Nov 03 20:32:54.338324 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:53.969454 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.347618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.347600 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 20:32:54.370236 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.370205 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 20:32:54.370564 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.370541 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"simulator-w4-3fbb1a-kserve-self-signed-certs\"" Nov 03 20:32:54.374867 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.374725 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.375639 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.375615 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"simulator-w3-9f46e5-kserve-self-signed-certs\"" Nov 03 20:32:54.375890 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.375871 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-32e482\"" Nov 03 20:32:54.376126 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.376108 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 20:32:54.376538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.376517 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 20:32:54.376791 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.376772 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-98529d\"" Nov 03 20:32:54.377166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.377047 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"distinct-w2-2b5703-kserve-self-signed-certs\"" Nov 03 20:32:54.377442 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.377423 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.377653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.377636 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.378002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.377982 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 20:32:54.378417 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.378397 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.383018 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.382986 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.388786 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.388741 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.389741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.389713 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.391421 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.391403 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.412709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.412627 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.423150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.422997 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.468230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.468136 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.468332 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.468270 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.468559 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.468536 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:32:54.470306 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.470280 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-3fbb1a\"" Nov 03 20:32:54.472079 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.472059 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 20:32:54.472316 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.472292 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 20:32:54.472462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.472445 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.472698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.472647 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.472788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.472771 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.473034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.473012 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 20:32:54.473103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.473094 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.473173 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.473155 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-2b5703\"" Nov 03 20:32:54.473221 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.473155 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 20:32:54.473221 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.473201 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 20:32:54.476259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.476036 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 20:32:54.476429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.476405 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.476533 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.476514 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.476961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.476940 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.477036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.476706 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 20:32:54.477036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.477019 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 20:32:54.477126 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.476448 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 20:32:54.477745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.477414 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0e1546\"" Nov 03 20:32:54.479850 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.479832 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 20:32:54.492837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.492687 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 20:32:54.492837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.492818 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 20:32:54.493007 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.492895 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.493054 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.493016 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e7b70d-serving-cert\"" Nov 03 20:32:54.493361 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.493184 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 20:32:54.493361 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.493241 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 20:32:54.513701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.502875 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 20:32:54.513701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.502997 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 20:32:54.513701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.503111 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 20:32:54.513701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.503180 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 20:32:54.513701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.503226 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 20:32:54.519392 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.519361 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.520418 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.520393 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.520889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.520859 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.550075 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.545479 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.564547 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:54.558496 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 20:32:54.578736 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:54.578045 5059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0e4ce7/agc-cred-5f4fe13bd610: secret "agc-cred-5f4fe13bd610" not found Nov 03 20:32:54.578736 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:54.578084 5059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2: secret "agc-cred-5f4fe13bd610" not found Nov 03 20:32:54.578736 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:54.578154 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0309a608-f6c4-45ca-991b-33ba73221a6c-provider-credentials podName:0309a608-f6c4-45ca-991b-33ba73221a6c nodeName:}" failed. No retries permitted until 2026-11-03 20:32:55.578124097 +0000 UTC m=+3369.835921348 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0309a608-f6c4-45ca-991b-33ba73221a6c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" (UID: "0309a608-f6c4-45ca-991b-33ba73221a6c") : secret "agc-cred-5f4fe13bd610" not found Nov 03 20:32:54.899618 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:54.899104 5059 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-plugins-e2e-extmod-b-0e4ce7: configmap "payload-processing-external-model-plugins-e2e-extmod-b-0e4ce7" not found Nov 03 20:32:54.899618 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:54.899183 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0309a608-f6c4-45ca-991b-33ba73221a6c-plugins-config-volume podName:0309a608-f6c4-45ca-991b-33ba73221a6c nodeName:}" failed. No retries permitted until 2026-11-03 20:32:55.899161739 +0000 UTC m=+3370.156958999 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0309a608-f6c4-45ca-991b-33ba73221a6c-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" (UID: "0309a608-f6c4-45ca-991b-33ba73221a6c") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-0e4ce7" not found Nov 03 20:32:55.186824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.186760 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38cf8370-729gp"] Nov 03 20:32:55.229648 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.229539 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38cf8370-729gp"] Nov 03 20:32:55.261679 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.261632 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce41531-n4559"] Nov 03 20:32:55.276046 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.276005 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce41531-n4559"] Nov 03 20:32:55.310948 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.310836 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d937bf0d-c87mk"] Nov 03 20:32:55.311204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.310903 5059 reflector.go:578] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 03 20:32:55.327366 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.327317 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d937bf0d-c87mk"] Nov 03 20:32:55.365339 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.365280 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" event={"ID":"492c7b4e-84c0-49a7-b12f-153a021f971b","Type":"ContainerStarted","Data":"9a314ded87df57e4578fee99a500049875bfdb3f70a6196625866f2a41d5c267"} Nov 03 20:32:55.369249 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.369220 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-fsbsp_1bcd4690-4753-4246-abb4-8668a0a56231/volume-data-source-validator/0.log" Nov 03 20:32:55.369394 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.369359 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fsbsp" event={"ID":"1bcd4690-4753-4246-abb4-8668a0a56231","Type":"ContainerStarted","Data":"34fbf9edd9a9b6fb620b51e1b7b72b6469ff59a49c409953c3ad6b93fa26cf2f"} Nov 03 20:32:55.373166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.373110 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84" event={"ID":"4a35329c-82fb-479f-abca-bff9ed082558","Type":"ContainerStarted","Data":"2e57d5a40799f051dc61748015ccfef4a18d3c2d9058dcbee05723c8f222c743"} Nov 03 20:32:55.374332 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.374297 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84" Nov 03 20:32:55.376618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.376594 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" event={"ID":"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac","Type":"ContainerStarted","Data":"f3a682944a052f5773970fd4a1c1414d244464e281630d9cbe3a0f7623116968"} Nov 03 20:32:55.376982 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.376957 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 20:32:55.382032 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.382010 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 20:32:55.384616 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.384592 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" event={"ID":"1731953b-1850-40d1-9ba4-cb08dc4ebc5d","Type":"ContainerStarted","Data":"da8dcf56301a5bf859cd1267d1436e748641fe4be7bf5c9b6d88b7d504191f65"} Nov 03 20:32:55.385078 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.385057 5059 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" secret="" err="secret \"e2e-inf-a-d2aa3b-openshift-default-dockercfg-g8c6q\" not found" Nov 03 20:32:55.387978 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.387957 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" event={"ID":"1072628f-f513-4b4e-8954-afee5a1aa733","Type":"ContainerStarted","Data":"c92f883ff6898e09b68a779d22ce3740c636fefbf8228b2a6a5e2048a78c601e"} Nov 03 20:32:55.388179 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.388159 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" Nov 03 20:32:55.390727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.390703 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mm85p" event={"ID":"532de14d-f2dd-4c66-ba6f-8b103f7c9375","Type":"ContainerStarted","Data":"6f53b40ec729940a8b447824fba4edc4a9b60468b059f5be6551e07af2e247ed"} Nov 03 20:32:55.395120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.395095 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6k9n9" event={"ID":"f485893c-6f14-449b-b837-8ac181315aeb","Type":"ContainerStarted","Data":"88b5b44b71c45a6eb5b61eaf4c105d6ffa0032eb4646d4eb2fa600f2d3805504"} Nov 03 20:32:55.399512 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.399494 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" event={"ID":"270c522c-ef72-46a7-9e65-3be2ebdbcfa6","Type":"ContainerStarted","Data":"95bb03881e42514f575533d04c3cf2c9897aba4b7ba9dea11e5683f2620cea59"} Nov 03 20:32:55.400167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.400150 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" Nov 03 20:32:55.402471 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.402451 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" event={"ID":"7b8992fc-fba6-4135-90f8-fb259e765dd5","Type":"ContainerStarted","Data":"79f4410301f54a1a4f3ead6bc5461974ff7f958c6320d7f9399fbe12307acdfb"} Nov 03 20:32:55.630851 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:55.630776 5059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0e4ce7/agc-cred-5f4fe13bd610: secret "agc-cred-5f4fe13bd610" not found Nov 03 20:32:55.630851 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:55.630806 5059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2: secret "agc-cred-5f4fe13bd610" not found Nov 03 20:32:55.631018 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:55.630862 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0309a608-f6c4-45ca-991b-33ba73221a6c-provider-credentials podName:0309a608-f6c4-45ca-991b-33ba73221a6c nodeName:}" failed. No retries permitted until 2026-11-03 20:32:57.630844344 +0000 UTC m=+3371.888641586 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0309a608-f6c4-45ca-991b-33ba73221a6c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" (UID: "0309a608-f6c4-45ca-991b-33ba73221a6c") : secret "agc-cred-5f4fe13bd610" not found Nov 03 20:32:55.877613 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.877567 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11283cc0-04dc-42b5-8821-9540534b6781" path="/var/lib/kubelet/pods/11283cc0-04dc-42b5-8821-9540534b6781/volumes" Nov 03 20:32:55.879877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.879842 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84c459f5-0958-4b2b-8527-c733819d0a56" path="/var/lib/kubelet/pods/84c459f5-0958-4b2b-8527-c733819d0a56/volumes" Nov 03 20:32:55.880515 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:55.880491 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9064e69a-d47d-4bad-83b5-ad17fd2dd0c3" path="/var/lib/kubelet/pods/9064e69a-d47d-4bad-83b5-ad17fd2dd0c3/volumes" Nov 03 20:32:55.936833 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:55.936751 5059 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-plugins-e2e-extmod-b-0e4ce7: configmap "payload-processing-external-model-plugins-e2e-extmod-b-0e4ce7" not found Nov 03 20:32:55.937241 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:55.937218 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0309a608-f6c4-45ca-991b-33ba73221a6c-plugins-config-volume podName:0309a608-f6c4-45ca-991b-33ba73221a6c nodeName:}" failed. No retries permitted until 2026-11-03 20:32:57.937192903 +0000 UTC m=+3372.194990159 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0309a608-f6c4-45ca-991b-33ba73221a6c-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" (UID: "0309a608-f6c4-45ca-991b-33ba73221a6c") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-0e4ce7" not found Nov 03 20:32:56.036827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.034160 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:32:56.041656 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.039574 5059 patch_prober.go:29] interesting pod/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Nov 03 20:32:56.041656 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.039624 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" podUID="1731953b-1850-40d1-9ba4-cb08dc4ebc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Nov 03 20:32:56.333597 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.333564 5059 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 20:32:56.345400 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.345360 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pj9xz","odh-ai-gateway-infra/maas-api-revoke-sub-5c6990cd-vnf4z","odh-ai-gateway-infra/maas-api-revoke-sub-da7869e3-kttw5","openshift-insights/periodic-gathering-ftqrx-rt56g","odh-ai-gateway-infra/maas-api-revoke-sub-85c49152-qxjf2","odh-ai-gateway-infra/maas-api-revoke-sub-b89f5007-8qfqc"] Nov 03 20:32:56.345571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.345451 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g","openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l","openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42","openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7","openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx","openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fsbsp","openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r","openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj","openshift-service-ca/service-ca-5fffc97bdf-brvgz","openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5","kube-system/kas-connection-checker-685895c7c9-jt54m","openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p","openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff","openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl","odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2","openshift-console/downloads-6dd789c8b7-c88f8","openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm","openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44","llm/echo-simulator-kserve-64b6f8f96c-d7wsq","e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55","openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f","llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5","openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c","openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4","llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp","openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx","cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mm85p","openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn","openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps","openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6","openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn","openshift-network-diagnostics/network-check-source-6f78d59f99-tvb96","llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l","openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9","openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc","opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5","openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95","openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm","openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7","e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w","odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts","openshift-image-registry/image-registry-6c57db574-fmm6v","openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r","openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qchj6","e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22","openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk","openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl","openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7","openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w","openshift-network-console/networking-console-plugin-86c478bb88-846mt","kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw","kuadrant-system/authorino-operator-7b7bb4bc77-nj52h","openshift-ingress/router-default-f7b87d695-cwlfq","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x","openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt","openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5","ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2","openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq","openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk","openshift-kube-storage-version-migrator/migrator-66df5d496b-pbgwk","e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4","openshift-console-operator/console-operator-7cd56b8dbc-hmc5z","openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b","openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn","openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h","openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk","openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d","openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7","openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8","openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj","kube-system/kas-connection-checker-685895c7c9-6hx8t","openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2","openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg","openshift-kube-storage-version-migrator/migrator-66df5d496b-6k9n9","openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8","openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x","openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd","openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd","kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84","kuadrant-system/authorino-66b59d468d-5bfd2","openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj","openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w","openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt","kube-system/kas-connection-checker-685895c7c9-bd8f8"] Nov 03 20:32:56.348541 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.348499 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="kube-system/kas-connection-checker-685895c7c9-bd8f8" podUID="0fc518fc-6302-4a1a-9ff9-f1e036a2199b" containerName="connection-checker" containerID="cri-o://13a83c66d1721d1a4304250d96ba6c4a613cfb0ba59048a8e63f219d8a6f74c8" gracePeriod=30 Nov 03 20:32:56.348867 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.348811 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" podUID="5ea38154-8a1f-4e1f-8fa0-50481ba4fec0" containerName="istio-proxy" containerID="cri-o://cc7a98566be4a3d14d59e43fa3c73400eb83547051245560b3c66cac856b8485" gracePeriod=30 Nov 03 20:32:56.349052 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.349026 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" podUID="05fceb9c-4a04-470f-a94a-055121319f82" containerName="istio-proxy" containerID="cri-o://8eba0c0d93805e32f585be9b5dffe0df1cf0f78d3928ac2e35a5a76d60ddd0e3" gracePeriod=30 Nov 03 20:32:56.349246 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.349220 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" podUID="a594e899-889e-4f7f-a8ba-b7e7a389ba8a" containerName="istio-proxy" containerID="cri-o://034deb3d094b43323ad58f8415dd48a1c89f624ddffe2676cc27ac841dc9e817" gracePeriod=30 Nov 03 20:32:56.349698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.349651 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" podUID="0f32ecf7-06f7-4098-9793-acf761a30c09" containerName="istio-proxy" containerID="cri-o://5ac8799b34596375075829e9ebddeeb7da35d54820c64c585e69600c30f1d815" gracePeriod=30 Nov 03 20:32:56.351698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.350602 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" podUID="474b2e1d-832b-4edd-8765-31e0bc77cc2a" containerName="istio-proxy" containerID="cri-o://3513472a5d02bb21de815edd2ec4ca696e8e46e17be5d839916c093e17bd9499" gracePeriod=30 Nov 03 20:32:56.351698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.350911 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" podUID="40b2621f-854e-44e0-9b01-c877e333f644" containerName="cluster-samples-operator" containerID="cri-o://b2569884782bd34d772a62814be138ac35a8f3a6ac0255af71fcef484fe98a3d" gracePeriod=30 Nov 03 20:32:56.351698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.351161 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" podUID="8adff4a5-5376-43ec-9b53-888705ae8236" containerName="istio-proxy" containerID="cri-o://3904c49bb7ab63c515d01a6052936b8d990a8f9bb352ab2810819de54dc7b74c" gracePeriod=30 Nov 03 20:32:56.351698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.351441 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" podUID="ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" containerName="istio-proxy" containerID="cri-o://8880ed12964f51d73ab467040324e0b739812c728f7782757374b9b0f025d69e" gracePeriod=30 Nov 03 20:32:56.351896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.351754 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" podUID="c9721811-d27d-4e13-a53d-4ae81e16fb96" containerName="istio-proxy" containerID="cri-o://b4dc2ca466f30e8024a88160d858421a5e6e6e115a118109fb3f8569f83e3ae2" gracePeriod=30 Nov 03 20:32:56.351943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.351918 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" podUID="ab8320f7-2354-433a-a8a7-4d91cc5d37d2" containerName="istio-proxy" containerID="cri-o://91f3fb2bb610fbeea4466047557a60f91aee6d1477483a4dc37e84c19840f464" gracePeriod=30 Nov 03 20:32:56.352107 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.352084 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="kube-system/kas-connection-checker-685895c7c9-jt54m" podUID="e2155512-2116-44f6-be5e-77cef72f9e14" containerName="connection-checker" containerID="cri-o://27b2092e2b1ae773885e5c47b4223f33ad58ad3523d44d0fdfca69cbbee17e22" gracePeriod=30 Nov 03 20:32:56.352357 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.352333 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" podUID="3b9d0847-df69-4b6a-b346-2cd206e42339" containerName="istio-proxy" containerID="cri-o://b5a63189fd2f2d22d3e6f03c32065113a3ccf85221d07eb70f923a647b5b8d38" gracePeriod=30 Nov 03 20:32:56.352514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.352494 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" podUID="b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" containerName="istio-proxy" containerID="cri-o://083bf41b34c7c17d7a323c71536ca53488c9b5d6024a331fe82c42a4a9894a43" gracePeriod=30 Nov 03 20:32:56.352691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.352645 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" podUID="4ea6e69e-3556-4ffe-bb9e-796e8795d747" containerName="maas-api" containerID="cri-o://c21a21bf6b505e70d50fabd813f6c6546130257160456e17e096e7eab6ae4f7d" gracePeriod=30 Nov 03 20:32:56.352830 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.352810 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/downloads-6dd789c8b7-c88f8" podUID="6a7937ca-c705-4091-967e-2e640c622a26" containerName="download-server" containerID="cri-o://cecbb61d72ca19f659a4e15f9281290fd0e1acefbff34b44775daf81cb4ed42c" gracePeriod=5 Nov 03 20:32:56.352984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.352965 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" podUID="171c7288-3157-4fe0-80ee-94cf5b2ca2dd" containerName="istio-proxy" containerID="cri-o://07485e5a1a9bd8f3e2278bef20b0f805688a601c1ec005e9ee1b9f8abb449cd3" gracePeriod=30 Nov 03 20:32:56.353499 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.353472 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" podUID="14a06bdf-e15d-406c-bb76-47ae5dd06b2d" containerName="istio-proxy" containerID="cri-o://cdc8ec4afacb060d845fdcbcf9f9a36756ac0c3cf27302051a5a1dae638f96f6" gracePeriod=30 Nov 03 20:32:56.353799 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.353774 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" podUID="64153a99-d5f7-4f92-a0c9-f79b39eba97c" containerName="discovery" containerID="cri-o://1929782c8e0a4859c99bc362063000c768666ab59352d7c235edbdf79f308a9e" gracePeriod=30 Nov 03 20:32:56.353954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.353934 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" podUID="70d45aa9-bfa8-43c6-8674-6f8f14a3097e" containerName="istio-proxy" containerID="cri-o://6a213e69e74bbe139a2688d93452e7d59e19f98c3c9a8e404648454c84993ae7" gracePeriod=30 Nov 03 20:32:56.354263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.354236 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" podUID="2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" containerName="istio-proxy" containerID="cri-o://6c9d6b78a3dbde46dbddcc1222495b4fce407ba8a3afc87ef4704aadd4524702" gracePeriod=30 Nov 03 20:32:56.354426 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.354406 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" podUID="08b3a993-1a64-4fe5-9577-b0f11a36779e" containerName="istio-proxy" containerID="cri-o://ab75e4ee856c98aa1f2cad9eecfea81d6df844e47094122246ef45af739d37d5" gracePeriod=30 Nov 03 20:32:56.354690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.354650 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" podUID="eac98813-7b4c-4c4a-8bf9-444bba2fa644" containerName="istio-proxy" containerID="cri-o://f0ca6f95c31b4ccad9131b19d71014e3f54afa9b79e7f53f9d77d4b861fb6af9" gracePeriod=30 Nov 03 20:32:56.354860 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.354838 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" podUID="8153e018-7f9a-4645-b102-06ff3913f682" containerName="istio-proxy" containerID="cri-o://3a0b8447bfda8a58216cc853554baf15762b4ccc77d0d98ce90bfe2224c3b122" gracePeriod=30 Nov 03 20:32:56.355013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.354990 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tvb96" podUID="0a985b50-c4c0-4a39-abed-29f53e2be747" containerName="check-endpoints" containerID="cri-o://b3dbf9416fa12ac2df38a62843937fa243f692c05fb2159a754be69427937d5d" gracePeriod=30 Nov 03 20:32:56.357166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.355365 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" podUID="677a5153-5fe3-4624-bf06-3c8dfc6bc72d" containerName="discovery" containerID="cri-o://daf533566a4042ce35f672f5974749c01a13dc2086255c8268966b3679ba35ea" gracePeriod=30 Nov 03 20:32:56.357166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.355792 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" podUID="c2d80995-5f1c-46db-8d71-a130cb92e279" containerName="istio-proxy" containerID="cri-o://39fd5a87217d3b8146ccfe78173093985a26712d776696de4ab3b763f50a762f" gracePeriod=30 Nov 03 20:32:56.357166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.355972 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" podUID="37e90ab0-e9c0-420c-a0f4-c7923aa00297" containerName="istio-proxy" containerID="cri-o://f8323dfa4e2c1a85ed81f607b727475c16361a359dbb2bb285fe28a219d7a6eb" gracePeriod=30 Nov 03 20:32:56.357166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.356136 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" podUID="cef4417c-3d04-4d65-a5bc-6ec1af6f09bb" containerName="istio-proxy" containerID="cri-o://c56ee037a40c322b89e375002cf24e08b31d3e1b30e1b41b0f4819b338d40ec8" gracePeriod=30 Nov 03 20:32:56.357166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.356217 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" podUID="34880975-a897-4ad4-af59-18617a93cdad" containerName="istio-proxy" containerID="cri-o://89e013c2e4eeadb512ea4a0824409f67321420e7c404d52597c8ba458a34976b" gracePeriod=30 Nov 03 20:32:56.357166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.356941 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" podUID="d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" containerName="istio-proxy" containerID="cri-o://e7118a165e1c692020e00ab701f84d22189feb4501a06e12ef7daecc27e93292" gracePeriod=30 Nov 03 20:32:56.357714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.357212 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" podUID="edcfb7ab-b14f-4d5c-b752-5b43a6eee163" containerName="istio-proxy" containerID="cri-o://bab3644b9638d4ac499f2df75a736f3f63d2b12e17964b2ccc440b6a34499ce5" gracePeriod=30 Nov 03 20:32:56.358649 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.357897 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" podUID="c8683a65-9898-4b4f-a692-f2da325a93b4" containerName="istio-proxy" containerID="cri-o://d36cbc617302444ca97907469c12558d0fc4a6b46d78b847f1d5ccbc843b16e6" gracePeriod=30 Nov 03 20:32:56.358649 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.358193 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" podUID="40b2621f-854e-44e0-9b01-c877e333f644" containerName="cluster-samples-operator-watch" containerID="cri-o://1a4ab42465871189364c0b8d0033b852cbd3488b01bc6dd2ae20a8268416653f" gracePeriod=30 Nov 03 20:32:56.359221 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.359162 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" podUID="f88ae6f3-f1d6-4aa3-ad33-589490696167" containerName="istio-proxy" containerID="cri-o://f35a41707de2c0253145d89c0f1f6796087f8f0d6d98dc9ca16c8dcbf8e24a1e" gracePeriod=30 Nov 03 20:32:56.359328 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.359187 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pbgwk" podUID="815b15b6-2c37-4ca3-9b38-e7e1328d0aa8" containerName="migrator" containerID="cri-o://c3bd71ffe2f7af1614a32a2b8d3f234a09fc878537716084e68bd018236a5528" gracePeriod=30 Nov 03 20:32:56.359620 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.359511 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" podUID="518169ac-2f8d-4645-ae47-210b67ca080a" containerName="istio-proxy" containerID="cri-o://fcb62eadee37b9a5aa9bdc20f77cd13d99c001430b049cbc4d5006aa5cc74354" gracePeriod=30 Nov 03 20:32:56.371752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.359526 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts" podUID="633c352c-f07e-4032-b962-23a148297e45" containerName="postgres" containerID="cri-o://e882b42c99b4f73bf517ede2116273cfd6eda2d61af90817e8c3c5603097f5b8" gracePeriod=30 Nov 03 20:32:56.375109 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.372995 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" podUID="fc57130b-c950-49f2-8a98-22c47bd98288" containerName="istio-proxy" containerID="cri-o://7598d2be586d16550aff9c80d082d04770669cb98c756f3811527543a1ac8d0f" gracePeriod=30 Nov 03 20:32:56.375109 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.373506 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" podUID="f0d3c590-f523-4b7e-9086-60df5fc585e8" containerName="istio-proxy" containerID="cri-o://1638b310e0522c5f2a20ee3d226436456869be0ba4a5f3e1f6c3bdebb4a48c87" gracePeriod=30 Nov 03 20:32:56.395817 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.394369 5059 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-hmc5z container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.20:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:32:56.395817 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.394431 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" podUID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.20:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:56.407454 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.407297 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" podUID="04a982f9-143c-406c-99d5-9ef73d5c571a" containerName="networking-console-plugin" containerID="cri-o://147857ae5dcfb3795e8ee333ec3863a15882c306754cc03d2571ba058fc49ce3" gracePeriod=30 Nov 03 20:32:56.407985 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.407907 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" podUID="8b648c6b-1f7d-42ca-9c3f-a435494bad03" containerName="istio-proxy" containerID="cri-o://1fe212cabed0fc214cbab7c5d208f7f12d710cdc5a5c86dc76e26a0430ec5af5" gracePeriod=30 Nov 03 20:32:56.408753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.408367 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nj52h" podUID="941b16dd-0550-4434-86f9-26e38efb68ee" containerName="manager" containerID="cri-o://ac8fdfdbcf156bbdd591e8f7042d8b960bce286b35806ac720a68b8da48a5940" gracePeriod=10 Nov 03 20:32:56.414301 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.414259 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" podUID="8d0db609-b271-4c98-a303-fb3ee71f152c" containerName="cluster-monitoring-operator" containerID="cri-o://6a679e526b0df571226e2d9d4986921d285477efff7809df408a7dcd93178490" gracePeriod=30 Nov 03 20:32:56.414537 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.414504 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" podUID="4946e3aa-a961-415a-b71d-95164d5fbca3" containerName="istio-proxy" containerID="cri-o://2ce7833679c135af351a934eeaaf542087c31098688f1a7b779e25f9a2bf4bc1" gracePeriod=30 Nov 03 20:32:56.414833 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.414795 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" podUID="40b88c7d-08e9-4203-9ebd-2c9f69b884dc" containerName="istio-proxy" containerID="cri-o://6739ff178c81c67d823d4e8761f1f61d4d648a31a61e02195fb24f7ce8ca0bdf" gracePeriod=30 Nov 03 20:32:56.415074 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.415046 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" podUID="d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" containerName="istio-proxy" containerID="cri-o://5c19ca06fbb1d449a708204a75d839cc487a9ea18245ce5649dd7dfdd316e86f" gracePeriod=30 Nov 03 20:32:56.415489 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.415416 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="kube-system/kas-connection-checker-685895c7c9-6hx8t" podUID="f9d935b2-aa51-474c-836d-a74f91efab07" containerName="connection-checker" containerID="cri-o://e7fcbceef75077181869de0f40c073966ec17006f2685b45f17586d4fcf3eeb3" gracePeriod=30 Nov 03 20:32:56.415715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.415691 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" podUID="4c609469-d7d7-4530-8d74-e34aa56dda8f" containerName="istio-proxy" containerID="cri-o://25b28b428ebc85ad2241009e2e20a1a14a4f98588d071bd9785777b228e75310" gracePeriod=30 Nov 03 20:32:56.416600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.416471 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" podUID="7909b42c-1572-4fe2-816a-447020594fe4" containerName="istio-proxy" containerID="cri-o://4cc323b3fee429ec7bc1ab6e19bfbfe5f3926b292a99afff5f656543c7dd8e48" gracePeriod=30 Nov 03 20:32:56.416990 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.416959 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/router-default-f7b87d695-cwlfq" podUID="603868b0-8628-4227-8471-1894e2e1de50" containerName="router" containerID="cri-o://a75a1e78b8fa43ee7161dbf1263337919ca861621fb782933da9fe9d87a6c4b1" gracePeriod=3600 Nov 03 20:32:56.417431 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.417396 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" podUID="9d60eb56-9852-4c25-ae50-8a932e752793" containerName="istio-proxy" containerID="cri-o://1800f61ab969d2f54568614237d89ac56416a680367326b1f485f3183aebbfe9" gracePeriod=30 Nov 03 20:32:56.417873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.417843 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-66b59d468d-5bfd2" podUID="c7fce352-1850-4048-be2e-fa458fd4d2f3" containerName="authorino" containerID="cri-o://8121a3441525651ca2c833f0f9236c162d2c111dd09ec30bb62b251a8accdd62" gracePeriod=30 Nov 03 20:32:56.419059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.419011 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" podUID="c99b0c19-004e-450c-852a-e38338635b61" containerName="service-ca-operator" containerID="cri-o://b742b7b0202501adeba598c21e37a746e39b623b622c6c629096721511b0d73e" gracePeriod=30 Nov 03 20:32:56.422721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.418245 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qchj6" podUID="ad491ce3-2f92-45a7-9b4f-278309989e64" containerName="prometheus-operator-admission-webhook" containerID="cri-o://d84ec766302d997e0dd2646c72025bbffe88f5776ae0c76a2bedd7ae24b833ab" gracePeriod=30 Nov 03 20:32:56.587060 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.584770 5059 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:32:56.587060 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.584816 5059 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:32:56.587060 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.584862 5059 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:32:56.587060 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.584884 5059 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-83.us-east-2.compute.internal" event="NodeReady" Nov 03 20:32:56.587060 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:56.585889 5059 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="3fd1b473caf0e5fcbbbf5a97f92155614da76dd2857a8df4e1895ff6ca147711" cmd=["/bin/sleep","15"] Nov 03 20:32:56.587060 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.585952 5059 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" Nov 03 20:32:56.587060 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:56.585971 5059 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" podUID="5ac27a9c-459f-4310-aef7-8dc643b46c6c" containerName="main" containerID="cri-o://3fd1b473caf0e5fcbbbf5a97f92155614da76dd2857a8df4e1895ff6ca147711" Nov 03 20:32:56.587060 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.586021 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" podUID="5ac27a9c-459f-4310-aef7-8dc643b46c6c" containerName="main" containerID="cri-o://3fd1b473caf0e5fcbbbf5a97f92155614da76dd2857a8df4e1895ff6ca147711" gracePeriod=60 Nov 03 20:32:56.587060 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.586179 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" event={"ID":"6785e24f-6fb3-43ab-a469-84d30d94b2f0","Type":"ContainerStarted","Data":"95024980a099f9ccdf78362bbb5aa84a698eaab1e6f9c5a70a9520e6a4b6e491"} Nov 03 20:32:56.613028 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.595644 5059 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" secret="" err="secret \"e2e-inf-a-d2aa3b-openshift-default-dockercfg-g8c6q\" not found" Nov 03 20:32:56.683802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.682752 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5"] Nov 03 20:32:56.689530 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.689014 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5"] Nov 03 20:32:56.689530 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.689096 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5" Nov 03 20:32:56.699410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.699379 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t"] Nov 03 20:32:56.710377 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.708757 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t" Nov 03 20:32:56.736595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.736008 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t"] Nov 03 20:32:56.758756 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.756921 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:32:56.760738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.759024 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:32:56.835337 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.835225 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9428a2-ab50-4e44-a725-e0da980bdb93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-266d0609-xqmv5\" (UID: \"6b9428a2-ab50-4e44-a725-e0da980bdb93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5" Nov 03 20:32:56.835337 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.835309 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9428a2-ab50-4e44-a725-e0da980bdb93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-266d0609-xqmv5\" (UID: \"6b9428a2-ab50-4e44-a725-e0da980bdb93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5" Nov 03 20:32:56.835531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.835343 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b25c2d0-60bb-4df1-8193-5f7f853c0a73-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t\" (UID: \"4b25c2d0-60bb-4df1-8193-5f7f853c0a73\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t" Nov 03 20:32:56.835531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.835418 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dshjq\" (UniqueName: \"kubernetes.io/projected/6b9428a2-ab50-4e44-a725-e0da980bdb93-kube-api-access-dshjq\") pod \"maas-api-revoke-sub-266d0609-xqmv5\" (UID: \"6b9428a2-ab50-4e44-a725-e0da980bdb93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5" Nov 03 20:32:56.835682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.835535 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-625k6\" (UniqueName: \"kubernetes.io/projected/4b25c2d0-60bb-4df1-8193-5f7f853c0a73-kube-api-access-625k6\") pod \"payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t\" (UID: \"4b25c2d0-60bb-4df1-8193-5f7f853c0a73\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t" Nov 03 20:32:56.938604 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.937322 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dshjq\" (UniqueName: \"kubernetes.io/projected/6b9428a2-ab50-4e44-a725-e0da980bdb93-kube-api-access-dshjq\") pod \"maas-api-revoke-sub-266d0609-xqmv5\" (UID: \"6b9428a2-ab50-4e44-a725-e0da980bdb93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5" Nov 03 20:32:56.938604 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.937468 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-625k6\" (UniqueName: \"kubernetes.io/projected/4b25c2d0-60bb-4df1-8193-5f7f853c0a73-kube-api-access-625k6\") pod \"payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t\" (UID: \"4b25c2d0-60bb-4df1-8193-5f7f853c0a73\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t" Nov 03 20:32:56.938604 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.937626 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9428a2-ab50-4e44-a725-e0da980bdb93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-266d0609-xqmv5\" (UID: \"6b9428a2-ab50-4e44-a725-e0da980bdb93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5" Nov 03 20:32:56.938604 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.937720 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9428a2-ab50-4e44-a725-e0da980bdb93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-266d0609-xqmv5\" (UID: \"6b9428a2-ab50-4e44-a725-e0da980bdb93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5" Nov 03 20:32:56.938604 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.937769 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b25c2d0-60bb-4df1-8193-5f7f853c0a73-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t\" (UID: \"4b25c2d0-60bb-4df1-8193-5f7f853c0a73\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t" Nov 03 20:32:56.945202 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.942867 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b25c2d0-60bb-4df1-8193-5f7f853c0a73-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t\" (UID: \"4b25c2d0-60bb-4df1-8193-5f7f853c0a73\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t" Nov 03 20:32:56.949920 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:56.949844 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9428a2-ab50-4e44-a725-e0da980bdb93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-266d0609-xqmv5\" (UID: \"6b9428a2-ab50-4e44-a725-e0da980bdb93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5" Nov 03 20:32:57.044364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.044231 5059 patch_prober.go:29] interesting pod/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Nov 03 20:32:57.044364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.044288 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" podUID="1731953b-1850-40d1-9ba4-cb08dc4ebc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Nov 03 20:32:57.173344 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.173312 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-625k6\" (UniqueName: \"kubernetes.io/projected/4b25c2d0-60bb-4df1-8193-5f7f853c0a73-kube-api-access-625k6\") pod \"payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t\" (UID: \"4b25c2d0-60bb-4df1-8193-5f7f853c0a73\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t" Nov 03 20:32:57.174877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.174825 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9428a2-ab50-4e44-a725-e0da980bdb93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-266d0609-xqmv5\" (UID: \"6b9428a2-ab50-4e44-a725-e0da980bdb93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5" Nov 03 20:32:57.233039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.232937 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dshjq\" (UniqueName: \"kubernetes.io/projected/6b9428a2-ab50-4e44-a725-e0da980bdb93-kube-api-access-dshjq\") pod \"maas-api-revoke-sub-266d0609-xqmv5\" (UID: \"6b9428a2-ab50-4e44-a725-e0da980bdb93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5" Nov 03 20:32:57.274329 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.274016 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5" Nov 03 20:32:57.287761 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.284824 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t" Nov 03 20:32:57.322649 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.322620 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-464r7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:32:57.322800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.322734 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" podUID="c8683a65-9898-4b4f-a692-f2da325a93b4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:32:57.333039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.333009 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:32:57.333184 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.333057 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" podUID="518169ac-2f8d-4645-ae47-210b67ca080a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:32:57.498709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.498648 5059 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:32:57.499152 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.498715 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" podUID="474b2e1d-832b-4edd-8765-31e0bc77cc2a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:32:57.538106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.538005 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:32:57.538345 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.538299 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" podUID="4c609469-d7d7-4530-8d74-e34aa56dda8f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:32:57.597892 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.597859 5059 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-hmc5z container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.20:8443/readyz\": context deadline exceeded" start-of-body= Nov 03 20:32:57.598107 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.598076 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" podUID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.20:8443/readyz\": context deadline exceeded" Nov 03 20:32:57.598427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.598397 5059 generic.go:356] "Generic (PLEG): container finished" podID="40b2621f-854e-44e0-9b01-c877e333f644" containerID="1a4ab42465871189364c0b8d0033b852cbd3488b01bc6dd2ae20a8268416653f" exitCode=0 Nov 03 20:32:57.598427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.598423 5059 generic.go:356] "Generic (PLEG): container finished" podID="40b2621f-854e-44e0-9b01-c877e333f644" containerID="b2569884782bd34d772a62814be138ac35a8f3a6ac0255af71fcef484fe98a3d" exitCode=0 Nov 03 20:32:57.598547 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.598499 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" event={"ID":"40b2621f-854e-44e0-9b01-c877e333f644","Type":"ContainerDied","Data":"1a4ab42465871189364c0b8d0033b852cbd3488b01bc6dd2ae20a8268416653f"} Nov 03 20:32:57.598606 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.598551 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" event={"ID":"40b2621f-854e-44e0-9b01-c877e333f644","Type":"ContainerDied","Data":"b2569884782bd34d772a62814be138ac35a8f3a6ac0255af71fcef484fe98a3d"} Nov 03 20:32:57.601401 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.601380 5059 generic.go:356] "Generic (PLEG): container finished" podID="c99b0c19-004e-450c-852a-e38338635b61" containerID="b742b7b0202501adeba598c21e37a746e39b623b622c6c629096721511b0d73e" exitCode=0 Nov 03 20:32:57.601654 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.601414 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" event={"ID":"c99b0c19-004e-450c-852a-e38338635b61","Type":"ContainerDied","Data":"b742b7b0202501adeba598c21e37a746e39b623b622c6c629096721511b0d73e"} Nov 03 20:32:57.601842 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.601828 5059 scope.go:122] "RemoveContainer" containerID="481852318efef15a99c90fc33e8f7d88cfa6b35f952bd0e4e6c4993664dcc592" Nov 03 20:32:57.607594 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.607124 5059 generic.go:356] "Generic (PLEG): container finished" podID="6a7937ca-c705-4091-967e-2e640c622a26" containerID="cecbb61d72ca19f659a4e15f9281290fd0e1acefbff34b44775daf81cb4ed42c" exitCode=0 Nov 03 20:32:57.607594 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.607209 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-c88f8" event={"ID":"6a7937ca-c705-4091-967e-2e640c622a26","Type":"ContainerDied","Data":"cecbb61d72ca19f659a4e15f9281290fd0e1acefbff34b44775daf81cb4ed42c"} Nov 03 20:32:57.609642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.609465 5059 generic.go:356] "Generic (PLEG): container finished" podID="677a5153-5fe3-4624-bf06-3c8dfc6bc72d" containerID="daf533566a4042ce35f672f5974749c01a13dc2086255c8268966b3679ba35ea" exitCode=0 Nov 03 20:32:57.609642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.609584 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" event={"ID":"677a5153-5fe3-4624-bf06-3c8dfc6bc72d","Type":"ContainerDied","Data":"daf533566a4042ce35f672f5974749c01a13dc2086255c8268966b3679ba35ea"} Nov 03 20:32:57.615447 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.615308 5059 generic.go:356] "Generic (PLEG): container finished" podID="ad491ce3-2f92-45a7-9b4f-278309989e64" containerID="d84ec766302d997e0dd2646c72025bbffe88f5776ae0c76a2bedd7ae24b833ab" exitCode=0 Nov 03 20:32:57.615447 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.615387 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qchj6" event={"ID":"ad491ce3-2f92-45a7-9b4f-278309989e64","Type":"ContainerDied","Data":"d84ec766302d997e0dd2646c72025bbffe88f5776ae0c76a2bedd7ae24b833ab"} Nov 03 20:32:57.617920 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.617750 5059 generic.go:356] "Generic (PLEG): container finished" podID="941b16dd-0550-4434-86f9-26e38efb68ee" containerID="ac8fdfdbcf156bbdd591e8f7042d8b960bce286b35806ac720a68b8da48a5940" exitCode=0 Nov 03 20:32:57.617920 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.617855 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nj52h" event={"ID":"941b16dd-0550-4434-86f9-26e38efb68ee","Type":"ContainerDied","Data":"ac8fdfdbcf156bbdd591e8f7042d8b960bce286b35806ac720a68b8da48a5940"} Nov 03 20:32:57.620346 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.620308 5059 generic.go:356] "Generic (PLEG): container finished" podID="4ea6e69e-3556-4ffe-bb9e-796e8795d747" containerID="c21a21bf6b505e70d50fabd813f6c6546130257160456e17e096e7eab6ae4f7d" exitCode=0 Nov 03 20:32:57.620671 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.620385 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" event={"ID":"4ea6e69e-3556-4ffe-bb9e-796e8795d747","Type":"ContainerDied","Data":"c21a21bf6b505e70d50fabd813f6c6546130257160456e17e096e7eab6ae4f7d"} Nov 03 20:32:57.623051 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.623003 5059 generic.go:356] "Generic (PLEG): container finished" podID="633c352c-f07e-4032-b962-23a148297e45" containerID="e882b42c99b4f73bf517ede2116273cfd6eda2d61af90817e8c3c5603097f5b8" exitCode=0 Nov 03 20:32:57.623230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.623197 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts" event={"ID":"633c352c-f07e-4032-b962-23a148297e45","Type":"ContainerDied","Data":"e882b42c99b4f73bf517ede2116273cfd6eda2d61af90817e8c3c5603097f5b8"} Nov 03 20:32:57.626832 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.626811 5059 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-lf9vc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:8080/ready\": dial tcp 10.132.0.99:8080: connect: connection refused" start-of-body= Nov 03 20:32:57.627008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.626986 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" podUID="677a5153-5fe3-4624-bf06-3c8dfc6bc72d" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.99:8080/ready\": dial tcp 10.132.0.99:8080: connect: connection refused" Nov 03 20:32:57.627130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.626916 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tvb96" event={"ID":"0a985b50-c4c0-4a39-abed-29f53e2be747","Type":"ContainerDied","Data":"b3dbf9416fa12ac2df38a62843937fa243f692c05fb2159a754be69427937d5d"} Nov 03 20:32:57.627231 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.626899 5059 generic.go:356] "Generic (PLEG): container finished" podID="0a985b50-c4c0-4a39-abed-29f53e2be747" containerID="b3dbf9416fa12ac2df38a62843937fa243f692c05fb2159a754be69427937d5d" exitCode=0 Nov 03 20:32:57.629339 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.629316 5059 generic.go:356] "Generic (PLEG): container finished" podID="04a982f9-143c-406c-99d5-9ef73d5c571a" containerID="147857ae5dcfb3795e8ee333ec3863a15882c306754cc03d2571ba058fc49ce3" exitCode=0 Nov 03 20:32:57.629489 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.629472 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" event={"ID":"04a982f9-143c-406c-99d5-9ef73d5c571a","Type":"ContainerDied","Data":"147857ae5dcfb3795e8ee333ec3863a15882c306754cc03d2571ba058fc49ce3"} Nov 03 20:32:57.632291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.632270 5059 generic.go:356] "Generic (PLEG): container finished" podID="8d0db609-b271-4c98-a303-fb3ee71f152c" containerID="6a679e526b0df571226e2d9d4986921d285477efff7809df408a7dcd93178490" exitCode=0 Nov 03 20:32:57.632396 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.632379 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" event={"ID":"8d0db609-b271-4c98-a303-fb3ee71f152c","Type":"ContainerDied","Data":"6a679e526b0df571226e2d9d4986921d285477efff7809df408a7dcd93178490"} Nov 03 20:32:57.636227 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.636206 5059 generic.go:356] "Generic (PLEG): container finished" podID="64153a99-d5f7-4f92-a0c9-f79b39eba97c" containerID="1929782c8e0a4859c99bc362063000c768666ab59352d7c235edbdf79f308a9e" exitCode=0 Nov 03 20:32:57.636481 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.636403 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" event={"ID":"64153a99-d5f7-4f92-a0c9-f79b39eba97c","Type":"ContainerDied","Data":"1929782c8e0a4859c99bc362063000c768666ab59352d7c235edbdf79f308a9e"} Nov 03 20:32:57.639863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.639397 5059 generic.go:356] "Generic (PLEG): container finished" podID="5ac27a9c-459f-4310-aef7-8dc643b46c6c" containerID="3fd1b473caf0e5fcbbbf5a97f92155614da76dd2857a8df4e1895ff6ca147711" exitCode=0 Nov 03 20:32:57.639863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.639490 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" event={"ID":"5ac27a9c-459f-4310-aef7-8dc643b46c6c","Type":"ContainerDied","Data":"3fd1b473caf0e5fcbbbf5a97f92155614da76dd2857a8df4e1895ff6ca147711"} Nov 03 20:32:57.643868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.642538 5059 generic.go:356] "Generic (PLEG): container finished" podID="815b15b6-2c37-4ca3-9b38-e7e1328d0aa8" containerID="c3bd71ffe2f7af1614a32a2b8d3f234a09fc878537716084e68bd018236a5528" exitCode=0 Nov 03 20:32:57.643868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.642722 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6k9n9" podUID="f485893c-6f14-449b-b837-8ac181315aeb" containerName="migrator" containerID="cri-o://88b5b44b71c45a6eb5b61eaf4c105d6ffa0032eb4646d4eb2fa600f2d3805504" gracePeriod=30 Nov 03 20:32:57.643868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.642828 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pbgwk" event={"ID":"815b15b6-2c37-4ca3-9b38-e7e1328d0aa8","Type":"ContainerDied","Data":"c3bd71ffe2f7af1614a32a2b8d3f234a09fc878537716084e68bd018236a5528"} Nov 03 20:32:57.643868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.642949 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" podUID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" containerName="console-operator" containerID="cri-o://f3a682944a052f5773970fd4a1c1414d244464e281630d9cbe3a0f7623116968" gracePeriod=30 Nov 03 20:32:57.643868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.643109 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" podUID="1072628f-f513-4b4e-8954-afee5a1aa733" containerName="manager" containerID="cri-o://c92f883ff6898e09b68a779d22ce3740c636fefbf8228b2a6a5e2048a78c601e" gracePeriod=10 Nov 03 20:32:57.643868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.643264 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84" podUID="4a35329c-82fb-479f-abca-bff9ed082558" containerName="manager" containerID="cri-o://2e57d5a40799f051dc61748015ccfef4a18d3c2d9058dcbee05723c8f222c743" gracePeriod=10 Nov 03 20:32:57.643868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.643401 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mm85p" podUID="532de14d-f2dd-4c66-ba6f-8b103f7c9375" containerName="cert-manager-operator" containerID="cri-o://6f53b40ec729940a8b447824fba4edc4a9b60468b059f5be6551e07af2e247ed" gracePeriod=10 Nov 03 20:32:57.643868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.643532 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" podUID="492c7b4e-84c0-49a7-b12f-153a021f971b" containerName="kube-storage-version-migrator-operator" containerID="cri-o://9a314ded87df57e4578fee99a500049875bfdb3f70a6196625866f2a41d5c267" gracePeriod=30 Nov 03 20:32:57.643868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.643675 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" podUID="7b8992fc-fba6-4135-90f8-fb259e765dd5" containerName="service-ca-controller" containerID="cri-o://79f4410301f54a1a4f3ead6bc5461974ff7f958c6320d7f9399fbe12307acdfb" gracePeriod=30 Nov 03 20:32:57.645245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.643806 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fsbsp" podUID="1bcd4690-4753-4246-abb4-8668a0a56231" containerName="volume-data-source-validator" containerID="cri-o://34fbf9edd9a9b6fb620b51e1b7b72b6469ff59a49c409953c3ad6b93fa26cf2f" gracePeriod=30 Nov 03 20:32:57.645245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.644529 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" podUID="270c522c-ef72-46a7-9e65-3be2ebdbcfa6" containerName="manager" containerID="cri-o://95bb03881e42514f575533d04c3cf2c9897aba4b7ba9dea11e5683f2620cea59" gracePeriod=10 Nov 03 20:32:57.645245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.644760 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" podUID="1731953b-1850-40d1-9ba4-cb08dc4ebc5d" containerName="istio-proxy" containerID="cri-o://da8dcf56301a5bf859cd1267d1436e748641fe4be7bf5c9b6d88b7d504191f65" gracePeriod=30 Nov 03 20:32:57.645245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.644928 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" podUID="6785e24f-6fb3-43ab-a469-84d30d94b2f0" containerName="manager" containerID="cri-o://95024980a099f9ccdf78362bbb5aa84a698eaab1e6f9c5a70a9520e6a4b6e491" gracePeriod=10 Nov 03 20:32:57.649403 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.649361 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" podUID="5ac27a9c-459f-4310-aef7-8dc643b46c6c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": dial tcp 10.132.0.50:8000: connect: connection refused" Nov 03 20:32:57.695716 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:57.687549 5059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0e4ce7/agc-cred-5f4fe13bd610: secret "agc-cred-5f4fe13bd610" not found Nov 03 20:32:57.695716 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:57.687576 5059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2: secret "agc-cred-5f4fe13bd610" not found Nov 03 20:32:57.695716 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:57.687633 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0309a608-f6c4-45ca-991b-33ba73221a6c-provider-credentials podName:0309a608-f6c4-45ca-991b-33ba73221a6c nodeName:}" failed. No retries permitted until 2026-11-03 20:33:01.687614099 +0000 UTC m=+3375.945411348 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0309a608-f6c4-45ca-991b-33ba73221a6c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" (UID: "0309a608-f6c4-45ca-991b-33ba73221a6c") : secret "agc-cred-5f4fe13bd610" not found Nov 03 20:32:57.775838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.775495 5059 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-hmc5z container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.20:8443/readyz\": read tcp 10.132.0.2:37822->10.132.0.20:8443: read: connection reset by peer" start-of-body= Nov 03 20:32:57.775838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:57.775566 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" podUID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.20:8443/readyz\": read tcp 10.132.0.2:37822->10.132.0.20:8443: read: connection reset by peer" Nov 03 20:32:58.090681 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:58.060576 5059 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-plugins-e2e-extmod-b-0e4ce7: configmap "payload-processing-external-model-plugins-e2e-extmod-b-0e4ce7" not found Nov 03 20:32:58.090681 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:32:58.060688 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0309a608-f6c4-45ca-991b-33ba73221a6c-plugins-config-volume podName:0309a608-f6c4-45ca-991b-33ba73221a6c nodeName:}" failed. No retries permitted until 2026-11-03 20:33:02.060647352 +0000 UTC m=+3376.318444606 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0309a608-f6c4-45ca-991b-33ba73221a6c-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" (UID: "0309a608-f6c4-45ca-991b-33ba73221a6c") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-0e4ce7" not found Nov 03 20:32:58.106058 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.106030 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d" path="/var/lib/kubelet/pods/34dc7a0b-ebbe-49b3-95de-0a77c7f63e6d/volumes" Nov 03 20:32:58.109098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.109076 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44c0c10d-2b49-481e-85d8-50d0a3b251a1" path="/var/lib/kubelet/pods/44c0c10d-2b49-481e-85d8-50d0a3b251a1/volumes" Nov 03 20:32:58.111025 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.111006 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b69fe94-91fc-44a0-a91b-b7470105d659" path="/var/lib/kubelet/pods/5b69fe94-91fc-44a0-a91b-b7470105d659/volumes" Nov 03 20:32:58.114426 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.114393 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ec89e5b-1640-4b12-9150-b40a3a353c8e" path="/var/lib/kubelet/pods/5ec89e5b-1640-4b12-9150-b40a3a353c8e/volumes" Nov 03 20:32:58.116718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.116700 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a60d7aa8-156a-4c7e-964c-3c20241c02ec" path="/var/lib/kubelet/pods/a60d7aa8-156a-4c7e-964c-3c20241c02ec/volumes" Nov 03 20:32:58.126414 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.126398 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3853f0d-7f5a-41a8-b2b9-dc1b730753f2" path="/var/lib/kubelet/pods/e3853f0d-7f5a-41a8-b2b9-dc1b730753f2/volumes" Nov 03 20:32:58.389635 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.388204 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 20:32:58.404050 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.404009 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qchj6" Nov 03 20:32:58.426034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.425955 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nj52h" Nov 03 20:32:58.430025 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.429997 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tvb96" Nov 03 20:32:58.433797 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.433773 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:32:58.437237 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.436813 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pbgwk" Nov 03 20:32:58.439242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.439217 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" Nov 03 20:32:58.443439 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.443421 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts" Nov 03 20:32:58.444656 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.444627 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:32:58.444656 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.444721 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" podUID="ab8320f7-2354-433a-a8a7-4d91cc5d37d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:32:58.445463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.445445 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-c88f8" Nov 03 20:32:58.447727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.447707 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 20:32:58.452166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.452146 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 20:32:58.454749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.454734 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" Nov 03 20:32:58.458221 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.458201 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" Nov 03 20:32:58.464090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.464068 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 20:32:58.471417 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.471396 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-dshm\") pod \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\" (UID: \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\") " Nov 03 20:32:58.471505 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.471452 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ac27a9c-459f-4310-aef7-8dc643b46c6c-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ac27a9c-459f-4310-aef7-8dc643b46c6c-tls-certs\") pod \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\" (UID: \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\") " Nov 03 20:32:58.471571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.471560 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-tmp-dir\") pod \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\" (UID: \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\") " Nov 03 20:32:58.471740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.471726 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-model-cache\") pod \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\" (UID: \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\") " Nov 03 20:32:58.471838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.471826 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-home\" (UniqueName: \"kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-home\") pod \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\" (UID: \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\") " Nov 03 20:32:58.471888 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.471877 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ac27a9c-459f-4310-aef7-8dc643b46c6c-kube-api-access-n58hn\" (UniqueName: \"kubernetes.io/projected/5ac27a9c-459f-4310-aef7-8dc643b46c6c-kube-api-access-n58hn\") pod \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\" (UID: \"5ac27a9c-459f-4310-aef7-8dc643b46c6c\") " Nov 03 20:32:58.471923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.471911 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad491ce3-2f92-45a7-9b4f-278309989e64-tls-certificates\" (UniqueName: \"kubernetes.io/secret/ad491ce3-2f92-45a7-9b4f-278309989e64-tls-certificates\") pod \"ad491ce3-2f92-45a7-9b4f-278309989e64\" (UID: \"ad491ce3-2f92-45a7-9b4f-278309989e64\") " Nov 03 20:32:58.473977 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.473950 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-home" pod "5ac27a9c-459f-4310-aef7-8dc643b46c6c" (UID: "5ac27a9c-459f-4310-aef7-8dc643b46c6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:58.473977 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.473972 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-tmp-dir" pod "5ac27a9c-459f-4310-aef7-8dc643b46c6c" (UID: "5ac27a9c-459f-4310-aef7-8dc643b46c6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:58.474436 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.474398 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-model-cache" pod "5ac27a9c-459f-4310-aef7-8dc643b46c6c" (UID: "5ac27a9c-459f-4310-aef7-8dc643b46c6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:58.478475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.478439 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ac27a9c-459f-4310-aef7-8dc643b46c6c-tls-certs" pod "5ac27a9c-459f-4310-aef7-8dc643b46c6c" (UID: "5ac27a9c-459f-4310-aef7-8dc643b46c6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:58.479305 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.479286 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-dshm" pod "5ac27a9c-459f-4310-aef7-8dc643b46c6c" (UID: "5ac27a9c-459f-4310-aef7-8dc643b46c6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:58.480444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.480421 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad491ce3-2f92-45a7-9b4f-278309989e64-tls-certificates" pod "ad491ce3-2f92-45a7-9b4f-278309989e64" (UID: "ad491ce3-2f92-45a7-9b4f-278309989e64"). InnerVolumeSpecName "tls-certificates". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:58.480516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.480439 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac27a9c-459f-4310-aef7-8dc643b46c6c-kube-api-access-n58hn" pod "5ac27a9c-459f-4310-aef7-8dc643b46c6c" (UID: "5ac27a9c-459f-4310-aef7-8dc643b46c6c"). InnerVolumeSpecName "kube-api-access-n58hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:58.573766 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.573741 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a985b50-c4c0-4a39-abed-29f53e2be747-kube-api-access-8tvmx\" (UniqueName: \"kubernetes.io/projected/0a985b50-c4c0-4a39-abed-29f53e2be747-kube-api-access-8tvmx\") pod \"0a985b50-c4c0-4a39-abed-29f53e2be747\" (UID: \"0a985b50-c4c0-4a39-abed-29f53e2be747\") " Nov 03 20:32:58.574152 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.573776 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/633c352c-f07e-4032-b962-23a148297e45-kube-api-access-p9rhb\" (UniqueName: \"kubernetes.io/projected/633c352c-f07e-4032-b962-23a148297e45-kube-api-access-p9rhb\") pod \"633c352c-f07e-4032-b962-23a148297e45\" (UID: \"633c352c-f07e-4032-b962-23a148297e45\") " Nov 03 20:32:58.574152 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.573846 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ea6e69e-3556-4ffe-bb9e-796e8795d747-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea6e69e-3556-4ffe-bb9e-796e8795d747-metrics-certs\") pod \"4ea6e69e-3556-4ffe-bb9e-796e8795d747\" (UID: \"4ea6e69e-3556-4ffe-bb9e-796e8795d747\") " Nov 03 20:32:58.574152 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.573879 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-kube-api-access-f7pbn\" (UniqueName: \"kubernetes.io/projected/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-kube-api-access-f7pbn\") pod \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " Nov 03 20:32:58.574152 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.573914 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls\") pod \"8d0db609-b271-4c98-a303-fb3ee71f152c\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " Nov 03 20:32:58.574152 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.573950 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d0db609-b271-4c98-a303-fb3ee71f152c-telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8d0db609-b271-4c98-a303-fb3ee71f152c-telemetry-config\") pod \"8d0db609-b271-4c98-a303-fb3ee71f152c\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " Nov 03 20:32:58.574152 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.573978 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-csr-ca-configmap\") pod \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " Nov 03 20:32:58.574152 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.574006 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-csr-dns-cert\") pod \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " Nov 03 20:32:58.574152 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.574038 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-config\") pod \"40b2621f-854e-44e0-9b01-c877e333f644\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " Nov 03 20:32:58.574152 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.574077 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c99b0c19-004e-450c-852a-e38338635b61-serving-cert\" (UniqueName: \"kubernetes.io/secret/c99b0c19-004e-450c-852a-e38338635b61-serving-cert\") pod \"c99b0c19-004e-450c-852a-e38338635b61\" (UID: \"c99b0c19-004e-450c-852a-e38338635b61\") " Nov 03 20:32:58.574152 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.574123 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ea6e69e-3556-4ffe-bb9e-796e8795d747-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea6e69e-3556-4ffe-bb9e-796e8795d747-openshift-service-ca\") pod \"4ea6e69e-3556-4ffe-bb9e-796e8795d747\" (UID: \"4ea6e69e-3556-4ffe-bb9e-796e8795d747\") " Nov 03 20:32:58.574152 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.574157 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64153a99-d5f7-4f92-a0c9-f79b39eba97c-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/64153a99-d5f7-4f92-a0c9-f79b39eba97c-local-certs\") pod \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " Nov 03 20:32:58.574998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.574196 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b2621f-854e-44e0-9b01-c877e333f644-kube-api-access-jc5qt\" (UniqueName: \"kubernetes.io/projected/40b2621f-854e-44e0-9b01-c877e333f644-kube-api-access-jc5qt\") pod \"40b2621f-854e-44e0-9b01-c877e333f644\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " Nov 03 20:32:58.574998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.574263 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-cacerts\" (UniqueName: \"kubernetes.io/secret/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-cacerts\") pod \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " Nov 03 20:32:58.574998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.574293 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c99b0c19-004e-450c-852a-e38338635b61-kube-api-access-xk5v5\" (UniqueName: \"kubernetes.io/projected/c99b0c19-004e-450c-852a-e38338635b61-kube-api-access-xk5v5\") pod \"c99b0c19-004e-450c-852a-e38338635b61\" (UID: \"c99b0c19-004e-450c-852a-e38338635b61\") " Nov 03 20:32:58.574998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.574333 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64153a99-d5f7-4f92-a0c9-f79b39eba97c-kube-api-access-p4rpm\" (UniqueName: \"kubernetes.io/projected/64153a99-d5f7-4f92-a0c9-f79b39eba97c-kube-api-access-p4rpm\") pod \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " Nov 03 20:32:58.574998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.574363 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea6e69e-3556-4ffe-bb9e-796e8795d747-kube-api-access-vf6rz\" (UniqueName: \"kubernetes.io/projected/4ea6e69e-3556-4ffe-bb9e-796e8795d747-kube-api-access-vf6rz\") pod \"4ea6e69e-3556-4ffe-bb9e-796e8795d747\" (UID: \"4ea6e69e-3556-4ffe-bb9e-796e8795d747\") " Nov 03 20:32:58.574998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.574404 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert\") pod \"04a982f9-143c-406c-99d5-9ef73d5c571a\" (UID: \"04a982f9-143c-406c-99d5-9ef73d5c571a\") " Nov 03 20:32:58.574998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.574440 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d0db609-b271-4c98-a303-fb3ee71f152c-kube-api-access-mkxhk\" (UniqueName: \"kubernetes.io/projected/8d0db609-b271-4c98-a303-fb3ee71f152c-kube-api-access-mkxhk\") pod \"8d0db609-b271-4c98-a303-fb3ee71f152c\" (UID: \"8d0db609-b271-4c98-a303-fb3ee71f152c\") " Nov 03 20:32:58.574998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.574500 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7937ca-c705-4091-967e-2e640c622a26-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a7937ca-c705-4091-967e-2e640c622a26-tmp\") pod \"6a7937ca-c705-4091-967e-2e640c622a26\" (UID: \"6a7937ca-c705-4091-967e-2e640c622a26\") " Nov 03 20:32:58.574998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.574556 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ea6e69e-3556-4ffe-bb9e-796e8795d747-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea6e69e-3556-4ffe-bb9e-796e8795d747-maas-api-tls\") pod \"4ea6e69e-3556-4ffe-bb9e-796e8795d747\" (UID: \"4ea6e69e-3556-4ffe-bb9e-796e8795d747\") " Nov 03 20:32:58.574998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.574572 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d0db609-b271-4c98-a303-fb3ee71f152c-telemetry-config" pod "8d0db609-b271-4c98-a303-fb3ee71f152c" (UID: "8d0db609-b271-4c98-a303-fb3ee71f152c"). InnerVolumeSpecName "telemetry-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:58.574998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.574652 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04a982f9-143c-406c-99d5-9ef73d5c571a-nginx-conf\" (UniqueName: \"kubernetes.io/configmap/04a982f9-143c-406c-99d5-9ef73d5c571a-nginx-conf\") pod \"04a982f9-143c-406c-99d5-9ef73d5c571a\" (UID: \"04a982f9-143c-406c-99d5-9ef73d5c571a\") " Nov 03 20:32:58.574998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.574726 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-token\" (UniqueName: \"kubernetes.io/projected/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-token\") pod \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " Nov 03 20:32:58.574998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.574835 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-token\" (UniqueName: \"kubernetes.io/projected/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-token\") pod \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " Nov 03 20:32:58.574998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.574955 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-csr-dns-cert\") pod \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " Nov 03 20:32:58.575655 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.575073 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c99b0c19-004e-450c-852a-e38338635b61-config\" (UniqueName: \"kubernetes.io/configmap/c99b0c19-004e-450c-852a-e38338635b61-config\") pod \"c99b0c19-004e-450c-852a-e38338635b61\" (UID: \"c99b0c19-004e-450c-852a-e38338635b61\") " Nov 03 20:32:58.575655 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.575108 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a7937ca-c705-4091-967e-2e640c622a26-kube-api-access-q7rw8\" (UniqueName: \"kubernetes.io/projected/6a7937ca-c705-4091-967e-2e640c622a26-kube-api-access-q7rw8\") pod \"6a7937ca-c705-4091-967e-2e640c622a26\" (UID: \"6a7937ca-c705-4091-967e-2e640c622a26\") " Nov 03 20:32:58.575655 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.575141 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64153a99-d5f7-4f92-a0c9-f79b39eba97c-cacerts\" (UniqueName: \"kubernetes.io/secret/64153a99-d5f7-4f92-a0c9-f79b39eba97c-cacerts\") pod \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " Nov 03 20:32:58.575655 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.575300 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/815b15b6-2c37-4ca3-9b38-e7e1328d0aa8-kube-api-access-mdss4\" (UniqueName: \"kubernetes.io/projected/815b15b6-2c37-4ca3-9b38-e7e1328d0aa8-kube-api-access-mdss4\") pod \"815b15b6-2c37-4ca3-9b38-e7e1328d0aa8\" (UID: \"815b15b6-2c37-4ca3-9b38-e7e1328d0aa8\") " Nov 03 20:32:58.575655 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.575339 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/941b16dd-0550-4434-86f9-26e38efb68ee-kube-api-access-p7dng\" (UniqueName: \"kubernetes.io/projected/941b16dd-0550-4434-86f9-26e38efb68ee-kube-api-access-p7dng\") pod \"941b16dd-0550-4434-86f9-26e38efb68ee\" (UID: \"941b16dd-0550-4434-86f9-26e38efb68ee\") " Nov 03 20:32:58.575655 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.575441 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-csr-ca-configmap\") pod \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " Nov 03 20:32:58.575655 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.575499 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-kubeconfig\") pod \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " Nov 03 20:32:58.575655 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.575540 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-local-certs\") pod \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " Nov 03 20:32:58.575655 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.575593 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls\") pod \"40b2621f-854e-44e0-9b01-c877e333f644\" (UID: \"40b2621f-854e-44e0-9b01-c877e333f644\") " Nov 03 20:32:58.577029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.575820 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-kubeconfig\") pod \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\" (UID: \"64153a99-d5f7-4f92-a0c9-f79b39eba97c\") " Nov 03 20:32:58.577029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.575895 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/633c352c-f07e-4032-b962-23a148297e45-data\" (UniqueName: \"kubernetes.io/empty-dir/633c352c-f07e-4032-b962-23a148297e45-data\") pod \"633c352c-f07e-4032-b962-23a148297e45\" (UID: \"633c352c-f07e-4032-b962-23a148297e45\") " Nov 03 20:32:58.578033 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.578007 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ac27a9c-459f-4310-aef7-8dc643b46c6c-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.578178 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.578037 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.578178 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.578052 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.578178 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.578068 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.578178 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.578082 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n58hn\" (UniqueName: \"kubernetes.io/projected/5ac27a9c-459f-4310-aef7-8dc643b46c6c-kube-api-access-n58hn\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.578178 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.578098 5059 reconciler_common.go:299] "Volume detached for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8d0db609-b271-4c98-a303-fb3ee71f152c-telemetry-config\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.578178 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.578112 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ad491ce3-2f92-45a7-9b4f-278309989e64-tls-certificates\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.578178 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.578125 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ac27a9c-459f-4310-aef7-8dc643b46c6c-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.579711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.579412 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-config" pod "40b2621f-854e-44e0-9b01-c877e333f644" (UID: "40b2621f-854e-44e0-9b01-c877e333f644"). InnerVolumeSpecName "samples-operator-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:58.588102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.586341 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-csr-ca-configmap" pod "64153a99-d5f7-4f92-a0c9-f79b39eba97c" (UID: "64153a99-d5f7-4f92-a0c9-f79b39eba97c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:58.591723 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.591693 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-csr-dns-cert" pod "64153a99-d5f7-4f92-a0c9-f79b39eba97c" (UID: "64153a99-d5f7-4f92-a0c9-f79b39eba97c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:58.597312 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.597283 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea6e69e-3556-4ffe-bb9e-796e8795d747-openshift-service-ca" pod "4ea6e69e-3556-4ffe-bb9e-796e8795d747" (UID: "4ea6e69e-3556-4ffe-bb9e-796e8795d747"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:58.597598 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.597575 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c99b0c19-004e-450c-852a-e38338635b61-config" pod "c99b0c19-004e-450c-852a-e38338635b61" (UID: "c99b0c19-004e-450c-852a-e38338635b61"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:58.623805 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.622625 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-csr-ca-configmap" pod "677a5153-5fe3-4624-bf06-3c8dfc6bc72d" (UID: "677a5153-5fe3-4624-bf06-3c8dfc6bc72d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:58.624071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.624034 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-local-certs" pod "677a5153-5fe3-4624-bf06-3c8dfc6bc72d" (UID: "677a5153-5fe3-4624-bf06-3c8dfc6bc72d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:58.648705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.647817 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c99b0c19-004e-450c-852a-e38338635b61-kube-api-access-xk5v5" pod "c99b0c19-004e-450c-852a-e38338635b61" (UID: "c99b0c19-004e-450c-852a-e38338635b61"). InnerVolumeSpecName "kube-api-access-xk5v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:58.657701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.655800 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-kubeconfig" pod "64153a99-d5f7-4f92-a0c9-f79b39eba97c" (UID: "64153a99-d5f7-4f92-a0c9-f79b39eba97c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:58.703531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.683974 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/633c352c-f07e-4032-b962-23a148297e45-kube-api-access-p9rhb" pod "633c352c-f07e-4032-b962-23a148297e45" (UID: "633c352c-f07e-4032-b962-23a148297e45"). InnerVolumeSpecName "kube-api-access-p9rhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:58.708565 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.706291 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea6e69e-3556-4ffe-bb9e-796e8795d747-maas-api-tls" pod "4ea6e69e-3556-4ffe-bb9e-796e8795d747" (UID: "4ea6e69e-3556-4ffe-bb9e-796e8795d747"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:58.708565 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.707245 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea6e69e-3556-4ffe-bb9e-796e8795d747-metrics-certs" pod "4ea6e69e-3556-4ffe-bb9e-796e8795d747" (UID: "4ea6e69e-3556-4ffe-bb9e-796e8795d747"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:58.708565 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.707931 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a982f9-143c-406c-99d5-9ef73d5c571a-nginx-conf" pod "04a982f9-143c-406c-99d5-9ef73d5c571a" (UID: "04a982f9-143c-406c-99d5-9ef73d5c571a"). InnerVolumeSpecName "nginx-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:58.711127 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.709082 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-token" pod "677a5153-5fe3-4624-bf06-3c8dfc6bc72d" (UID: "677a5153-5fe3-4624-bf06-3c8dfc6bc72d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:58.720253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.720212 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-token\" (UniqueName: \"kubernetes.io/projected/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-token\") pod \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\" (UID: \"677a5153-5fe3-4624-bf06-3c8dfc6bc72d\") " Nov 03 20:32:58.723333 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:58.720456 5059 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/677a5153-5fe3-4624-bf06-3c8dfc6bc72d/volumes/kubernetes.io~projected/istio-token Nov 03 20:32:58.723333 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.723284 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-token" pod "677a5153-5fe3-4624-bf06-3c8dfc6bc72d" (UID: "677a5153-5fe3-4624-bf06-3c8dfc6bc72d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:58.723333 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.721605 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7937ca-c705-4091-967e-2e640c622a26-tmp" pod "6a7937ca-c705-4091-967e-2e640c622a26" (UID: "6a7937ca-c705-4091-967e-2e640c622a26"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:58.723757 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.723731 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ea6e69e-3556-4ffe-bb9e-796e8795d747-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea6e69e-3556-4ffe-bb9e-796e8795d747-metrics-certs\") pod \"4ea6e69e-3556-4ffe-bb9e-796e8795d747\" (UID: \"4ea6e69e-3556-4ffe-bb9e-796e8795d747\") " Nov 03 20:32:58.724463 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:58.723908 5059 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/4ea6e69e-3556-4ffe-bb9e-796e8795d747/volumes/kubernetes.io~secret/metrics-certs Nov 03 20:32:58.724463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.724437 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea6e69e-3556-4ffe-bb9e-796e8795d747-metrics-certs" pod "4ea6e69e-3556-4ffe-bb9e-796e8795d747" (UID: "4ea6e69e-3556-4ffe-bb9e-796e8795d747"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:58.725639 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.725617 5059 reconciler_common.go:299] "Volume detached for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/04a982f9-143c-406c-99d5-9ef73d5c571a-nginx-conf\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.725755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.725742 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.725907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.725894 5059 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c99b0c19-004e-450c-852a-e38338635b61-config\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.725985 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.725974 5059 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-csr-ca-configmap\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.726062 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.726051 5059 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-local-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.726130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.726120 5059 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-kubeconfig\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.726216 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.726204 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9rhb\" (UniqueName: \"kubernetes.io/projected/633c352c-f07e-4032-b962-23a148297e45-kube-api-access-p9rhb\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.726286 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.726275 5059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea6e69e-3556-4ffe-bb9e-796e8795d747-metrics-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.726359 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.726348 5059 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-csr-ca-configmap\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.726450 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.726439 5059 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-csr-dns-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.726522 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.726510 5059 reconciler_common.go:299] "Volume detached for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-config\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.726597 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.726586 5059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea6e69e-3556-4ffe-bb9e-796e8795d747-openshift-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.726681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.726657 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk5v5\" (UniqueName: \"kubernetes.io/projected/c99b0c19-004e-450c-852a-e38338635b61-kube-api-access-xk5v5\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.726755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.726745 5059 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a7937ca-c705-4091-967e-2e640c622a26-tmp\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.726824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.726814 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea6e69e-3556-4ffe-bb9e-796e8795d747-maas-api-tls\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.728455 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.728332 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/815b15b6-2c37-4ca3-9b38-e7e1328d0aa8-kube-api-access-mdss4" pod "815b15b6-2c37-4ca3-9b38-e7e1328d0aa8" (UID: "815b15b6-2c37-4ca3-9b38-e7e1328d0aa8"). InnerVolumeSpecName "kube-api-access-mdss4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:58.728455 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.728430 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a985b50-c4c0-4a39-abed-29f53e2be747-kube-api-access-8tvmx" pod "0a985b50-c4c0-4a39-abed-29f53e2be747" (UID: "0a985b50-c4c0-4a39-abed-29f53e2be747"). InnerVolumeSpecName "kube-api-access-8tvmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:58.742571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.732776 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64153a99-d5f7-4f92-a0c9-f79b39eba97c-kube-api-access-p4rpm" pod "64153a99-d5f7-4f92-a0c9-f79b39eba97c" (UID: "64153a99-d5f7-4f92-a0c9-f79b39eba97c"). InnerVolumeSpecName "kube-api-access-p4rpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:58.800937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.800831 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-5fffc97bdf-brvgz_7b8992fc-fba6-4135-90f8-fb259e765dd5/service-ca-controller/1.log" Nov 03 20:32:58.801848 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.801588 5059 generic.go:356] "Generic (PLEG): container finished" podID="7b8992fc-fba6-4135-90f8-fb259e765dd5" containerID="79f4410301f54a1a4f3ead6bc5461974ff7f958c6320d7f9399fbe12307acdfb" exitCode=255 Nov 03 20:32:58.801848 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.801713 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" event={"ID":"7b8992fc-fba6-4135-90f8-fb259e765dd5","Type":"ContainerDied","Data":"79f4410301f54a1a4f3ead6bc5461974ff7f958c6320d7f9399fbe12307acdfb"} Nov 03 20:32:58.801848 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.801769 5059 scope.go:122] "RemoveContainer" containerID="a9bfcc2a7fbcc7ee6fea838b485c6e2a9dbb15881176ff436fbfec03c0df0eaf" Nov 03 20:32:58.804596 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.804086 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:32:58.804596 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.804137 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" podUID="f0d3c590-f523-4b7e-9086-60df5fc585e8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:32:58.814188 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.813869 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" event={"ID":"40b2621f-854e-44e0-9b01-c877e333f644","Type":"ContainerDied","Data":"d8b3f14031a53eecbdb2db39f15186180f1b7059e2868a65f8b7a4fe42d2380a"} Nov 03 20:32:58.814188 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.813992 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx" Nov 03 20:32:58.821945 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.821887 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-csr-dns-cert" pod "677a5153-5fe3-4624-bf06-3c8dfc6bc72d" (UID: "677a5153-5fe3-4624-bf06-3c8dfc6bc72d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:58.832675 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.822173 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" event={"ID":"c99b0c19-004e-450c-852a-e38338635b61","Type":"ContainerDied","Data":"22cc4b81bd4c7dc1a411b6935fbe5de145ecdb4a6cb45704d4810326ae0a896e"} Nov 03 20:32:58.832675 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.822257 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl" Nov 03 20:32:58.832675 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.822464 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-cacerts" pod "677a5153-5fe3-4624-bf06-3c8dfc6bc72d" (UID: "677a5153-5fe3-4624-bf06-3c8dfc6bc72d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:58.832675 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.830320 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tvmx\" (UniqueName: \"kubernetes.io/projected/0a985b50-c4c0-4a39-abed-29f53e2be747-kube-api-access-8tvmx\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.832675 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.830339 5059 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-cacerts\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.832675 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.830349 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4rpm\" (UniqueName: \"kubernetes.io/projected/64153a99-d5f7-4f92-a0c9-f79b39eba97c-kube-api-access-p4rpm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.832675 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.830358 5059 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-csr-dns-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.832675 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.830369 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdss4\" (UniqueName: \"kubernetes.io/projected/815b15b6-2c37-4ca3-9b38-e7e1328d0aa8-kube-api-access-mdss4\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.832675 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.830925 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/633c352c-f07e-4032-b962-23a148297e45-data" pod "633c352c-f07e-4032-b962-23a148297e45" (UID: "633c352c-f07e-4032-b962-23a148297e45"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.834478 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-wh95x_492c7b4e-84c0-49a7-b12f-153a021f971b/kube-storage-version-migrator-operator/2.log" Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.835043 5059 generic.go:356] "Generic (PLEG): container finished" podID="492c7b4e-84c0-49a7-b12f-153a021f971b" containerID="9a314ded87df57e4578fee99a500049875bfdb3f70a6196625866f2a41d5c267" exitCode=255 Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.835156 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" event={"ID":"492c7b4e-84c0-49a7-b12f-153a021f971b","Type":"ContainerDied","Data":"9a314ded87df57e4578fee99a500049875bfdb3f70a6196625866f2a41d5c267"} Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.838229 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-fsbsp_1bcd4690-4753-4246-abb4-8668a0a56231/volume-data-source-validator/1.log" Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.839573 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-fsbsp_1bcd4690-4753-4246-abb4-8668a0a56231/volume-data-source-validator/0.log" Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.839603 5059 generic.go:356] "Generic (PLEG): container finished" podID="1bcd4690-4753-4246-abb4-8668a0a56231" containerID="34fbf9edd9a9b6fb620b51e1b7b72b6469ff59a49c409953c3ad6b93fa26cf2f" exitCode=2 Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.839713 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fsbsp" event={"ID":"1bcd4690-4753-4246-abb4-8668a0a56231","Type":"ContainerDied","Data":"34fbf9edd9a9b6fb620b51e1b7b72b6469ff59a49c409953c3ad6b93fa26cf2f"} Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.842118 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-c88f8" event={"ID":"6a7937ca-c705-4091-967e-2e640c622a26","Type":"ContainerDied","Data":"18aa1249881644f7f3beb3555c25607ffb06ef4df7000b71ce35b76f7c4cf22a"} Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.842208 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-c88f8" Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.846061 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" event={"ID":"677a5153-5fe3-4624-bf06-3c8dfc6bc72d","Type":"ContainerDied","Data":"b393f72f0eb96740f96ac2b965ee7b282d6d33d6db11a8993b42ebddaee9a7fc"} Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.846143 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc" Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.846909 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls" pod "8d0db609-b271-4c98-a303-fb3ee71f152c" (UID: "8d0db609-b271-4c98-a303-fb3ee71f152c"). InnerVolumeSpecName "cluster-monitoring-operator-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.847677 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64153a99-d5f7-4f92-a0c9-f79b39eba97c-cacerts" pod "64153a99-d5f7-4f92-a0c9-f79b39eba97c" (UID: "64153a99-d5f7-4f92-a0c9-f79b39eba97c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.848893 5059 generic.go:356] "Generic (PLEG): container finished" podID="4a35329c-82fb-479f-abca-bff9ed082558" containerID="2e57d5a40799f051dc61748015ccfef4a18d3c2d9058dcbee05723c8f222c743" exitCode=0 Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.848962 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84" event={"ID":"4a35329c-82fb-479f-abca-bff9ed082558","Type":"ContainerDied","Data":"2e57d5a40799f051dc61748015ccfef4a18d3c2d9058dcbee05723c8f222c743"} Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.853796 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-token" pod "64153a99-d5f7-4f92-a0c9-f79b39eba97c" (UID: "64153a99-d5f7-4f92-a0c9-f79b39eba97c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.853897 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert" pod "04a982f9-143c-406c-99d5-9ef73d5c571a" (UID: "04a982f9-143c-406c-99d5-9ef73d5c571a"). InnerVolumeSpecName "networking-console-plugin-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.853905 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64153a99-d5f7-4f92-a0c9-f79b39eba97c-local-certs" pod "64153a99-d5f7-4f92-a0c9-f79b39eba97c" (UID: "64153a99-d5f7-4f92-a0c9-f79b39eba97c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.853955 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls" pod "40b2621f-854e-44e0-9b01-c877e333f644" (UID: "40b2621f-854e-44e0-9b01-c877e333f644"). InnerVolumeSpecName "samples-operator-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.854017 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c99b0c19-004e-450c-852a-e38338635b61-serving-cert" pod "c99b0c19-004e-450c-852a-e38338635b61" (UID: "c99b0c19-004e-450c-852a-e38338635b61"). InnerVolumeSpecName "serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.854156 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d0db609-b271-4c98-a303-fb3ee71f152c-kube-api-access-mkxhk" pod "8d0db609-b271-4c98-a303-fb3ee71f152c" (UID: "8d0db609-b271-4c98-a303-fb3ee71f152c"). InnerVolumeSpecName "kube-api-access-mkxhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.855730 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941b16dd-0550-4434-86f9-26e38efb68ee-kube-api-access-p7dng" pod "941b16dd-0550-4434-86f9-26e38efb68ee" (UID: "941b16dd-0550-4434-86f9-26e38efb68ee"). InnerVolumeSpecName "kube-api-access-p7dng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.856443 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-kubeconfig" pod "677a5153-5fe3-4624-bf06-3c8dfc6bc72d" (UID: "677a5153-5fe3-4624-bf06-3c8dfc6bc72d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.856490 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7937ca-c705-4091-967e-2e640c622a26-kube-api-access-q7rw8" pod "6a7937ca-c705-4091-967e-2e640c622a26" (UID: "6a7937ca-c705-4091-967e-2e640c622a26"). InnerVolumeSpecName "kube-api-access-q7rw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.856547 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-kube-api-access-f7pbn" pod "677a5153-5fe3-4624-bf06-3c8dfc6bc72d" (UID: "677a5153-5fe3-4624-bf06-3c8dfc6bc72d"). InnerVolumeSpecName "kube-api-access-f7pbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.857044 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/3.log" Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.857451 5059 generic.go:356] "Generic (PLEG): container finished" podID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" containerID="f3a682944a052f5773970fd4a1c1414d244464e281630d9cbe3a0f7623116968" exitCode=255 Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.857524 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" event={"ID":"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac","Type":"ContainerDied","Data":"f3a682944a052f5773970fd4a1c1414d244464e281630d9cbe3a0f7623116968"} Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.860547 5059 generic.go:356] "Generic (PLEG): container finished" podID="6785e24f-6fb3-43ab-a469-84d30d94b2f0" containerID="95024980a099f9ccdf78362bbb5aa84a698eaab1e6f9c5a70a9520e6a4b6e491" exitCode=0 Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.860631 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" event={"ID":"6785e24f-6fb3-43ab-a469-84d30d94b2f0","Type":"ContainerDied","Data":"95024980a099f9ccdf78362bbb5aa84a698eaab1e6f9c5a70a9520e6a4b6e491"} Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.863209 5059 generic.go:356] "Generic (PLEG): container finished" podID="c7fce352-1850-4048-be2e-fa458fd4d2f3" containerID="8121a3441525651ca2c833f0f9236c162d2c111dd09ec30bb62b251a8accdd62" exitCode=0 Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.863274 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66b59d468d-5bfd2" event={"ID":"c7fce352-1850-4048-be2e-fa458fd4d2f3","Type":"ContainerDied","Data":"8121a3441525651ca2c833f0f9236c162d2c111dd09ec30bb62b251a8accdd62"} Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.864767 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b2621f-854e-44e0-9b01-c877e333f644-kube-api-access-jc5qt" pod "40b2621f-854e-44e0-9b01-c877e333f644" (UID: "40b2621f-854e-44e0-9b01-c877e333f644"). InnerVolumeSpecName "kube-api-access-jc5qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.866123 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qchj6" event={"ID":"ad491ce3-2f92-45a7-9b4f-278309989e64","Type":"ContainerDied","Data":"55e4a9cfd3117a07fe7fd35c39dc1e744a4e090608b00a10f19636b940937dce"} Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.866211 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qchj6" Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.874262 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nj52h" event={"ID":"941b16dd-0550-4434-86f9-26e38efb68ee","Type":"ContainerDied","Data":"856432fd57e3ebd630fbb3bc3ed8e855f477cf255bc3d852e79fb8e43e9b5835"} Nov 03 20:32:58.874453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.874378 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nj52h" Nov 03 20:32:58.910014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.882697 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" event={"ID":"4ea6e69e-3556-4ffe-bb9e-796e8795d747","Type":"ContainerDied","Data":"ddbc5f90543f820fd5f5ea6b8587849321d7d4c87b3a8a2ea777eb765a071395"} Nov 03 20:32:58.910014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.882913 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2" Nov 03 20:32:58.910014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.885185 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts" event={"ID":"633c352c-f07e-4032-b962-23a148297e45","Type":"ContainerDied","Data":"885f346c447707f93bb08eda59e11a44a56b8387e22202e8922cad337479c936"} Nov 03 20:32:58.910014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.885263 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts" Nov 03 20:32:58.910014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.895270 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tvb96" event={"ID":"0a985b50-c4c0-4a39-abed-29f53e2be747","Type":"ContainerDied","Data":"574c6906b17ca7382d033e3c1490a3c3ffef5f8642f16e2bde9dbd6349e080a2"} Nov 03 20:32:58.910014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.895346 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tvb96" Nov 03 20:32:58.910014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.900461 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" event={"ID":"04a982f9-143c-406c-99d5-9ef73d5c571a","Type":"ContainerDied","Data":"887eb13a4950ee22ec72790db18fd5bfedfc6957e0af69fc8c65d2c77f870323"} Nov 03 20:32:58.910014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.900528 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-846mt" Nov 03 20:32:58.910014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.908575 5059 generic.go:356] "Generic (PLEG): container finished" podID="1072628f-f513-4b4e-8954-afee5a1aa733" containerID="c92f883ff6898e09b68a779d22ce3740c636fefbf8228b2a6a5e2048a78c601e" exitCode=0 Nov 03 20:32:58.910014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.908697 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" event={"ID":"1072628f-f513-4b4e-8954-afee5a1aa733","Type":"ContainerDied","Data":"c92f883ff6898e09b68a779d22ce3740c636fefbf8228b2a6a5e2048a78c601e"} Nov 03 20:32:58.915739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.915174 5059 generic.go:356] "Generic (PLEG): container finished" podID="532de14d-f2dd-4c66-ba6f-8b103f7c9375" containerID="6f53b40ec729940a8b447824fba4edc4a9b60468b059f5be6551e07af2e247ed" exitCode=0 Nov 03 20:32:58.915739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.915260 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mm85p" event={"ID":"532de14d-f2dd-4c66-ba6f-8b103f7c9375","Type":"ContainerDied","Data":"6f53b40ec729940a8b447824fba4edc4a9b60468b059f5be6551e07af2e247ed"} Nov 03 20:32:58.938100 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.924457 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" Nov 03 20:32:58.938100 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.927711 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b" event={"ID":"8d0db609-b271-4c98-a303-fb3ee71f152c","Type":"ContainerDied","Data":"bc5f2d96ab72bf6de58c0862a759ae4375b7675d1de97b000932e8d79181cd70"} Nov 03 20:32:58.947538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.941261 5059 reconciler_common.go:299] "Volume detached for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/04a982f9-143c-406c-99d5-9ef73d5c571a-networking-console-plugin-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.947538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.941293 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkxhk\" (UniqueName: \"kubernetes.io/projected/8d0db609-b271-4c98-a303-fb3ee71f152c-kube-api-access-mkxhk\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.947538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.941310 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64153a99-d5f7-4f92-a0c9-f79b39eba97c-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.947538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.941325 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7rw8\" (UniqueName: \"kubernetes.io/projected/6a7937ca-c705-4091-967e-2e640c622a26-kube-api-access-q7rw8\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.947538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.941341 5059 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/64153a99-d5f7-4f92-a0c9-f79b39eba97c-cacerts\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.947538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.941357 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7dng\" (UniqueName: \"kubernetes.io/projected/941b16dd-0550-4434-86f9-26e38efb68ee-kube-api-access-p7dng\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.947538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.941371 5059 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-istio-kubeconfig\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.947538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.941387 5059 reconciler_common.go:299] "Volume detached for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40b2621f-854e-44e0-9b01-c877e333f644-samples-operator-tls\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.947538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.941400 5059 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/633c352c-f07e-4032-b962-23a148297e45-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.947538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.941418 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7pbn\" (UniqueName: \"kubernetes.io/projected/677a5153-5fe3-4624-bf06-3c8dfc6bc72d-kube-api-access-f7pbn\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.947538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.941432 5059 reconciler_common.go:299] "Volume detached for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d0db609-b271-4c98-a303-fb3ee71f152c-cluster-monitoring-operator-tls\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.947538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.941447 5059 reconciler_common.go:299] "Volume detached for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c99b0c19-004e-450c-852a-e38338635b61-serving-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.947538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.941462 5059 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/64153a99-d5f7-4f92-a0c9-f79b39eba97c-local-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.947538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.941477 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc5qt\" (UniqueName: \"kubernetes.io/projected/40b2621f-854e-44e0-9b01-c877e333f644-kube-api-access-jc5qt\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.948702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.948622 5059 generic.go:356] "Generic (PLEG): container finished" podID="f485893c-6f14-449b-b837-8ac181315aeb" containerID="88b5b44b71c45a6eb5b61eaf4c105d6ffa0032eb4646d4eb2fa600f2d3805504" exitCode=0 Nov 03 20:32:58.948789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.948727 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6k9n9" event={"ID":"f485893c-6f14-449b-b837-8ac181315aeb","Type":"ContainerDied","Data":"88b5b44b71c45a6eb5b61eaf4c105d6ffa0032eb4646d4eb2fa600f2d3805504"} Nov 03 20:32:58.949541 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.949312 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea6e69e-3556-4ffe-bb9e-796e8795d747-kube-api-access-vf6rz" pod "4ea6e69e-3556-4ffe-bb9e-796e8795d747" (UID: "4ea6e69e-3556-4ffe-bb9e-796e8795d747"). InnerVolumeSpecName "kube-api-access-vf6rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:58.954778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.950574 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" Nov 03 20:32:58.955047 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.954977 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" Nov 03 20:32:58.970552 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.970297 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c" event={"ID":"64153a99-d5f7-4f92-a0c9-f79b39eba97c","Type":"ContainerDied","Data":"8ba1f76672e6305c0b0bdd96ef45512c44b85b7829f85b511af9bea1f49228c9"} Nov 03 20:32:58.971958 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.971903 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" event={"ID":"270c522c-ef72-46a7-9e65-3be2ebdbcfa6","Type":"ContainerDied","Data":"95bb03881e42514f575533d04c3cf2c9897aba4b7ba9dea11e5683f2620cea59"} Nov 03 20:32:58.972137 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.971870 5059 generic.go:356] "Generic (PLEG): container finished" podID="270c522c-ef72-46a7-9e65-3be2ebdbcfa6" containerID="95bb03881e42514f575533d04c3cf2c9897aba4b7ba9dea11e5683f2620cea59" exitCode=0 Nov 03 20:32:58.976421 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.975018 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" event={"ID":"5ac27a9c-459f-4310-aef7-8dc643b46c6c","Type":"ContainerDied","Data":"504e87c07e2e0fb48957e4f70d79e1670ae33f82206e608d6d77f1eb0478480d"} Nov 03 20:32:58.976421 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.975112 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-d7wsq" Nov 03 20:32:58.977398 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.977377 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pbgwk" event={"ID":"815b15b6-2c37-4ca3-9b38-e7e1328d0aa8","Type":"ContainerDied","Data":"6c41b8db3ecc464cfdadc0046d8144c354dc482d49d376e427a16f417211c4bc"} Nov 03 20:32:58.977468 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.977439 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pbgwk" Nov 03 20:32:58.993840 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:58.993646 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84" Nov 03 20:32:59.006625 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.006594 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-fsbsp_1bcd4690-4753-4246-abb4-8668a0a56231/volume-data-source-validator/1.log" Nov 03 20:32:59.009098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.008236 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-fsbsp_1bcd4690-4753-4246-abb4-8668a0a56231/volume-data-source-validator/0.log" Nov 03 20:32:59.012093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.012071 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fsbsp" Nov 03 20:32:59.017206 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.017183 5059 scope.go:122] "RemoveContainer" containerID="1a4ab42465871189364c0b8d0033b852cbd3488b01bc6dd2ae20a8268416653f" Nov 03 20:32:59.027061 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.027038 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66b59d468d-5bfd2" Nov 03 20:32:59.035018 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.034982 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" Nov 03 20:32:59.038186 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.038154 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6k9n9" Nov 03 20:32:59.038414 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.038394 5059 scope.go:122] "RemoveContainer" containerID="b2569884782bd34d772a62814be138ac35a8f3a6ac0255af71fcef484fe98a3d" Nov 03 20:32:59.044156 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.043856 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 20:32:59.045969 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.045941 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf6rz\" (UniqueName: \"kubernetes.io/projected/4ea6e69e-3556-4ffe-bb9e-796e8795d747-kube-api-access-vf6rz\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.046437 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.046420 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/3.log" Nov 03 20:32:59.063864 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.063774 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 20:32:59.087520 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.087496 5059 scope.go:122] "RemoveContainer" containerID="b742b7b0202501adeba598c21e37a746e39b623b622c6c629096721511b0d73e" Nov 03 20:32:59.088259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.088229 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-wh95x_492c7b4e-84c0-49a7-b12f-153a021f971b/kube-storage-version-migrator-operator/2.log" Nov 03 20:32:59.088365 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.088231 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-5fffc97bdf-brvgz_7b8992fc-fba6-4135-90f8-fb259e765dd5/service-ca-controller/1.log" Nov 03 20:32:59.092709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.092687 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" Nov 03 20:32:59.095754 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.095731 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mm85p" Nov 03 20:32:59.098550 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.098532 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" Nov 03 20:32:59.105003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.103599 5059 scope.go:122] "RemoveContainer" containerID="8b68d8a54774f3a932e8c394d04b88681ee2fd3757c1b50ef500cf35cb4119f9" Nov 03 20:32:59.120155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.120049 5059 scope.go:122] "RemoveContainer" containerID="56aa53c1d031f7ab6c282efd0400e4a7abf4371a2d9df0ad33beb6d199334684" Nov 03 20:32:59.148578 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.148543 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bcd4690-4753-4246-abb4-8668a0a56231-kube-api-access-8wvnz\" (UniqueName: \"kubernetes.io/projected/1bcd4690-4753-4246-abb4-8668a0a56231-kube-api-access-8wvnz\") pod \"1bcd4690-4753-4246-abb4-8668a0a56231\" (UID: \"1bcd4690-4753-4246-abb4-8668a0a56231\") " Nov 03 20:32:59.148737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.148633 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1072628f-f513-4b4e-8954-afee5a1aa733-kube-api-access-w522z\" (UniqueName: \"kubernetes.io/projected/1072628f-f513-4b4e-8954-afee5a1aa733-kube-api-access-w522z\") pod \"1072628f-f513-4b4e-8954-afee5a1aa733\" (UID: \"1072628f-f513-4b4e-8954-afee5a1aa733\") " Nov 03 20:32:59.148737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.148709 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f485893c-6f14-449b-b837-8ac181315aeb-kube-api-access-86bkx\" (UniqueName: \"kubernetes.io/projected/f485893c-6f14-449b-b837-8ac181315aeb-kube-api-access-86bkx\") pod \"f485893c-6f14-449b-b837-8ac181315aeb\" (UID: \"f485893c-6f14-449b-b837-8ac181315aeb\") " Nov 03 20:32:59.148835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.148740 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6785e24f-6fb3-43ab-a469-84d30d94b2f0-apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6785e24f-6fb3-43ab-a469-84d30d94b2f0-apiservice-cert\") pod \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\" (UID: \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\") " Nov 03 20:32:59.148835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.148813 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6785e24f-6fb3-43ab-a469-84d30d94b2f0-kube-api-access-4n4kg\" (UniqueName: \"kubernetes.io/projected/6785e24f-6fb3-43ab-a469-84d30d94b2f0-kube-api-access-4n4kg\") pod \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\" (UID: \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\") " Nov 03 20:32:59.148911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.148838 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1072628f-f513-4b4e-8954-afee5a1aa733-cert\" (UniqueName: \"kubernetes.io/secret/1072628f-f513-4b4e-8954-afee5a1aa733-cert\") pod \"1072628f-f513-4b4e-8954-afee5a1aa733\" (UID: \"1072628f-f513-4b4e-8954-afee5a1aa733\") " Nov 03 20:32:59.148911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.148876 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7fce352-1850-4048-be2e-fa458fd4d2f3-tls-cert\" (UniqueName: \"kubernetes.io/secret/c7fce352-1850-4048-be2e-fa458fd4d2f3-tls-cert\") pod \"c7fce352-1850-4048-be2e-fa458fd4d2f3\" (UID: \"c7fce352-1850-4048-be2e-fa458fd4d2f3\") " Nov 03 20:32:59.148988 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.148940 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a35329c-82fb-479f-abca-bff9ed082558-kube-api-access-p2454\" (UniqueName: \"kubernetes.io/projected/4a35329c-82fb-479f-abca-bff9ed082558-kube-api-access-p2454\") pod \"4a35329c-82fb-479f-abca-bff9ed082558\" (UID: \"4a35329c-82fb-479f-abca-bff9ed082558\") " Nov 03 20:32:59.149095 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.149077 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6785e24f-6fb3-43ab-a469-84d30d94b2f0-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6785e24f-6fb3-43ab-a469-84d30d94b2f0-tmp\") pod \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\" (UID: \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\") " Nov 03 20:32:59.149140 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.149130 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6785e24f-6fb3-43ab-a469-84d30d94b2f0-webhook-cert\" (UniqueName: \"kubernetes.io/secret/6785e24f-6fb3-43ab-a469-84d30d94b2f0-webhook-cert\") pod \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\" (UID: \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\") " Nov 03 20:32:59.149178 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.149166 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/270c522c-ef72-46a7-9e65-3be2ebdbcfa6-kube-api-access-rwhh5\" (UniqueName: \"kubernetes.io/projected/270c522c-ef72-46a7-9e65-3be2ebdbcfa6-kube-api-access-rwhh5\") pod \"270c522c-ef72-46a7-9e65-3be2ebdbcfa6\" (UID: \"270c522c-ef72-46a7-9e65-3be2ebdbcfa6\") " Nov 03 20:32:59.149296 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.149278 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1072628f-f513-4b4e-8954-afee5a1aa733-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1072628f-f513-4b4e-8954-afee5a1aa733-metrics-certs\") pod \"1072628f-f513-4b4e-8954-afee5a1aa733\" (UID: \"1072628f-f513-4b4e-8954-afee5a1aa733\") " Nov 03 20:32:59.149431 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.149416 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7fce352-1850-4048-be2e-fa458fd4d2f3-kube-api-access-62z44\" (UniqueName: \"kubernetes.io/projected/c7fce352-1850-4048-be2e-fa458fd4d2f3-kube-api-access-62z44\") pod \"c7fce352-1850-4048-be2e-fa458fd4d2f3\" (UID: \"c7fce352-1850-4048-be2e-fa458fd4d2f3\") " Nov 03 20:32:59.149508 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.149494 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6785e24f-6fb3-43ab-a469-84d30d94b2f0-manifests\" (UniqueName: \"kubernetes.io/empty-dir/6785e24f-6fb3-43ab-a469-84d30d94b2f0-manifests\") pod \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\" (UID: \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\") " Nov 03 20:32:59.149560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.149546 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1072628f-f513-4b4e-8954-afee5a1aa733-manager-config\" (UniqueName: \"kubernetes.io/configmap/1072628f-f513-4b4e-8954-afee5a1aa733-manager-config\") pod \"1072628f-f513-4b4e-8954-afee5a1aa733\" (UID: \"1072628f-f513-4b4e-8954-afee5a1aa733\") " Nov 03 20:32:59.149626 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.149611 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6785e24f-6fb3-43ab-a469-84d30d94b2f0-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6785e24f-6fb3-43ab-a469-84d30d94b2f0-metrics-cert\") pod \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\" (UID: \"6785e24f-6fb3-43ab-a469-84d30d94b2f0\") " Nov 03 20:32:59.163389 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.163327 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1072628f-f513-4b4e-8954-afee5a1aa733-metrics-certs" pod "1072628f-f513-4b4e-8954-afee5a1aa733" (UID: "1072628f-f513-4b4e-8954-afee5a1aa733"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:59.166737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.165495 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7fce352-1850-4048-be2e-fa458fd4d2f3-kube-api-access-62z44" pod "c7fce352-1850-4048-be2e-fa458fd4d2f3" (UID: "c7fce352-1850-4048-be2e-fa458fd4d2f3"). InnerVolumeSpecName "kube-api-access-62z44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:59.167162 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.167125 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6785e24f-6fb3-43ab-a469-84d30d94b2f0-tmp" pod "6785e24f-6fb3-43ab-a469-84d30d94b2f0" (UID: "6785e24f-6fb3-43ab-a469-84d30d94b2f0"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:59.168060 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.168022 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bcd4690-4753-4246-abb4-8668a0a56231-kube-api-access-8wvnz" pod "1bcd4690-4753-4246-abb4-8668a0a56231" (UID: "1bcd4690-4753-4246-abb4-8668a0a56231"). InnerVolumeSpecName "kube-api-access-8wvnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:59.169332 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.169005 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a35329c-82fb-479f-abca-bff9ed082558-kube-api-access-p2454" pod "4a35329c-82fb-479f-abca-bff9ed082558" (UID: "4a35329c-82fb-479f-abca-bff9ed082558"). InnerVolumeSpecName "kube-api-access-p2454". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:59.173380 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.173340 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6785e24f-6fb3-43ab-a469-84d30d94b2f0-kube-api-access-4n4kg" pod "6785e24f-6fb3-43ab-a469-84d30d94b2f0" (UID: "6785e24f-6fb3-43ab-a469-84d30d94b2f0"). InnerVolumeSpecName "kube-api-access-4n4kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:59.174623 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.174534 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1072628f-f513-4b4e-8954-afee5a1aa733-cert" pod "1072628f-f513-4b4e-8954-afee5a1aa733" (UID: "1072628f-f513-4b4e-8954-afee5a1aa733"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:59.175318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.175188 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f485893c-6f14-449b-b837-8ac181315aeb-kube-api-access-86bkx" pod "f485893c-6f14-449b-b837-8ac181315aeb" (UID: "f485893c-6f14-449b-b837-8ac181315aeb"). InnerVolumeSpecName "kube-api-access-86bkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:59.175458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.175432 5059 scope.go:122] "RemoveContainer" containerID="cecbb61d72ca19f659a4e15f9281290fd0e1acefbff34b44775daf81cb4ed42c" Nov 03 20:32:59.179030 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.179002 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1072628f-f513-4b4e-8954-afee5a1aa733-kube-api-access-w522z" pod "1072628f-f513-4b4e-8954-afee5a1aa733" (UID: "1072628f-f513-4b4e-8954-afee5a1aa733"). InnerVolumeSpecName "kube-api-access-w522z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:59.180177 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.180144 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/270c522c-ef72-46a7-9e65-3be2ebdbcfa6-kube-api-access-rwhh5" pod "270c522c-ef72-46a7-9e65-3be2ebdbcfa6" (UID: "270c522c-ef72-46a7-9e65-3be2ebdbcfa6"). InnerVolumeSpecName "kube-api-access-rwhh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:59.193745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.187030 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1072628f-f513-4b4e-8954-afee5a1aa733-manager-config" pod "1072628f-f513-4b4e-8954-afee5a1aa733" (UID: "1072628f-f513-4b4e-8954-afee5a1aa733"). InnerVolumeSpecName "manager-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:59.193745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.187745 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7fce352-1850-4048-be2e-fa458fd4d2f3-tls-cert" pod "c7fce352-1850-4048-be2e-fa458fd4d2f3" (UID: "c7fce352-1850-4048-be2e-fa458fd4d2f3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:59.193745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.190385 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6785e24f-6fb3-43ab-a469-84d30d94b2f0-metrics-cert" pod "6785e24f-6fb3-43ab-a469-84d30d94b2f0" (UID: "6785e24f-6fb3-43ab-a469-84d30d94b2f0"). InnerVolumeSpecName "metrics-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:59.193745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.190939 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6785e24f-6fb3-43ab-a469-84d30d94b2f0-webhook-cert" pod "6785e24f-6fb3-43ab-a469-84d30d94b2f0" (UID: "6785e24f-6fb3-43ab-a469-84d30d94b2f0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:59.196689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.196631 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6785e24f-6fb3-43ab-a469-84d30d94b2f0-apiservice-cert" pod "6785e24f-6fb3-43ab-a469-84d30d94b2f0" (UID: "6785e24f-6fb3-43ab-a469-84d30d94b2f0"). InnerVolumeSpecName "apiservice-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:59.198626 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.198606 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6785e24f-6fb3-43ab-a469-84d30d94b2f0-manifests" pod "6785e24f-6fb3-43ab-a469-84d30d94b2f0" (UID: "6785e24f-6fb3-43ab-a469-84d30d94b2f0"). InnerVolumeSpecName "manifests". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:59.227058 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.222846 5059 scope.go:122] "RemoveContainer" containerID="daf533566a4042ce35f672f5974749c01a13dc2086255c8268966b3679ba35ea" Nov 03 20:32:59.243714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.243687 5059 scope.go:122] "RemoveContainer" containerID="667c8d1cf790b30fc65fdbeb2860e35758fd0e7eb1773d3f331c5e009252f137" Nov 03 20:32:59.252373 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.252333 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-serving-cert\") pod \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\" (UID: \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\") " Nov 03 20:32:59.252495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.252417 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/492c7b4e-84c0-49a7-b12f-153a021f971b-kube-api-access-hgtxr\" (UniqueName: \"kubernetes.io/projected/492c7b4e-84c0-49a7-b12f-153a021f971b-kube-api-access-hgtxr\") pod \"492c7b4e-84c0-49a7-b12f-153a021f971b\" (UID: \"492c7b4e-84c0-49a7-b12f-153a021f971b\") " Nov 03 20:32:59.252495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.252470 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b8992fc-fba6-4135-90f8-fb259e765dd5-kube-api-access-7rnvd\" (UniqueName: \"kubernetes.io/projected/7b8992fc-fba6-4135-90f8-fb259e765dd5-kube-api-access-7rnvd\") pod \"7b8992fc-fba6-4135-90f8-fb259e765dd5\" (UID: \"7b8992fc-fba6-4135-90f8-fb259e765dd5\") " Nov 03 20:32:59.252639 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.252621 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-config\" (UniqueName: \"kubernetes.io/configmap/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-config\") pod \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\" (UID: \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\") " Nov 03 20:32:59.252733 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.252712 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-kube-api-access-tv87l\" (UniqueName: \"kubernetes.io/projected/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-kube-api-access-tv87l\") pod \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\" (UID: \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\") " Nov 03 20:32:59.252806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.252786 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/492c7b4e-84c0-49a7-b12f-153a021f971b-config\" (UniqueName: \"kubernetes.io/configmap/492c7b4e-84c0-49a7-b12f-153a021f971b-config\") pod \"492c7b4e-84c0-49a7-b12f-153a021f971b\" (UID: \"492c7b4e-84c0-49a7-b12f-153a021f971b\") " Nov 03 20:32:59.252897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.252882 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/492c7b4e-84c0-49a7-b12f-153a021f971b-serving-cert\" (UniqueName: \"kubernetes.io/secret/492c7b4e-84c0-49a7-b12f-153a021f971b-serving-cert\") pod \"492c7b4e-84c0-49a7-b12f-153a021f971b\" (UID: \"492c7b4e-84c0-49a7-b12f-153a021f971b\") " Nov 03 20:32:59.252970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.252955 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b8992fc-fba6-4135-90f8-fb259e765dd5-signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7b8992fc-fba6-4135-90f8-fb259e765dd5-signing-cabundle\") pod \"7b8992fc-fba6-4135-90f8-fb259e765dd5\" (UID: \"7b8992fc-fba6-4135-90f8-fb259e765dd5\") " Nov 03 20:32:59.253102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.253084 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/532de14d-f2dd-4c66-ba6f-8b103f7c9375-kube-api-access-zmzpr\" (UniqueName: \"kubernetes.io/projected/532de14d-f2dd-4c66-ba6f-8b103f7c9375-kube-api-access-zmzpr\") pod \"532de14d-f2dd-4c66-ba6f-8b103f7c9375\" (UID: \"532de14d-f2dd-4c66-ba6f-8b103f7c9375\") " Nov 03 20:32:59.253169 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.253139 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532de14d-f2dd-4c66-ba6f-8b103f7c9375-tmp\" (UniqueName: \"kubernetes.io/empty-dir/532de14d-f2dd-4c66-ba6f-8b103f7c9375-tmp\") pod \"532de14d-f2dd-4c66-ba6f-8b103f7c9375\" (UID: \"532de14d-f2dd-4c66-ba6f-8b103f7c9375\") " Nov 03 20:32:59.253275 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.253259 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-trusted-ca\") pod \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\" (UID: \"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac\") " Nov 03 20:32:59.253331 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.253310 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b8992fc-fba6-4135-90f8-fb259e765dd5-signing-key\" (UniqueName: \"kubernetes.io/secret/7b8992fc-fba6-4135-90f8-fb259e765dd5-signing-key\") pod \"7b8992fc-fba6-4135-90f8-fb259e765dd5\" (UID: \"7b8992fc-fba6-4135-90f8-fb259e765dd5\") " Nov 03 20:32:59.260709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.254225 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wvnz\" (UniqueName: \"kubernetes.io/projected/1bcd4690-4753-4246-abb4-8668a0a56231-kube-api-access-8wvnz\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.260709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.254256 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w522z\" (UniqueName: \"kubernetes.io/projected/1072628f-f513-4b4e-8954-afee5a1aa733-kube-api-access-w522z\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.260709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.254272 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86bkx\" (UniqueName: \"kubernetes.io/projected/f485893c-6f14-449b-b837-8ac181315aeb-kube-api-access-86bkx\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.260709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.254287 5059 reconciler_common.go:299] "Volume detached for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6785e24f-6fb3-43ab-a469-84d30d94b2f0-apiservice-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.260709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.254301 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n4kg\" (UniqueName: \"kubernetes.io/projected/6785e24f-6fb3-43ab-a469-84d30d94b2f0-kube-api-access-4n4kg\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.260709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.255727 5059 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1072628f-f513-4b4e-8954-afee5a1aa733-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.260709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.255755 5059 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7fce352-1850-4048-be2e-fa458fd4d2f3-tls-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.260709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.255768 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2454\" (UniqueName: \"kubernetes.io/projected/4a35329c-82fb-479f-abca-bff9ed082558-kube-api-access-p2454\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.260709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.255781 5059 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6785e24f-6fb3-43ab-a469-84d30d94b2f0-tmp\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.260709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.255794 5059 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6785e24f-6fb3-43ab-a469-84d30d94b2f0-webhook-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.260709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.255806 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwhh5\" (UniqueName: \"kubernetes.io/projected/270c522c-ef72-46a7-9e65-3be2ebdbcfa6-kube-api-access-rwhh5\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.260709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.255818 5059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1072628f-f513-4b4e-8954-afee5a1aa733-metrics-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.260709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.255830 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62z44\" (UniqueName: \"kubernetes.io/projected/c7fce352-1850-4048-be2e-fa458fd4d2f3-kube-api-access-62z44\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.260709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.255843 5059 reconciler_common.go:299] "Volume detached for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6785e24f-6fb3-43ab-a469-84d30d94b2f0-manifests\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.260709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.255856 5059 reconciler_common.go:299] "Volume detached for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1072628f-f513-4b4e-8954-afee5a1aa733-manager-config\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.260709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.255869 5059 reconciler_common.go:299] "Volume detached for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6785e24f-6fb3-43ab-a469-84d30d94b2f0-metrics-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.260709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.256961 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492c7b4e-84c0-49a7-b12f-153a021f971b-config" pod "492c7b4e-84c0-49a7-b12f-153a021f971b" (UID: "492c7b4e-84c0-49a7-b12f-153a021f971b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:59.260709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.258645 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b8992fc-fba6-4135-90f8-fb259e765dd5-signing-cabundle" pod "7b8992fc-fba6-4135-90f8-fb259e765dd5" (UID: "7b8992fc-fba6-4135-90f8-fb259e765dd5"). InnerVolumeSpecName "signing-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:59.260709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.259774 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532de14d-f2dd-4c66-ba6f-8b103f7c9375-tmp" pod "532de14d-f2dd-4c66-ba6f-8b103f7c9375" (UID: "532de14d-f2dd-4c66-ba6f-8b103f7c9375"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:59.260709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.259811 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-trusted-ca" pod "f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" (UID: "f8b140c9-9e09-4cbd-847b-82ee1e3be6ac"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:59.260709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.260194 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-config" pod "f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" (UID: "f8b140c9-9e09-4cbd-847b-82ee1e3be6ac"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:59.261408 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.261093 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b8992fc-fba6-4135-90f8-fb259e765dd5-signing-key" pod "7b8992fc-fba6-4135-90f8-fb259e765dd5" (UID: "7b8992fc-fba6-4135-90f8-fb259e765dd5"). InnerVolumeSpecName "signing-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:59.263913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.263885 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/532de14d-f2dd-4c66-ba6f-8b103f7c9375-kube-api-access-zmzpr" pod "532de14d-f2dd-4c66-ba6f-8b103f7c9375" (UID: "532de14d-f2dd-4c66-ba6f-8b103f7c9375"). InnerVolumeSpecName "kube-api-access-zmzpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:59.264896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.264802 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-kube-api-access-tv87l" pod "f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" (UID: "f8b140c9-9e09-4cbd-847b-82ee1e3be6ac"). InnerVolumeSpecName "kube-api-access-tv87l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:59.266878 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.266851 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/492c7b4e-84c0-49a7-b12f-153a021f971b-serving-cert" pod "492c7b4e-84c0-49a7-b12f-153a021f971b" (UID: "492c7b4e-84c0-49a7-b12f-153a021f971b"). InnerVolumeSpecName "serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:59.267034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.266977 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8992fc-fba6-4135-90f8-fb259e765dd5-kube-api-access-7rnvd" pod "7b8992fc-fba6-4135-90f8-fb259e765dd5" (UID: "7b8992fc-fba6-4135-90f8-fb259e765dd5"). InnerVolumeSpecName "kube-api-access-7rnvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:59.267034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.267015 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-serving-cert" pod "f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" (UID: "f8b140c9-9e09-4cbd-847b-82ee1e3be6ac"). InnerVolumeSpecName "serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:59.268572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.268547 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492c7b4e-84c0-49a7-b12f-153a021f971b-kube-api-access-hgtxr" pod "492c7b4e-84c0-49a7-b12f-153a021f971b" (UID: "492c7b4e-84c0-49a7-b12f-153a021f971b"). InnerVolumeSpecName "kube-api-access-hgtxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:59.272939 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.272830 5059 scope.go:122] "RemoveContainer" containerID="b7138eb33c5d6fa9eeb22093ad7d51deb0e28bca02ed9a1262e5df9e2a221d24" Nov 03 20:32:59.287907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.287882 5059 scope.go:122] "RemoveContainer" containerID="24306abe36f01f202cf4225862e6256f61b8116c2b16568f30c02b610183da1c" Nov 03 20:32:59.301857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.301831 5059 scope.go:122] "RemoveContainer" containerID="d84ec766302d997e0dd2646c72025bbffe88f5776ae0c76a2bedd7ae24b833ab" Nov 03 20:32:59.329115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.329081 5059 scope.go:122] "RemoveContainer" containerID="ac8fdfdbcf156bbdd591e8f7042d8b960bce286b35806ac720a68b8da48a5940" Nov 03 20:32:59.358640 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.358601 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rnvd\" (UniqueName: \"kubernetes.io/projected/7b8992fc-fba6-4135-90f8-fb259e765dd5-kube-api-access-7rnvd\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.358640 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.358640 5059 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-config\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.358858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.358671 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv87l\" (UniqueName: \"kubernetes.io/projected/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-kube-api-access-tv87l\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.358858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.358685 5059 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/492c7b4e-84c0-49a7-b12f-153a021f971b-config\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.358858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.358700 5059 reconciler_common.go:299] "Volume detached for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/492c7b4e-84c0-49a7-b12f-153a021f971b-serving-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.358858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.358714 5059 reconciler_common.go:299] "Volume detached for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7b8992fc-fba6-4135-90f8-fb259e765dd5-signing-cabundle\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.358858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.358727 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmzpr\" (UniqueName: \"kubernetes.io/projected/532de14d-f2dd-4c66-ba6f-8b103f7c9375-kube-api-access-zmzpr\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.358858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.358743 5059 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/532de14d-f2dd-4c66-ba6f-8b103f7c9375-tmp\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.358858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.358757 5059 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-trusted-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.358858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.358770 5059 reconciler_common.go:299] "Volume detached for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7b8992fc-fba6-4135-90f8-fb259e765dd5-signing-key\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.358858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.358784 5059 reconciler_common.go:299] "Volume detached for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac-serving-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.358858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.358799 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgtxr\" (UniqueName: \"kubernetes.io/projected/492c7b4e-84c0-49a7-b12f-153a021f971b-kube-api-access-hgtxr\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.364453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.364422 5059 scope.go:122] "RemoveContainer" containerID="c21a21bf6b505e70d50fabd813f6c6546130257160456e17e096e7eab6ae4f7d" Nov 03 20:32:59.389642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.389599 5059 scope.go:122] "RemoveContainer" containerID="e882b42c99b4f73bf517ede2116273cfd6eda2d61af90817e8c3c5603097f5b8" Nov 03 20:32:59.424045 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.424007 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:32:59.424187 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.424089 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" podUID="d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:32:59.429467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.429444 5059 scope.go:122] "RemoveContainer" containerID="b3dbf9416fa12ac2df38a62843937fa243f692c05fb2159a754be69427937d5d" Nov 03 20:32:59.453621 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.453594 5059 scope.go:122] "RemoveContainer" containerID="147857ae5dcfb3795e8ee333ec3863a15882c306754cc03d2571ba058fc49ce3" Nov 03 20:32:59.467937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.467917 5059 scope.go:122] "RemoveContainer" containerID="cadce9a446527c8c3f13706fda86af3620c7a7c77a7b43d35d1a7f7d6579e0e8" Nov 03 20:32:59.484975 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.484940 5059 scope.go:122] "RemoveContainer" containerID="111ab0145bc09ea2159df08b45a75e0a16c06fb447f17cd20b906e1b1a7c091f" Nov 03 20:32:59.501691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.501645 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4"] Nov 03 20:32:59.502862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.502838 5059 cpu_manager.go:414] "containerMap: removing container" podUID="4a35329c-82fb-479f-abca-bff9ed082558" containerName="manager" Nov 03 20:32:59.502964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.502863 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a35329c-82fb-479f-abca-bff9ed082558" containerName="manager" Nov 03 20:32:59.502964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.502880 5059 cpu_manager.go:414] "containerMap: removing container" podUID="04a982f9-143c-406c-99d5-9ef73d5c571a" containerName="networking-console-plugin" Nov 03 20:32:59.502964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.502890 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a982f9-143c-406c-99d5-9ef73d5c571a" containerName="networking-console-plugin" Nov 03 20:32:59.502964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.502906 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1072628f-f513-4b4e-8954-afee5a1aa733" containerName="manager" Nov 03 20:32:59.502964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.502920 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1072628f-f513-4b4e-8954-afee5a1aa733" containerName="manager" Nov 03 20:32:59.502964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.502941 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1072628f-f513-4b4e-8954-afee5a1aa733" containerName="manager" Nov 03 20:32:59.502964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.502948 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1072628f-f513-4b4e-8954-afee5a1aa733" containerName="manager" Nov 03 20:32:59.502964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.502960 5059 cpu_manager.go:414] "containerMap: removing container" podUID="7b8992fc-fba6-4135-90f8-fb259e765dd5" containerName="service-ca-controller" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.502968 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b8992fc-fba6-4135-90f8-fb259e765dd5" containerName="service-ca-controller" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.502982 5059 cpu_manager.go:414] "containerMap: removing container" podUID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" containerName="console-operator" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.502991 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" containerName="console-operator" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503001 5059 cpu_manager.go:414] "containerMap: removing container" podUID="40b2621f-854e-44e0-9b01-c877e333f644" containerName="cluster-samples-operator-watch" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503008 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b2621f-854e-44e0-9b01-c877e333f644" containerName="cluster-samples-operator-watch" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503038 5059 cpu_manager.go:414] "containerMap: removing container" podUID="64153a99-d5f7-4f92-a0c9-f79b39eba97c" containerName="discovery" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503046 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64153a99-d5f7-4f92-a0c9-f79b39eba97c" containerName="discovery" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503061 5059 cpu_manager.go:414] "containerMap: removing container" podUID="ad491ce3-2f92-45a7-9b4f-278309989e64" containerName="prometheus-operator-admission-webhook" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503070 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad491ce3-2f92-45a7-9b4f-278309989e64" containerName="prometheus-operator-admission-webhook" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503088 5059 cpu_manager.go:414] "containerMap: removing container" podUID="492c7b4e-84c0-49a7-b12f-153a021f971b" containerName="kube-storage-version-migrator-operator" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503097 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="492c7b4e-84c0-49a7-b12f-153a021f971b" containerName="kube-storage-version-migrator-operator" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503109 5059 cpu_manager.go:414] "containerMap: removing container" podUID="4a35329c-82fb-479f-abca-bff9ed082558" containerName="manager" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503117 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a35329c-82fb-479f-abca-bff9ed082558" containerName="manager" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503130 5059 cpu_manager.go:414] "containerMap: removing container" podUID="532de14d-f2dd-4c66-ba6f-8b103f7c9375" containerName="cert-manager-operator" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503138 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="532de14d-f2dd-4c66-ba6f-8b103f7c9375" containerName="cert-manager-operator" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503153 5059 cpu_manager.go:414] "containerMap: removing container" podUID="677a5153-5fe3-4624-bf06-3c8dfc6bc72d" containerName="discovery" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503162 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="677a5153-5fe3-4624-bf06-3c8dfc6bc72d" containerName="discovery" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503182 5059 cpu_manager.go:414] "containerMap: removing container" podUID="492c7b4e-84c0-49a7-b12f-153a021f971b" containerName="kube-storage-version-migrator-operator" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503190 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="492c7b4e-84c0-49a7-b12f-153a021f971b" containerName="kube-storage-version-migrator-operator" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503201 5059 cpu_manager.go:414] "containerMap: removing container" podUID="532de14d-f2dd-4c66-ba6f-8b103f7c9375" containerName="cert-manager-operator" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503208 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="532de14d-f2dd-4c66-ba6f-8b103f7c9375" containerName="cert-manager-operator" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503219 5059 cpu_manager.go:414] "containerMap: removing container" podUID="941b16dd-0550-4434-86f9-26e38efb68ee" containerName="manager" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503226 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="941b16dd-0550-4434-86f9-26e38efb68ee" containerName="manager" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503239 5059 cpu_manager.go:414] "containerMap: removing container" podUID="5ac27a9c-459f-4310-aef7-8dc643b46c6c" containerName="main" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503246 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac27a9c-459f-4310-aef7-8dc643b46c6c" containerName="main" Nov 03 20:32:59.503239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503255 5059 cpu_manager.go:414] "containerMap: removing container" podUID="40b2621f-854e-44e0-9b01-c877e333f644" containerName="cluster-samples-operator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503263 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b2621f-854e-44e0-9b01-c877e333f644" containerName="cluster-samples-operator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503279 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c99b0c19-004e-450c-852a-e38338635b61" containerName="service-ca-operator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503286 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c99b0c19-004e-450c-852a-e38338635b61" containerName="service-ca-operator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503299 5059 cpu_manager.go:414] "containerMap: removing container" podUID="f485893c-6f14-449b-b837-8ac181315aeb" containerName="migrator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503306 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f485893c-6f14-449b-b837-8ac181315aeb" containerName="migrator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503321 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1bcd4690-4753-4246-abb4-8668a0a56231" containerName="volume-data-source-validator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503328 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bcd4690-4753-4246-abb4-8668a0a56231" containerName="volume-data-source-validator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503342 5059 cpu_manager.go:414] "containerMap: removing container" podUID="6785e24f-6fb3-43ab-a469-84d30d94b2f0" containerName="manager" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503348 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6785e24f-6fb3-43ab-a469-84d30d94b2f0" containerName="manager" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503361 5059 cpu_manager.go:414] "containerMap: removing container" podUID="8d0db609-b271-4c98-a303-fb3ee71f152c" containerName="cluster-monitoring-operator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503369 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d0db609-b271-4c98-a303-fb3ee71f152c" containerName="cluster-monitoring-operator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503384 5059 cpu_manager.go:414] "containerMap: removing container" podUID="815b15b6-2c37-4ca3-9b38-e7e1328d0aa8" containerName="migrator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503391 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="815b15b6-2c37-4ca3-9b38-e7e1328d0aa8" containerName="migrator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503404 5059 cpu_manager.go:414] "containerMap: removing container" podUID="0a985b50-c4c0-4a39-abed-29f53e2be747" containerName="check-endpoints" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503412 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a985b50-c4c0-4a39-abed-29f53e2be747" containerName="check-endpoints" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503423 5059 cpu_manager.go:414] "containerMap: removing container" podUID="6a7937ca-c705-4091-967e-2e640c622a26" containerName="download-server" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503430 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a7937ca-c705-4091-967e-2e640c622a26" containerName="download-server" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503442 5059 cpu_manager.go:414] "containerMap: removing container" podUID="6785e24f-6fb3-43ab-a469-84d30d94b2f0" containerName="manager" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503449 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6785e24f-6fb3-43ab-a469-84d30d94b2f0" containerName="manager" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503460 5059 cpu_manager.go:414] "containerMap: removing container" podUID="633c352c-f07e-4032-b962-23a148297e45" containerName="postgres" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503467 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="633c352c-f07e-4032-b962-23a148297e45" containerName="postgres" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503486 5059 cpu_manager.go:414] "containerMap: removing container" podUID="4ea6e69e-3556-4ffe-bb9e-796e8795d747" containerName="maas-api" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503493 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea6e69e-3556-4ffe-bb9e-796e8795d747" containerName="maas-api" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503508 5059 cpu_manager.go:414] "containerMap: removing container" podUID="6785e24f-6fb3-43ab-a469-84d30d94b2f0" containerName="copy-manifests" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503515 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6785e24f-6fb3-43ab-a469-84d30d94b2f0" containerName="copy-manifests" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503527 5059 cpu_manager.go:414] "containerMap: removing container" podUID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" containerName="console-operator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503532 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" containerName="console-operator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503537 5059 cpu_manager.go:414] "containerMap: removing container" podUID="270c522c-ef72-46a7-9e65-3be2ebdbcfa6" containerName="manager" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503541 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="270c522c-ef72-46a7-9e65-3be2ebdbcfa6" containerName="manager" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503547 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c99b0c19-004e-450c-852a-e38338635b61" containerName="service-ca-operator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503552 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c99b0c19-004e-450c-852a-e38338635b61" containerName="service-ca-operator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503559 5059 cpu_manager.go:414] "containerMap: removing container" podUID="f485893c-6f14-449b-b837-8ac181315aeb" containerName="migrator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503563 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f485893c-6f14-449b-b837-8ac181315aeb" containerName="migrator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503571 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c7fce352-1850-4048-be2e-fa458fd4d2f3" containerName="authorino" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503576 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7fce352-1850-4048-be2e-fa458fd4d2f3" containerName="authorino" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503589 5059 cpu_manager.go:414] "containerMap: removing container" podUID="270c522c-ef72-46a7-9e65-3be2ebdbcfa6" containerName="manager" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503596 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="270c522c-ef72-46a7-9e65-3be2ebdbcfa6" containerName="manager" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503773 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="270c522c-ef72-46a7-9e65-3be2ebdbcfa6" containerName="manager" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503788 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b8992fc-fba6-4135-90f8-fb259e765dd5" containerName="service-ca-controller" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503801 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a35329c-82fb-479f-abca-bff9ed082558" containerName="manager" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503811 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a982f9-143c-406c-99d5-9ef73d5c571a" containerName="networking-console-plugin" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503819 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1072628f-f513-4b4e-8954-afee5a1aa733" containerName="manager" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503828 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1072628f-f513-4b4e-8954-afee5a1aa733" containerName="manager" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503838 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b8992fc-fba6-4135-90f8-fb259e765dd5" containerName="service-ca-controller" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503847 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" containerName="console-operator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503854 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b2621f-854e-44e0-9b01-c877e333f644" containerName="cluster-samples-operator-watch" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503866 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="64153a99-d5f7-4f92-a0c9-f79b39eba97c" containerName="discovery" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503876 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad491ce3-2f92-45a7-9b4f-278309989e64" containerName="prometheus-operator-admission-webhook" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503888 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="492c7b4e-84c0-49a7-b12f-153a021f971b" containerName="kube-storage-version-migrator-operator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503898 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a35329c-82fb-479f-abca-bff9ed082558" containerName="manager" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503908 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="532de14d-f2dd-4c66-ba6f-8b103f7c9375" containerName="cert-manager-operator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503918 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="677a5153-5fe3-4624-bf06-3c8dfc6bc72d" containerName="discovery" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503928 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="492c7b4e-84c0-49a7-b12f-153a021f971b" containerName="kube-storage-version-migrator-operator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503939 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="532de14d-f2dd-4c66-ba6f-8b103f7c9375" containerName="cert-manager-operator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503950 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="941b16dd-0550-4434-86f9-26e38efb68ee" containerName="manager" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503962 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ac27a9c-459f-4310-aef7-8dc643b46c6c" containerName="main" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503972 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b2621f-854e-44e0-9b01-c877e333f644" containerName="cluster-samples-operator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503982 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c99b0c19-004e-450c-852a-e38338635b61" containerName="service-ca-operator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503990 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f485893c-6f14-449b-b837-8ac181315aeb" containerName="migrator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.503999 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bcd4690-4753-4246-abb4-8668a0a56231" containerName="volume-data-source-validator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.504010 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6785e24f-6fb3-43ab-a469-84d30d94b2f0" containerName="manager" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.504020 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d0db609-b271-4c98-a303-fb3ee71f152c" containerName="cluster-monitoring-operator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.504030 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="815b15b6-2c37-4ca3-9b38-e7e1328d0aa8" containerName="migrator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.504041 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" containerName="console-operator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.504051 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a985b50-c4c0-4a39-abed-29f53e2be747" containerName="check-endpoints" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.504062 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a7937ca-c705-4091-967e-2e640c622a26" containerName="download-server" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.504070 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6785e24f-6fb3-43ab-a469-84d30d94b2f0" containerName="manager" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.504078 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="633c352c-f07e-4032-b962-23a148297e45" containerName="postgres" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.504120 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea6e69e-3556-4ffe-bb9e-796e8795d747" containerName="maas-api" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.504130 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" containerName="console-operator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.504140 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c99b0c19-004e-450c-852a-e38338635b61" containerName="service-ca-operator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.504150 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bcd4690-4753-4246-abb4-8668a0a56231" containerName="volume-data-source-validator" Nov 03 20:32:59.504397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.504159 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7fce352-1850-4048-be2e-fa458fd4d2f3" containerName="authorino" Nov 03 20:32:59.507365 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.507342 5059 scope.go:122] "RemoveContainer" containerID="6a679e526b0df571226e2d9d4986921d285477efff7809df408a7dcd93178490" Nov 03 20:32:59.510580 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.510555 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4" Nov 03 20:32:59.533093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.522154 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4"] Nov 03 20:32:59.533093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.522304 5059 scope.go:122] "RemoveContainer" containerID="5d874b39931c2408a3ab4748545ce2ab475106ee717ad14248ee3ca358ef09d0" Nov 03 20:32:59.623229 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:59.623094 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b9428a2_ab50_4e44_a725_e0da980bdb93.slice/crio-525f6d3424c4cc255ff7e85b227e91e40f3d49c08a0875d0fdde86eb52893da8 WatchSource:0}: Error finding container 525f6d3424c4cc255ff7e85b227e91e40f3d49c08a0875d0fdde86eb52893da8: Status 404 returned error can't find the container with id 525f6d3424c4cc255ff7e85b227e91e40f3d49c08a0875d0fdde86eb52893da8 Nov 03 20:32:59.623723 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:32:59.623694 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b25c2d0_60bb_4df1_8193_5f7f853c0a73.slice/crio-d1aa1348af3cbccddd516d7cce315ccc0828f8681c485afd95e08448f2ce5bae WatchSource:0}: Error finding container d1aa1348af3cbccddd516d7cce315ccc0828f8681c485afd95e08448f2ce5bae: Status 404 returned error can't find the container with id d1aa1348af3cbccddd516d7cce315ccc0828f8681c485afd95e08448f2ce5bae Nov 03 20:32:59.626936 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.626907 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5"] Nov 03 20:32:59.660827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.660739 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t"] Nov 03 20:32:59.681199 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.681164 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1326708b-4642-4992-91ea-8a590863a703-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bdd0b59-9wxx4\" (UID: \"1326708b-4642-4992-91ea-8a590863a703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4" Nov 03 20:32:59.681832 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.681798 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nkdn\" (UniqueName: \"kubernetes.io/projected/1326708b-4642-4992-91ea-8a590863a703-kube-api-access-9nkdn\") pod \"maas-api-revoke-sub-4bdd0b59-9wxx4\" (UID: \"1326708b-4642-4992-91ea-8a590863a703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4" Nov 03 20:32:59.684241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.684215 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1326708b-4642-4992-91ea-8a590863a703-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bdd0b59-9wxx4\" (UID: \"1326708b-4642-4992-91ea-8a590863a703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4" Nov 03 20:32:59.723067 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.718399 5059 scope.go:122] "RemoveContainer" containerID="1929782c8e0a4859c99bc362063000c768666ab59352d7c235edbdf79f308a9e" Nov 03 20:32:59.733812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.730377 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8"] Nov 03 20:32:59.733812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.731432 5059 cpu_manager.go:414] "containerMap: removing container" podUID="7b8992fc-fba6-4135-90f8-fb259e765dd5" containerName="service-ca-controller" Nov 03 20:32:59.733812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.731454 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b8992fc-fba6-4135-90f8-fb259e765dd5" containerName="service-ca-controller" Nov 03 20:32:59.733812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.731493 5059 cpu_manager.go:414] "containerMap: removing container" podUID="492c7b4e-84c0-49a7-b12f-153a021f971b" containerName="kube-storage-version-migrator-operator" Nov 03 20:32:59.733812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.731501 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="492c7b4e-84c0-49a7-b12f-153a021f971b" containerName="kube-storage-version-migrator-operator" Nov 03 20:32:59.733812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.731533 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1bcd4690-4753-4246-abb4-8668a0a56231" containerName="volume-data-source-validator" Nov 03 20:32:59.733812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.731542 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bcd4690-4753-4246-abb4-8668a0a56231" containerName="volume-data-source-validator" Nov 03 20:32:59.733812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.731581 5059 cpu_manager.go:414] "containerMap: removing container" podUID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" containerName="console-operator" Nov 03 20:32:59.733812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.731588 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" containerName="console-operator" Nov 03 20:32:59.733812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.731771 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="270c522c-ef72-46a7-9e65-3be2ebdbcfa6" containerName="manager" Nov 03 20:32:59.733812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.731794 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="492c7b4e-84c0-49a7-b12f-153a021f971b" containerName="kube-storage-version-migrator-operator" Nov 03 20:32:59.733812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.731808 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f485893c-6f14-449b-b837-8ac181315aeb" containerName="migrator" Nov 03 20:32:59.746679 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.743486 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr"] Nov 03 20:32:59.748578 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.748555 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h"] Nov 03 20:32:59.751543 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.751518 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8" Nov 03 20:32:59.752223 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.752019 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:32:59.753176 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.753157 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls"] Nov 03 20:32:59.754164 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.753266 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h" Nov 03 20:32:59.774857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.774831 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls" Nov 03 20:32:59.775332 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.774655 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8"] Nov 03 20:32:59.780778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.777077 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h"] Nov 03 20:32:59.785930 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.785907 5059 scope.go:122] "RemoveContainer" containerID="95bb03881e42514f575533d04c3cf2c9897aba4b7ba9dea11e5683f2620cea59" Nov 03 20:32:59.793430 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.793404 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1326708b-4642-4992-91ea-8a590863a703-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bdd0b59-9wxx4\" (UID: \"1326708b-4642-4992-91ea-8a590863a703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4" Nov 03 20:32:59.793645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.793622 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nkdn\" (UniqueName: \"kubernetes.io/projected/1326708b-4642-4992-91ea-8a590863a703-kube-api-access-9nkdn\") pod \"maas-api-revoke-sub-4bdd0b59-9wxx4\" (UID: \"1326708b-4642-4992-91ea-8a590863a703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4" Nov 03 20:32:59.794036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.793926 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1326708b-4642-4992-91ea-8a590863a703-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bdd0b59-9wxx4\" (UID: \"1326708b-4642-4992-91ea-8a590863a703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4" Nov 03 20:32:59.796344 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.796326 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr"] Nov 03 20:32:59.798703 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.798540 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1326708b-4642-4992-91ea-8a590863a703-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bdd0b59-9wxx4\" (UID: \"1326708b-4642-4992-91ea-8a590863a703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4" Nov 03 20:32:59.805715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.805643 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls"] Nov 03 20:32:59.820102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.820072 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1326708b-4642-4992-91ea-8a590863a703-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bdd0b59-9wxx4\" (UID: \"1326708b-4642-4992-91ea-8a590863a703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4" Nov 03 20:32:59.827343 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.827071 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nkdn\" (UniqueName: \"kubernetes.io/projected/1326708b-4642-4992-91ea-8a590863a703-kube-api-access-9nkdn\") pod \"maas-api-revoke-sub-4bdd0b59-9wxx4\" (UID: \"1326708b-4642-4992-91ea-8a590863a703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4" Nov 03 20:32:59.827343 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.827203 5059 scope.go:122] "RemoveContainer" containerID="a4f29644d81f5639b20431cb580b7160e30c9a7f0f42d00f2d940270329ea0f4" Nov 03 20:32:59.854917 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.854896 5059 scope.go:122] "RemoveContainer" containerID="3fd1b473caf0e5fcbbbf5a97f92155614da76dd2857a8df4e1895ff6ca147711" Nov 03 20:32:59.880864 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.878812 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4"] Nov 03 20:32:59.888619 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.888597 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4" Nov 03 20:32:59.890623 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.890598 5059 scope.go:122] "RemoveContainer" containerID="c3bd71ffe2f7af1614a32a2b8d3f234a09fc878537716084e68bd018236a5528" Nov 03 20:32:59.899971 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.899826 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-envoy\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:32:59.903168 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.902756 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-workload-certs\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:32:59.903168 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.902797 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-credential-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:32:59.903168 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.902822 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8gkn\" (UniqueName: \"kubernetes.io/projected/8760fa0e-a78b-4db8-b71d-bf4086ddbfbd-kube-api-access-w8gkn\") pod \"payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls\" (UID: \"8760fa0e-a78b-4db8-b71d-bf4086ddbfbd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls" Nov 03 20:32:59.903168 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.902855 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-podinfo\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:32:59.903168 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.902880 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8\" (UID: \"ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8" Nov 03 20:32:59.903168 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.902897 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-data\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:32:59.903168 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.902914 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8760fa0e-a78b-4db8-b71d-bf4086ddbfbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls\" (UID: \"8760fa0e-a78b-4db8-b71d-bf4086ddbfbd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls" Nov 03 20:32:59.903168 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.902944 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s7n4\" (UniqueName: \"kubernetes.io/projected/ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988-kube-api-access-4s7n4\") pod \"payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8\" (UID: \"ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8" Nov 03 20:32:59.903168 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.902967 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-token\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:32:59.903168 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.903008 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2n95\" (UniqueName: \"kubernetes.io/projected/d931dc13-4399-48dc-80e0-b0b304c5dda0-kube-api-access-s2n95\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:32:59.903168 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.903066 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nskbm\" (UniqueName: \"kubernetes.io/projected/2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1-kube-api-access-nskbm\") pod \"payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h\" (UID: \"2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h" Nov 03 20:32:59.903168 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.903156 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h\" (UID: \"2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h" Nov 03 20:32:59.903168 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.903185 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-workload-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:32:59.904016 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.903225 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d931dc13-4399-48dc-80e0-b0b304c5dda0-istiod-ca-cert\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:32:59.908577 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:32:59.908553 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4"] Nov 03 20:33:00.042724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.037980 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4" Nov 03 20:33:00.044417 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.038813 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nskbm\" (UniqueName: \"kubernetes.io/projected/2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1-kube-api-access-nskbm\") pod \"payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h\" (UID: \"2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h" Nov 03 20:33:00.044612 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.044514 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z79fs\" (UniqueName: \"kubernetes.io/projected/dc10f92e-e902-48c4-9cd5-4f768c4611be-kube-api-access-z79fs\") pod \"maas-api-revoke-sub-abad4031-r5fd4\" (UID: \"dc10f92e-e902-48c4-9cd5-4f768c4611be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4" Nov 03 20:33:00.044612 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.044594 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h\" (UID: \"2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h" Nov 03 20:33:00.044818 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.044631 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-workload-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:33:00.044818 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.044739 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d931dc13-4399-48dc-80e0-b0b304c5dda0-istiod-ca-cert\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:33:00.044959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.044856 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc10f92e-e902-48c4-9cd5-4f768c4611be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-r5fd4\" (UID: \"dc10f92e-e902-48c4-9cd5-4f768c4611be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4" Nov 03 20:33:00.045120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.045013 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-envoy\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:33:00.045120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.045088 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-workload-certs\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:33:00.045227 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.045144 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-credential-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:33:00.045227 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.045182 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8gkn\" (UniqueName: \"kubernetes.io/projected/8760fa0e-a78b-4db8-b71d-bf4086ddbfbd-kube-api-access-w8gkn\") pod \"payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls\" (UID: \"8760fa0e-a78b-4db8-b71d-bf4086ddbfbd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls" Nov 03 20:33:00.045427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.045249 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-podinfo\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:33:00.045427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.045291 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8\" (UID: \"ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8" Nov 03 20:33:00.045427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.045323 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-data\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:33:00.045427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.045353 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8760fa0e-a78b-4db8-b71d-bf4086ddbfbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls\" (UID: \"8760fa0e-a78b-4db8-b71d-bf4086ddbfbd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls" Nov 03 20:33:00.045427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.045411 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s7n4\" (UniqueName: \"kubernetes.io/projected/ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988-kube-api-access-4s7n4\") pod \"payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8\" (UID: \"ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8" Nov 03 20:33:00.045685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.045452 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-token\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:33:00.047269 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.046023 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2n95\" (UniqueName: \"kubernetes.io/projected/d931dc13-4399-48dc-80e0-b0b304c5dda0-kube-api-access-s2n95\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:33:00.047269 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.046087 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc10f92e-e902-48c4-9cd5-4f768c4611be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-r5fd4\" (UID: \"dc10f92e-e902-48c4-9cd5-4f768c4611be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4" Nov 03 20:33:00.047911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.047885 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h\" (UID: \"2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h" Nov 03 20:33:00.049075 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.049042 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-workload-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:33:00.049976 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.049755 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d931dc13-4399-48dc-80e0-b0b304c5dda0-istiod-ca-cert\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:33:00.051382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.050916 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8\" (UID: \"ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8" Nov 03 20:33:00.051382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.050979 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8760fa0e-a78b-4db8-b71d-bf4086ddbfbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls\" (UID: \"8760fa0e-a78b-4db8-b71d-bf4086ddbfbd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls" Nov 03 20:33:00.051382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.051262 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-data\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:33:00.052534 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.052145 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-credential-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:33:00.052534 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.052317 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-workload-certs\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:33:00.064531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.064328 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-pbgwk"] Nov 03 20:33:00.067705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.067518 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-podinfo\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:33:00.079161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.079131 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-envoy\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:33:00.085308 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.082074 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s7n4\" (UniqueName: \"kubernetes.io/projected/ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988-kube-api-access-4s7n4\") pod \"payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8\" (UID: \"ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8" Nov 03 20:33:00.085446 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.083883 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8gkn\" (UniqueName: \"kubernetes.io/projected/8760fa0e-a78b-4db8-b71d-bf4086ddbfbd-kube-api-access-w8gkn\") pod \"payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls\" (UID: \"8760fa0e-a78b-4db8-b71d-bf4086ddbfbd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls" Nov 03 20:33:00.085576 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.084759 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-fsbsp_1bcd4690-4753-4246-abb4-8668a0a56231/volume-data-source-validator/1.log" Nov 03 20:33:00.085850 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.085060 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-token\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:33:00.085975 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.085928 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2n95\" (UniqueName: \"kubernetes.io/projected/d931dc13-4399-48dc-80e0-b0b304c5dda0-kube-api-access-s2n95\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:33:00.086274 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.086062 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fsbsp" event={"ID":"1bcd4690-4753-4246-abb4-8668a0a56231","Type":"ContainerDied","Data":"3c720fc599e07edb9f53c3258393229ef983d842fab015bf43f5f59b3342d40f"} Nov 03 20:33:00.086274 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.086120 5059 scope.go:122] "RemoveContainer" containerID="34fbf9edd9a9b6fb620b51e1b7b72b6469ff59a49c409953c3ad6b93fa26cf2f" Nov 03 20:33:00.086274 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.086031 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fsbsp" Nov 03 20:33:00.092330 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.092302 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-pbgwk"] Nov 03 20:33:00.099978 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.099117 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nskbm\" (UniqueName: \"kubernetes.io/projected/2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1-kube-api-access-nskbm\") pod \"payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h\" (UID: \"2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h" Nov 03 20:33:00.101983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.101953 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-fzr95"] Nov 03 20:33:00.139545 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.139300 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:33:00.150438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.147802 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc10f92e-e902-48c4-9cd5-4f768c4611be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-r5fd4\" (UID: \"dc10f92e-e902-48c4-9cd5-4f768c4611be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4" Nov 03 20:33:00.150438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.148069 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc10f92e-e902-48c4-9cd5-4f768c4611be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-r5fd4\" (UID: \"dc10f92e-e902-48c4-9cd5-4f768c4611be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4" Nov 03 20:33:00.150438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.148164 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79fs\" (UniqueName: \"kubernetes.io/projected/dc10f92e-e902-48c4-9cd5-4f768c4611be-kube-api-access-z79fs\") pod \"maas-api-revoke-sub-abad4031-r5fd4\" (UID: \"dc10f92e-e902-48c4-9cd5-4f768c4611be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4" Nov 03 20:33:00.150438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.149232 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc10f92e-e902-48c4-9cd5-4f768c4611be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-r5fd4\" (UID: \"dc10f92e-e902-48c4-9cd5-4f768c4611be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4" Nov 03 20:33:00.150708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.150516 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8" Nov 03 20:33:00.150943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.150918 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-fzr95"] Nov 03 20:33:00.151034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.151025 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-fzr95" Nov 03 20:33:00.155205 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.154630 5059 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:33:00.155205 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.154695 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" podUID="70d45aa9-bfa8-43c6-8674-6f8f14a3097e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:33:00.155205 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.154820 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc10f92e-e902-48c4-9cd5-4f768c4611be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-r5fd4\" (UID: \"dc10f92e-e902-48c4-9cd5-4f768c4611be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4" Nov 03 20:33:00.155205 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.155006 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h" Nov 03 20:33:00.162520 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.162492 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84" event={"ID":"4a35329c-82fb-479f-abca-bff9ed082558","Type":"ContainerDied","Data":"43465fcec0008095aec34e37b1b670001c5a5d5b990377e6dd6dce8357fd8e9d"} Nov 03 20:33:00.162993 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.162677 5059 scope.go:122] "RemoveContainer" containerID="2e57d5a40799f051dc61748015ccfef4a18d3c2d9058dcbee05723c8f222c743" Nov 03 20:33:00.163260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.163240 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84" Nov 03 20:33:00.167225 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.166865 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 20:33:00.167225 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.166931 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-74fgb\"" Nov 03 20:33:00.167225 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.166990 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 03 20:33:00.167225 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.167045 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 03 20:33:00.167225 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.167064 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 20:33:00.172957 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.170868 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j"] Nov 03 20:33:00.176629 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.175422 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hmc5z_f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/console-operator/3.log" Nov 03 20:33:00.176629 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.175574 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" Nov 03 20:33:00.179918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.179166 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-hmc5z" event={"ID":"f8b140c9-9e09-4cbd-847b-82ee1e3be6ac","Type":"ContainerDied","Data":"38325d12703d972e1ae36cba1ea5a54367988c1d6b25ab7c5727d1b58e368726"} Nov 03 20:33:00.179918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.179206 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2"] Nov 03 20:33:00.179918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.179288 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.184095 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.184043 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" event={"ID":"6785e24f-6fb3-43ab-a469-84d30d94b2f0","Type":"ContainerDied","Data":"0d86ec4f25a9afa09a34a67ce566ad22504c2a23ba3bf73a57cc65382e5607c3"} Nov 03 20:33:00.185528 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.185481 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-q5tr2"] Nov 03 20:33:00.186044 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.185773 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5" Nov 03 20:33:00.190533 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.190451 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls" Nov 03 20:33:00.205681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.203263 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j"] Nov 03 20:33:00.227965 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.227899 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z79fs\" (UniqueName: \"kubernetes.io/projected/dc10f92e-e902-48c4-9cd5-4f768c4611be-kube-api-access-z79fs\") pod \"maas-api-revoke-sub-abad4031-r5fd4\" (UID: \"dc10f92e-e902-48c4-9cd5-4f768c4611be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4" Nov 03 20:33:00.252237 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.252164 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66b59d468d-5bfd2" Nov 03 20:33:00.252350 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.252298 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66b59d468d-5bfd2" event={"ID":"c7fce352-1850-4048-be2e-fa458fd4d2f3","Type":"ContainerDied","Data":"54963a6f8070c198ca5235d825b024fa48be551889691ecb18a8d272bc1c0398"} Nov 03 20:33:00.261185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.261151 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn58z\" (UniqueName: \"kubernetes.io/projected/191c7922-07dd-4c9c-89d2-d804f638854d-kube-api-access-tn58z\") pod \"odh-model-controller-769875db56-fzr95\" (UID: \"191c7922-07dd-4c9c-89d2-d804f638854d\") " pod="opendatahub/odh-model-controller-769875db56-fzr95" Nov 03 20:33:00.261279 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.261243 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.261334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.261297 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/191c7922-07dd-4c9c-89d2-d804f638854d-metrics-certs\") pod \"odh-model-controller-769875db56-fzr95\" (UID: \"191c7922-07dd-4c9c-89d2-d804f638854d\") " pod="opendatahub/odh-model-controller-769875db56-fzr95" Nov 03 20:33:00.261383 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.261366 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.261433 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.261394 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/191c7922-07dd-4c9c-89d2-d804f638854d-cert\") pod \"odh-model-controller-769875db56-fzr95\" (UID: \"191c7922-07dd-4c9c-89d2-d804f638854d\") " pod="opendatahub/odh-model-controller-769875db56-fzr95" Nov 03 20:33:00.268280 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.268251 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.268461 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.268443 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.269382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.268575 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.269382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.268695 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.269382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.268724 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.269382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.268797 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.269899 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.269872 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r4ct\" (UniqueName: \"kubernetes.io/projected/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-kube-api-access-4r4ct\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.276356 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.276236 5059 scope.go:122] "RemoveContainer" containerID="f3a682944a052f5773970fd4a1c1414d244464e281630d9cbe3a0f7623116968" Nov 03 20:33:00.363788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.363757 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-lz2v2"] Nov 03 20:33:00.382212 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.382124 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r4ct\" (UniqueName: \"kubernetes.io/projected/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-kube-api-access-4r4ct\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.382212 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.382207 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn58z\" (UniqueName: \"kubernetes.io/projected/191c7922-07dd-4c9c-89d2-d804f638854d-kube-api-access-tn58z\") pod \"odh-model-controller-769875db56-fzr95\" (UID: \"191c7922-07dd-4c9c-89d2-d804f638854d\") " pod="opendatahub/odh-model-controller-769875db56-fzr95" Nov 03 20:33:00.382403 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.382284 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.382403 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.382336 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/191c7922-07dd-4c9c-89d2-d804f638854d-metrics-certs\") pod \"odh-model-controller-769875db56-fzr95\" (UID: \"191c7922-07dd-4c9c-89d2-d804f638854d\") " pod="opendatahub/odh-model-controller-769875db56-fzr95" Nov 03 20:33:00.382569 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.382414 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.382569 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.382441 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/191c7922-07dd-4c9c-89d2-d804f638854d-cert\") pod \"odh-model-controller-769875db56-fzr95\" (UID: \"191c7922-07dd-4c9c-89d2-d804f638854d\") " pod="opendatahub/odh-model-controller-769875db56-fzr95" Nov 03 20:33:00.382684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.382598 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.382731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.382681 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.382731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.382706 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.382821 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.382796 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.382863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.382821 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.382904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.382888 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.387844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.385769 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.387844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.386027 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.387844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.386526 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.387844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.386891 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.387844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.387350 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.395106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.393848 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-d7wsq"] Nov 03 20:33:00.395106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.393912 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.395106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.393951 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lz2v2" Nov 03 20:33:00.412413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.402108 5059 scope.go:122] "RemoveContainer" containerID="95024980a099f9ccdf78362bbb5aa84a698eaab1e6f9c5a70a9520e6a4b6e491" Nov 03 20:33:00.412413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.402390 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-lz2v2"] Nov 03 20:33:00.412413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.406397 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t" event={"ID":"4b25c2d0-60bb-4df1-8193-5f7f853c0a73","Type":"ContainerStarted","Data":"bd910959fcbe106c88812aae26cf4e57d7855ae92a08f802492c9b663a34b390"} Nov 03 20:33:00.412413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.406449 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t" event={"ID":"4b25c2d0-60bb-4df1-8193-5f7f853c0a73","Type":"ContainerStarted","Data":"d1aa1348af3cbccddd516d7cce315ccc0828f8681c485afd95e08448f2ce5bae"} Nov 03 20:33:00.412413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.406747 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/191c7922-07dd-4c9c-89d2-d804f638854d-cert\") pod \"odh-model-controller-769875db56-fzr95\" (UID: \"191c7922-07dd-4c9c-89d2-d804f638854d\") " pod="opendatahub/odh-model-controller-769875db56-fzr95" Nov 03 20:33:00.412413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.412227 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4" Nov 03 20:33:00.413214 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.413179 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-d7wsq"] Nov 03 20:33:00.413453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.413435 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t" Nov 03 20:33:00.414627 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.414239 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 20:33:00.414627 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.414277 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-w84zh\"" Nov 03 20:33:00.465529 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.465280 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r4ct\" (UniqueName: \"kubernetes.io/projected/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-kube-api-access-4r4ct\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.467597 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.467573 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" Nov 03 20:33:00.468869 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.468577 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7" event={"ID":"1072628f-f513-4b4e-8954-afee5a1aa733","Type":"ContainerDied","Data":"2a05791624cadc6e4f146b5ee449504344b96fd16c4bce70bef096744cab69b4"} Nov 03 20:33:00.469086 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.469069 5059 scope.go:122] "RemoveContainer" containerID="3eedb7b821c263449d6f28d722073293785400637faa42e6848b5007c54e889e" Nov 03 20:33:00.475078 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.474979 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/191c7922-07dd-4c9c-89d2-d804f638854d-metrics-certs\") pod \"odh-model-controller-769875db56-fzr95\" (UID: \"191c7922-07dd-4c9c-89d2-d804f638854d\") " pod="opendatahub/odh-model-controller-769875db56-fzr95" Nov 03 20:33:00.481897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.481088 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.481897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.481707 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn58z\" (UniqueName: \"kubernetes.io/projected/191c7922-07dd-4c9c-89d2-d804f638854d-kube-api-access-tn58z\") pod \"odh-model-controller-769875db56-fzr95\" (UID: \"191c7922-07dd-4c9c-89d2-d804f638854d\") " pod="opendatahub/odh-model-controller-769875db56-fzr95" Nov 03 20:33:00.483084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.482886 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.485451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.485381 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b4d89c5d-1730-4077-a442-d3924153133f-data\") pod \"postgres-7cdbf6dc44-lz2v2\" (UID: \"b4d89c5d-1730-4077-a442-d3924153133f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lz2v2" Nov 03 20:33:00.486249 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.485643 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhwmd\" (UniqueName: \"kubernetes.io/projected/b4d89c5d-1730-4077-a442-d3924153133f-kube-api-access-jhwmd\") pod \"postgres-7cdbf6dc44-lz2v2\" (UID: \"b4d89c5d-1730-4077-a442-d3924153133f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lz2v2" Nov 03 20:33:00.518130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.517217 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mm85p" event={"ID":"532de14d-f2dd-4c66-ba6f-8b103f7c9375","Type":"ContainerDied","Data":"059483529a6e51a7600c519628a33dc2f35af25b5257d5868669225c1c3bef09"} Nov 03 20:33:00.518130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.517445 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mm85p" Nov 03 20:33:00.525189 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.525166 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6k9n9" event={"ID":"f485893c-6f14-449b-b837-8ac181315aeb","Type":"ContainerDied","Data":"57dc8ba289eb00c068e01a84058a62327d180d9194361eb4011c73b22e8d29f2"} Nov 03 20:33:00.526956 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.525482 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6k9n9" Nov 03 20:33:00.535279 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.535209 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4"] Nov 03 20:33:00.538596 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.536923 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" event={"ID":"270c522c-ef72-46a7-9e65-3be2ebdbcfa6","Type":"ContainerDied","Data":"20bf4fd0a808dbaefe1d4e12aae0124a790da36e28b26676cb2056a8cd542a42"} Nov 03 20:33:00.538596 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.537112 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw" Nov 03 20:33:00.544048 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.544026 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5" event={"ID":"6b9428a2-ab50-4e44-a725-e0da980bdb93","Type":"ContainerStarted","Data":"525f6d3424c4cc255ff7e85b227e91e40f3d49c08a0875d0fdde86eb52893da8"} Nov 03 20:33:00.553722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.553693 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:00.607901 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.588135 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b4d89c5d-1730-4077-a442-d3924153133f-data\") pod \"postgres-7cdbf6dc44-lz2v2\" (UID: \"b4d89c5d-1730-4077-a442-d3924153133f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lz2v2" Nov 03 20:33:00.607901 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.588392 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhwmd\" (UniqueName: \"kubernetes.io/projected/b4d89c5d-1730-4077-a442-d3924153133f-kube-api-access-jhwmd\") pod \"postgres-7cdbf6dc44-lz2v2\" (UID: \"b4d89c5d-1730-4077-a442-d3924153133f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lz2v2" Nov 03 20:33:00.607901 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.597127 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4"] Nov 03 20:33:00.636146 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.636079 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b4d89c5d-1730-4077-a442-d3924153133f-data\") pod \"postgres-7cdbf6dc44-lz2v2\" (UID: \"b4d89c5d-1730-4077-a442-d3924153133f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lz2v2" Nov 03 20:33:00.641613 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.640706 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" Nov 03 20:33:00.657048 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.656779 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l"] Nov 03 20:33:00.682253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.682194 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4"] Nov 03 20:33:00.724073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.690844 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.724073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.695045 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d098ef77-addb-4077-b6d4-f0ae477409b8-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wz2w4\" (UID: \"d098ef77-addb-4077-b6d4-f0ae477409b8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" Nov 03 20:33:00.724073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.695097 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d098ef77-addb-4077-b6d4-f0ae477409b8-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wz2w4\" (UID: \"d098ef77-addb-4077-b6d4-f0ae477409b8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" Nov 03 20:33:00.724073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.695128 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d098ef77-addb-4077-b6d4-f0ae477409b8-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wz2w4\" (UID: \"d098ef77-addb-4077-b6d4-f0ae477409b8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" Nov 03 20:33:00.724073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.695170 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d098ef77-addb-4077-b6d4-f0ae477409b8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wz2w4\" (UID: \"d098ef77-addb-4077-b6d4-f0ae477409b8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" Nov 03 20:33:00.724073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.695203 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22tms\" (UniqueName: \"kubernetes.io/projected/d098ef77-addb-4077-b6d4-f0ae477409b8-kube-api-access-22tms\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wz2w4\" (UID: \"d098ef77-addb-4077-b6d4-f0ae477409b8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" Nov 03 20:33:00.724073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.695519 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d098ef77-addb-4077-b6d4-f0ae477409b8-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wz2w4\" (UID: \"d098ef77-addb-4077-b6d4-f0ae477409b8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" Nov 03 20:33:00.724073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.697486 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz"] Nov 03 20:33:00.724073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.699782 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.724073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.700578 5059 scope.go:122] "RemoveContainer" containerID="8121a3441525651ca2c833f0f9236c162d2c111dd09ec30bb62b251a8accdd62" Nov 03 20:33:00.787920 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.787292 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.787920 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.787716 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-fzr95" Nov 03 20:33:00.792731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.791977 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9"] Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.815318 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-5fffc97bdf-brvgz_7b8992fc-fba6-4135-90f8-fb259e765dd5/service-ca-controller/1.log" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.815374 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" event={"ID":"7b8992fc-fba6-4135-90f8-fb259e765dd5","Type":"ContainerDied","Data":"7f52997d73f0c127088919f5e5f96fae8499e63d1b444d7b1b7b4b0d1a3ad067"} Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.815480 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.815545 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-brvgz" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.816597 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-podinfo\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.816650 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.817465 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-data\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.817502 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.817552 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.817579 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.821867 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.821983 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.822015 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.822044 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq8k2\" (UniqueName: \"kubernetes.io/projected/6752fea6-4adc-47e6-858c-101dcb6d6a84-kube-api-access-qq8k2\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.822097 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.822149 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d098ef77-addb-4077-b6d4-f0ae477409b8-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wz2w4\" (UID: \"d098ef77-addb-4077-b6d4-f0ae477409b8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.822189 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-workload-socket\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.822219 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d098ef77-addb-4077-b6d4-f0ae477409b8-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wz2w4\" (UID: \"d098ef77-addb-4077-b6d4-f0ae477409b8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.822248 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d098ef77-addb-4077-b6d4-f0ae477409b8-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wz2w4\" (UID: \"d098ef77-addb-4077-b6d4-f0ae477409b8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.822291 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d098ef77-addb-4077-b6d4-f0ae477409b8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wz2w4\" (UID: \"d098ef77-addb-4077-b6d4-f0ae477409b8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.822321 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22tms\" (UniqueName: \"kubernetes.io/projected/d098ef77-addb-4077-b6d4-f0ae477409b8-kube-api-access-22tms\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wz2w4\" (UID: \"d098ef77-addb-4077-b6d4-f0ae477409b8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.822366 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.822430 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-token\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.822477 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb4f711-0bcc-46f4-86b9-795c66613832-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.822567 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b5t4\" (UniqueName: \"kubernetes.io/projected/4b43d3ba-a346-45d6-8077-8a52b42de081-kube-api-access-5b5t4\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.822615 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.822653 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-workload-certs\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.822755 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b43d3ba-a346-45d6-8077-8a52b42de081-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.822803 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.822845 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.822881 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.822954 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d098ef77-addb-4077-b6d4-f0ae477409b8-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wz2w4\" (UID: \"d098ef77-addb-4077-b6d4-f0ae477409b8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.822987 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-credential-socket\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.823013 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6752fea6-4adc-47e6-858c-101dcb6d6a84-istiod-ca-cert\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.823040 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.837134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.823076 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n26xp\" (UniqueName: \"kubernetes.io/projected/9bb4f711-0bcc-46f4-86b9-795c66613832-kube-api-access-n26xp\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.838984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.823124 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-envoy\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.838984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.823591 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d098ef77-addb-4077-b6d4-f0ae477409b8-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wz2w4\" (UID: \"d098ef77-addb-4077-b6d4-f0ae477409b8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" Nov 03 20:33:00.838984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.823841 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d098ef77-addb-4077-b6d4-f0ae477409b8-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wz2w4\" (UID: \"d098ef77-addb-4077-b6d4-f0ae477409b8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" Nov 03 20:33:00.838984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.824055 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d098ef77-addb-4077-b6d4-f0ae477409b8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wz2w4\" (UID: \"d098ef77-addb-4077-b6d4-f0ae477409b8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" Nov 03 20:33:00.838984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.829712 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d098ef77-addb-4077-b6d4-f0ae477409b8-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wz2w4\" (UID: \"d098ef77-addb-4077-b6d4-f0ae477409b8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" Nov 03 20:33:00.838984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.830874 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-wh95x_492c7b4e-84c0-49a7-b12f-153a021f971b/kube-storage-version-migrator-operator/2.log" Nov 03 20:33:00.838984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.830928 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" event={"ID":"492c7b4e-84c0-49a7-b12f-153a021f971b","Type":"ContainerDied","Data":"1af68c1e1deeba4a098d6b6a340842f1096a672a7c9f458990ee87e7fd746ec3"} Nov 03 20:33:00.838984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.831054 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x" Nov 03 20:33:00.838984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.832274 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm"] Nov 03 20:33:00.838984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.836578 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d098ef77-addb-4077-b6d4-f0ae477409b8-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wz2w4\" (UID: \"d098ef77-addb-4077-b6d4-f0ae477409b8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" Nov 03 20:33:00.839424 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.839279 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhwmd\" (UniqueName: \"kubernetes.io/projected/b4d89c5d-1730-4077-a442-d3924153133f-kube-api-access-jhwmd\") pod \"postgres-7cdbf6dc44-lz2v2\" (UID: \"b4d89c5d-1730-4077-a442-d3924153133f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lz2v2" Nov 03 20:33:00.839983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.839781 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc"] Nov 03 20:33:00.839983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.839868 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm" Nov 03 20:33:00.841267 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.841221 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22tms\" (UniqueName: \"kubernetes.io/projected/d098ef77-addb-4077-b6d4-f0ae477409b8-kube-api-access-22tms\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-wz2w4\" (UID: \"d098ef77-addb-4077-b6d4-f0ae477409b8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" Nov 03 20:33:00.847277 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.846824 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf9vc"] Nov 03 20:33:00.852308 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.852289 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" Nov 03 20:33:00.853951 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:33:00.853922 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd931dc13_4399_48dc_80e0_b0b304c5dda0.slice/crio-31b6388801c2c480e702c4985e3f6638dbac4103cebd0a5bf86216cac78f3f2f WatchSource:0}: Error finding container 31b6388801c2c480e702c4985e3f6638dbac4103cebd0a5bf86216cac78f3f2f: Status 404 returned error can't find the container with id 31b6388801c2c480e702c4985e3f6638dbac4103cebd0a5bf86216cac78f3f2f Nov 03 20:33:00.854544 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.854521 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l"] Nov 03 20:33:00.916718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.916615 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4"] Nov 03 20:33:00.926321 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.925615 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.926321 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.925935 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.926321 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.925978 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.926321 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.926008 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq8k2\" (UniqueName: \"kubernetes.io/projected/6752fea6-4adc-47e6-858c-101dcb6d6a84-kube-api-access-qq8k2\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.926321 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.926068 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.926321 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.926143 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-workload-socket\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.926321 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.926248 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.927841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.926769 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-token\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.927841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.926831 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb4f711-0bcc-46f4-86b9-795c66613832-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.927841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.926924 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b5t4\" (UniqueName: \"kubernetes.io/projected/4b43d3ba-a346-45d6-8077-8a52b42de081-kube-api-access-5b5t4\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.927841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.926981 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.927841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.927021 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-workload-certs\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.927841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.927086 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b43d3ba-a346-45d6-8077-8a52b42de081-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.927841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.927133 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.927841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.927177 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.927841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.927219 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.927841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.927304 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-credential-socket\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.927841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.927330 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6752fea6-4adc-47e6-858c-101dcb6d6a84-istiod-ca-cert\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.927841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.927364 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd9d29fb-d764-4eac-a744-dbdecde12ff0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9\" (UID: \"fd9d29fb-d764-4eac-a744-dbdecde12ff0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9" Nov 03 20:33:00.927841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.927396 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kvn9\" (UniqueName: \"kubernetes.io/projected/7dc48f99-393e-4af4-b942-1ed63909b588-kube-api-access-5kvn9\") pod \"payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm\" (UID: \"7dc48f99-393e-4af4-b942-1ed63909b588\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm" Nov 03 20:33:00.927841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.927427 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.927841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.927468 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n26xp\" (UniqueName: \"kubernetes.io/projected/9bb4f711-0bcc-46f4-86b9-795c66613832-kube-api-access-n26xp\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.927841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.927521 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-envoy\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.927841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.927566 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-podinfo\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.927841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.927626 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.933093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.929509 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-workload-certs\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.933093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.931165 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb"] Nov 03 20:33:00.933093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.927656 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-data\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.934962 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.933718 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.934962 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.933782 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6752fea6-4adc-47e6-858c-101dcb6d6a84-istiod-ca-cert\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.934962 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.933961 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-workload-socket\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.934962 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.934213 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.935249 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b43d3ba-a346-45d6-8077-8a52b42de081-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.935416 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.935908 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-credential-socket\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.936189 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc48f99-393e-4af4-b942-1ed63909b588-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm\" (UID: \"7dc48f99-393e-4af4-b942-1ed63909b588\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm" Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.936759 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.937471 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.937487 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.937555 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.937599 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.938034 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-data\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.938725 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.938743 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.938986 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.938988 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.939579 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb4f711-0bcc-46f4-86b9-795c66613832-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.939770 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9"] Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.939812 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.939845 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccx9j\" (UniqueName: \"kubernetes.io/projected/fd9d29fb-d764-4eac-a744-dbdecde12ff0-kube-api-access-ccx9j\") pod \"payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9\" (UID: \"fd9d29fb-d764-4eac-a744-dbdecde12ff0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9" Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.939868 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb" Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.939874 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.940233 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.940482 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.940733 5059 scope.go:122] "RemoveContainer" containerID="c92f883ff6898e09b68a779d22ce3740c636fefbf8228b2a6a5e2048a78c601e" Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.941843 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-envoy\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.942281 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.944253 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.945204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.945103 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n26xp\" (UniqueName: \"kubernetes.io/projected/9bb4f711-0bcc-46f4-86b9-795c66613832-kube-api-access-n26xp\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:00.947488 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.946351 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-podinfo\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.947488 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.947374 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm"] Nov 03 20:33:00.954137 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.952586 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq8k2\" (UniqueName: \"kubernetes.io/projected/6752fea6-4adc-47e6-858c-101dcb6d6a84-kube-api-access-qq8k2\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.955963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.955856 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-token\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:00.955963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.955901 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz"] Nov 03 20:33:00.957206 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.957091 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b5t4\" (UniqueName: \"kubernetes.io/projected/4b43d3ba-a346-45d6-8077-8a52b42de081-kube-api-access-5b5t4\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.961491 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.961173 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:00.972942 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:00.961949 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:01.034031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.033879 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb"] Nov 03 20:33:01.049347 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.049297 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd9d29fb-d764-4eac-a744-dbdecde12ff0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9\" (UID: \"fd9d29fb-d764-4eac-a744-dbdecde12ff0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9" Nov 03 20:33:01.049528 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.049511 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kvn9\" (UniqueName: \"kubernetes.io/projected/7dc48f99-393e-4af4-b942-1ed63909b588-kube-api-access-5kvn9\") pod \"payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm\" (UID: \"7dc48f99-393e-4af4-b942-1ed63909b588\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm" Nov 03 20:33:01.049784 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.049768 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc48f99-393e-4af4-b942-1ed63909b588-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm\" (UID: \"7dc48f99-393e-4af4-b942-1ed63909b588\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm" Nov 03 20:33:01.049974 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.049960 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccx9j\" (UniqueName: \"kubernetes.io/projected/fd9d29fb-d764-4eac-a744-dbdecde12ff0-kube-api-access-ccx9j\") pod \"payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9\" (UID: \"fd9d29fb-d764-4eac-a744-dbdecde12ff0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9" Nov 03 20:33:01.050347 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.050329 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26576b40-fd9c-4298-9129-5b341d2c0b3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb\" (UID: \"26576b40-fd9c-4298-9129-5b341d2c0b3d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb" Nov 03 20:33:01.050608 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.050578 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htk2g\" (UniqueName: \"kubernetes.io/projected/26576b40-fd9c-4298-9129-5b341d2c0b3d-kube-api-access-htk2g\") pod \"payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb\" (UID: \"26576b40-fd9c-4298-9129-5b341d2c0b3d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb" Nov 03 20:33:01.055538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.055493 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd9d29fb-d764-4eac-a744-dbdecde12ff0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9\" (UID: \"fd9d29fb-d764-4eac-a744-dbdecde12ff0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9" Nov 03 20:33:01.062013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.058015 5059 scope.go:122] "RemoveContainer" containerID="6f53b40ec729940a8b447824fba4edc4a9b60468b059f5be6551e07af2e247ed" Nov 03 20:33:01.062013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.058172 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr"] Nov 03 20:33:01.071066 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.071037 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lz2v2" Nov 03 20:33:01.092989 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.086220 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kvn9\" (UniqueName: \"kubernetes.io/projected/7dc48f99-393e-4af4-b942-1ed63909b588-kube-api-access-5kvn9\") pod \"payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm\" (UID: \"7dc48f99-393e-4af4-b942-1ed63909b588\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm" Nov 03 20:33:01.112863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.111104 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc48f99-393e-4af4-b942-1ed63909b588-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm\" (UID: \"7dc48f99-393e-4af4-b942-1ed63909b588\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm" Nov 03 20:33:01.122904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.118766 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccx9j\" (UniqueName: \"kubernetes.io/projected/fd9d29fb-d764-4eac-a744-dbdecde12ff0-kube-api-access-ccx9j\") pod \"payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9\" (UID: \"fd9d29fb-d764-4eac-a744-dbdecde12ff0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9" Nov 03 20:33:01.123132 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.121285 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:33:01.123286 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.123260 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" podUID="9d60eb56-9852-4c25-ae50-8a932e752793" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:33:01.135143 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.135120 5059 patch_prober.go:29] interesting pod/router-default-f7b87d695-cwlfq container/router namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]backend-proxy-http ok Nov 03 20:33:01.135143 ip-10-0-132-83 kubenswrapper[5059]: [+]has-synced ok Nov 03 20:33:01.135143 ip-10-0-132-83 kubenswrapper[5059]: [-]process-running failed: reason withheld Nov 03 20:33:01.135143 ip-10-0-132-83 kubenswrapper[5059]: healthz check failed Nov 03 20:33:01.135424 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.135386 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-f7b87d695-cwlfq" podUID="603868b0-8628-4227-8471-1894e2e1de50" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Nov 03 20:33:01.138633 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.138597 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm" Nov 03 20:33:01.144919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.144891 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4"] Nov 03 20:33:01.152849 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.152211 5059 scope.go:122] "RemoveContainer" containerID="88b5b44b71c45a6eb5b61eaf4c105d6ffa0032eb4646d4eb2fa600f2d3805504" Nov 03 20:33:01.158216 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.158187 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26576b40-fd9c-4298-9129-5b341d2c0b3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb\" (UID: \"26576b40-fd9c-4298-9129-5b341d2c0b3d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb" Nov 03 20:33:01.158341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.158319 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htk2g\" (UniqueName: \"kubernetes.io/projected/26576b40-fd9c-4298-9129-5b341d2c0b3d-kube-api-access-htk2g\") pod \"payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb\" (UID: \"26576b40-fd9c-4298-9129-5b341d2c0b3d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb" Nov 03 20:33:01.160737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.160703 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26576b40-fd9c-4298-9129-5b341d2c0b3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb\" (UID: \"26576b40-fd9c-4298-9129-5b341d2c0b3d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb" Nov 03 20:33:01.222040 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.208769 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r"] Nov 03 20:33:01.222040 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.212015 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:01.222040 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.213141 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:01.234618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.234389 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:01.259709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.251391 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9" Nov 03 20:33:01.259709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.252056 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" Nov 03 20:33:01.292197 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.260753 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htk2g\" (UniqueName: \"kubernetes.io/projected/26576b40-fd9c-4298-9129-5b341d2c0b3d-kube-api-access-htk2g\") pod \"payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb\" (UID: \"26576b40-fd9c-4298-9129-5b341d2c0b3d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb" Nov 03 20:33:01.346688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.344165 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs"] Nov 03 20:33:01.350710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.350338 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.352290 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.351121 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz"] Nov 03 20:33:01.358742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.353347 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44" podUID="26375297-81f4-427f-94c3-d56f7a6a11fb" containerName="payload-pre-processing" containerID="cri-o://93172d79a58c2d8458ec95cdf845557fbb10e251f6cf98dda510ae64f827325b" gracePeriod=25 Nov 03 20:33:01.358742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.353520 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p" podUID="ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51" containerName="payload-processing" containerID="cri-o://04da4e840db9b36ab74f969a24bc9303a68de3283b026a9e0deacd6cef05ae43" gracePeriod=25 Nov 03 20:33:01.361049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.360759 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2" podUID="cfcb49c5-4381-411e-9d64-d9dcd6181239" containerName="payload-pre-processing" containerID="cri-o://06775cb2b6311ab6024c5cf0da37eef3631ba5e53f2a44ad31bd33c7d778d26f" gracePeriod=25 Nov 03 20:33:01.361049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.360901 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj" podUID="06b78cd0-94df-40f9-a1ed-87c1751675d3" containerName="payload-pre-processing" containerID="cri-o://e188f63192d3371f9c88f1fcfd31fbc6a5bb4e3b477faa591137913f4bacb386" gracePeriod=25 Nov 03 20:33:01.361049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.360978 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt" podUID="c92b1be0-0890-4db6-ac75-968b6e5a3434" containerName="payload-processing" containerID="cri-o://ddfb36a61a4890d032f40a96fa7dc33ada9d7745461eee81dc563f215eda9eae" gracePeriod=25 Nov 03 20:33:01.361049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.361048 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7" podUID="2c363281-0226-420f-a963-5922d5ca609e" containerName="payload-pre-processing" containerID="cri-o://a1545bd75166fa5a492a8e6723253de1761cd417b14f627fbdab40e21ab83652" gracePeriod=25 Nov 03 20:33:01.361280 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.361115 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn" podUID="25ceab3d-6344-4512-a471-d91aec0cb48c" containerName="payload-processing" containerID="cri-o://b13a3b8cfa23ddc9815549092296b81b15bde977434b7a3b30d1385e768e8fb0" gracePeriod=25 Nov 03 20:33:01.361280 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.361187 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9" podUID="890f130b-cdeb-43da-acd4-1654673271cc" containerName="payload-processing" containerID="cri-o://80563e166a6378b34d09a54f08d0ed8f1e665107abe95933f0fa76b31fc439e7" gracePeriod=25 Nov 03 20:33:01.361280 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.361248 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps" podUID="2faa14fa-ec86-4633-a968-c3f761d13443" containerName="payload-processing" containerID="cri-o://d610230fd1e961d1b794119c73b53766e612967c8196f325df7359012d12af34" gracePeriod=25 Nov 03 20:33:01.368084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.366088 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst"] Nov 03 20:33:01.373676 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.373634 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8" podUID="705dea2b-ad85-4b1c-a3c6-a9f705ac9039" containerName="payload-processing" containerID="cri-o://c1c608afee5d8326b20645c4ef715824387423915b54182a5c217c34ffe83cf2" gracePeriod=25 Nov 03 20:33:01.373945 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.373926 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.374192 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.374177 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst" Nov 03 20:33:01.376098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.376041 5059 scope.go:122] "RemoveContainer" containerID="79f4410301f54a1a4f3ead6bc5461974ff7f958c6320d7f9399fbe12307acdfb" Nov 03 20:33:01.379653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.377695 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn"] Nov 03 20:33:01.379653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.379052 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-98529d-dockercfg-rgksk\"" Nov 03 20:33:01.387865 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.387847 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.402155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.402133 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn"] Nov 03 20:33:01.403793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.403771 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb" Nov 03 20:33:01.413552 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.413527 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" podUID="0309a608-f6c4-45ca-991b-33ba73221a6c" containerName="payload-processing" containerID="cri-o://6e4d31c85e8cd1b2b8591eeb7383bfd43ae2ffe368b7b0f552663b7b668eb233" gracePeriod=25 Nov 03 20:33:01.413809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.413788 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq" podUID="0d960e4e-9539-4a53-9df7-1c520ab7d708" containerName="payload-pre-processing" containerID="cri-o://4436c68602fe345d6bb4fb9dfe6e2ac4dec763399336a2db74a54312609ee767" gracePeriod=25 Nov 03 20:33:01.419207 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.419049 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl" podUID="9c3d72b6-73c4-402f-8308-ee5d87a9dd83" containerName="payload-pre-processing" containerID="cri-o://3d68de58f4e860642d4d4255a14fe00815183f88da48e5f851f0a0c70e971419" gracePeriod=25 Nov 03 20:33:01.425509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.425479 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b2ec36-7e99-46e0-9671-1c500c402e45-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r\" (UID: \"60b2ec36-7e99-46e0-9671-1c500c402e45\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" Nov 03 20:33:01.425607 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.425588 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b2ec36-7e99-46e0-9671-1c500c402e45-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r\" (UID: \"60b2ec36-7e99-46e0-9671-1c500c402e45\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" Nov 03 20:33:01.425675 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.425624 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgpr7\" (UniqueName: \"kubernetes.io/projected/60b2ec36-7e99-46e0-9671-1c500c402e45-kube-api-access-lgpr7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r\" (UID: \"60b2ec36-7e99-46e0-9671-1c500c402e45\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" Nov 03 20:33:01.425756 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.425738 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b2ec36-7e99-46e0-9671-1c500c402e45-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r\" (UID: \"60b2ec36-7e99-46e0-9671-1c500c402e45\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" Nov 03 20:33:01.425840 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.425824 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b2ec36-7e99-46e0-9671-1c500c402e45-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r\" (UID: \"60b2ec36-7e99-46e0-9671-1c500c402e45\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" Nov 03 20:33:01.425889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.425868 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b2ec36-7e99-46e0-9671-1c500c402e45-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r\" (UID: \"60b2ec36-7e99-46e0-9671-1c500c402e45\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" Nov 03 20:33:01.427286 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:01.427258 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d45aa9_bfa8_43c6_8674_6f8f14a3097e.slice/crio-6a213e69e74bbe139a2688d93452e7d59e19f98c3c9a8e404648454c84993ae7.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:33:01.432473 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:33:01.429425 5059 logging.go:55] [core] [Channel #6805 SubChannel #6806]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.70:50052", ServerName: "10.132.0.70:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.70:50052: connect: connection refused" Nov 03 20:33:01.433799 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.433782 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn" Nov 03 20:33:01.529271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.528369 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-podinfo\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.529271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.528429 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-token\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.529271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.528466 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77396fff-1778-4f09-9e81-05e715fc015e-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.529271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.528491 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3627ae-66c6-44a4-aa60-480bd54008f0-istiod-ca-cert\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.529271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.528576 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77396fff-1778-4f09-9e81-05e715fc015e-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.529271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.528617 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istiod-ca-cert\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.529271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.528697 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75tbk\" (UniqueName: \"kubernetes.io/projected/977cec40-92df-410a-a3b6-68520b98aaf2-kube-api-access-75tbk\") pod \"payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst\" (UID: \"977cec40-92df-410a-a3b6-68520b98aaf2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst" Nov 03 20:33:01.529271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.528738 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4698\" (UniqueName: \"kubernetes.io/projected/cca72a7d-a69d-4e05-ac4d-da20e15a0243-kube-api-access-r4698\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.529271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.528772 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77396fff-1778-4f09-9e81-05e715fc015e-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.529271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.528811 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-workload-certs\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.529271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.528914 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-data\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.529271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.528944 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/977cec40-92df-410a-a3b6-68520b98aaf2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst\" (UID: \"977cec40-92df-410a-a3b6-68520b98aaf2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst" Nov 03 20:33:01.529271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.528986 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-podinfo\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.529271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.529006 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77396fff-1778-4f09-9e81-05e715fc015e-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.529271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.529042 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-credential-socket\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.529271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.529096 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b2ec36-7e99-46e0-9671-1c500c402e45-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r\" (UID: \"60b2ec36-7e99-46e0-9671-1c500c402e45\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" Nov 03 20:33:01.529271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.529181 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-credential-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.529271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.529232 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-envoy\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.529271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.529254 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-workload-certs\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.530320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.529299 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b2ec36-7e99-46e0-9671-1c500c402e45-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r\" (UID: \"60b2ec36-7e99-46e0-9671-1c500c402e45\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" Nov 03 20:33:01.530320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.529325 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgpr7\" (UniqueName: \"kubernetes.io/projected/60b2ec36-7e99-46e0-9671-1c500c402e45-kube-api-access-lgpr7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r\" (UID: \"60b2ec36-7e99-46e0-9671-1c500c402e45\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" Nov 03 20:33:01.530320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.529387 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-data\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.530320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.529452 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77396fff-1778-4f09-9e81-05e715fc015e-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.530320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.529486 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-token\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.530320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.529553 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b2ec36-7e99-46e0-9671-1c500c402e45-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r\" (UID: \"60b2ec36-7e99-46e0-9671-1c500c402e45\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" Nov 03 20:33:01.530320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.529618 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwqd5\" (UniqueName: \"kubernetes.io/projected/77396fff-1778-4f09-9e81-05e715fc015e-kube-api-access-bwqd5\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.530320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.529672 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77396fff-1778-4f09-9e81-05e715fc015e-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.530320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.529725 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b2ec36-7e99-46e0-9671-1c500c402e45-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r\" (UID: \"60b2ec36-7e99-46e0-9671-1c500c402e45\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" Nov 03 20:33:01.530320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.529765 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-workload-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.530320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.529826 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b2ec36-7e99-46e0-9671-1c500c402e45-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r\" (UID: \"60b2ec36-7e99-46e0-9671-1c500c402e45\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" Nov 03 20:33:01.530320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.529873 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5bf8\" (UniqueName: \"kubernetes.io/projected/4b3627ae-66c6-44a4-aa60-480bd54008f0-kube-api-access-s5bf8\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.530320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.530173 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-envoy\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.530320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.530228 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77396fff-1778-4f09-9e81-05e715fc015e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.530320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.530272 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77396fff-1778-4f09-9e81-05e715fc015e-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.530320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.530312 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-workload-socket\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.531133 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.531035 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b2ec36-7e99-46e0-9671-1c500c402e45-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r\" (UID: \"60b2ec36-7e99-46e0-9671-1c500c402e45\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" Nov 03 20:33:01.531422 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.531038 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b2ec36-7e99-46e0-9671-1c500c402e45-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r\" (UID: \"60b2ec36-7e99-46e0-9671-1c500c402e45\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" Nov 03 20:33:01.532016 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.531996 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b2ec36-7e99-46e0-9671-1c500c402e45-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r\" (UID: \"60b2ec36-7e99-46e0-9671-1c500c402e45\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" Nov 03 20:33:01.533722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.533635 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst"] Nov 03 20:33:01.535301 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.535274 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b2ec36-7e99-46e0-9671-1c500c402e45-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r\" (UID: \"60b2ec36-7e99-46e0-9671-1c500c402e45\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" Nov 03 20:33:01.538190 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.537910 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b2ec36-7e99-46e0-9671-1c500c402e45-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r\" (UID: \"60b2ec36-7e99-46e0-9671-1c500c402e45\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" Nov 03 20:33:01.546805 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.546686 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r"] Nov 03 20:33:01.558026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.557930 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn"] Nov 03 20:33:01.566719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.566627 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn"] Nov 03 20:33:01.576791 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.576752 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz"] Nov 03 20:33:01.583222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.583159 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs"] Nov 03 20:33:01.597638 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.596381 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgpr7\" (UniqueName: \"kubernetes.io/projected/60b2ec36-7e99-46e0-9671-1c500c402e45-kube-api-access-lgpr7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r\" (UID: \"60b2ec36-7e99-46e0-9671-1c500c402e45\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.666654 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.667467 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5bf8\" (UniqueName: \"kubernetes.io/projected/4b3627ae-66c6-44a4-aa60-480bd54008f0-kube-api-access-s5bf8\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.667552 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-envoy\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.667595 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77396fff-1778-4f09-9e81-05e715fc015e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.667625 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77396fff-1778-4f09-9e81-05e715fc015e-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.667655 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-workload-socket\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.667731 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcrnp\" (UniqueName: \"kubernetes.io/projected/c9ead1ac-0ace-4761-9fde-b7f97a9dfa31-kube-api-access-wcrnp\") pod \"payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn\" (UID: \"c9ead1ac-0ace-4761-9fde-b7f97a9dfa31\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.667755 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-podinfo\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.667803 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-token\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.667824 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77396fff-1778-4f09-9e81-05e715fc015e-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.667849 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3627ae-66c6-44a4-aa60-480bd54008f0-istiod-ca-cert\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.667920 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77396fff-1778-4f09-9e81-05e715fc015e-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.667955 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istiod-ca-cert\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.668017 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75tbk\" (UniqueName: \"kubernetes.io/projected/977cec40-92df-410a-a3b6-68520b98aaf2-kube-api-access-75tbk\") pod \"payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst\" (UID: \"977cec40-92df-410a-a3b6-68520b98aaf2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.668056 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4698\" (UniqueName: \"kubernetes.io/projected/cca72a7d-a69d-4e05-ac4d-da20e15a0243-kube-api-access-r4698\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.668075 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77396fff-1778-4f09-9e81-05e715fc015e-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.668103 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-workload-certs\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.668192 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-data\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.668225 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/977cec40-92df-410a-a3b6-68520b98aaf2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst\" (UID: \"977cec40-92df-410a-a3b6-68520b98aaf2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.668257 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-podinfo\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.668277 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77396fff-1778-4f09-9e81-05e715fc015e-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.668310 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-credential-socket\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.668405 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-credential-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.668442 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-envoy\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.668463 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-workload-certs\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.668521 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-data\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.668574 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77396fff-1778-4f09-9e81-05e715fc015e-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.668606 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-token\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.668647 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9ead1ac-0ace-4761-9fde-b7f97a9dfa31-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn\" (UID: \"c9ead1ac-0ace-4761-9fde-b7f97a9dfa31\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.670456 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwqd5\" (UniqueName: \"kubernetes.io/projected/77396fff-1778-4f09-9e81-05e715fc015e-kube-api-access-bwqd5\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.670498 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77396fff-1778-4f09-9e81-05e715fc015e-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.670550 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-workload-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.671100 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-workload-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.695022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.673108 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-workload-certs\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.711702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.673776 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77396fff-1778-4f09-9e81-05e715fc015e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.711702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.674022 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77396fff-1778-4f09-9e81-05e715fc015e-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.711702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.674264 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-workload-socket\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.711702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.688601 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-podinfo\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.711702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.692097 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77396fff-1778-4f09-9e81-05e715fc015e-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.711702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.693412 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3627ae-66c6-44a4-aa60-480bd54008f0-istiod-ca-cert\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.711702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.693643 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77396fff-1778-4f09-9e81-05e715fc015e-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.711702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.694234 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istiod-ca-cert\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.711702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.696415 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-workload-certs\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.711702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.696706 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-data\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.711702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.696985 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77396fff-1778-4f09-9e81-05e715fc015e-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.711702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.697252 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-credential-socket\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.711702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.697385 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-data\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.711702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.698232 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/977cec40-92df-410a-a3b6-68520b98aaf2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst\" (UID: \"977cec40-92df-410a-a3b6-68520b98aaf2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst" Nov 03 20:33:01.711702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.698472 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-credential-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.711702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.700001 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77396fff-1778-4f09-9e81-05e715fc015e-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.733587 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.733542 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75tbk\" (UniqueName: \"kubernetes.io/projected/977cec40-92df-410a-a3b6-68520b98aaf2-kube-api-access-75tbk\") pod \"payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst\" (UID: \"977cec40-92df-410a-a3b6-68520b98aaf2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst" Nov 03 20:33:01.743277 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.740018 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-token\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.743277 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.742984 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-token\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.749545 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.748242 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl"] Nov 03 20:33:01.750089 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.750024 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-envoy\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.755302 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.755145 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-xzpgl"] Nov 03 20:33:01.770471 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.770412 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-846mt"] Nov 03 20:33:01.777164 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.774832 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwqd5\" (UniqueName: \"kubernetes.io/projected/77396fff-1778-4f09-9e81-05e715fc015e-kube-api-access-bwqd5\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.777164 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.775233 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4698\" (UniqueName: \"kubernetes.io/projected/cca72a7d-a69d-4e05-ac4d-da20e15a0243-kube-api-access-r4698\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.777164 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.775853 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5bf8\" (UniqueName: \"kubernetes.io/projected/4b3627ae-66c6-44a4-aa60-480bd54008f0-kube-api-access-s5bf8\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.777164 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.776715 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9ead1ac-0ace-4761-9fde-b7f97a9dfa31-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn\" (UID: \"c9ead1ac-0ace-4761-9fde-b7f97a9dfa31\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn" Nov 03 20:33:01.777164 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.777003 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcrnp\" (UniqueName: \"kubernetes.io/projected/c9ead1ac-0ace-4761-9fde-b7f97a9dfa31-kube-api-access-wcrnp\") pod \"payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn\" (UID: \"c9ead1ac-0ace-4761-9fde-b7f97a9dfa31\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn" Nov 03 20:33:01.782282 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.778056 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77396fff-1778-4f09-9e81-05e715fc015e-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.787153 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.787126 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77396fff-1778-4f09-9e81-05e715fc015e-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn\" (UID: \"77396fff-1778-4f09-9e81-05e715fc015e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:01.787340 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.787322 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-envoy\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:01.791420 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.791391 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9ead1ac-0ace-4761-9fde-b7f97a9dfa31-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn\" (UID: \"c9ead1ac-0ace-4761-9fde-b7f97a9dfa31\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn" Nov 03 20:33:01.792498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.792472 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-podinfo\") pod \"e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:01.798035 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.797976 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcrnp\" (UniqueName: \"kubernetes.io/projected/c9ead1ac-0ace-4761-9fde-b7f97a9dfa31-kube-api-access-wcrnp\") pod \"payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn\" (UID: \"c9ead1ac-0ace-4761-9fde-b7f97a9dfa31\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn" Nov 03 20:33:01.798035 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.798030 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c"] Nov 03 20:33:01.810207 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:01.810178 5059 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0e4ce7/agc-cred-5f4fe13bd610: secret "agc-cred-5f4fe13bd610" not found Nov 03 20:33:01.810291 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:01.810217 5059 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2: secret "agc-cred-5f4fe13bd610" not found Nov 03 20:33:01.810291 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:01.810290 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0309a608-f6c4-45ca-991b-33ba73221a6c-provider-credentials podName:0309a608-f6c4-45ca-991b-33ba73221a6c nodeName:}" failed. No retries permitted until 2026-11-03 20:33:09.810262177 +0000 UTC m=+3384.068059417 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0309a608-f6c4-45ca-991b-33ba73221a6c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" (UID: "0309a608-f6c4-45ca-991b-33ba73221a6c") : secret "agc-cred-5f4fe13bd610" not found Nov 03 20:33:01.825238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.825022 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-846mt"] Nov 03 20:33:01.825238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.825117 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c" Nov 03 20:33:01.834314 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.834079 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c"] Nov 03 20:33:01.860113 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.856213 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 20:33:01.860113 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.856268 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" podUID="8153e018-7f9a-4645-b102-06ff3913f682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 20:33:01.888165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.887026 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece17ed8-afcd-41ab-85e0-adb35875ecd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c\" (UID: \"ece17ed8-afcd-41ab-85e0-adb35875ecd6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c" Nov 03 20:33:01.888165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.887405 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtgkq\" (UniqueName: \"kubernetes.io/projected/ece17ed8-afcd-41ab-85e0-adb35875ecd6-kube-api-access-dtgkq\") pod \"payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c\" (UID: \"ece17ed8-afcd-41ab-85e0-adb35875ecd6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c" Nov 03 20:33:01.892194 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.891471 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04a982f9-143c-406c-99d5-9ef73d5c571a" path="/var/lib/kubelet/pods/04a982f9-143c-406c-99d5-9ef73d5c571a/volumes" Nov 03 20:33:01.898126 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.896275 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea6e69e-3556-4ffe-bb9e-796e8795d747" path="/var/lib/kubelet/pods/4ea6e69e-3556-4ffe-bb9e-796e8795d747/volumes" Nov 03 20:33:01.905014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.904986 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ac27a9c-459f-4310-aef7-8dc643b46c6c" path="/var/lib/kubelet/pods/5ac27a9c-459f-4310-aef7-8dc643b46c6c/volumes" Nov 03 20:33:01.905894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.905872 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="677a5153-5fe3-4624-bf06-3c8dfc6bc72d" path="/var/lib/kubelet/pods/677a5153-5fe3-4624-bf06-3c8dfc6bc72d/volumes" Nov 03 20:33:01.907103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.907082 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="815b15b6-2c37-4ca3-9b38-e7e1328d0aa8" path="/var/lib/kubelet/pods/815b15b6-2c37-4ca3-9b38-e7e1328d0aa8/volumes" Nov 03 20:33:01.907885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.907865 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c99b0c19-004e-450c-852a-e38338635b61" path="/var/lib/kubelet/pods/c99b0c19-004e-450c-852a-e38338635b61/volumes" Nov 03 20:33:01.915914 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.915880 5059 generic.go:356] "Generic (PLEG): container finished" podID="70d45aa9-bfa8-43c6-8674-6f8f14a3097e" containerID="6a213e69e74bbe139a2688d93452e7d59e19f98c3c9a8e404648454c84993ae7" exitCode=0 Nov 03 20:33:01.916222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.916199 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" event={"ID":"70d45aa9-bfa8-43c6-8674-6f8f14a3097e","Type":"ContainerDied","Data":"6a213e69e74bbe139a2688d93452e7d59e19f98c3c9a8e404648454c84993ae7"} Nov 03 20:33:01.928409 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.919458 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" event={"ID":"d931dc13-4399-48dc-80e0-b0b304c5dda0","Type":"ContainerStarted","Data":"31b6388801c2c480e702c4985e3f6638dbac4103cebd0a5bf86216cac78f3f2f"} Nov 03 20:33:01.958182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.954933 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5"] Nov 03 20:33:01.981882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.981695 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:01.982067 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.982045 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts"] Nov 03 20:33:01.991742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.991638 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtgkq\" (UniqueName: \"kubernetes.io/projected/ece17ed8-afcd-41ab-85e0-adb35875ecd6-kube-api-access-dtgkq\") pod \"payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c\" (UID: \"ece17ed8-afcd-41ab-85e0-adb35875ecd6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c" Nov 03 20:33:01.991946 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.991921 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece17ed8-afcd-41ab-85e0-adb35875ecd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c\" (UID: \"ece17ed8-afcd-41ab-85e0-adb35875ecd6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c" Nov 03 20:33:01.993529 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.993112 5059 generic.go:356] "Generic (PLEG): container finished" podID="d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" containerID="e7118a165e1c692020e00ab701f84d22189feb4501a06e12ef7daecc27e93292" exitCode=0 Nov 03 20:33:01.993529 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.993221 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" event={"ID":"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840","Type":"ContainerDied","Data":"e7118a165e1c692020e00ab701f84d22189feb4501a06e12ef7daecc27e93292"} Nov 03 20:33:01.993529 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:01.993222 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece17ed8-afcd-41ab-85e0-adb35875ecd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c\" (UID: \"ece17ed8-afcd-41ab-85e0-adb35875ecd6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c" Nov 03 20:33:02.006605 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.006364 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:02.010397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.010331 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-x8zts"] Nov 03 20:33:02.017692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.017603 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5"] Nov 03 20:33:02.052518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.048610 5059 generic.go:356] "Generic (PLEG): container finished" podID="08b3a993-1a64-4fe5-9577-b0f11a36779e" containerID="ab75e4ee856c98aa1f2cad9eecfea81d6df844e47094122246ef45af739d37d5" exitCode=0 Nov 03 20:33:02.052518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.049164 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" event={"ID":"08b3a993-1a64-4fe5-9577-b0f11a36779e","Type":"ContainerDied","Data":"ab75e4ee856c98aa1f2cad9eecfea81d6df844e47094122246ef45af739d37d5"} Nov 03 20:33:02.056040 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.055852 5059 generic.go:356] "Generic (PLEG): container finished" podID="f88ae6f3-f1d6-4aa3-ad33-589490696167" containerID="f35a41707de2c0253145d89c0f1f6796087f8f0d6d98dc9ca16c8dcbf8e24a1e" exitCode=0 Nov 03 20:33:02.056040 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.055966 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" event={"ID":"f88ae6f3-f1d6-4aa3-ad33-589490696167","Type":"ContainerDied","Data":"f35a41707de2c0253145d89c0f1f6796087f8f0d6d98dc9ca16c8dcbf8e24a1e"} Nov 03 20:33:02.060166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.059815 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q"] Nov 03 20:33:02.066827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.065639 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.074420 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.074382 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2"] Nov 03 20:33:02.097601 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.097257 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-token\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.097601 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.097324 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5585d8d-d156-418c-ae87-012781bc0e8c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.097601 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.097460 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-workload-certs\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.097873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.097850 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-envoy\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.098011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.097993 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41778c8b-1334-4a0a-afbc-2d377d514e8d-istiod-ca-cert\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.109124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.099925 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-data\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.109124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.099960 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-podinfo\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.109124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.099986 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wcqx\" (UniqueName: \"kubernetes.io/projected/41778c8b-1334-4a0a-afbc-2d377d514e8d-kube-api-access-4wcqx\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.109124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.100013 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5585d8d-d156-418c-ae87-012781bc0e8c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.109124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.100104 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95lks\" (UniqueName: \"kubernetes.io/projected/e5585d8d-d156-418c-ae87-012781bc0e8c-kube-api-access-95lks\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.109124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.100129 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5585d8d-d156-418c-ae87-012781bc0e8c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.109124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.100216 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5585d8d-d156-418c-ae87-012781bc0e8c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.109124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.100279 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5585d8d-d156-418c-ae87-012781bc0e8c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.109124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.100346 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5585d8d-d156-418c-ae87-012781bc0e8c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.109124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.100380 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-workload-socket\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.109124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.100463 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5585d8d-d156-418c-ae87-012781bc0e8c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.109124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.100489 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5585d8d-d156-418c-ae87-012781bc0e8c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.109124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.100512 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-credential-socket\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.109124 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:02.100720 5059 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-plugins-e2e-extmod-b-0e4ce7: configmap "payload-processing-external-model-plugins-e2e-extmod-b-0e4ce7" not found Nov 03 20:33:02.109124 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:02.100771 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0309a608-f6c4-45ca-991b-33ba73221a6c-plugins-config-volume podName:0309a608-f6c4-45ca-991b-33ba73221a6c nodeName:}" failed. No retries permitted until 2026-11-03 20:33:10.100756145 +0000 UTC m=+3384.358553381 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0309a608-f6c4-45ca-991b-33ba73221a6c-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" (UID: "0309a608-f6c4-45ca-991b-33ba73221a6c") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-0e4ce7" not found Nov 03 20:33:02.109124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.103908 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w"] Nov 03 20:33:02.109124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.107765 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.114058 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.114034 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcrh4"] Nov 03 20:33:02.114524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.114504 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.118515 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.118462 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtgkq\" (UniqueName: \"kubernetes.io/projected/ece17ed8-afcd-41ab-85e0-adb35875ecd6-kube-api-access-dtgkq\") pod \"payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c\" (UID: \"ece17ed8-afcd-41ab-85e0-adb35875ecd6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c" Nov 03 20:33:02.125547 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.125376 5059 generic.go:356] "Generic (PLEG): container finished" podID="890f130b-cdeb-43da-acd4-1654673271cc" containerID="80563e166a6378b34d09a54f08d0ed8f1e665107abe95933f0fa76b31fc439e7" exitCode=0 Nov 03 20:33:02.151615 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.142096 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9" event={"ID":"890f130b-cdeb-43da-acd4-1654673271cc","Type":"ContainerDied","Data":"80563e166a6378b34d09a54f08d0ed8f1e665107abe95933f0fa76b31fc439e7"} Nov 03 20:33:02.151615 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.142149 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q"] Nov 03 20:33:02.151615 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.142262 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcrh4" Nov 03 20:33:02.158218 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.157626 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 20:33:02.158218 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.158092 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 20:33:02.191773 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.190395 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5" event={"ID":"6b9428a2-ab50-4e44-a725-e0da980bdb93","Type":"ContainerStarted","Data":"b6f8babda48abb99f0cb3d851b8caae4d8cd12ad7eda8750e60f7605358c12ba"} Nov 03 20:33:02.195270 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.194554 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-lw7w7\"" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.203225 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41778c8b-1334-4a0a-afbc-2d377d514e8d-istiod-ca-cert\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.203344 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-data\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.203373 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-podinfo\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.203399 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wcqx\" (UniqueName: \"kubernetes.io/projected/41778c8b-1334-4a0a-afbc-2d377d514e8d-kube-api-access-4wcqx\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.203429 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fa410e-25d7-4800-a22b-9dea6ccca3ea-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.203456 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5585d8d-d156-418c-ae87-012781bc0e8c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.203494 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fa410e-25d7-4800-a22b-9dea6ccca3ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.203540 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426d1445-9e48-4516-a920-fca04d918f22-istiod-ca-cert\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.203567 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fa410e-25d7-4800-a22b-9dea6ccca3ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.203594 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fa410e-25d7-4800-a22b-9dea6ccca3ea-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.203639 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95lks\" (UniqueName: \"kubernetes.io/projected/e5585d8d-d156-418c-ae87-012781bc0e8c-kube-api-access-95lks\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.203691 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5585d8d-d156-418c-ae87-012781bc0e8c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.203770 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-istio-data\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.203812 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5585d8d-d156-418c-ae87-012781bc0e8c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.203838 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fa410e-25d7-4800-a22b-9dea6ccca3ea-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.203901 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5585d8d-d156-418c-ae87-012781bc0e8c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.203928 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-istio-envoy\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.203968 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fa410e-25d7-4800-a22b-9dea6ccca3ea-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.204020 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5585d8d-d156-418c-ae87-012781bc0e8c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.204057 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-workload-socket\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.204082 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-credential-socket\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.204168 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5585d8d-d156-418c-ae87-012781bc0e8c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.204197 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5585d8d-d156-418c-ae87-012781bc0e8c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.204219 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-credential-socket\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.204261 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fa410e-25d7-4800-a22b-9dea6ccca3ea-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.204302 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-token\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.204329 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jqnx\" (UniqueName: \"kubernetes.io/projected/426d1445-9e48-4516-a920-fca04d918f22-kube-api-access-6jqnx\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.204353 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdjg2\" (UniqueName: \"kubernetes.io/projected/81fa410e-25d7-4800-a22b-9dea6ccca3ea-kube-api-access-wdjg2\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.204409 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5585d8d-d156-418c-ae87-012781bc0e8c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.204446 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxfgk\" (UniqueName: \"kubernetes.io/projected/f44d81a0-55a2-4021-b34f-3400deb30c82-kube-api-access-hxfgk\") pod \"limitador-operator-controller-manager-5778bb4449-dcrh4\" (UID: \"f44d81a0-55a2-4021-b34f-3400deb30c82\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcrh4" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.204496 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426d1445-9e48-4516-a920-fca04d918f22-istio-podinfo\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.204540 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-workload-socket\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.207954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.204565 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-workload-certs\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.209516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.204697 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-workload-certs\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.209516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.204794 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fa410e-25d7-4800-a22b-9dea6ccca3ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.209516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.204891 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-envoy\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.209516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.204915 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426d1445-9e48-4516-a920-fca04d918f22-istio-token\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.209516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.205904 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41778c8b-1334-4a0a-afbc-2d377d514e8d-istiod-ca-cert\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.209516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.206197 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-data\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.209516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.206613 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-workload-socket\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.209516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.207023 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5585d8d-d156-418c-ae87-012781bc0e8c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.209516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.207481 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-credential-socket\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.209516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.207788 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5585d8d-d156-418c-ae87-012781bc0e8c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.209516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.207807 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5585d8d-d156-418c-ae87-012781bc0e8c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.209516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.208533 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-workload-certs\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.209516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.208802 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5585d8d-d156-418c-ae87-012781bc0e8c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.211153 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.210964 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5585d8d-d156-418c-ae87-012781bc0e8c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.281285 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.230024 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c"] Nov 03 20:33:02.281285 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.253709 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcrh4"] Nov 03 20:33:02.281285 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.253912 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5585d8d-d156-418c-ae87-012781bc0e8c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.281285 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.261528 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5585d8d-d156-418c-ae87-012781bc0e8c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.304551 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.304461 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w"] Nov 03 20:33:02.304821 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.304802 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l5g9c"] Nov 03 20:33:02.330741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.329501 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-workload-socket\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.330741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.329546 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-workload-certs\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.330741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.329718 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fa410e-25d7-4800-a22b-9dea6ccca3ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.330741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.329792 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426d1445-9e48-4516-a920-fca04d918f22-istio-token\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.330741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.329933 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fa410e-25d7-4800-a22b-9dea6ccca3ea-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.330741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.329976 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fa410e-25d7-4800-a22b-9dea6ccca3ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.330741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.330025 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426d1445-9e48-4516-a920-fca04d918f22-istiod-ca-cert\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.330741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.330054 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fa410e-25d7-4800-a22b-9dea6ccca3ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.330741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.330081 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fa410e-25d7-4800-a22b-9dea6ccca3ea-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.330741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.330195 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-istio-data\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.330741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.330233 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fa410e-25d7-4800-a22b-9dea6ccca3ea-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.330741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.330294 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-istio-envoy\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.330741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.330386 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fa410e-25d7-4800-a22b-9dea6ccca3ea-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.330741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.330448 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-credential-socket\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.330741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.330564 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fa410e-25d7-4800-a22b-9dea6ccca3ea-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.330741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.330605 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jqnx\" (UniqueName: \"kubernetes.io/projected/426d1445-9e48-4516-a920-fca04d918f22-kube-api-access-6jqnx\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.330741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.330633 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdjg2\" (UniqueName: \"kubernetes.io/projected/81fa410e-25d7-4800-a22b-9dea6ccca3ea-kube-api-access-wdjg2\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.415683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.394754 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxfgk\" (UniqueName: \"kubernetes.io/projected/f44d81a0-55a2-4021-b34f-3400deb30c82-kube-api-access-hxfgk\") pod \"limitador-operator-controller-manager-5778bb4449-dcrh4\" (UID: \"f44d81a0-55a2-4021-b34f-3400deb30c82\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcrh4" Nov 03 20:33:02.415683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.394834 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426d1445-9e48-4516-a920-fca04d918f22-istio-podinfo\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.415683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.338137 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fa410e-25d7-4800-a22b-9dea6ccca3ea-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.415683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.339680 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fa410e-25d7-4800-a22b-9dea6ccca3ea-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.415683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.340074 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fa410e-25d7-4800-a22b-9dea6ccca3ea-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.415683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.355504 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-workload-socket\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.415683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.355750 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-workload-certs\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.415683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.355808 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-istio-data\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.415683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.356311 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-credential-socket\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.415683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.336562 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-envoy\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.415683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.337162 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-podinfo\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.415683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.396872 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fa410e-25d7-4800-a22b-9dea6ccca3ea-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.415683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.410285 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fa410e-25d7-4800-a22b-9dea6ccca3ea-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.415683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.413151 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4" event={"ID":"1326708b-4642-4992-91ea-8a590863a703","Type":"ContainerStarted","Data":"9bf7e288190b11ae0ba4b37a69f007f930e0f2afcf8152bdfbdbfa5e3df689c6"} Nov 03 20:33:02.415683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.413212 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4" event={"ID":"1326708b-4642-4992-91ea-8a590863a703","Type":"ContainerStarted","Data":"b79ef057628756807c50f783b8a0e808e8ec00460d8b0941097f753f392a1bf3"} Nov 03 20:33:02.420432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.417130 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fa410e-25d7-4800-a22b-9dea6ccca3ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.420432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.417757 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426d1445-9e48-4516-a920-fca04d918f22-istiod-ca-cert\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.431528 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.431387 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426d1445-9e48-4516-a920-fca04d918f22-istio-token\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.431528 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.431453 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2"] Nov 03 20:33:02.434402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.432031 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9" podUID="890f130b-cdeb-43da-acd4-1654673271cc" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.70:50052\" within 1s: context deadline exceeded" Nov 03 20:33:02.434402 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:33:02.432160 5059 logging.go:55] [core] [Channel #6805 SubChannel #6806]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.70:50052", ServerName: "10.132.0.70:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.70:50052: operation was canceled" Nov 03 20:33:02.434619 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.434548 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxfgk\" (UniqueName: \"kubernetes.io/projected/f44d81a0-55a2-4021-b34f-3400deb30c82-kube-api-access-hxfgk\") pod \"limitador-operator-controller-manager-5778bb4449-dcrh4\" (UID: \"f44d81a0-55a2-4021-b34f-3400deb30c82\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcrh4" Nov 03 20:33:02.443527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.439061 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426d1445-9e48-4516-a920-fca04d918f22-istio-podinfo\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.443527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.441565 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jqnx\" (UniqueName: \"kubernetes.io/projected/426d1445-9e48-4516-a920-fca04d918f22-kube-api-access-6jqnx\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.547354 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.512769 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95lks\" (UniqueName: \"kubernetes.io/projected/e5585d8d-d156-418c-ae87-012781bc0e8c-kube-api-access-95lks\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.547354 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.513025 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fa410e-25d7-4800-a22b-9dea6ccca3ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.547354 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.513248 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5585d8d-d156-418c-ae87-012781bc0e8c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68d996c557dng2q\" (UID: \"e5585d8d-d156-418c-ae87-012781bc0e8c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:02.547354 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.520094 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-istio-envoy\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:02.547354 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.520767 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdjg2\" (UniqueName: \"kubernetes.io/projected/81fa410e-25d7-4800-a22b-9dea6ccca3ea-kube-api-access-wdjg2\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.547354 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.535224 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-token\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.547354 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.535507 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wcqx\" (UniqueName: \"kubernetes.io/projected/41778c8b-1334-4a0a-afbc-2d377d514e8d-kube-api-access-4wcqx\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:02.547354 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.535771 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fa410e-25d7-4800-a22b-9dea6ccca3ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-5sk9w\" (UID: \"81fa410e-25d7-4800-a22b-9dea6ccca3ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:02.590605 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:33:02.589884 5059 logging.go:55] [core] [Channel #6807 SubChannel #6808]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.71:50052", ServerName: "10.132.0.71:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.71:50052: connect: connection refused" Nov 03 20:33:02.659769 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.642052 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx"] Nov 03 20:33:02.659769 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.653782 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-86qxx"] Nov 03 20:33:02.699049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.693506 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q"] Nov 03 20:33:02.751309 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.731505 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q" Nov 03 20:33:02.813749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.813462 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" start-of-body= Nov 03 20:33:02.813749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.813527 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" podUID="05fceb9c-4a04-470f-a94a-055121319f82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" Nov 03 20:33:02.917763 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.840713 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q"] Nov 03 20:33:02.993717 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.893360 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbg8q\" (UniqueName: \"kubernetes.io/projected/8e9aefe5-8450-4eb9-8d8b-df89be81ebb3-kube-api-access-dbg8q\") pod \"payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q\" (UID: \"8e9aefe5-8450-4eb9-8d8b-df89be81ebb3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q" Nov 03 20:33:02.993717 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:02.920463 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9aefe5-8450-4eb9-8d8b-df89be81ebb3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q\" (UID: \"8e9aefe5-8450-4eb9-8d8b-df89be81ebb3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q" Nov 03 20:33:03.007287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.007120 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h"] Nov 03 20:33:03.020816 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.020710 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8"] Nov 03 20:33:03.024277 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.024084 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbg8q\" (UniqueName: \"kubernetes.io/projected/8e9aefe5-8450-4eb9-8d8b-df89be81ebb3-kube-api-access-dbg8q\") pod \"payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q\" (UID: \"8e9aefe5-8450-4eb9-8d8b-df89be81ebb3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q" Nov 03 20:33:03.024630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.024489 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9aefe5-8450-4eb9-8d8b-df89be81ebb3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q\" (UID: \"8e9aefe5-8450-4eb9-8d8b-df89be81ebb3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q" Nov 03 20:33:03.026488 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.025894 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9aefe5-8450-4eb9-8d8b-df89be81ebb3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q\" (UID: \"8e9aefe5-8450-4eb9-8d8b-df89be81ebb3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q" Nov 03 20:33:03.124166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.120817 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbg8q\" (UniqueName: \"kubernetes.io/projected/8e9aefe5-8450-4eb9-8d8b-df89be81ebb3-kube-api-access-dbg8q\") pod \"payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q\" (UID: \"8e9aefe5-8450-4eb9-8d8b-df89be81ebb3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q" Nov 03 20:33:03.153437 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.144334 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:33:03.153437 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.144390 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" podUID="eac98813-7b4c-4c4a-8bf9-444bba2fa644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 20:33:03.202139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.202065 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/downloads-6dd789c8b7-c88f8"] Nov 03 20:33:03.221084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.213720 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/downloads-6dd789c8b7-c88f8"] Nov 03 20:33:03.438103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.431447 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vznnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 20:33:03.438103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.431547 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" podUID="171c7288-3157-4fe0-80ee-94cf5b2ca2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 20:33:03.509265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.507992 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn" podUID="25ceab3d-6344-4512-a471-d91aec0cb48c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.71:50052\" within 1s: context deadline exceeded" Nov 03 20:33:03.552915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.552816 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls"] Nov 03 20:33:03.605725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.605372 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b"] Nov 03 20:33:03.615313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.614898 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-h9f7b"] Nov 03 20:33:03.831287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.786006 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7krsk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 20:33:03.831287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.786065 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" podUID="4946e3aa-a961-415a-b71d-95164d5fbca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 20:33:03.831950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.831464 5059 generic.go:356] "Generic (PLEG): container finished" podID="d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" containerID="5c19ca06fbb1d449a708204a75d839cc487a9ea18245ce5649dd7dfdd316e86f" exitCode=0 Nov 03 20:33:03.831950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.831567 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" event={"ID":"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b","Type":"ContainerDied","Data":"5c19ca06fbb1d449a708204a75d839cc487a9ea18245ce5649dd7dfdd316e86f"} Nov 03 20:33:03.837697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.835863 5059 generic.go:356] "Generic (PLEG): container finished" podID="f0d3c590-f523-4b7e-9086-60df5fc585e8" containerID="1638b310e0522c5f2a20ee3d226436456869be0ba4a5f3e1f6c3bdebb4a48c87" exitCode=0 Nov 03 20:33:03.837697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.835939 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" event={"ID":"f0d3c590-f523-4b7e-9086-60df5fc585e8","Type":"ContainerDied","Data":"1638b310e0522c5f2a20ee3d226436456869be0ba4a5f3e1f6c3bdebb4a48c87"} Nov 03 20:33:03.837697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.837674 5059 generic.go:356] "Generic (PLEG): container finished" podID="cfcb49c5-4381-411e-9d64-d9dcd6181239" containerID="06775cb2b6311ab6024c5cf0da37eef3631ba5e53f2a44ad31bd33c7d778d26f" exitCode=0 Nov 03 20:33:03.837931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.837752 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2" event={"ID":"cfcb49c5-4381-411e-9d64-d9dcd6181239","Type":"ContainerDied","Data":"06775cb2b6311ab6024c5cf0da37eef3631ba5e53f2a44ad31bd33c7d778d26f"} Nov 03 20:33:03.894030 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.842507 5059 generic.go:356] "Generic (PLEG): container finished" podID="a594e899-889e-4f7f-a8ba-b7e7a389ba8a" containerID="034deb3d094b43323ad58f8415dd48a1c89f624ddffe2676cc27ac841dc9e817" exitCode=0 Nov 03 20:33:03.894030 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.842608 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" event={"ID":"a594e899-889e-4f7f-a8ba-b7e7a389ba8a","Type":"ContainerDied","Data":"034deb3d094b43323ad58f8415dd48a1c89f624ddffe2676cc27ac841dc9e817"} Nov 03 20:33:03.894030 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.844843 5059 generic.go:356] "Generic (PLEG): container finished" podID="474b2e1d-832b-4edd-8765-31e0bc77cc2a" containerID="3513472a5d02bb21de815edd2ec4ca696e8e46e17be5d839916c093e17bd9499" exitCode=0 Nov 03 20:33:03.894030 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.844912 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" event={"ID":"474b2e1d-832b-4edd-8765-31e0bc77cc2a","Type":"ContainerDied","Data":"3513472a5d02bb21de815edd2ec4ca696e8e46e17be5d839916c093e17bd9499"} Nov 03 20:33:03.903686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.902976 5059 generic.go:356] "Generic (PLEG): container finished" podID="edcfb7ab-b14f-4d5c-b752-5b43a6eee163" containerID="bab3644b9638d4ac499f2df75a736f3f63d2b12e17964b2ccc440b6a34499ce5" exitCode=0 Nov 03 20:33:03.906042 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.906016 5059 generic.go:356] "Generic (PLEG): container finished" podID="0309a608-f6c4-45ca-991b-33ba73221a6c" containerID="6e4d31c85e8cd1b2b8591eeb7383bfd43ae2ffe368b7b0f552663b7b668eb233" exitCode=0 Nov 03 20:33:03.921109 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.919135 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40b2621f-854e-44e0-9b01-c877e333f644" path="/var/lib/kubelet/pods/40b2621f-854e-44e0-9b01-c877e333f644/volumes" Nov 03 20:33:03.921109 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.920650 5059 generic.go:356] "Generic (PLEG): container finished" podID="c8683a65-9898-4b4f-a692-f2da325a93b4" containerID="d36cbc617302444ca97907469c12558d0fc4a6b46d78b847f1d5ccbc843b16e6" exitCode=0 Nov 03 20:33:03.964723 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.926588 5059 generic.go:356] "Generic (PLEG): container finished" podID="1326708b-4642-4992-91ea-8a590863a703" containerID="9bf7e288190b11ae0ba4b37a69f007f930e0f2afcf8152bdfbdbfa5e3df689c6" exitCode=22 Nov 03 20:33:03.964723 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.927224 5059 scope.go:122] "RemoveContainer" containerID="9bf7e288190b11ae0ba4b37a69f007f930e0f2afcf8152bdfbdbfa5e3df689c6" Nov 03 20:33:03.964723 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.935944 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="633c352c-f07e-4032-b962-23a148297e45" path="/var/lib/kubelet/pods/633c352c-f07e-4032-b962-23a148297e45/volumes" Nov 03 20:33:03.964723 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:03.947065 5059 generic.go:356] "Generic (PLEG): container finished" podID="518169ac-2f8d-4645-ae47-210b67ca080a" containerID="fcb62eadee37b9a5aa9bdc20f77cd13d99c001430b049cbc4d5006aa5cc74354" exitCode=0 Nov 03 20:33:04.105085 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.023239 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64153a99-d5f7-4f92-a0c9-f79b39eba97c" path="/var/lib/kubelet/pods/64153a99-d5f7-4f92-a0c9-f79b39eba97c/volumes" Nov 03 20:33:04.106915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.106629 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a7937ca-c705-4091-967e-2e640c622a26" path="/var/lib/kubelet/pods/6a7937ca-c705-4091-967e-2e640c622a26/volumes" Nov 03 20:33:04.109083 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.108197 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d0db609-b271-4c98-a303-fb3ee71f152c" path="/var/lib/kubelet/pods/8d0db609-b271-4c98-a303-fb3ee71f152c/volumes" Nov 03 20:33:04.109083 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.108774 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" event={"ID":"edcfb7ab-b14f-4d5c-b752-5b43a6eee163","Type":"ContainerDied","Data":"bab3644b9638d4ac499f2df75a736f3f63d2b12e17964b2ccc440b6a34499ce5"} Nov 03 20:33:04.109083 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.108847 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qchj6"] Nov 03 20:33:04.109083 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.108878 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qchj6"] Nov 03 20:33:04.109083 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.108903 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" event={"ID":"0309a608-f6c4-45ca-991b-33ba73221a6c","Type":"ContainerDied","Data":"6e4d31c85e8cd1b2b8591eeb7383bfd43ae2ffe368b7b0f552663b7b668eb233"} Nov 03 20:33:04.109083 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.108937 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" event={"ID":"c8683a65-9898-4b4f-a692-f2da325a93b4","Type":"ContainerDied","Data":"d36cbc617302444ca97907469c12558d0fc4a6b46d78b847f1d5ccbc843b16e6"} Nov 03 20:33:04.109083 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.108964 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4" event={"ID":"1326708b-4642-4992-91ea-8a590863a703","Type":"ContainerDied","Data":"9bf7e288190b11ae0ba4b37a69f007f930e0f2afcf8152bdfbdbfa5e3df689c6"} Nov 03 20:33:04.109083 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.108991 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" event={"ID":"518169ac-2f8d-4645-ae47-210b67ca080a","Type":"ContainerDied","Data":"fcb62eadee37b9a5aa9bdc20f77cd13d99c001430b049cbc4d5006aa5cc74354"} Nov 03 20:33:04.115498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.111586 5059 generic.go:356] "Generic (PLEG): container finished" podID="8b648c6b-1f7d-42ca-9c3f-a435494bad03" containerID="1fe212cabed0fc214cbab7c5d208f7f12d710cdc5a5c86dc76e26a0430ec5af5" exitCode=0 Nov 03 20:33:04.115498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.111705 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" event={"ID":"8b648c6b-1f7d-42ca-9c3f-a435494bad03","Type":"ContainerDied","Data":"1fe212cabed0fc214cbab7c5d208f7f12d710cdc5a5c86dc76e26a0430ec5af5"} Nov 03 20:33:04.115498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.114082 5059 generic.go:356] "Generic (PLEG): container finished" podID="171c7288-3157-4fe0-80ee-94cf5b2ca2dd" containerID="07485e5a1a9bd8f3e2278bef20b0f805688a601c1ec005e9ee1b9f8abb449cd3" exitCode=0 Nov 03 20:33:04.115498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.114176 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" event={"ID":"171c7288-3157-4fe0-80ee-94cf5b2ca2dd","Type":"ContainerDied","Data":"07485e5a1a9bd8f3e2278bef20b0f805688a601c1ec005e9ee1b9f8abb449cd3"} Nov 03 20:33:04.119701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.118470 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2"] Nov 03 20:33:04.202653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.145754 5059 generic.go:356] "Generic (PLEG): container finished" podID="cef4417c-3d04-4d65-a5bc-6ec1af6f09bb" containerID="c56ee037a40c322b89e375002cf24e08b31d3e1b30e1b41b0f4819b338d40ec8" exitCode=0 Nov 03 20:33:04.202653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.150034 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" event={"ID":"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb","Type":"ContainerDied","Data":"c56ee037a40c322b89e375002cf24e08b31d3e1b30e1b41b0f4819b338d40ec8"} Nov 03 20:33:04.202653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.150156 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.202653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.152725 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2"] Nov 03 20:33:04.235931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.233361 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.235931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.233438 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.235931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.233586 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.235931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.233647 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.235931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.233818 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09d624de-215a-40be-9b38-0ba5a08743ef-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.235931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.233883 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.235931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.233932 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2lp6\" (UniqueName: \"kubernetes.io/projected/09d624de-215a-40be-9b38-0ba5a08743ef-kube-api-access-c2lp6\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.235931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.233963 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09d624de-215a-40be-9b38-0ba5a08743ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.235931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.234024 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09d624de-215a-40be-9b38-0ba5a08743ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.235931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.235059 5059 generic.go:356] "Generic (PLEG): container finished" podID="ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" containerID="8880ed12964f51d73ab467040324e0b739812c728f7782757374b9b0f025d69e" exitCode=0 Nov 03 20:33:04.235931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.235157 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" event={"ID":"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c","Type":"ContainerDied","Data":"8880ed12964f51d73ab467040324e0b739812c728f7782757374b9b0f025d69e"} Nov 03 20:33:04.301248 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.249917 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47"] Nov 03 20:33:04.321964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.321631 5059 generic.go:356] "Generic (PLEG): container finished" podID="26375297-81f4-427f-94c3-d56f7a6a11fb" containerID="93172d79a58c2d8458ec95cdf845557fbb10e251f6cf98dda510ae64f827325b" exitCode=0 Nov 03 20:33:04.321964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.321814 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:33:04.321964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.321858 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" podUID="14a06bdf-e15d-406c-bb76-47ae5dd06b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:04.347185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.343444 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.347185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.343530 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.347185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.343692 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.347185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.343755 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.347185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.343904 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09d624de-215a-40be-9b38-0ba5a08743ef-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.347185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.343972 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.347185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.344020 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2lp6\" (UniqueName: \"kubernetes.io/projected/09d624de-215a-40be-9b38-0ba5a08743ef-kube-api-access-c2lp6\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.347185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.344056 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09d624de-215a-40be-9b38-0ba5a08743ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.347185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.344119 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09d624de-215a-40be-9b38-0ba5a08743ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.347185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.346075 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.347185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.346366 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.347185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.346620 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.347185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.346884 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.356706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.348630 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09d624de-215a-40be-9b38-0ba5a08743ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.405438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.403562 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09d624de-215a-40be-9b38-0ba5a08743ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.405438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.404277 5059 generic.go:356] "Generic (PLEG): container finished" podID="2c363281-0226-420f-a963-5922d5ca609e" containerID="a1545bd75166fa5a492a8e6723253de1761cd417b14f627fbdab40e21ab83652" exitCode=0 Nov 03 20:33:04.406197 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.405718 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47"] Nov 03 20:33:04.406197 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.405782 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44" event={"ID":"26375297-81f4-427f-94c3-d56f7a6a11fb","Type":"ContainerDied","Data":"93172d79a58c2d8458ec95cdf845557fbb10e251f6cf98dda510ae64f827325b"} Nov 03 20:33:04.406197 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.405831 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7" event={"ID":"2c363281-0226-420f-a963-5922d5ca609e","Type":"ContainerDied","Data":"a1545bd75166fa5a492a8e6723253de1761cd417b14f627fbdab40e21ab83652"} Nov 03 20:33:04.406197 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.405966 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47" Nov 03 20:33:04.406748 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.406394 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09d624de-215a-40be-9b38-0ba5a08743ef-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.428221 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.411800 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.428221 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.413482 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2lp6\" (UniqueName: \"kubernetes.io/projected/09d624de-215a-40be-9b38-0ba5a08743ef-kube-api-access-c2lp6\") pod \"maas-default-gateway-openshift-default-855f9574cf-78xj2\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:04.428221 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.413740 5059 generic.go:356] "Generic (PLEG): container finished" podID="4946e3aa-a961-415a-b71d-95164d5fbca3" containerID="2ce7833679c135af351a934eeaaf542087c31098688f1a7b779e25f9a2bf4bc1" exitCode=0 Nov 03 20:33:04.428221 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.413837 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" event={"ID":"4946e3aa-a961-415a-b71d-95164d5fbca3","Type":"ContainerDied","Data":"2ce7833679c135af351a934eeaaf542087c31098688f1a7b779e25f9a2bf4bc1"} Nov 03 20:33:04.428221 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.426117 5059 generic.go:356] "Generic (PLEG): container finished" podID="2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" containerID="6c9d6b78a3dbde46dbddcc1222495b4fce407ba8a3afc87ef4704aadd4524702" exitCode=0 Nov 03 20:33:04.428221 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.426201 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" event={"ID":"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193","Type":"ContainerDied","Data":"6c9d6b78a3dbde46dbddcc1222495b4fce407ba8a3afc87ef4704aadd4524702"} Nov 03 20:33:04.438984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.430564 5059 generic.go:356] "Generic (PLEG): container finished" podID="25ceab3d-6344-4512-a471-d91aec0cb48c" containerID="b13a3b8cfa23ddc9815549092296b81b15bde977434b7a3b30d1385e768e8fb0" exitCode=0 Nov 03 20:33:04.438984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.430699 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn" event={"ID":"25ceab3d-6344-4512-a471-d91aec0cb48c","Type":"ContainerDied","Data":"b13a3b8cfa23ddc9815549092296b81b15bde977434b7a3b30d1385e768e8fb0"} Nov 03 20:33:04.438984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.432774 5059 generic.go:356] "Generic (PLEG): container finished" podID="0d960e4e-9539-4a53-9df7-1c520ab7d708" containerID="4436c68602fe345d6bb4fb9dfe6e2ac4dec763399336a2db74a54312609ee767" exitCode=0 Nov 03 20:33:04.438984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.432842 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq" event={"ID":"0d960e4e-9539-4a53-9df7-1c520ab7d708","Type":"ContainerDied","Data":"4436c68602fe345d6bb4fb9dfe6e2ac4dec763399336a2db74a54312609ee767"} Nov 03 20:33:04.438984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.435080 5059 generic.go:356] "Generic (PLEG): container finished" podID="ab8320f7-2354-433a-a8a7-4d91cc5d37d2" containerID="91f3fb2bb610fbeea4466047557a60f91aee6d1477483a4dc37e84c19840f464" exitCode=0 Nov 03 20:33:04.438984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.435149 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" event={"ID":"ab8320f7-2354-433a-a8a7-4d91cc5d37d2","Type":"ContainerDied","Data":"91f3fb2bb610fbeea4466047557a60f91aee6d1477483a4dc37e84c19840f464"} Nov 03 20:33:04.438984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.436888 5059 generic.go:356] "Generic (PLEG): container finished" podID="9c3d72b6-73c4-402f-8308-ee5d87a9dd83" containerID="3d68de58f4e860642d4d4255a14fe00815183f88da48e5f851f0a0c70e971419" exitCode=0 Nov 03 20:33:04.438984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.436953 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl" event={"ID":"9c3d72b6-73c4-402f-8308-ee5d87a9dd83","Type":"ContainerDied","Data":"3d68de58f4e860642d4d4255a14fe00815183f88da48e5f851f0a0c70e971419"} Nov 03 20:33:04.438984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.438792 5059 generic.go:356] "Generic (PLEG): container finished" podID="ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51" containerID="04da4e840db9b36ab74f969a24bc9303a68de3283b026a9e0deacd6cef05ae43" exitCode=0 Nov 03 20:33:04.438984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.438851 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p" event={"ID":"ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51","Type":"ContainerDied","Data":"04da4e840db9b36ab74f969a24bc9303a68de3283b026a9e0deacd6cef05ae43"} Nov 03 20:33:04.442445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.442320 5059 generic.go:356] "Generic (PLEG): container finished" podID="4c609469-d7d7-4530-8d74-e34aa56dda8f" containerID="25b28b428ebc85ad2241009e2e20a1a14a4f98588d071bd9785777b228e75310" exitCode=0 Nov 03 20:33:04.442445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.442389 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" event={"ID":"4c609469-d7d7-4530-8d74-e34aa56dda8f","Type":"ContainerDied","Data":"25b28b428ebc85ad2241009e2e20a1a14a4f98588d071bd9785777b228e75310"} Nov 03 20:33:04.446802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.446220 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c29d53f-b738-49ec-9b15-1c88e5950008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-gqz47\" (UID: \"6c29d53f-b738-49ec-9b15-1c88e5950008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47" Nov 03 20:33:04.446802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.446364 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxxr7\" (UniqueName: \"kubernetes.io/projected/6c29d53f-b738-49ec-9b15-1c88e5950008-kube-api-access-rxxr7\") pod \"maas-api-revoke-sub-5bac6f1e-gqz47\" (UID: \"6c29d53f-b738-49ec-9b15-1c88e5950008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47" Nov 03 20:33:04.446802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.446405 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c29d53f-b738-49ec-9b15-1c88e5950008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-gqz47\" (UID: \"6c29d53f-b738-49ec-9b15-1c88e5950008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47" Nov 03 20:33:04.465826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.448564 5059 generic.go:356] "Generic (PLEG): container finished" podID="c92b1be0-0890-4db6-ac75-968b6e5a3434" containerID="ddfb36a61a4890d032f40a96fa7dc33ada9d7745461eee81dc563f215eda9eae" exitCode=0 Nov 03 20:33:04.465826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.448628 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt" event={"ID":"c92b1be0-0890-4db6-ac75-968b6e5a3434","Type":"ContainerDied","Data":"ddfb36a61a4890d032f40a96fa7dc33ada9d7745461eee81dc563f215eda9eae"} Nov 03 20:33:04.465826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.451166 5059 generic.go:356] "Generic (PLEG): container finished" podID="5ea38154-8a1f-4e1f-8fa0-50481ba4fec0" containerID="cc7a98566be4a3d14d59e43fa3c73400eb83547051245560b3c66cac856b8485" exitCode=0 Nov 03 20:33:04.465826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.451230 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" event={"ID":"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0","Type":"ContainerDied","Data":"cc7a98566be4a3d14d59e43fa3c73400eb83547051245560b3c66cac856b8485"} Nov 03 20:33:04.465826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.453415 5059 generic.go:356] "Generic (PLEG): container finished" podID="1731953b-1850-40d1-9ba4-cb08dc4ebc5d" containerID="da8dcf56301a5bf859cd1267d1436e748641fe4be7bf5c9b6d88b7d504191f65" exitCode=0 Nov 03 20:33:04.465826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.453477 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" event={"ID":"1731953b-1850-40d1-9ba4-cb08dc4ebc5d","Type":"ContainerDied","Data":"da8dcf56301a5bf859cd1267d1436e748641fe4be7bf5c9b6d88b7d504191f65"} Nov 03 20:33:04.465826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.455599 5059 generic.go:356] "Generic (PLEG): container finished" podID="06b78cd0-94df-40f9-a1ed-87c1751675d3" containerID="e188f63192d3371f9c88f1fcfd31fbc6a5bb4e3b477faa591137913f4bacb386" exitCode=0 Nov 03 20:33:04.465826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.455691 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj" event={"ID":"06b78cd0-94df-40f9-a1ed-87c1751675d3","Type":"ContainerDied","Data":"e188f63192d3371f9c88f1fcfd31fbc6a5bb4e3b477faa591137913f4bacb386"} Nov 03 20:33:04.465826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.457211 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" event={"ID":"70d45aa9-bfa8-43c6-8674-6f8f14a3097e","Type":"ContainerDied","Data":"f5d4f211d31c87466d57af904dc64201bfbfa2b8af799af82610b17982f3e0d7"} Nov 03 20:33:04.465826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.457248 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5d4f211d31c87466d57af904dc64201bfbfa2b8af799af82610b17982f3e0d7" Nov 03 20:33:04.501741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.499023 5059 generic.go:356] "Generic (PLEG): container finished" podID="0f32ecf7-06f7-4098-9793-acf761a30c09" containerID="5ac8799b34596375075829e9ebddeeb7da35d54820c64c585e69600c30f1d815" exitCode=0 Nov 03 20:33:04.501741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.499136 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" event={"ID":"0f32ecf7-06f7-4098-9793-acf761a30c09","Type":"ContainerDied","Data":"5ac8799b34596375075829e9ebddeeb7da35d54820c64c585e69600c30f1d815"} Nov 03 20:33:04.517594 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.516707 5059 generic.go:356] "Generic (PLEG): container finished" podID="8153e018-7f9a-4645-b102-06ff3913f682" containerID="3a0b8447bfda8a58216cc853554baf15762b4ccc77d0d98ce90bfe2224c3b122" exitCode=0 Nov 03 20:33:04.517594 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.516795 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" event={"ID":"8153e018-7f9a-4645-b102-06ff3913f682","Type":"ContainerDied","Data":"3a0b8447bfda8a58216cc853554baf15762b4ccc77d0d98ce90bfe2224c3b122"} Nov 03 20:33:04.519206 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.518743 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls" event={"ID":"8760fa0e-a78b-4db8-b71d-bf4086ddbfbd","Type":"ContainerStarted","Data":"4959b0c7f42848c673b202788a758f9f274e1f3de52fca202971872398c3d9c7"} Nov 03 20:33:04.521227 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.520723 5059 generic.go:356] "Generic (PLEG): container finished" podID="37e90ab0-e9c0-420c-a0f4-c7923aa00297" containerID="f8323dfa4e2c1a85ed81f607b727475c16361a359dbb2bb285fe28a219d7a6eb" exitCode=0 Nov 03 20:33:04.521227 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.520804 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" event={"ID":"37e90ab0-e9c0-420c-a0f4-c7923aa00297","Type":"ContainerDied","Data":"f8323dfa4e2c1a85ed81f607b727475c16361a359dbb2bb285fe28a219d7a6eb"} Nov 03 20:33:04.552787 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.527397 5059 generic.go:356] "Generic (PLEG): container finished" podID="c2d80995-5f1c-46db-8d71-a130cb92e279" containerID="39fd5a87217d3b8146ccfe78173093985a26712d776696de4ab3b763f50a762f" exitCode=0 Nov 03 20:33:04.552787 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.527477 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" event={"ID":"c2d80995-5f1c-46db-8d71-a130cb92e279","Type":"ContainerDied","Data":"39fd5a87217d3b8146ccfe78173093985a26712d776696de4ab3b763f50a762f"} Nov 03 20:33:04.552787 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.534855 5059 generic.go:356] "Generic (PLEG): container finished" podID="34880975-a897-4ad4-af59-18617a93cdad" containerID="89e013c2e4eeadb512ea4a0824409f67321420e7c404d52597c8ba458a34976b" exitCode=0 Nov 03 20:33:04.552787 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.534930 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" event={"ID":"34880975-a897-4ad4-af59-18617a93cdad","Type":"ContainerDied","Data":"89e013c2e4eeadb512ea4a0824409f67321420e7c404d52597c8ba458a34976b"} Nov 03 20:33:04.552787 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.536690 5059 generic.go:356] "Generic (PLEG): container finished" podID="8adff4a5-5376-43ec-9b53-888705ae8236" containerID="3904c49bb7ab63c515d01a6052936b8d990a8f9bb352ab2810819de54dc7b74c" exitCode=0 Nov 03 20:33:04.552787 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.536760 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" event={"ID":"8adff4a5-5376-43ec-9b53-888705ae8236","Type":"ContainerDied","Data":"3904c49bb7ab63c515d01a6052936b8d990a8f9bb352ab2810819de54dc7b74c"} Nov 03 20:33:04.552787 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.539432 5059 generic.go:356] "Generic (PLEG): container finished" podID="705dea2b-ad85-4b1c-a3c6-a9f705ac9039" containerID="c1c608afee5d8326b20645c4ef715824387423915b54182a5c217c34ffe83cf2" exitCode=0 Nov 03 20:33:04.552787 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.539508 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8" event={"ID":"705dea2b-ad85-4b1c-a3c6-a9f705ac9039","Type":"ContainerDied","Data":"c1c608afee5d8326b20645c4ef715824387423915b54182a5c217c34ffe83cf2"} Nov 03 20:33:04.574710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.556106 5059 generic.go:356] "Generic (PLEG): container finished" podID="2faa14fa-ec86-4633-a968-c3f761d13443" containerID="d610230fd1e961d1b794119c73b53766e612967c8196f325df7359012d12af34" exitCode=0 Nov 03 20:33:04.574710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.556200 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps" event={"ID":"2faa14fa-ec86-4633-a968-c3f761d13443","Type":"ContainerDied","Data":"d610230fd1e961d1b794119c73b53766e612967c8196f325df7359012d12af34"} Nov 03 20:33:04.611058 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.609682 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c29d53f-b738-49ec-9b15-1c88e5950008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-gqz47\" (UID: \"6c29d53f-b738-49ec-9b15-1c88e5950008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47" Nov 03 20:33:04.611058 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.609897 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxxr7\" (UniqueName: \"kubernetes.io/projected/6c29d53f-b738-49ec-9b15-1c88e5950008-kube-api-access-rxxr7\") pod \"maas-api-revoke-sub-5bac6f1e-gqz47\" (UID: \"6c29d53f-b738-49ec-9b15-1c88e5950008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47" Nov 03 20:33:04.611058 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.609950 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c29d53f-b738-49ec-9b15-1c88e5950008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-gqz47\" (UID: \"6c29d53f-b738-49ec-9b15-1c88e5950008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47" Nov 03 20:33:04.613304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.612850 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c29d53f-b738-49ec-9b15-1c88e5950008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-gqz47\" (UID: \"6c29d53f-b738-49ec-9b15-1c88e5950008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47" Nov 03 20:33:04.642014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.638414 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxxr7\" (UniqueName: \"kubernetes.io/projected/6c29d53f-b738-49ec-9b15-1c88e5950008-kube-api-access-rxxr7\") pod \"maas-api-revoke-sub-5bac6f1e-gqz47\" (UID: \"6c29d53f-b738-49ec-9b15-1c88e5950008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47" Nov 03 20:33:04.654707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.653441 5059 generic.go:356] "Generic (PLEG): container finished" podID="c9721811-d27d-4e13-a53d-4ae81e16fb96" containerID="b4dc2ca466f30e8024a88160d858421a5e6e6e115a118109fb3f8569f83e3ae2" exitCode=0 Nov 03 20:33:04.654707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.653533 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" event={"ID":"c9721811-d27d-4e13-a53d-4ae81e16fb96","Type":"ContainerDied","Data":"b4dc2ca466f30e8024a88160d858421a5e6e6e115a118109fb3f8569f83e3ae2"} Nov 03 20:33:04.690819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.664601 5059 generic.go:356] "Generic (PLEG): container finished" podID="9d60eb56-9852-4c25-ae50-8a932e752793" containerID="1800f61ab969d2f54568614237d89ac56416a680367326b1f485f3183aebbfe9" exitCode=0 Nov 03 20:33:04.690819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.664702 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" event={"ID":"9d60eb56-9852-4c25-ae50-8a932e752793","Type":"ContainerDied","Data":"1800f61ab969d2f54568614237d89ac56416a680367326b1f485f3183aebbfe9"} Nov 03 20:33:04.718837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.700942 5059 generic.go:356] "Generic (PLEG): container finished" podID="eac98813-7b4c-4c4a-8bf9-444bba2fa644" containerID="f0ca6f95c31b4ccad9131b19d71014e3f54afa9b79e7f53f9d77d4b861fb6af9" exitCode=0 Nov 03 20:33:04.718837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.701029 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" event={"ID":"eac98813-7b4c-4c4a-8bf9-444bba2fa644","Type":"ContainerDied","Data":"f0ca6f95c31b4ccad9131b19d71014e3f54afa9b79e7f53f9d77d4b861fb6af9"} Nov 03 20:33:04.718837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.704712 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" event={"ID":"d931dc13-4399-48dc-80e0-b0b304c5dda0","Type":"ContainerStarted","Data":"51e5dd7d49f3191cbe26fa7fea29e5e5bd18af15599c15f1e53810285e8f9294"} Nov 03 20:33:04.718837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.706742 5059 generic.go:356] "Generic (PLEG): container finished" podID="fc57130b-c950-49f2-8a98-22c47bd98288" containerID="7598d2be586d16550aff9c80d082d04770669cb98c756f3811527543a1ac8d0f" exitCode=0 Nov 03 20:33:04.718837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.706845 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" event={"ID":"fc57130b-c950-49f2-8a98-22c47bd98288","Type":"ContainerDied","Data":"7598d2be586d16550aff9c80d082d04770669cb98c756f3811527543a1ac8d0f"} Nov 03 20:33:04.718837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.709495 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4" podStartSLOduration=39.709478844 podStartE2EDuration="39.709478844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:04.429631523 +0000 UTC m=+3378.687428780" watchObservedRunningTime="2026-11-03 20:33:04.709478844 +0000 UTC m=+3378.967276103" Nov 03 20:33:04.718837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.716880 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c29d53f-b738-49ec-9b15-1c88e5950008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-gqz47\" (UID: \"6c29d53f-b738-49ec-9b15-1c88e5950008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47" Nov 03 20:33:04.725638 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.720769 5059 generic.go:356] "Generic (PLEG): container finished" podID="b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" containerID="083bf41b34c7c17d7a323c71536ca53488c9b5d6024a331fe82c42a4a9894a43" exitCode=0 Nov 03 20:33:04.725638 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.720854 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" event={"ID":"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3","Type":"ContainerDied","Data":"083bf41b34c7c17d7a323c71536ca53488c9b5d6024a331fe82c42a4a9894a43"} Nov 03 20:33:04.730966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.728215 5059 generic.go:356] "Generic (PLEG): container finished" podID="7909b42c-1572-4fe2-816a-447020594fe4" containerID="4cc323b3fee429ec7bc1ab6e19bfbfe5f3926b292a99afff5f656543c7dd8e48" exitCode=0 Nov 03 20:33:04.730966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.728269 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" event={"ID":"7909b42c-1572-4fe2-816a-447020594fe4","Type":"ContainerDied","Data":"4cc323b3fee429ec7bc1ab6e19bfbfe5f3926b292a99afff5f656543c7dd8e48"} Nov 03 20:33:04.730966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.730615 5059 generic.go:356] "Generic (PLEG): container finished" podID="40b88c7d-08e9-4203-9ebd-2c9f69b884dc" containerID="6739ff178c81c67d823d4e8761f1f61d4d648a31a61e02195fb24f7ce8ca0bdf" exitCode=0 Nov 03 20:33:04.730966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.730894 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" event={"ID":"40b88c7d-08e9-4203-9ebd-2c9f69b884dc","Type":"ContainerDied","Data":"6739ff178c81c67d823d4e8761f1f61d4d648a31a61e02195fb24f7ce8ca0bdf"} Nov 03 20:33:04.738228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.732975 5059 generic.go:356] "Generic (PLEG): container finished" podID="05fceb9c-4a04-470f-a94a-055121319f82" containerID="8eba0c0d93805e32f585be9b5dffe0df1cf0f78d3928ac2e35a5a76d60ddd0e3" exitCode=0 Nov 03 20:33:04.738228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.733051 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" event={"ID":"05fceb9c-4a04-470f-a94a-055121319f82","Type":"ContainerDied","Data":"8eba0c0d93805e32f585be9b5dffe0df1cf0f78d3928ac2e35a5a76d60ddd0e3"} Nov 03 20:33:04.738228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.734819 5059 generic.go:356] "Generic (PLEG): container finished" podID="3b9d0847-df69-4b6a-b346-2cd206e42339" containerID="b5a63189fd2f2d22d3e6f03c32065113a3ccf85221d07eb70f923a647b5b8d38" exitCode=0 Nov 03 20:33:04.738228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.734877 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" event={"ID":"3b9d0847-df69-4b6a-b346-2cd206e42339","Type":"ContainerDied","Data":"b5a63189fd2f2d22d3e6f03c32065113a3ccf85221d07eb70f923a647b5b8d38"} Nov 03 20:33:04.738228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.737631 5059 generic.go:356] "Generic (PLEG): container finished" podID="14a06bdf-e15d-406c-bb76-47ae5dd06b2d" containerID="cdc8ec4afacb060d845fdcbcf9f9a36756ac0c3cf27302051a5a1dae638f96f6" exitCode=0 Nov 03 20:33:04.738228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.738208 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" event={"ID":"14a06bdf-e15d-406c-bb76-47ae5dd06b2d","Type":"ContainerDied","Data":"cdc8ec4afacb060d845fdcbcf9f9a36756ac0c3cf27302051a5a1dae638f96f6"} Nov 03 20:33:04.744255 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.742607 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8" event={"ID":"ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988","Type":"ContainerStarted","Data":"0aaedbc7dc2a65872890219273178402f55efd0bc2d6336437d7741bab1dffcb"} Nov 03 20:33:04.828713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.828631 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs"] Nov 03 20:33:04.844781 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.844649 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7"] Nov 03 20:33:04.849858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.849835 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:04.855413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.854567 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:04.869853 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.865746 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-nj52h"] Nov 03 20:33:04.914355 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.914295 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-nj52h"] Nov 03 20:33:04.928837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.928717 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs"] Nov 03 20:33:04.941653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:04.941569 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7"] Nov 03 20:33:05.025086 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.025024 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-tvb96"] Nov 03 20:33:05.027900 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.027823 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-tvb96"] Nov 03 20:33:05.049554 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.041252 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-workload-certs\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.053356 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.053331 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.053484 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.053470 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.053705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.053646 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.053979 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.053958 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-workload-socket\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.054110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.054095 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.054297 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.054278 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggjjf\" (UniqueName: \"kubernetes.io/projected/a2c58648-4022-4bfb-be9f-170a6222dfd8-kube-api-access-ggjjf\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.054474 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.054444 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.054654 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.054638 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8vhp\" (UniqueName: \"kubernetes.io/projected/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-kube-api-access-r8vhp\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.054751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.047963 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-brvgz"] Nov 03 20:33:05.054946 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.054783 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-token\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.055074 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.055058 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-credential-socket\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.055241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.055227 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.055331 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.055316 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c58648-4022-4bfb-be9f-170a6222dfd8-istiod-ca-cert\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.055451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.055434 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-podinfo\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.055577 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.055563 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-envoy\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.055759 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.055739 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.056012 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.055970 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.056113 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.056099 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-data\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.056512 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.056489 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-brvgz"] Nov 03 20:33:05.121382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.117330 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h"] Nov 03 20:33:05.126372 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.126338 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h"] Nov 03 20:33:05.126483 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.126450 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h" Nov 03 20:33:05.139954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.138439 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66b59d468d-5bfd2"] Nov 03 20:33:05.143378 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.143328 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:33:05.144448 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.144427 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body= Nov 03 20:33:05.144573 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.144461 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" podUID="d931dc13-4399-48dc-80e0-b0b304c5dda0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" Nov 03 20:33:05.144692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.144646 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-66b59d468d-5bfd2"] Nov 03 20:33:05.148805 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.148784 5059 scope.go:122] "RemoveContainer" containerID="9a314ded87df57e4578fee99a500049875bfdb3f70a6196625866f2a41d5c267" Nov 03 20:33:05.150605 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.150557 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:05.150762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.150731 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:05.150859 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.150835 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:05.197792 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.197772 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fsbsp"] Nov 03 20:33:05.205551 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.205524 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fsbsp"] Nov 03 20:33:05.213140 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.213105 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-workload-socket\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.213254 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.213163 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.213307 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.213248 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggjjf\" (UniqueName: \"kubernetes.io/projected/a2c58648-4022-4bfb-be9f-170a6222dfd8-kube-api-access-ggjjf\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.213370 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.213314 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.213424 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.213395 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8vhp\" (UniqueName: \"kubernetes.io/projected/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-kube-api-access-r8vhp\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.213473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.213434 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28034528-e342-417a-a9df-050d7ea91453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb827799-ndv2h\" (UID: \"28034528-e342-417a-a9df-050d7ea91453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h" Nov 03 20:33:05.213526 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.213467 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-token\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.213526 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.213506 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-credential-socket\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.213636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.213613 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.213815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.213793 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c58648-4022-4bfb-be9f-170a6222dfd8-istiod-ca-cert\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.213965 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.213947 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-podinfo\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.214120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.214104 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-envoy\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.214319 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.214300 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.214438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.214423 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28034528-e342-417a-a9df-050d7ea91453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb827799-ndv2h\" (UID: \"28034528-e342-417a-a9df-050d7ea91453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h" Nov 03 20:33:05.214713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.214692 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.214846 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.214830 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-data\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.215045 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.215027 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-workload-certs\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.215187 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.215171 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.215312 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.215295 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.215513 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.215494 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmvvg\" (UniqueName: \"kubernetes.io/projected/28034528-e342-417a-a9df-050d7ea91453-kube-api-access-kmvvg\") pod \"maas-api-revoke-sub-fb827799-ndv2h\" (UID: \"28034528-e342-417a-a9df-050d7ea91453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h" Nov 03 20:33:05.216525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.215606 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-workload-socket\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.216525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.215653 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw"] Nov 03 20:33:05.216525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.215651 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.216525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.216002 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.217793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.217158 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.217793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.217393 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.217793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.217690 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-credential-socket\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.221443 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.219437 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-data\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.221443 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.220254 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-workload-certs\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.221443 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.220893 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:05.221443 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.221354 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c58648-4022-4bfb-be9f-170a6222dfd8-istiod-ca-cert\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.221700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.221545 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-envoy\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.222959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.222536 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.222959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.222641 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.222959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.222897 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.223806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.223721 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.235802 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:33:05.226716 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d89c5d_1730_4077_a442_d3924153133f.slice/crio-943a18316f71f4f98be3cf9cddcb4e3f27ad47f648f647d8c47512985cb630d2 WatchSource:0}: Error finding container 943a18316f71f4f98be3cf9cddcb4e3f27ad47f648f647d8c47512985cb630d2: Status 404 returned error can't find the container with id 943a18316f71f4f98be3cf9cddcb4e3f27ad47f648f647d8c47512985cb630d2 Nov 03 20:33:05.235802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.227557 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-slpzw"] Nov 03 20:33:05.235802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.228021 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-podinfo\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.235802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.231134 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5" podStartSLOduration=9.231117565 podStartE2EDuration="9.231117565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:04.211695695 +0000 UTC m=+3378.469492955" watchObservedRunningTime="2026-11-03 20:33:05.231117565 +0000 UTC m=+3379.488914823" Nov 03 20:33:05.237293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.237235 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8vhp\" (UniqueName: \"kubernetes.io/projected/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-kube-api-access-r8vhp\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.245324 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.241600 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-token\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.245650 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.245415 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-cmtzs\" (UID: \"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:05.247697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.247317 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggjjf\" (UniqueName: \"kubernetes.io/projected/a2c58648-4022-4bfb-be9f-170a6222dfd8-kube-api-access-ggjjf\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:05.250322 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.250081 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst" Nov 03 20:33:05.251725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.251507 5059 scope.go:122] "RemoveContainer" containerID="2f57092ac51573f4a21c3d1c3f99cddf2880faf5135c68f41abacbcf61bc2cfa" Nov 03 20:33:05.287737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.285940 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-hmc5z"] Nov 03 20:33:05.295435 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.295404 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-hmc5z"] Nov 03 20:33:05.319689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.319558 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmvvg\" (UniqueName: \"kubernetes.io/projected/28034528-e342-417a-a9df-050d7ea91453-kube-api-access-kmvvg\") pod \"maas-api-revoke-sub-fb827799-ndv2h\" (UID: \"28034528-e342-417a-a9df-050d7ea91453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h" Nov 03 20:33:05.320604 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.319927 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28034528-e342-417a-a9df-050d7ea91453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb827799-ndv2h\" (UID: \"28034528-e342-417a-a9df-050d7ea91453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h" Nov 03 20:33:05.320604 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.320212 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28034528-e342-417a-a9df-050d7ea91453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb827799-ndv2h\" (UID: \"28034528-e342-417a-a9df-050d7ea91453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h" Nov 03 20:33:05.322133 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.322090 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28034528-e342-417a-a9df-050d7ea91453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb827799-ndv2h\" (UID: \"28034528-e342-417a-a9df-050d7ea91453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h" Nov 03 20:33:05.323046 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.323023 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28034528-e342-417a-a9df-050d7ea91453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb827799-ndv2h\" (UID: \"28034528-e342-417a-a9df-050d7ea91453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h" Nov 03 20:33:05.337680 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.337637 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmvvg\" (UniqueName: \"kubernetes.io/projected/28034528-e342-417a-a9df-050d7ea91453-kube-api-access-kmvvg\") pod \"maas-api-revoke-sub-fb827799-ndv2h\" (UID: \"28034528-e342-417a-a9df-050d7ea91453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h" Nov 03 20:33:05.341728 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.341498 5059 scope.go:122] "RemoveContainer" containerID="3f07fce3b6aeb311a72b724e58639936894c5cc3f1b194d161e8c7b428a68157" Nov 03 20:33:05.358894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.358837 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84"] Nov 03 20:33:05.367194 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.367138 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-c2m84"] Nov 03 20:33:05.398082 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.396984 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t" podStartSLOduration=63.39696941 podStartE2EDuration="1m3.39696941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:04.865021884 +0000 UTC m=+3379.122819155" watchObservedRunningTime="2026-11-03 20:33:05.39696941 +0000 UTC m=+3379.654766666" Nov 03 20:33:05.415600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.415576 5059 scope.go:122] "RemoveContainer" containerID="6f8ee91ca327bbf5a0c3704b9e3ce856daebe10db7f411298f5e612e7a7a2a8f" Nov 03 20:33:05.418611 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.418589 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7"] Nov 03 20:33:05.423620 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.423596 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4"] Nov 03 20:33:05.428787 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.428736 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4"] Nov 03 20:33:05.434520 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.434438 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d9f95b69-9vwq7"] Nov 03 20:33:05.440426 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.440404 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j"] Nov 03 20:33:05.450813 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.450763 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:05.452301 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.452284 5059 scope.go:122] "RemoveContainer" containerID="efcfe340defa1e3223df36397fde5c79fbe4b5c7832e90583c68ba8293d3e314" Nov 03 20:33:05.452629 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.452572 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-lz2v2"] Nov 03 20:33:05.474227 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.474146 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-fzr95"] Nov 03 20:33:05.491702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.491267 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn" Nov 03 20:33:05.496278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.496116 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c" Nov 03 20:33:05.504635 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.504613 5059 scope.go:122] "RemoveContainer" containerID="6eff9a042806518844e626a38bbf88994cc517a23f4a2adcbbff67af78d07b64" Nov 03 20:33:05.520341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.520312 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6k9n9"] Nov 03 20:33:05.524380 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.524355 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6k9n9"] Nov 03 20:33:05.543250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.543225 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:05.544871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.544851 5059 scope.go:122] "RemoveContainer" containerID="45c3ba404ea0796dd18bc58ed44508e09f4ede955131e48779814d8f243781bd" Nov 03 20:33:05.546902 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.546876 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x"] Nov 03 20:33:05.551889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.551868 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:05.552881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.552829 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wh95x"] Nov 03 20:33:05.589200 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.589044 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:33:05.597493 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.597461 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:05.599800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.599774 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mm85p"] Nov 03 20:33:05.605102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.605085 5059 scope.go:122] "RemoveContainer" containerID="e696c1151a11239c9d6d9c393f5e0ca1dfa476058f6f2240ee0c35998ef6a3d3" Nov 03 20:33:05.607771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.607732 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mm85p"] Nov 03 20:33:05.650200 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.650020 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:05.733188 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.733157 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5"] Nov 03 20:33:05.742059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.742033 5059 scope.go:122] "RemoveContainer" containerID="083613d8d2a66dbf18be20acf40c9c2e55284f3701c7490d658626a11a1dff74" Nov 03 20:33:05.743948 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.743918 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-9b8p5"] Nov 03 20:33:05.799174 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.798205 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" event={"ID":"0f32ecf7-06f7-4098-9793-acf761a30c09","Type":"ContainerDied","Data":"58f0c433bf4f049cef0f62c8ae114894d5f9997d1c364fb2311ace0372d8fe70"} Nov 03 20:33:05.799174 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.798269 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58f0c433bf4f049cef0f62c8ae114894d5f9997d1c364fb2311ace0372d8fe70" Nov 03 20:33:05.800625 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.800598 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" event={"ID":"8153e018-7f9a-4645-b102-06ff3913f682","Type":"ContainerDied","Data":"93c48ae9de73afe2cd0d526c232d81ef7b3edc4ee33e2b54bfcc9f04887312a8"} Nov 03 20:33:05.800740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.800651 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93c48ae9de73afe2cd0d526c232d81ef7b3edc4ee33e2b54bfcc9f04887312a8" Nov 03 20:33:05.802916 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.802890 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls" event={"ID":"8760fa0e-a78b-4db8-b71d-bf4086ddbfbd","Type":"ContainerStarted","Data":"cef80931e6aff9be1967d6d3105b1fc9be14a0fbc1100b81828051cdb49d5711"} Nov 03 20:33:05.804404 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.804378 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls" Nov 03 20:33:05.806940 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.806862 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" event={"ID":"37e90ab0-e9c0-420c-a0f4-c7923aa00297","Type":"ContainerDied","Data":"1be2351effe8da8c655ee906e4857030a8714afdf23aa891ad55cfcc73d26f5d"} Nov 03 20:33:05.806940 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.806904 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1be2351effe8da8c655ee906e4857030a8714afdf23aa891ad55cfcc73d26f5d" Nov 03 20:33:05.809674 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.809567 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" event={"ID":"c2d80995-5f1c-46db-8d71-a130cb92e279","Type":"ContainerDied","Data":"7f4aac55cec05c0b92f14350c76f0428c0ca2ed33f0cd6f1220537cebab730b1"} Nov 03 20:33:05.809674 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.809607 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f4aac55cec05c0b92f14350c76f0428c0ca2ed33f0cd6f1220537cebab730b1" Nov 03 20:33:05.819096 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.819013 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" event={"ID":"34880975-a897-4ad4-af59-18617a93cdad","Type":"ContainerDied","Data":"2617ba973814cb2eac509072a9b1fdcceb26d9b42e09f34eb13eaf2e3aa6e749"} Nov 03 20:33:05.819096 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.819056 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2617ba973814cb2eac509072a9b1fdcceb26d9b42e09f34eb13eaf2e3aa6e749" Nov 03 20:33:05.821993 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.821935 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" event={"ID":"8adff4a5-5376-43ec-9b53-888705ae8236","Type":"ContainerDied","Data":"ca0dc432182cad3fc0a9b45ad885449be4d6c0dc42dd143b8c84906cb422e240"} Nov 03 20:33:05.821993 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.821981 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca0dc432182cad3fc0a9b45ad885449be4d6c0dc42dd143b8c84906cb422e240" Nov 03 20:33:05.826054 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.824295 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8" event={"ID":"705dea2b-ad85-4b1c-a3c6-a9f705ac9039","Type":"ContainerDied","Data":"47d752e074652163b0738c275ed94995338fa980323fecafc676c3c822967ee7"} Nov 03 20:33:05.826054 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.824333 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47d752e074652163b0738c275ed94995338fa980323fecafc676c3c822967ee7" Nov 03 20:33:05.826054 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.825983 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps" event={"ID":"2faa14fa-ec86-4633-a968-c3f761d13443","Type":"ContainerDied","Data":"c44513f85db26c84eec581e4101bcb4caf3fe658b6bb76d6c6fe7f404623dd5d"} Nov 03 20:33:05.826054 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.826019 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c44513f85db26c84eec581e4101bcb4caf3fe658b6bb76d6c6fe7f404623dd5d" Nov 03 20:33:05.828898 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.828839 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" event={"ID":"c9721811-d27d-4e13-a53d-4ae81e16fb96","Type":"ContainerDied","Data":"d7ff0c8bb110c798a57bd78b4c449469b094a557594b7ef3ed9540f8a707fec2"} Nov 03 20:33:05.828898 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.828870 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7ff0c8bb110c798a57bd78b4c449469b094a557594b7ef3ed9540f8a707fec2" Nov 03 20:33:05.839193 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.839099 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" event={"ID":"9d60eb56-9852-4c25-ae50-8a932e752793","Type":"ContainerDied","Data":"b8f814c27581a58bb22d904094ae0a72ad87eb9aa97e7d3369103603db7a97ce"} Nov 03 20:33:05.839193 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.839150 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8f814c27581a58bb22d904094ae0a72ad87eb9aa97e7d3369103603db7a97ce" Nov 03 20:33:05.843677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.841380 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" podStartSLOduration=63.841350006 podStartE2EDuration="1m3.841350006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:05.808195159 +0000 UTC m=+3380.065992416" watchObservedRunningTime="2026-11-03 20:33:05.841350006 +0000 UTC m=+3380.099147263" Nov 03 20:33:05.846955 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.846858 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" event={"ID":"eac98813-7b4c-4c4a-8bf9-444bba2fa644","Type":"ContainerDied","Data":"139eb572a391573df18b064fe99053fa177236d3732ef6a9676a36d718e52a4a"} Nov 03 20:33:05.846955 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.846911 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="139eb572a391573df18b064fe99053fa177236d3732ef6a9676a36d718e52a4a" Nov 03 20:33:05.912764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.909768 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a985b50-c4c0-4a39-abed-29f53e2be747" path="/var/lib/kubelet/pods/0a985b50-c4c0-4a39-abed-29f53e2be747/volumes" Nov 03 20:33:05.912764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.910807 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1072628f-f513-4b4e-8954-afee5a1aa733" path="/var/lib/kubelet/pods/1072628f-f513-4b4e-8954-afee5a1aa733/volumes" Nov 03 20:33:05.912764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.911550 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bcd4690-4753-4246-abb4-8668a0a56231" path="/var/lib/kubelet/pods/1bcd4690-4753-4246-abb4-8668a0a56231/volumes" Nov 03 20:33:05.912764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.912201 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="270c522c-ef72-46a7-9e65-3be2ebdbcfa6" path="/var/lib/kubelet/pods/270c522c-ef72-46a7-9e65-3be2ebdbcfa6/volumes" Nov 03 20:33:05.917965 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.917337 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls" podStartSLOduration=63.917320452 podStartE2EDuration="1m3.917320452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:05.91133668 +0000 UTC m=+3380.169133939" watchObservedRunningTime="2026-11-03 20:33:05.917320452 +0000 UTC m=+3380.175117739" Nov 03 20:33:05.936194 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.936163 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="492c7b4e-84c0-49a7-b12f-153a021f971b" path="/var/lib/kubelet/pods/492c7b4e-84c0-49a7-b12f-153a021f971b/volumes" Nov 03 20:33:05.937306 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.937283 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a35329c-82fb-479f-abca-bff9ed082558" path="/var/lib/kubelet/pods/4a35329c-82fb-479f-abca-bff9ed082558/volumes" Nov 03 20:33:05.941624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.941498 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:33:05.941624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.941570 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" podUID="2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:05.944868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.944842 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="532de14d-f2dd-4c66-ba6f-8b103f7c9375" path="/var/lib/kubelet/pods/532de14d-f2dd-4c66-ba6f-8b103f7c9375/volumes" Nov 03 20:33:05.968424 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.956390 5059 scope.go:122] "RemoveContainer" containerID="f0c0cf57d91e43bac1c932a3dc7cf04c1c2e7f51acbb2554396c4f939b1e52cd" Nov 03 20:33:05.968424 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.960744 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6785e24f-6fb3-43ab-a469-84d30d94b2f0" path="/var/lib/kubelet/pods/6785e24f-6fb3-43ab-a469-84d30d94b2f0/volumes" Nov 03 20:33:05.968424 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.964900 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b8992fc-fba6-4135-90f8-fb259e765dd5" path="/var/lib/kubelet/pods/7b8992fc-fba6-4135-90f8-fb259e765dd5/volumes" Nov 03 20:33:05.968424 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.965313 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcrh4" Nov 03 20:33:05.968424 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.966340 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="941b16dd-0550-4434-86f9-26e38efb68ee" path="/var/lib/kubelet/pods/941b16dd-0550-4434-86f9-26e38efb68ee/volumes" Nov 03 20:33:05.981721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.981692 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad491ce3-2f92-45a7-9b4f-278309989e64" path="/var/lib/kubelet/pods/ad491ce3-2f92-45a7-9b4f-278309989e64/volumes" Nov 03 20:33:05.982851 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:05.982829 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7fce352-1850-4048-be2e-fa458fd4d2f3" path="/var/lib/kubelet/pods/c7fce352-1850-4048-be2e-fa458fd4d2f3/volumes" Nov 03 20:33:06.041908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.041875 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f485893c-6f14-449b-b837-8ac181315aeb" path="/var/lib/kubelet/pods/f485893c-6f14-449b-b837-8ac181315aeb/volumes" Nov 03 20:33:06.043651 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.043628 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8b140c9-9e09-4cbd-847b-82ee1e3be6ac" path="/var/lib/kubelet/pods/f8b140c9-9e09-4cbd-847b-82ee1e3be6ac/volumes" Nov 03 20:33:06.044329 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.044307 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q" Nov 03 20:33:06.044724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.044697 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" event={"ID":"fc57130b-c950-49f2-8a98-22c47bd98288","Type":"ContainerDied","Data":"55e7a86a0d53f29a6fa9de295bc44317de3712a5ba1053f7449d5d580aac5ed8"} Nov 03 20:33:06.044867 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.044852 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55e7a86a0d53f29a6fa9de295bc44317de3712a5ba1053f7449d5d580aac5ed8" Nov 03 20:33:06.044947 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.044935 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" event={"ID":"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840","Type":"ContainerDied","Data":"a2e2318cab978244150236d259a7997e616a413482cbe8845c7e1d51252684fb"} Nov 03 20:33:06.045031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.045020 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2e2318cab978244150236d259a7997e616a413482cbe8845c7e1d51252684fb" Nov 03 20:33:06.045110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.045099 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" event={"ID":"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3","Type":"ContainerDied","Data":"d978f9fb5358f9acf3231902c969f661ecdcd235d6162a4233dd2e7376cf7eb7"} Nov 03 20:33:06.045201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.045188 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d978f9fb5358f9acf3231902c969f661ecdcd235d6162a4233dd2e7376cf7eb7" Nov 03 20:33:06.129393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.129347 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" event={"ID":"7909b42c-1572-4fe2-816a-447020594fe4","Type":"ContainerDied","Data":"0f34b01f77f5c5dfd37d65b1c806d09a59df2776b37e895036af224434dd2faa"} Nov 03 20:33:06.129612 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.129591 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f34b01f77f5c5dfd37d65b1c806d09a59df2776b37e895036af224434dd2faa" Nov 03 20:33:06.132155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.132132 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 20:33:06.133632 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.133606 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" event={"ID":"08b3a993-1a64-4fe5-9577-b0f11a36779e","Type":"ContainerDied","Data":"945b49aab43d7640ffe6f9eabe8651bd8c94dc095e160eb5fb327ca5d696ada1"} Nov 03 20:33:06.134081 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.134056 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="945b49aab43d7640ffe6f9eabe8651bd8c94dc095e160eb5fb327ca5d696ada1" Nov 03 20:33:06.134881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.134862 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 20:33:06.137831 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.137811 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" event={"ID":"f88ae6f3-f1d6-4aa3-ad33-589490696167","Type":"ContainerDied","Data":"307295e491d089d6a319631a56952ba3f148983cbd35b208c9b1fa12d1f37be9"} Nov 03 20:33:06.139076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.139056 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="307295e491d089d6a319631a56952ba3f148983cbd35b208c9b1fa12d1f37be9" Nov 03 20:33:06.139400 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.139364 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-g6xz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:33:06.139533 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.139513 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" podUID="0f32ecf7-06f7-4098-9793-acf761a30c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:06.143822 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.143796 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" event={"ID":"40b88c7d-08e9-4203-9ebd-2c9f69b884dc","Type":"ContainerDied","Data":"d5fbecd3af8991e6c4ec43085889d7697d506a848fcc7a66a3f398d621b52d17"} Nov 03 20:33:06.143970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.143953 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5fbecd3af8991e6c4ec43085889d7697d506a848fcc7a66a3f398d621b52d17" Nov 03 20:33:06.151823 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.145797 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body= Nov 03 20:33:06.166063 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.157061 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" podUID="d931dc13-4399-48dc-80e0-b0b304c5dda0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" Nov 03 20:33:06.230309 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.208852 5059 scope.go:122] "RemoveContainer" containerID="01046b5c62acd48be6ebabf851d13a922cd16c44da94bb90f68ca3cae9c5ee8f" Nov 03 20:33:06.230309 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.229468 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw"] Nov 03 20:33:06.230934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.230878 5059 cpu_manager.go:414] "containerMap: removing container" podUID="70d45aa9-bfa8-43c6-8674-6f8f14a3097e" containerName="istio-proxy" Nov 03 20:33:06.230934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.230896 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70d45aa9-bfa8-43c6-8674-6f8f14a3097e" containerName="istio-proxy" Nov 03 20:33:06.231091 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.230941 5059 cpu_manager.go:414] "containerMap: removing container" podUID="edcfb7ab-b14f-4d5c-b752-5b43a6eee163" containerName="istio-proxy" Nov 03 20:33:06.231091 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.230950 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edcfb7ab-b14f-4d5c-b752-5b43a6eee163" containerName="istio-proxy" Nov 03 20:33:06.242991 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.231256 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="edcfb7ab-b14f-4d5c-b752-5b43a6eee163" containerName="istio-proxy" Nov 03 20:33:06.242991 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.231295 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="70d45aa9-bfa8-43c6-8674-6f8f14a3097e" containerName="istio-proxy" Nov 03 20:33:06.251247 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.251220 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw"] Nov 03 20:33:06.251548 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.251523 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lz2v2" event={"ID":"b4d89c5d-1730-4077-a442-d3924153133f","Type":"ContainerStarted","Data":"943a18316f71f4f98be3cf9cddcb4e3f27ad47f648f647d8c47512985cb630d2"} Nov 03 20:33:06.251813 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.251785 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.258179 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.256389 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:33:06.258179 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.256438 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" podUID="a594e899-889e-4f7f-a8ba-b7e7a389ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:06.307806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.299884 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:33:06.307806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.301742 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:33:06.307806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.303346 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:33:06.307806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.304908 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 20:33:06.307806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.306490 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:33:06.315682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.313336 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-podinfo\") pod \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " Nov 03 20:33:06.315682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.313381 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-credential-socket\") pod \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " Nov 03 20:33:06.315682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.313419 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-token\" (UniqueName: \"kubernetes.io/projected/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-token\") pod \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " Nov 03 20:33:06.315682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.313554 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istiod-ca-cert\") pod \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " Nov 03 20:33:06.315682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.313612 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-data\") pod \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " Nov 03 20:33:06.315682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.313728 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-kube-api-access-lnfhk\" (UniqueName: \"kubernetes.io/projected/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-kube-api-access-lnfhk\") pod \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " Nov 03 20:33:06.315682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.313798 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-envoy\") pod \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " Nov 03 20:33:06.315682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.313866 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-envoy\") pod \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " Nov 03 20:33:06.315682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.313908 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-credential-socket\") pod \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " Nov 03 20:33:06.315682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.314014 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-podinfo\") pod \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " Nov 03 20:33:06.315682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.314091 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-workload-socket\") pod \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " Nov 03 20:33:06.315682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.314210 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-workload-certs\") pod \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " Nov 03 20:33:06.315682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.314280 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-workload-socket\") pod \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\" (UID: \"edcfb7ab-b14f-4d5c-b752-5b43a6eee163\") " Nov 03 20:33:06.315682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.314348 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-workload-certs\") pod \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " Nov 03 20:33:06.315682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.314385 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-token\" (UniqueName: \"kubernetes.io/projected/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-token\") pod \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " Nov 03 20:33:06.315682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.314423 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-data\") pod \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " Nov 03 20:33:06.315682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.314496 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istiod-ca-cert\") pod \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " Nov 03 20:33:06.315682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.314580 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-kube-api-access-kdlh2\" (UniqueName: \"kubernetes.io/projected/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-kube-api-access-kdlh2\") pod \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\" (UID: \"70d45aa9-bfa8-43c6-8674-6f8f14a3097e\") " Nov 03 20:33:06.324076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.319579 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-credential-socket" pod "70d45aa9-bfa8-43c6-8674-6f8f14a3097e" (UID: "70d45aa9-bfa8-43c6-8674-6f8f14a3097e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.324076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.320138 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-workload-socket" pod "70d45aa9-bfa8-43c6-8674-6f8f14a3097e" (UID: "70d45aa9-bfa8-43c6-8674-6f8f14a3097e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.324076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.320369 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-workload-certs" pod "edcfb7ab-b14f-4d5c-b752-5b43a6eee163" (UID: "edcfb7ab-b14f-4d5c-b752-5b43a6eee163"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.324076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.320999 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" event={"ID":"05fceb9c-4a04-470f-a94a-055121319f82","Type":"ContainerDied","Data":"9f155b483f4b2676e0c5a139fb683e66a580a3b42972f576a5c67b9f37065425"} Nov 03 20:33:06.324076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.321049 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f155b483f4b2676e0c5a139fb683e66a580a3b42972f576a5c67b9f37065425" Nov 03 20:33:06.324076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.323590 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-podinfo" pod "edcfb7ab-b14f-4d5c-b752-5b43a6eee163" (UID: "edcfb7ab-b14f-4d5c-b752-5b43a6eee163"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:06.324076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.323648 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-credential-socket" pod "edcfb7ab-b14f-4d5c-b752-5b43a6eee163" (UID: "edcfb7ab-b14f-4d5c-b752-5b43a6eee163"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.324076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.323647 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-token" pod "edcfb7ab-b14f-4d5c-b752-5b43a6eee163" (UID: "edcfb7ab-b14f-4d5c-b752-5b43a6eee163"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.324076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.323908 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-workload-socket" pod "edcfb7ab-b14f-4d5c-b752-5b43a6eee163" (UID: "edcfb7ab-b14f-4d5c-b752-5b43a6eee163"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.324076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.323981 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-podinfo" pod "70d45aa9-bfa8-43c6-8674-6f8f14a3097e" (UID: "70d45aa9-bfa8-43c6-8674-6f8f14a3097e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.325729 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-workload-certs" pod "70d45aa9-bfa8-43c6-8674-6f8f14a3097e" (UID: "70d45aa9-bfa8-43c6-8674-6f8f14a3097e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.326186 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g"] Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.326428 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istiod-ca-cert" pod "edcfb7ab-b14f-4d5c-b752-5b43a6eee163" (UID: "edcfb7ab-b14f-4d5c-b752-5b43a6eee163"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.326542 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-token" pod "70d45aa9-bfa8-43c6-8674-6f8f14a3097e" (UID: "70d45aa9-bfa8-43c6-8674-6f8f14a3097e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.327610 5059 cpu_manager.go:414] "containerMap: removing container" podUID="37e90ab0-e9c0-420c-a0f4-c7923aa00297" containerName="istio-proxy" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.327637 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37e90ab0-e9c0-420c-a0f4-c7923aa00297" containerName="istio-proxy" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.327676 5059 cpu_manager.go:414] "containerMap: removing container" podUID="d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" containerName="istio-proxy" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.327685 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" containerName="istio-proxy" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.327738 5059 cpu_manager.go:414] "containerMap: removing container" podUID="518169ac-2f8d-4645-ae47-210b67ca080a" containerName="istio-proxy" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.327748 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="518169ac-2f8d-4645-ae47-210b67ca080a" containerName="istio-proxy" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.327769 5059 cpu_manager.go:414] "containerMap: removing container" podUID="518169ac-2f8d-4645-ae47-210b67ca080a" containerName="istio-proxy" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.327777 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="518169ac-2f8d-4645-ae47-210b67ca080a" containerName="istio-proxy" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.327785 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-data" pod "70d45aa9-bfa8-43c6-8674-6f8f14a3097e" (UID: "70d45aa9-bfa8-43c6-8674-6f8f14a3097e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.327814 5059 cpu_manager.go:414] "containerMap: removing container" podUID="8b648c6b-1f7d-42ca-9c3f-a435494bad03" containerName="istio-proxy" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.327822 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b648c6b-1f7d-42ca-9c3f-a435494bad03" containerName="istio-proxy" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.327841 5059 cpu_manager.go:414] "containerMap: removing container" podUID="f88ae6f3-f1d6-4aa3-ad33-589490696167" containerName="istio-proxy" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.327849 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f88ae6f3-f1d6-4aa3-ad33-589490696167" containerName="istio-proxy" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.328056 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="37e90ab0-e9c0-420c-a0f4-c7923aa00297" containerName="istio-proxy" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.328076 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" containerName="istio-proxy" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.328098 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="518169ac-2f8d-4645-ae47-210b67ca080a" containerName="istio-proxy" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.328111 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="518169ac-2f8d-4645-ae47-210b67ca080a" containerName="istio-proxy" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.328129 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b648c6b-1f7d-42ca-9c3f-a435494bad03" containerName="istio-proxy" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.328138 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b648c6b-1f7d-42ca-9c3f-a435494bad03" containerName="istio-proxy" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.328147 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f88ae6f3-f1d6-4aa3-ad33-589490696167" containerName="istio-proxy" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.328596 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-data" pod "edcfb7ab-b14f-4d5c-b752-5b43a6eee163" (UID: "edcfb7ab-b14f-4d5c-b752-5b43a6eee163"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.330195 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.333160 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istiod-ca-cert" pod "70d45aa9-bfa8-43c6-8674-6f8f14a3097e" (UID: "70d45aa9-bfa8-43c6-8674-6f8f14a3097e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.335336 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.335360 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.335374 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.335388 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.335401 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.335415 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.335428 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.335717 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.335735 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.335750 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.335768 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.335785 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.335800 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.340338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.335814 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.343213 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.343176 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9" Nov 03 20:33:06.347996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.347271 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g"] Nov 03 20:33:06.347996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.347330 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" event={"ID":"3b9d0847-df69-4b6a-b346-2cd206e42339","Type":"ContainerDied","Data":"b129ab4014dd05316589eb057a3b903e1c83703402f17902eb6f41cc97ab87a3"} Nov 03 20:33:06.347996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.347375 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" event={"ID":"14a06bdf-e15d-406c-bb76-47ae5dd06b2d","Type":"ContainerDied","Data":"145f74cb397cea989e22d486f3d9a839d7f366e5a050980469a0ae282b447e35"} Nov 03 20:33:06.347996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.347401 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="145f74cb397cea989e22d486f3d9a839d7f366e5a050980469a0ae282b447e35" Nov 03 20:33:06.347996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.347480 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.350923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.350877 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:33:06.353722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.352626 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8" event={"ID":"ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988","Type":"ContainerStarted","Data":"314d646649bd50357a3898f7794abc5ffb6d6c3c1a641d460a6c9fc3147b1d2c"} Nov 03 20:33:06.353722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.353643 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8" Nov 03 20:33:06.357223 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.356201 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h" event={"ID":"2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1","Type":"ContainerStarted","Data":"d40d4645400fbdd61191d0029f2cb502ccdd5240fb067cb03ff331ea2b1ebb45"} Nov 03 20:33:06.357223 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.356246 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h" event={"ID":"2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1","Type":"ContainerStarted","Data":"b3c1602d6af6a906f6869be53ebabaedf926eb70cc1fd5c92e659b441752317a"} Nov 03 20:33:06.357223 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.357162 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h" Nov 03 20:33:06.372038 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.372010 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:33:06.397851 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.389092 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-kube-api-access-lnfhk" pod "edcfb7ab-b14f-4d5c-b752-5b43a6eee163" (UID: "edcfb7ab-b14f-4d5c-b752-5b43a6eee163"). InnerVolumeSpecName "kube-api-access-lnfhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.402812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.402761 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-envoy" pod "edcfb7ab-b14f-4d5c-b752-5b43a6eee163" (UID: "edcfb7ab-b14f-4d5c-b752-5b43a6eee163"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.409870 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.407061 5059 scope.go:122] "RemoveContainer" containerID="073b6545188a48162ae3968a113b770a3a9d7b5615b65392e4809e1551e74cc2" Nov 03 20:33:06.409870 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.407397 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-kube-api-access-kdlh2" pod "70d45aa9-bfa8-43c6-8674-6f8f14a3097e" (UID: "70d45aa9-bfa8-43c6-8674-6f8f14a3097e"). InnerVolumeSpecName "kube-api-access-kdlh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.414776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.410220 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth"] Nov 03 20:33:06.414776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.411515 5059 cpu_manager.go:414] "containerMap: removing container" podUID="8b648c6b-1f7d-42ca-9c3f-a435494bad03" containerName="istio-proxy" Nov 03 20:33:06.414776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.411533 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b648c6b-1f7d-42ca-9c3f-a435494bad03" containerName="istio-proxy" Nov 03 20:33:06.414776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.411549 5059 cpu_manager.go:414] "containerMap: removing container" podUID="3b9d0847-df69-4b6a-b346-2cd206e42339" containerName="istio-proxy" Nov 03 20:33:06.414776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.411557 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b9d0847-df69-4b6a-b346-2cd206e42339" containerName="istio-proxy" Nov 03 20:33:06.414776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.411578 5059 cpu_manager.go:414] "containerMap: removing container" podUID="3b9d0847-df69-4b6a-b346-2cd206e42339" containerName="istio-proxy" Nov 03 20:33:06.414776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.411585 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b9d0847-df69-4b6a-b346-2cd206e42339" containerName="istio-proxy" Nov 03 20:33:06.414776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.411594 5059 cpu_manager.go:414] "containerMap: removing container" podUID="8adff4a5-5376-43ec-9b53-888705ae8236" containerName="istio-proxy" Nov 03 20:33:06.414776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.411602 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8adff4a5-5376-43ec-9b53-888705ae8236" containerName="istio-proxy" Nov 03 20:33:06.414776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.411615 5059 cpu_manager.go:414] "containerMap: removing container" podUID="d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" containerName="istio-proxy" Nov 03 20:33:06.414776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.411622 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" containerName="istio-proxy" Nov 03 20:33:06.414776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.411639 5059 cpu_manager.go:414] "containerMap: removing container" podUID="890f130b-cdeb-43da-acd4-1654673271cc" containerName="payload-processing" Nov 03 20:33:06.414776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.411646 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="890f130b-cdeb-43da-acd4-1654673271cc" containerName="payload-processing" Nov 03 20:33:06.414776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.411687 5059 cpu_manager.go:414] "containerMap: removing container" podUID="05fceb9c-4a04-470f-a94a-055121319f82" containerName="istio-proxy" Nov 03 20:33:06.414776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.411694 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05fceb9c-4a04-470f-a94a-055121319f82" containerName="istio-proxy" Nov 03 20:33:06.414776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.411883 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b9d0847-df69-4b6a-b346-2cd206e42339" containerName="istio-proxy" Nov 03 20:33:06.414776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.411903 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b9d0847-df69-4b6a-b346-2cd206e42339" containerName="istio-proxy" Nov 03 20:33:06.414776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.411914 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8adff4a5-5376-43ec-9b53-888705ae8236" containerName="istio-proxy" Nov 03 20:33:06.414776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.411928 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" containerName="istio-proxy" Nov 03 20:33:06.414776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.411939 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="890f130b-cdeb-43da-acd4-1654673271cc" containerName="payload-processing" Nov 03 20:33:06.414776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.411950 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="05fceb9c-4a04-470f-a94a-055121319f82" containerName="istio-proxy" Nov 03 20:33:06.414776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.413138 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-envoy" pod "70d45aa9-bfa8-43c6-8674-6f8f14a3097e" (UID: "70d45aa9-bfa8-43c6-8674-6f8f14a3097e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.419505 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.418961 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9" Nov 03 20:33:06.431213 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.431192 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 20:33:06.434693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.434656 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.439961 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth"] Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.440001 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" event={"ID":"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b","Type":"ContainerDied","Data":"7e98325901ed5796a82a8f7bf31137a3ae839778e2b1640d33607242ae0eec8f"} Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.440035 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e98325901ed5796a82a8f7bf31137a3ae839778e2b1640d33607242ae0eec8f" Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.440048 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9" event={"ID":"890f130b-cdeb-43da-acd4-1654673271cc","Type":"ContainerDied","Data":"ef2df6bf30fd3d11eec890703aa6deaf10a3cd2e8b8333e43dae23ca4c91de09"} Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.440136 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.441109 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-workload-certs\") pod \"3b9d0847-df69-4b6a-b346-2cd206e42339\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.441161 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-data\") pod \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.441190 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-token\" (UniqueName: \"kubernetes.io/projected/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-token\") pod \"f88ae6f3-f1d6-4aa3-ad33-589490696167\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.441233 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-istio-data\") pod \"518169ac-2f8d-4645-ae47-210b67ca080a\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.441271 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-data\") pod \"f88ae6f3-f1d6-4aa3-ad33-589490696167\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.441304 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-workload-certs\") pod \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.441332 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b648c6b-1f7d-42ca-9c3f-a435494bad03-kube-api-access-7qcvm\" (UniqueName: \"kubernetes.io/projected/8b648c6b-1f7d-42ca-9c3f-a435494bad03-kube-api-access-7qcvm\") pod \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.441413 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-credential-socket\") pod \"f88ae6f3-f1d6-4aa3-ad33-589490696167\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.441447 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-podinfo\") pod \"3b9d0847-df69-4b6a-b346-2cd206e42339\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.441477 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37e90ab0-e9c0-420c-a0f4-c7923aa00297-kube-api-access-7xfpq\" (UniqueName: \"kubernetes.io/projected/37e90ab0-e9c0-420c-a0f4-c7923aa00297-kube-api-access-7xfpq\") pod \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.441512 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-workload-certs\") pod \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.441547 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/518169ac-2f8d-4645-ae47-210b67ca080a-istio-token\" (UniqueName: \"kubernetes.io/projected/518169ac-2f8d-4645-ae47-210b67ca080a-istio-token\") pod \"518169ac-2f8d-4645-ae47-210b67ca080a\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.441578 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f88ae6f3-f1d6-4aa3-ad33-589490696167-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f88ae6f3-f1d6-4aa3-ad33-589490696167-istiod-ca-cert\") pod \"f88ae6f3-f1d6-4aa3-ad33-589490696167\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.441620 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-workload-socket\") pod \"518169ac-2f8d-4645-ae47-210b67ca080a\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.441721 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f88ae6f3-f1d6-4aa3-ad33-589490696167-kube-api-access-vkjmc\" (UniqueName: \"kubernetes.io/projected/f88ae6f3-f1d6-4aa3-ad33-589490696167-kube-api-access-vkjmc\") pod \"f88ae6f3-f1d6-4aa3-ad33-589490696167\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.441756 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-envoy\") pod \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.441788 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-data\") pod \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.441857 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/518169ac-2f8d-4645-ae47-210b67ca080a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/518169ac-2f8d-4645-ae47-210b67ca080a-istio-podinfo\") pod \"518169ac-2f8d-4645-ae47-210b67ca080a\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.441925 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-podinfo\") pod \"f88ae6f3-f1d6-4aa3-ad33-589490696167\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.441954 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istiod-ca-cert\") pod \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.441985 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-envoy\") pod \"f88ae6f3-f1d6-4aa3-ad33-589490696167\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.442017 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istiod-ca-cert\") pod \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.442049 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/518169ac-2f8d-4645-ae47-210b67ca080a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/518169ac-2f8d-4645-ae47-210b67ca080a-istiod-ca-cert\") pod \"518169ac-2f8d-4645-ae47-210b67ca080a\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.442169 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-workload-socket\") pod \"3b9d0847-df69-4b6a-b346-2cd206e42339\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.442252 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-envoy\") pod \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.442294 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-token\" (UniqueName: \"kubernetes.io/projected/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-token\") pod \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.442321 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-data\") pod \"3b9d0847-df69-4b6a-b346-2cd206e42339\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.442400 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-token\" (UniqueName: \"kubernetes.io/projected/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-token\") pod \"3b9d0847-df69-4b6a-b346-2cd206e42339\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.442521 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-podinfo\") pod \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.442552 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-istio-envoy\") pod \"518169ac-2f8d-4645-ae47-210b67ca080a\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.442584 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-token\" (UniqueName: \"kubernetes.io/projected/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-token\") pod \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.442624 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-workload-certs\") pod \"f88ae6f3-f1d6-4aa3-ad33-589490696167\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.442706 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-credential-socket\") pod \"3b9d0847-df69-4b6a-b346-2cd206e42339\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.442753 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-credential-socket\") pod \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " Nov 03 20:33:06.479913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.442788 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-podinfo\") pod \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.442841 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-data\") pod \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.442918 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-token\" (UniqueName: \"kubernetes.io/projected/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-token\") pod \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.442964 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-credential-socket\") pod \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.443010 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-credential-socket\") pod \"518169ac-2f8d-4645-ae47-210b67ca080a\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.443070 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istiod-ca-cert\") pod \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.443104 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-data" pod "f88ae6f3-f1d6-4aa3-ad33-589490696167" (UID: "f88ae6f3-f1d6-4aa3-ad33-589490696167"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.443157 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-envoy\") pod \"3b9d0847-df69-4b6a-b346-2cd206e42339\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.443191 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-podinfo\") pod \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.443308 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/518169ac-2f8d-4645-ae47-210b67ca080a-kube-api-access-fqlb8\" (UniqueName: \"kubernetes.io/projected/518169ac-2f8d-4645-ae47-210b67ca080a-kube-api-access-fqlb8\") pod \"518169ac-2f8d-4645-ae47-210b67ca080a\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.443338 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-envoy\") pod \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.443381 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-workload-socket\") pod \"f88ae6f3-f1d6-4aa3-ad33-589490696167\" (UID: \"f88ae6f3-f1d6-4aa3-ad33-589490696167\") " Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.443418 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-credential-socket\") pod \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.443487 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-workload-socket\") pod \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.443546 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-workload-certs\") pod \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.443618 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b9d0847-df69-4b6a-b346-2cd206e42339-kube-api-access-6zzxr\" (UniqueName: \"kubernetes.io/projected/3b9d0847-df69-4b6a-b346-2cd206e42339-kube-api-access-6zzxr\") pod \"3b9d0847-df69-4b6a-b346-2cd206e42339\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.443648 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-workload-socket\") pod \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\" (UID: \"37e90ab0-e9c0-420c-a0f4-c7923aa00297\") " Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.443698 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-workload-socket\") pod \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\" (UID: \"8b648c6b-1f7d-42ca-9c3f-a435494bad03\") " Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.443758 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-kube-api-access-sq82f\" (UniqueName: \"kubernetes.io/projected/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-kube-api-access-sq82f\") pod \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\" (UID: \"d912f1fd-b76b-4bb6-a5d9-ccc8b4536840\") " Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.443813 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b9d0847-df69-4b6a-b346-2cd206e42339-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b9d0847-df69-4b6a-b346-2cd206e42339-istiod-ca-cert\") pod \"3b9d0847-df69-4b6a-b346-2cd206e42339\" (UID: \"3b9d0847-df69-4b6a-b346-2cd206e42339\") " Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.443909 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-workload-certs\") pod \"518169ac-2f8d-4645-ae47-210b67ca080a\" (UID: \"518169ac-2f8d-4645-ae47-210b67ca080a\") " Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.444537 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-workload-certs\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.444826 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-token\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.444873 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-podinfo\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.444929 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-credential-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.444966 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-workload-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.445203 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e997605a-f2cc-4ffe-be15-d03219b3b25b-istiod-ca-cert\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.445341 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-data\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.445382 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-envoy\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.445478 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6clpf\" (UniqueName: \"kubernetes.io/projected/e997605a-f2cc-4ffe-be15-d03219b3b25b-kube-api-access-6clpf\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.445829 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.445851 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.445867 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdlh2\" (UniqueName: \"kubernetes.io/projected/70d45aa9-bfa8-43c6-8674-6f8f14a3097e-kube-api-access-kdlh2\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.445884 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnfhk\" (UniqueName: \"kubernetes.io/projected/edcfb7ab-b14f-4d5c-b752-5b43a6eee163-kube-api-access-lnfhk\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.442582 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" event={"ID":"f0d3c590-f523-4b7e-9086-60df5fc585e8","Type":"ContainerDied","Data":"af5e6ffa1ae4d694d732a2b4e736590cfab71fec562f7e7542592fa7e65d20bf"} Nov 03 20:33:06.481877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.454326 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af5e6ffa1ae4d694d732a2b4e736590cfab71fec562f7e7542592fa7e65d20bf" Nov 03 20:33:06.524403 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.524097 5059 scope.go:122] "RemoveContainer" containerID="05da4fb16c45de93fc800ccb459f5ce97d6b836ada2d5d7bf411997c2312e7b3" Nov 03 20:33:06.530121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.529910 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 20:33:06.530310 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.530288 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz"] Nov 03 20:33:06.532306 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.532275 5059 cpu_manager.go:414] "containerMap: removing container" podUID="474b2e1d-832b-4edd-8765-31e0bc77cc2a" containerName="istio-proxy" Nov 03 20:33:06.532434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.532421 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="474b2e1d-832b-4edd-8765-31e0bc77cc2a" containerName="istio-proxy" Nov 03 20:33:06.532534 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.532521 5059 cpu_manager.go:414] "containerMap: removing container" podUID="b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" containerName="istio-proxy" Nov 03 20:33:06.532866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.532851 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" containerName="istio-proxy" Nov 03 20:33:06.533014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.533002 5059 cpu_manager.go:414] "containerMap: removing container" podUID="cef4417c-3d04-4d65-a5bc-6ec1af6f09bb" containerName="istio-proxy" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.536109 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cef4417c-3d04-4d65-a5bc-6ec1af6f09bb" containerName="istio-proxy" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.536159 5059 cpu_manager.go:414] "containerMap: removing container" podUID="b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" containerName="istio-proxy" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.536169 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" containerName="istio-proxy" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.536399 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" containerName="istio-proxy" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.536415 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="cef4417c-3d04-4d65-a5bc-6ec1af6f09bb" containerName="istio-proxy" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.536439 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="474b2e1d-832b-4edd-8765-31e0bc77cc2a" containerName="istio-proxy" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.443415 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-workload-certs" pod "3b9d0847-df69-4b6a-b346-2cd206e42339" (UID: "3b9d0847-df69-4b6a-b346-2cd206e42339"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.443898 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-data" pod "8b648c6b-1f7d-42ca-9c3f-a435494bad03" (UID: "8b648c6b-1f7d-42ca-9c3f-a435494bad03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.449444 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-istio-data" pod "518169ac-2f8d-4645-ae47-210b67ca080a" (UID: "518169ac-2f8d-4645-ae47-210b67ca080a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.449827 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-credential-socket" pod "d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" (UID: "d912f1fd-b76b-4bb6-a5d9-ccc8b4536840"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.457299 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-credential-socket" pod "8b648c6b-1f7d-42ca-9c3f-a435494bad03" (UID: "8b648c6b-1f7d-42ca-9c3f-a435494bad03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.457510 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-data" pod "3b9d0847-df69-4b6a-b346-2cd206e42339" (UID: "3b9d0847-df69-4b6a-b346-2cd206e42339"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.457533 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-workload-certs" pod "f88ae6f3-f1d6-4aa3-ad33-589490696167" (UID: "f88ae6f3-f1d6-4aa3-ad33-589490696167"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.463219 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/518169ac-2f8d-4645-ae47-210b67ca080a-istio-token" pod "518169ac-2f8d-4645-ae47-210b67ca080a" (UID: "518169ac-2f8d-4645-ae47-210b67ca080a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.465834 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-credential-socket" pod "3b9d0847-df69-4b6a-b346-2cd206e42339" (UID: "3b9d0847-df69-4b6a-b346-2cd206e42339"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.467286 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-workload-certs" pod "8b648c6b-1f7d-42ca-9c3f-a435494bad03" (UID: "8b648c6b-1f7d-42ca-9c3f-a435494bad03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.470754 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-credential-socket" pod "518169ac-2f8d-4645-ae47-210b67ca080a" (UID: "518169ac-2f8d-4645-ae47-210b67ca080a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.471061 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istiod-ca-cert" pod "d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" (UID: "d912f1fd-b76b-4bb6-a5d9-ccc8b4536840"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.497932 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-workload-certs" pod "37e90ab0-e9c0-420c-a0f4-c7923aa00297" (UID: "37e90ab0-e9c0-420c-a0f4-c7923aa00297"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.511310 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f88ae6f3-f1d6-4aa3-ad33-589490696167-istiod-ca-cert" pod "f88ae6f3-f1d6-4aa3-ad33-589490696167" (UID: "f88ae6f3-f1d6-4aa3-ad33-589490696167"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.511305 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-credential-socket" pod "f88ae6f3-f1d6-4aa3-ad33-589490696167" (UID: "f88ae6f3-f1d6-4aa3-ad33-589490696167"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.511608 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-workload-socket" pod "518169ac-2f8d-4645-ae47-210b67ca080a" (UID: "518169ac-2f8d-4645-ae47-210b67ca080a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.521175 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-podinfo" pod "37e90ab0-e9c0-420c-a0f4-c7923aa00297" (UID: "37e90ab0-e9c0-420c-a0f4-c7923aa00297"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.521312 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37e90ab0-e9c0-420c-a0f4-c7923aa00297-kube-api-access-7xfpq" pod "37e90ab0-e9c0-420c-a0f4-c7923aa00297" (UID: "37e90ab0-e9c0-420c-a0f4-c7923aa00297"). InnerVolumeSpecName "kube-api-access-7xfpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.521754 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-data" pod "37e90ab0-e9c0-420c-a0f4-c7923aa00297" (UID: "37e90ab0-e9c0-420c-a0f4-c7923aa00297"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.524681 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-workload-socket" pod "3b9d0847-df69-4b6a-b346-2cd206e42339" (UID: "3b9d0847-df69-4b6a-b346-2cd206e42339"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.525477 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/518169ac-2f8d-4645-ae47-210b67ca080a-istiod-ca-cert" pod "518169ac-2f8d-4645-ae47-210b67ca080a" (UID: "518169ac-2f8d-4645-ae47-210b67ca080a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.530288 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istiod-ca-cert" pod "37e90ab0-e9c0-420c-a0f4-c7923aa00297" (UID: "37e90ab0-e9c0-420c-a0f4-c7923aa00297"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.530575 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-token" pod "8b648c6b-1f7d-42ca-9c3f-a435494bad03" (UID: "8b648c6b-1f7d-42ca-9c3f-a435494bad03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.532240 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/518169ac-2f8d-4645-ae47-210b67ca080a-istio-podinfo" pod "518169ac-2f8d-4645-ae47-210b67ca080a" (UID: "518169ac-2f8d-4645-ae47-210b67ca080a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.532478 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-token" pod "f88ae6f3-f1d6-4aa3-ad33-589490696167" (UID: "f88ae6f3-f1d6-4aa3-ad33-589490696167"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.532990 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-workload-certs" pod "518169ac-2f8d-4645-ae47-210b67ca080a" (UID: "518169ac-2f8d-4645-ae47-210b67ca080a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.535178 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-data" pod "d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" (UID: "d912f1fd-b76b-4bb6-a5d9-ccc8b4536840"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.535376 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-workload-socket" pod "f88ae6f3-f1d6-4aa3-ad33-589490696167" (UID: "f88ae6f3-f1d6-4aa3-ad33-589490696167"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.536595 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b9d0847-df69-4b6a-b346-2cd206e42339-istiod-ca-cert" pod "3b9d0847-df69-4b6a-b346-2cd206e42339" (UID: "3b9d0847-df69-4b6a-b346-2cd206e42339"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.537235 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-workload-socket" pod "d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" (UID: "d912f1fd-b76b-4bb6-a5d9-ccc8b4536840"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.537267 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istiod-ca-cert" pod "8b648c6b-1f7d-42ca-9c3f-a435494bad03" (UID: "8b648c6b-1f7d-42ca-9c3f-a435494bad03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.537359 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-envoy" pod "37e90ab0-e9c0-420c-a0f4-c7923aa00297" (UID: "37e90ab0-e9c0-420c-a0f4-c7923aa00297"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.537460 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-workload-certs" pod "d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" (UID: "d912f1fd-b76b-4bb6-a5d9-ccc8b4536840"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.537493 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-envoy" pod "3b9d0847-df69-4b6a-b346-2cd206e42339" (UID: "3b9d0847-df69-4b6a-b346-2cd206e42339"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.537592 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-token" pod "3b9d0847-df69-4b6a-b346-2cd206e42339" (UID: "3b9d0847-df69-4b6a-b346-2cd206e42339"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.538776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.537647 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-podinfo" pod "3b9d0847-df69-4b6a-b346-2cd206e42339" (UID: "3b9d0847-df69-4b6a-b346-2cd206e42339"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:06.540639 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.538217 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-credential-socket" pod "37e90ab0-e9c0-420c-a0f4-c7923aa00297" (UID: "37e90ab0-e9c0-420c-a0f4-c7923aa00297"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.540639 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.535913 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj" Nov 03 20:33:06.540639 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.539514 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-envoy" pod "f88ae6f3-f1d6-4aa3-ad33-589490696167" (UID: "f88ae6f3-f1d6-4aa3-ad33-589490696167"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.540639 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.540034 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-workload-socket" pod "37e90ab0-e9c0-420c-a0f4-c7923aa00297" (UID: "37e90ab0-e9c0-420c-a0f4-c7923aa00297"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.540639 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.540231 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-workload-socket" pod "8b648c6b-1f7d-42ca-9c3f-a435494bad03" (UID: "8b648c6b-1f7d-42ca-9c3f-a435494bad03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.541772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.541012 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-token" pod "37e90ab0-e9c0-420c-a0f4-c7923aa00297" (UID: "37e90ab0-e9c0-420c-a0f4-c7923aa00297"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.541772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.541071 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/518169ac-2f8d-4645-ae47-210b67ca080a-kube-api-access-fqlb8" pod "518169ac-2f8d-4645-ae47-210b67ca080a" (UID: "518169ac-2f8d-4645-ae47-210b67ca080a"). InnerVolumeSpecName "kube-api-access-fqlb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.547094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.543339 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 20:33:06.547094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.544565 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b9d0847-df69-4b6a-b346-2cd206e42339-kube-api-access-6zzxr" pod "3b9d0847-df69-4b6a-b346-2cd206e42339" (UID: "3b9d0847-df69-4b6a-b346-2cd206e42339"). InnerVolumeSpecName "kube-api-access-6zzxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.547094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.545754 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-kube-api-access-sq82f" pod "d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" (UID: "d912f1fd-b76b-4bb6-a5d9-ccc8b4536840"). InnerVolumeSpecName "kube-api-access-sq82f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.547094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.546427 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/890f130b-cdeb-43da-acd4-1654673271cc-kube-api-access-j788w\" (UniqueName: \"kubernetes.io/projected/890f130b-cdeb-43da-acd4-1654673271cc-kube-api-access-j788w\") pod \"890f130b-cdeb-43da-acd4-1654673271cc\" (UID: \"890f130b-cdeb-43da-acd4-1654673271cc\") " Nov 03 20:33:06.547094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.546496 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-envoy\") pod \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " Nov 03 20:33:06.547094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.546973 5059 generic.go:356] "Generic (PLEG): container finished" podID="6b9428a2-ab50-4e44-a725-e0da980bdb93" containerID="b6f8babda48abb99f0cb3d851b8caae4d8cd12ad7eda8750e60f7605358c12ba" exitCode=22 Nov 03 20:33:06.548758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.547566 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-kube-api-access-7mbqm\" (UniqueName: \"kubernetes.io/projected/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-kube-api-access-7mbqm\") pod \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " Nov 03 20:33:06.548758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.547650 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-envoy\") pod \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " Nov 03 20:33:06.548758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.547881 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istiod-ca-cert\") pod \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " Nov 03 20:33:06.548758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.547984 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-istio-envoy\") pod \"8adff4a5-5376-43ec-9b53-888705ae8236\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " Nov 03 20:33:06.548758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548018 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05fceb9c-4a04-470f-a94a-055121319f82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05fceb9c-4a04-470f-a94a-055121319f82-istio-podinfo\") pod \"05fceb9c-4a04-470f-a94a-055121319f82\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " Nov 03 20:33:06.548758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548074 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05fceb9c-4a04-470f-a94a-055121319f82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05fceb9c-4a04-470f-a94a-055121319f82-istiod-ca-cert\") pod \"05fceb9c-4a04-470f-a94a-055121319f82\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " Nov 03 20:33:06.548758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548181 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-workload-socket\") pod \"8adff4a5-5376-43ec-9b53-888705ae8236\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " Nov 03 20:33:06.548758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548217 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8adff4a5-5376-43ec-9b53-888705ae8236-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8adff4a5-5376-43ec-9b53-888705ae8236-istio-podinfo\") pod \"8adff4a5-5376-43ec-9b53-888705ae8236\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " Nov 03 20:33:06.548758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548243 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz"] Nov 03 20:33:06.548758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548253 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-token\" (UniqueName: \"kubernetes.io/projected/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-token\") pod \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " Nov 03 20:33:06.548758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548283 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-credential-socket\") pod \"8adff4a5-5376-43ec-9b53-888705ae8236\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " Nov 03 20:33:06.548758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548287 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2" event={"ID":"cfcb49c5-4381-411e-9d64-d9dcd6181239","Type":"ContainerDied","Data":"6eb6c3a433e2fe458436cce8e74c7f5d996106a0fc51f6eab2da16d2f07bc977"} Nov 03 20:33:06.548758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548315 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-workload-socket\") pod \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " Nov 03 20:33:06.548758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548325 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6eb6c3a433e2fe458436cce8e74c7f5d996106a0fc51f6eab2da16d2f07bc977" Nov 03 20:33:06.548758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548341 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5" event={"ID":"6b9428a2-ab50-4e44-a725-e0da980bdb93","Type":"ContainerDied","Data":"b6f8babda48abb99f0cb3d851b8caae4d8cd12ad7eda8750e60f7605358c12ba"} Nov 03 20:33:06.548758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548362 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-workload-certs\") pod \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " Nov 03 20:33:06.548758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548419 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-istio-envoy\") pod \"05fceb9c-4a04-470f-a94a-055121319f82\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " Nov 03 20:33:06.548758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548450 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-istio-data\") pod \"8adff4a5-5376-43ec-9b53-888705ae8236\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " Nov 03 20:33:06.548758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548511 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8adff4a5-5376-43ec-9b53-888705ae8236-kube-api-access-dw5v4\" (UniqueName: \"kubernetes.io/projected/8adff4a5-5376-43ec-9b53-888705ae8236-kube-api-access-dw5v4\") pod \"8adff4a5-5376-43ec-9b53-888705ae8236\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " Nov 03 20:33:06.548758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548545 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-data\") pod \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " Nov 03 20:33:06.548758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548571 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istiod-ca-cert\") pod \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " Nov 03 20:33:06.548758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548704 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-workload-socket\") pod \"05fceb9c-4a04-470f-a94a-055121319f82\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " Nov 03 20:33:06.548758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548712 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.548758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548733 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-token\" (UniqueName: \"kubernetes.io/projected/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-token\") pod \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " Nov 03 20:33:06.549866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548783 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-workload-certs\") pod \"05fceb9c-4a04-470f-a94a-055121319f82\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " Nov 03 20:33:06.549866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548842 5059 scope.go:122] "RemoveContainer" containerID="b6f8babda48abb99f0cb3d851b8caae4d8cd12ad7eda8750e60f7605358c12ba" Nov 03 20:33:06.549866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548847 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-data\") pod \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " Nov 03 20:33:06.549866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548886 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05fceb9c-4a04-470f-a94a-055121319f82-kube-api-access-ld86p\" (UniqueName: \"kubernetes.io/projected/05fceb9c-4a04-470f-a94a-055121319f82-kube-api-access-ld86p\") pod \"05fceb9c-4a04-470f-a94a-055121319f82\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " Nov 03 20:33:06.549866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548894 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-envoy" pod "474b2e1d-832b-4edd-8765-31e0bc77cc2a" (UID: "474b2e1d-832b-4edd-8765-31e0bc77cc2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.549866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548946 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/474b2e1d-832b-4edd-8765-31e0bc77cc2a-kube-api-access-hdvl4\" (UniqueName: \"kubernetes.io/projected/474b2e1d-832b-4edd-8765-31e0bc77cc2a-kube-api-access-hdvl4\") pod \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " Nov 03 20:33:06.549866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.548980 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-workload-socket\") pod \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " Nov 03 20:33:06.549866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.549023 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-podinfo\") pod \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " Nov 03 20:33:06.549866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.549063 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-istio-data\") pod \"05fceb9c-4a04-470f-a94a-055121319f82\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " Nov 03 20:33:06.549866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.549125 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/890f130b-cdeb-43da-acd4-1654673271cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/890f130b-cdeb-43da-acd4-1654673271cc-plugins-config-volume\") pod \"890f130b-cdeb-43da-acd4-1654673271cc\" (UID: \"890f130b-cdeb-43da-acd4-1654673271cc\") " Nov 03 20:33:06.549866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.549164 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8adff4a5-5376-43ec-9b53-888705ae8236-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8adff4a5-5376-43ec-9b53-888705ae8236-istiod-ca-cert\") pod \"8adff4a5-5376-43ec-9b53-888705ae8236\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " Nov 03 20:33:06.549866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.549218 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-credential-socket\") pod \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " Nov 03 20:33:06.549866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.549254 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-podinfo\") pod \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " Nov 03 20:33:06.549866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.549368 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-credential-socket\") pod \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\" (UID: \"474b2e1d-832b-4edd-8765-31e0bc77cc2a\") " Nov 03 20:33:06.549866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.549388 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:33:06.549866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.549404 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8adff4a5-5376-43ec-9b53-888705ae8236-istio-token\" (UniqueName: \"kubernetes.io/projected/8adff4a5-5376-43ec-9b53-888705ae8236-istio-token\") pod \"8adff4a5-5376-43ec-9b53-888705ae8236\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " Nov 03 20:33:06.549866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.549483 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-credential-socket\") pod \"05fceb9c-4a04-470f-a94a-055121319f82\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " Nov 03 20:33:06.549866 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.549795 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/890f130b-cdeb-43da-acd4-1654673271cc-plugins-config-volume" pod "890f130b-cdeb-43da-acd4-1654673271cc" (UID: "890f130b-cdeb-43da-acd4-1654673271cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:06.550729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.550147 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05fceb9c-4a04-470f-a94a-055121319f82-istio-token\" (UniqueName: \"kubernetes.io/projected/05fceb9c-4a04-470f-a94a-055121319f82-istio-token\") pod \"05fceb9c-4a04-470f-a94a-055121319f82\" (UID: \"05fceb9c-4a04-470f-a94a-055121319f82\") " Nov 03 20:33:06.550729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.550224 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-workload-certs\") pod \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\" (UID: \"b962b15a-12b8-4b36-bc1d-ce5ed903f1d3\") " Nov 03 20:33:06.550729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.550346 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-workload-certs\") pod \"8adff4a5-5376-43ec-9b53-888705ae8236\" (UID: \"8adff4a5-5376-43ec-9b53-888705ae8236\") " Nov 03 20:33:06.550869 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.550802 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-workload-socket" pod "474b2e1d-832b-4edd-8765-31e0bc77cc2a" (UID: "474b2e1d-832b-4edd-8765-31e0bc77cc2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.553546 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.552195 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-workload-certs\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.553546 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.552351 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-workload-certs\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.553546 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.552485 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-envoy\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.553546 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.552529 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istiod-ca-cert\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.553546 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.552609 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-data\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.553546 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.552828 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-token\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.553546 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.552864 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-credential-socket\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.553546 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.552906 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-podinfo\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.553546 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.552969 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-credential-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.553546 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.553010 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-workload-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.553546 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.553195 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-workload-socket\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.553546 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.553261 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-token\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.553546 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.553355 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e997605a-f2cc-4ffe-be15-d03219b3b25b-istiod-ca-cert\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.553546 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.553485 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-data\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.555458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.554656 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-istio-data" pod "05fceb9c-4a04-470f-a94a-055121319f82" (UID: "05fceb9c-4a04-470f-a94a-055121319f82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.553522 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-data\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.555611 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-envoy\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.555698 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfncw\" (UniqueName: \"kubernetes.io/projected/b1573256-23a1-4a26-9972-87bc9b27f2c1-kube-api-access-kfncw\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.555765 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6clpf\" (UniqueName: \"kubernetes.io/projected/e997605a-f2cc-4ffe-be15-d03219b3b25b-kube-api-access-6clpf\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.555831 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-credential-socket\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.555961 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-podinfo\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556051 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-workload-certs\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556127 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd5rh\" (UniqueName: \"kubernetes.io/projected/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-kube-api-access-wd5rh\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556266 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-workload-socket\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556370 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-envoy\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556415 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-token\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556519 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-podinfo\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556549 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1573256-23a1-4a26-9972-87bc9b27f2c1-istiod-ca-cert\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556728 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/518169ac-2f8d-4645-ae47-210b67ca080a-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556750 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556767 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556782 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556799 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556812 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556827 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556842 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556858 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556873 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556888 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556903 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556919 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556936 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556950 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556965 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556981 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.556997 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/890f130b-cdeb-43da-acd4-1654673271cc-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557013 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557028 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqlb8\" (UniqueName: \"kubernetes.io/projected/518169ac-2f8d-4645-ae47-210b67ca080a-kube-api-access-fqlb8\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557042 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557080 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557094 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557108 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557123 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zzxr\" (UniqueName: \"kubernetes.io/projected/3b9d0847-df69-4b6a-b346-2cd206e42339-kube-api-access-6zzxr\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557139 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557154 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557170 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq82f\" (UniqueName: \"kubernetes.io/projected/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-kube-api-access-sq82f\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557184 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b9d0847-df69-4b6a-b346-2cd206e42339-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557202 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557217 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.594685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557232 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b9d0847-df69-4b6a-b346-2cd206e42339-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557248 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557263 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557280 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557297 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557310 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37e90ab0-e9c0-420c-a0f4-c7923aa00297-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557326 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557343 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b9d0847-df69-4b6a-b346-2cd206e42339-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557359 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xfpq\" (UniqueName: \"kubernetes.io/projected/37e90ab0-e9c0-420c-a0f4-c7923aa00297-kube-api-access-7xfpq\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557374 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557389 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/518169ac-2f8d-4645-ae47-210b67ca080a-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557404 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f88ae6f3-f1d6-4aa3-ad33-589490696167-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557419 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557433 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557449 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/518169ac-2f8d-4645-ae47-210b67ca080a-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557464 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557479 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557495 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37e90ab0-e9c0-420c-a0f4-c7923aa00297-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557595 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-podinfo" pod "474b2e1d-832b-4edd-8765-31e0bc77cc2a" (UID: "474b2e1d-832b-4edd-8765-31e0bc77cc2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557629 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474b2e1d-832b-4edd-8765-31e0bc77cc2a-kube-api-access-hdvl4" pod "474b2e1d-832b-4edd-8765-31e0bc77cc2a" (UID: "474b2e1d-832b-4edd-8765-31e0bc77cc2a"). InnerVolumeSpecName "kube-api-access-hdvl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.557873 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/890f130b-cdeb-43da-acd4-1654673271cc-kube-api-access-j788w" pod "890f130b-cdeb-43da-acd4-1654673271cc" (UID: "890f130b-cdeb-43da-acd4-1654673271cc"). InnerVolumeSpecName "kube-api-access-j788w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.560614 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-workload-certs\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.560902 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-credential-socket" pod "b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" (UID: "b962b15a-12b8-4b36-bc1d-ce5ed903f1d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.573232 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e997605a-f2cc-4ffe-be15-d03219b3b25b-istiod-ca-cert\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.573514 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-data\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.586205 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-credential-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.597558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.586453 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-workload-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.601188 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.599056 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8adff4a5-5376-43ec-9b53-888705ae8236-istiod-ca-cert" pod "8adff4a5-5376-43ec-9b53-888705ae8236" (UID: "8adff4a5-5376-43ec-9b53-888705ae8236"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:06.604751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.601598 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" event={"ID":"a594e899-889e-4f7f-a8ba-b7e7a389ba8a","Type":"ContainerDied","Data":"72e807611fccf6e4b9d3a050087a30f7d4a435174a7369909a9e64cfc19552bf"} Nov 03 20:33:06.604751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.601656 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72e807611fccf6e4b9d3a050087a30f7d4a435174a7369909a9e64cfc19552bf" Nov 03 20:33:06.604751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.601971 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:06.604751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.603253 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-credential-socket" pod "05fceb9c-4a04-470f-a94a-055121319f82" (UID: "05fceb9c-4a04-470f-a94a-055121319f82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.604751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.603458 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-credential-socket" pod "474b2e1d-832b-4edd-8765-31e0bc77cc2a" (UID: "474b2e1d-832b-4edd-8765-31e0bc77cc2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.606474 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.606447 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8adff4a5-5376-43ec-9b53-888705ae8236-istio-token" pod "8adff4a5-5376-43ec-9b53-888705ae8236" (UID: "8adff4a5-5376-43ec-9b53-888705ae8236"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.608474 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.607980 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-workload-certs" pod "b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" (UID: "b962b15a-12b8-4b36-bc1d-ce5ed903f1d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.608474 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.608206 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-workload-socket" pod "b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" (UID: "b962b15a-12b8-4b36-bc1d-ce5ed903f1d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.608769 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.608715 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-data" pod "b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" (UID: "b962b15a-12b8-4b36-bc1d-ce5ed903f1d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.608960 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.608937 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-workload-certs" pod "474b2e1d-832b-4edd-8765-31e0bc77cc2a" (UID: "474b2e1d-832b-4edd-8765-31e0bc77cc2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.611505 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.611477 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istiod-ca-cert" pod "b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" (UID: "b962b15a-12b8-4b36-bc1d-ce5ed903f1d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:06.616002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.612098 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-workload-certs" pod "8adff4a5-5376-43ec-9b53-888705ae8236" (UID: "8adff4a5-5376-43ec-9b53-888705ae8236"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.616002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.612114 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-istio-data" pod "8adff4a5-5376-43ec-9b53-888705ae8236" (UID: "8adff4a5-5376-43ec-9b53-888705ae8236"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.616002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.612553 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-workload-socket" pod "05fceb9c-4a04-470f-a94a-055121319f82" (UID: "05fceb9c-4a04-470f-a94a-055121319f82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.616002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.612761 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b648c6b-1f7d-42ca-9c3f-a435494bad03-kube-api-access-7qcvm" pod "8b648c6b-1f7d-42ca-9c3f-a435494bad03" (UID: "8b648c6b-1f7d-42ca-9c3f-a435494bad03"). InnerVolumeSpecName "kube-api-access-7qcvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.616002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.612830 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-podinfo" pod "d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" (UID: "d912f1fd-b76b-4bb6-a5d9-ccc8b4536840"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:06.616002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.612900 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f88ae6f3-f1d6-4aa3-ad33-589490696167-kube-api-access-vkjmc" pod "f88ae6f3-f1d6-4aa3-ad33-589490696167" (UID: "f88ae6f3-f1d6-4aa3-ad33-589490696167"). InnerVolumeSpecName "kube-api-access-vkjmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.616002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.612951 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-podinfo" pod "8b648c6b-1f7d-42ca-9c3f-a435494bad03" (UID: "8b648c6b-1f7d-42ca-9c3f-a435494bad03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:06.616002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.613000 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-envoy" pod "d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" (UID: "d912f1fd-b76b-4bb6-a5d9-ccc8b4536840"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.616002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.614749 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-workload-certs" pod "05fceb9c-4a04-470f-a94a-055121319f82" (UID: "05fceb9c-4a04-470f-a94a-055121319f82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.616002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.614868 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istiod-ca-cert" pod "474b2e1d-832b-4edd-8765-31e0bc77cc2a" (UID: "474b2e1d-832b-4edd-8765-31e0bc77cc2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:06.619960 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.619702 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-workload-socket" pod "8adff4a5-5376-43ec-9b53-888705ae8236" (UID: "8adff4a5-5376-43ec-9b53-888705ae8236"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.620165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.620138 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-data" pod "474b2e1d-832b-4edd-8765-31e0bc77cc2a" (UID: "474b2e1d-832b-4edd-8765-31e0bc77cc2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.632774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.622434 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-istio-envoy" pod "05fceb9c-4a04-470f-a94a-055121319f82" (UID: "05fceb9c-4a04-470f-a94a-055121319f82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.632774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.622554 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-token" pod "b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" (UID: "b962b15a-12b8-4b36-bc1d-ce5ed903f1d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.632774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.623213 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-podinfo\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.632774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.623331 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05fceb9c-4a04-470f-a94a-055121319f82-istio-token" pod "05fceb9c-4a04-470f-a94a-055121319f82" (UID: "05fceb9c-4a04-470f-a94a-055121319f82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.632774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.623392 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-podinfo" pod "f88ae6f3-f1d6-4aa3-ad33-589490696167" (UID: "f88ae6f3-f1d6-4aa3-ad33-589490696167"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:06.632774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.623425 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s"] Nov 03 20:33:06.632774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.624394 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-credential-socket" pod "8adff4a5-5376-43ec-9b53-888705ae8236" (UID: "8adff4a5-5376-43ec-9b53-888705ae8236"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.632774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.624474 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05fceb9c-4a04-470f-a94a-055121319f82-istiod-ca-cert" pod "05fceb9c-4a04-470f-a94a-055121319f82" (UID: "05fceb9c-4a04-470f-a94a-055121319f82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:06.632774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.625336 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" Nov 03 20:33:06.632774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.627124 5059 cpu_manager.go:414] "containerMap: removing container" podUID="06b78cd0-94df-40f9-a1ed-87c1751675d3" containerName="payload-pre-processing" Nov 03 20:33:06.632774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.627144 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06b78cd0-94df-40f9-a1ed-87c1751675d3" containerName="payload-pre-processing" Nov 03 20:33:06.632774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.627186 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c8683a65-9898-4b4f-a692-f2da325a93b4" containerName="istio-proxy" Nov 03 20:33:06.632774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.627194 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8683a65-9898-4b4f-a692-f2da325a93b4" containerName="istio-proxy" Nov 03 20:33:06.632774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.627211 5059 cpu_manager.go:414] "containerMap: removing container" podUID="40b88c7d-08e9-4203-9ebd-2c9f69b884dc" containerName="istio-proxy" Nov 03 20:33:06.632774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.627219 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b88c7d-08e9-4203-9ebd-2c9f69b884dc" containerName="istio-proxy" Nov 03 20:33:06.632774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.627399 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="06b78cd0-94df-40f9-a1ed-87c1751675d3" containerName="payload-pre-processing" Nov 03 20:33:06.632774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.627415 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" containerName="istio-proxy" Nov 03 20:33:06.632774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.627429 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8683a65-9898-4b4f-a692-f2da325a93b4" containerName="istio-proxy" Nov 03 20:33:06.632774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.627438 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b88c7d-08e9-4203-9ebd-2c9f69b884dc" containerName="istio-proxy" Nov 03 20:33:06.632774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.628447 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05fceb9c-4a04-470f-a94a-055121319f82-kube-api-access-ld86p" pod "05fceb9c-4a04-470f-a94a-055121319f82" (UID: "05fceb9c-4a04-470f-a94a-055121319f82"). InnerVolumeSpecName "kube-api-access-ld86p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.632774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.628521 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-istio-envoy" pod "518169ac-2f8d-4645-ae47-210b67ca080a" (UID: "518169ac-2f8d-4645-ae47-210b67ca080a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.632774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.628733 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-envoy" pod "8b648c6b-1f7d-42ca-9c3f-a435494bad03" (UID: "8b648c6b-1f7d-42ca-9c3f-a435494bad03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.632774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.629483 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6clpf\" (UniqueName: \"kubernetes.io/projected/e997605a-f2cc-4ffe-be15-d03219b3b25b-kube-api-access-6clpf\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.632774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.630635 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-token\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.641473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.634185 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-envoy" pod "b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" (UID: "b962b15a-12b8-4b36-bc1d-ce5ed903f1d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.641473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.634251 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05fceb9c-4a04-470f-a94a-055121319f82-istio-podinfo" pod "05fceb9c-4a04-470f-a94a-055121319f82" (UID: "05fceb9c-4a04-470f-a94a-055121319f82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:06.641473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.634326 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-token" pod "d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" (UID: "d912f1fd-b76b-4bb6-a5d9-ccc8b4536840"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.641473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.634375 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-istio-envoy" pod "8adff4a5-5376-43ec-9b53-888705ae8236" (UID: "8adff4a5-5376-43ec-9b53-888705ae8236"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.641473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.634431 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8adff4a5-5376-43ec-9b53-888705ae8236-kube-api-access-dw5v4" pod "8adff4a5-5376-43ec-9b53-888705ae8236" (UID: "8adff4a5-5376-43ec-9b53-888705ae8236"). InnerVolumeSpecName "kube-api-access-dw5v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.641473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.636434 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-token" pod "474b2e1d-832b-4edd-8765-31e0bc77cc2a" (UID: "474b2e1d-832b-4edd-8765-31e0bc77cc2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.641473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.637591 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-envoy\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:06.641473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.640550 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-podinfo" pod "b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" (UID: "b962b15a-12b8-4b36-bc1d-ce5ed903f1d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:06.643560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.642281 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s"] Nov 03 20:33:06.643560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.642325 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc" event={"ID":"474b2e1d-832b-4edd-8765-31e0bc77cc2a","Type":"ContainerDied","Data":"15b8f003915ecc6ccbabe46fd4f8d1f9ea8654fc469c49b72288ac914824c2e8"} Nov 03 20:33:06.643560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.642436 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:06.647117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.647061 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8adff4a5-5376-43ec-9b53-888705ae8236-istio-podinfo" pod "8adff4a5-5376-43ec-9b53-888705ae8236" (UID: "8adff4a5-5376-43ec-9b53-888705ae8236"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:06.655462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.647921 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" Nov 03 20:33:06.655462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.647952 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk" event={"ID":"edcfb7ab-b14f-4d5c-b752-5b43a6eee163","Type":"ContainerDied","Data":"05d1f7c6803b28af0b4c887a8576d44944ba6af0d944a9770f2f845a5a03f004"} Nov 03 20:33:06.655462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.651706 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" event={"ID":"0309a608-f6c4-45ca-991b-33ba73221a6c","Type":"ContainerDied","Data":"9e5ee2cf73166662f5459bdbc4c34f404a24c7301d0c98128d4de9fb5bd8e3e0"} Nov 03 20:33:06.655462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.651746 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e5ee2cf73166662f5459bdbc4c34f404a24c7301d0c98128d4de9fb5bd8e3e0" Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.656607 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" event={"ID":"c8683a65-9898-4b4f-a692-f2da325a93b4","Type":"ContainerDied","Data":"58ce0025686f1eadbf4bc094248fadb9cc6bcf4ba93427714d48ab1519673f95"} Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.656716 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7" Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.713867 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-envoy\") pod \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.713923 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istiod-ca-cert\") pod \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.713968 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-workload-certs\") pod \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.714002 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-data\") pod \"c8683a65-9898-4b4f-a692-f2da325a93b4\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.714056 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-data\") pod \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.714101 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-envoy\") pod \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.714169 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-workload-socket\") pod \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.714413 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-credential-socket\") pod \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.714451 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-kube-api-access-6bdv6\" (UniqueName: \"kubernetes.io/projected/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-kube-api-access-6bdv6\") pod \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.714558 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-kube-api-access-q5dsd\" (UniqueName: \"kubernetes.io/projected/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-kube-api-access-q5dsd\") pod \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.714591 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-workload-socket\") pod \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.714635 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-data\") pod \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.714720 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-credential-socket\") pod \"c8683a65-9898-4b4f-a692-f2da325a93b4\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.714807 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-token\" (UniqueName: \"kubernetes.io/projected/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-token\") pod \"c8683a65-9898-4b4f-a692-f2da325a93b4\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.714841 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istiod-ca-cert\") pod \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.714897 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-credential-socket\") pod \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.714934 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-token\" (UniqueName: \"kubernetes.io/projected/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-token\") pod \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.715011 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-podinfo\") pod \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.715223 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-workload-certs\") pod \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.715281 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8683a65-9898-4b4f-a692-f2da325a93b4-kube-api-access-67fhl\" (UniqueName: \"kubernetes.io/projected/c8683a65-9898-4b4f-a692-f2da325a93b4-kube-api-access-67fhl\") pod \"c8683a65-9898-4b4f-a692-f2da325a93b4\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.715323 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-podinfo\") pod \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\" (UID: \"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb\") " Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.715375 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-workload-socket\") pod \"c8683a65-9898-4b4f-a692-f2da325a93b4\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " Nov 03 20:33:06.715944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.715416 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06b78cd0-94df-40f9-a1ed-87c1751675d3-kube-api-access-rmx4m\" (UniqueName: \"kubernetes.io/projected/06b78cd0-94df-40f9-a1ed-87c1751675d3-kube-api-access-rmx4m\") pod \"06b78cd0-94df-40f9-a1ed-87c1751675d3\" (UID: \"06b78cd0-94df-40f9-a1ed-87c1751675d3\") " Nov 03 20:33:06.717964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.717033 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06b78cd0-94df-40f9-a1ed-87c1751675d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b78cd0-94df-40f9-a1ed-87c1751675d3-plugins-config-volume\") pod \"06b78cd0-94df-40f9-a1ed-87c1751675d3\" (UID: \"06b78cd0-94df-40f9-a1ed-87c1751675d3\") " Nov 03 20:33:06.717964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.717099 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-envoy\") pod \"c8683a65-9898-4b4f-a692-f2da325a93b4\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " Nov 03 20:33:06.717964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.717142 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-workload-certs\") pod \"c8683a65-9898-4b4f-a692-f2da325a93b4\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " Nov 03 20:33:06.717964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.717150 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-kube-api-access-7mbqm" pod "b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" (UID: "b962b15a-12b8-4b36-bc1d-ce5ed903f1d3"). InnerVolumeSpecName "kube-api-access-7mbqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.717964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.717235 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8683a65-9898-4b4f-a692-f2da325a93b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8683a65-9898-4b4f-a692-f2da325a93b4-istiod-ca-cert\") pod \"c8683a65-9898-4b4f-a692-f2da325a93b4\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " Nov 03 20:33:06.717964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.717278 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-token\" (UniqueName: \"kubernetes.io/projected/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-token\") pod \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\" (UID: \"40b88c7d-08e9-4203-9ebd-2c9f69b884dc\") " Nov 03 20:33:06.717964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.717362 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-podinfo\") pod \"c8683a65-9898-4b4f-a692-f2da325a93b4\" (UID: \"c8683a65-9898-4b4f-a692-f2da325a93b4\") " Nov 03 20:33:06.717964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.717459 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-workload-certs" pod "c8683a65-9898-4b4f-a692-f2da325a93b4" (UID: "c8683a65-9898-4b4f-a692-f2da325a93b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.718963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.718859 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-token\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.718963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.718934 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-workload-certs\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.719351 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.719079 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-data\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.719351 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.719145 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfncw\" (UniqueName: \"kubernetes.io/projected/b1573256-23a1-4a26-9972-87bc9b27f2c1-kube-api-access-kfncw\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.719351 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.719263 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-credential-socket\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.719591 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.719355 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-podinfo\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.719591 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.719392 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-istio-envoy\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.719591 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.719549 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-workload-certs\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.719813 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.719601 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxmp4\" (UniqueName: \"kubernetes.io/projected/abce531c-32e5-4fee-b6fa-7b489da29704-kube-api-access-bxmp4\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.719813 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.719640 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd5rh\" (UniqueName: \"kubernetes.io/projected/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-kube-api-access-wd5rh\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.719813 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.719686 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-credential-socket\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.719813 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.719734 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abce531c-32e5-4fee-b6fa-7b489da29704-istio-podinfo\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.719813 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.719785 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abce531c-32e5-4fee-b6fa-7b489da29704-istio-token\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.720250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.719875 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-workload-socket\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.720250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.719945 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-workload-socket\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.720250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.719991 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-envoy\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.720250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.720084 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-token\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.720250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.720176 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-istio-data\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.720250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.720213 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-podinfo\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.720250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.720244 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1573256-23a1-4a26-9972-87bc9b27f2c1-istiod-ca-cert\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.720632 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.720352 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-workload-certs\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.720632 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.720461 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-envoy\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.720632 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.720501 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istiod-ca-cert\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.720632 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.720581 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-data\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.721031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.720642 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-credential-socket\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.721031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.720974 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abce531c-32e5-4fee-b6fa-7b489da29704-istiod-ca-cert\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.721240 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.721155 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-workload-socket\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.722039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.721945 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05fceb9c-4a04-470f-a94a-055121319f82-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.722039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.721973 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkjmc\" (UniqueName: \"kubernetes.io/projected/f88ae6f3-f1d6-4aa3-ad33-589490696167-kube-api-access-vkjmc\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.722039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.721993 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05fceb9c-4a04-470f-a94a-055121319f82-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.722039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722010 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.722039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722033 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f88ae6f3-f1d6-4aa3-ad33-589490696167-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.722429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722051 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.722429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722071 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8adff4a5-5376-43ec-9b53-888705ae8236-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.722429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722088 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.722429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722110 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.722429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722126 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.722429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722142 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.722429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722165 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.722429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722185 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.722429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722202 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw5v4\" (UniqueName: \"kubernetes.io/projected/8adff4a5-5376-43ec-9b53-888705ae8236-kube-api-access-dw5v4\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.722429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722216 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.722429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722232 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.722429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722251 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/518169ac-2f8d-4645-ae47-210b67ca080a-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.722429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722265 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.722429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722280 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.722429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722297 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.722429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722371 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-credential-socket" pod "c8683a65-9898-4b4f-a692-f2da325a93b4" (UID: "c8683a65-9898-4b4f-a692-f2da325a93b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722317 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722506 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722524 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722540 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld86p\" (UniqueName: \"kubernetes.io/projected/05fceb9c-4a04-470f-a94a-055121319f82-kube-api-access-ld86p\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722561 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdvl4\" (UniqueName: \"kubernetes.io/projected/474b2e1d-832b-4edd-8765-31e0bc77cc2a-kube-api-access-hdvl4\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722578 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722595 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722611 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8adff4a5-5376-43ec-9b53-888705ae8236-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722635 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722654 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722710 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722729 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/474b2e1d-832b-4edd-8765-31e0bc77cc2a-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722754 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8adff4a5-5376-43ec-9b53-888705ae8236-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722770 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b648c6b-1f7d-42ca-9c3f-a435494bad03-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722772 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istiod-ca-cert" pod "cef4417c-3d04-4d65-a5bc-6ec1af6f09bb" (UID: "cef4417c-3d04-4d65-a5bc-6ec1af6f09bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722788 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fceb9c-4a04-470f-a94a-055121319f82-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722805 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05fceb9c-4a04-470f-a94a-055121319f82-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722825 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722840 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722855 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j788w\" (UniqueName: \"kubernetes.io/projected/890f130b-cdeb-43da-acd4-1654673271cc-kube-api-access-j788w\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722874 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mbqm\" (UniqueName: \"kubernetes.io/projected/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-kube-api-access-7mbqm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722890 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722905 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qcvm\" (UniqueName: \"kubernetes.io/projected/8b648c6b-1f7d-42ca-9c3f-a435494bad03-kube-api-access-7qcvm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.722918 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/474b2e1d-832b-4edd-8765-31e0bc77cc2a-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.723086 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8adff4a5-5376-43ec-9b53-888705ae8236-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.723410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.723118 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-workload-certs" pod "40b88c7d-08e9-4203-9ebd-2c9f69b884dc" (UID: "40b88c7d-08e9-4203-9ebd-2c9f69b884dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.725235 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.723508 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-data" pod "c8683a65-9898-4b4f-a692-f2da325a93b4" (UID: "c8683a65-9898-4b4f-a692-f2da325a93b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.725235 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.723933 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-data" pod "40b88c7d-08e9-4203-9ebd-2c9f69b884dc" (UID: "40b88c7d-08e9-4203-9ebd-2c9f69b884dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.730173 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.729206 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istiod-ca-cert" pod "40b88c7d-08e9-4203-9ebd-2c9f69b884dc" (UID: "40b88c7d-08e9-4203-9ebd-2c9f69b884dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:06.730173 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.729671 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-credential-socket" pod "cef4417c-3d04-4d65-a5bc-6ec1af6f09bb" (UID: "cef4417c-3d04-4d65-a5bc-6ec1af6f09bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.731254 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.731232 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m"] Nov 03 20:33:06.751028 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.741957 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" Nov 03 20:33:06.752789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.732449 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8683a65-9898-4b4f-a692-f2da325a93b4-istiod-ca-cert" pod "c8683a65-9898-4b4f-a692-f2da325a93b4" (UID: "c8683a65-9898-4b4f-a692-f2da325a93b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:06.752789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.740905 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-workload-certs" pod "cef4417c-3d04-4d65-a5bc-6ec1af6f09bb" (UID: "cef4417c-3d04-4d65-a5bc-6ec1af6f09bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.752789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.741182 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-workload-socket" pod "c8683a65-9898-4b4f-a692-f2da325a93b4" (UID: "c8683a65-9898-4b4f-a692-f2da325a93b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.752789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.742383 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-token" pod "40b88c7d-08e9-4203-9ebd-2c9f69b884dc" (UID: "40b88c7d-08e9-4203-9ebd-2c9f69b884dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.752789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.742463 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-token" pod "c8683a65-9898-4b4f-a692-f2da325a93b4" (UID: "c8683a65-9898-4b4f-a692-f2da325a93b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.752789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.743972 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06b78cd0-94df-40f9-a1ed-87c1751675d3-plugins-config-volume" pod "06b78cd0-94df-40f9-a1ed-87c1751675d3" (UID: "06b78cd0-94df-40f9-a1ed-87c1751675d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:06.752789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.744748 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-podinfo" pod "cef4417c-3d04-4d65-a5bc-6ec1af6f09bb" (UID: "cef4417c-3d04-4d65-a5bc-6ec1af6f09bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:06.752789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.747766 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-workload-socket" pod "cef4417c-3d04-4d65-a5bc-6ec1af6f09bb" (UID: "cef4417c-3d04-4d65-a5bc-6ec1af6f09bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.752789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.748863 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-credential-socket" pod "40b88c7d-08e9-4203-9ebd-2c9f69b884dc" (UID: "40b88c7d-08e9-4203-9ebd-2c9f69b884dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.752789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.749119 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-workload-socket" pod "40b88c7d-08e9-4203-9ebd-2c9f69b884dc" (UID: "40b88c7d-08e9-4203-9ebd-2c9f69b884dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.752789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.749814 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-data" pod "cef4417c-3d04-4d65-a5bc-6ec1af6f09bb" (UID: "cef4417c-3d04-4d65-a5bc-6ec1af6f09bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.771700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.764520 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-workload-socket\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.771700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.765279 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-envoy\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.780485 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.780453 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1573256-23a1-4a26-9972-87bc9b27f2c1-istiod-ca-cert\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.780838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.780810 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-workload-certs\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.781562 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.781536 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-data\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.781740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.781712 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istiod-ca-cert\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.785694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.782092 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-data\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.786822 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.786798 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-credential-socket\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.825986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.806296 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-token" pod "cef4417c-3d04-4d65-a5bc-6ec1af6f09bb" (UID: "cef4417c-3d04-4d65-a5bc-6ec1af6f09bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.845071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.782650 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-credential-socket\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.845071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.811107 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-workload-socket\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.845071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.816118 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" Nov 03 20:33:06.845071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.839544 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-token\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.845071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.840396 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b78cd0-94df-40f9-a1ed-87c1751675d3-kube-api-access-rmx4m" pod "06b78cd0-94df-40f9-a1ed-87c1751675d3" (UID: "06b78cd0-94df-40f9-a1ed-87c1751675d3"). InnerVolumeSpecName "kube-api-access-rmx4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.845071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.840557 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8683a65-9898-4b4f-a692-f2da325a93b4-kube-api-access-67fhl" pod "c8683a65-9898-4b4f-a692-f2da325a93b4" (UID: "c8683a65-9898-4b4f-a692-f2da325a93b4"). InnerVolumeSpecName "kube-api-access-67fhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.845071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.840623 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-envoy" pod "c8683a65-9898-4b4f-a692-f2da325a93b4" (UID: "c8683a65-9898-4b4f-a692-f2da325a93b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.845071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.841016 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-kube-api-access-q5dsd" pod "40b88c7d-08e9-4203-9ebd-2c9f69b884dc" (UID: "40b88c7d-08e9-4203-9ebd-2c9f69b884dc"). InnerVolumeSpecName "kube-api-access-q5dsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.845071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.841079 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-envoy" pod "cef4417c-3d04-4d65-a5bc-6ec1af6f09bb" (UID: "cef4417c-3d04-4d65-a5bc-6ec1af6f09bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.845528 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.845484 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-token\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.845816 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg" event={"ID":"518169ac-2f8d-4645-ae47-210b67ca080a","Type":"ContainerDied","Data":"6ffdd1a6aa4c61b216d8c84eba61449574d8e149612f72a033532ef642609540"} Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.845874 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp"] Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.829020 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-envoy\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.849808 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-kube-api-access-6bdv6" pod "cef4417c-3d04-4d65-a5bc-6ec1af6f09bb" (UID: "cef4417c-3d04-4d65-a5bc-6ec1af6f09bb"). InnerVolumeSpecName "kube-api-access-6bdv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.850504 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9069949b-78ef-42cc-99b0-1feffc84b679-istio-podinfo\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.851648 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-workload-certs\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.855424 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-podinfo\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.855570 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-istio-envoy\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.855639 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxmp4\" (UniqueName: \"kubernetes.io/projected/abce531c-32e5-4fee-b6fa-7b489da29704-kube-api-access-bxmp4\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.855696 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-credential-socket\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.855740 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abce531c-32e5-4fee-b6fa-7b489da29704-istio-podinfo\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.855787 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abce531c-32e5-4fee-b6fa-7b489da29704-istio-token\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.855819 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-workload-certs\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.855914 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-workload-socket\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.855981 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-istio-envoy\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.856023 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-istio-data\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.856072 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-istio-data\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.856184 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9069949b-78ef-42cc-99b0-1feffc84b679-istiod-ca-cert\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.856473 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abce531c-32e5-4fee-b6fa-7b489da29704-istiod-ca-cert\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.856515 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9069949b-78ef-42cc-99b0-1feffc84b679-istio-token\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.856598 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-workload-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.856652 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-credential-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.856750 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhlgt\" (UniqueName: \"kubernetes.io/projected/9069949b-78ef-42cc-99b0-1feffc84b679-kube-api-access-mhlgt\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.856832 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-workload-certs\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857054 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5dsd\" (UniqueName: \"kubernetes.io/projected/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-kube-api-access-q5dsd\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857074 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857091 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857119 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857144 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857160 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857176 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857198 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857212 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857228 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67fhl\" (UniqueName: \"kubernetes.io/projected/c8683a65-9898-4b4f-a692-f2da325a93b4-kube-api-access-67fhl\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857245 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857264 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857280 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmx4m\" (UniqueName: \"kubernetes.io/projected/06b78cd0-94df-40f9-a1ed-87c1751675d3-kube-api-access-rmx4m\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857295 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b78cd0-94df-40f9-a1ed-87c1751675d3-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857310 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857328 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8683a65-9898-4b4f-a692-f2da325a93b4-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.875244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857345 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.944348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857361 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.944348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857379 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.944348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857394 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.944348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857408 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.944348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857423 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.944348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857441 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.944348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857454 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.944348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.857469 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bdv6\" (UniqueName: \"kubernetes.io/projected/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb-kube-api-access-6bdv6\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:06.944348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.858012 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-workload-certs\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.944348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.929885 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-envoy" pod "40b88c7d-08e9-4203-9ebd-2c9f69b884dc" (UID: "40b88c7d-08e9-4203-9ebd-2c9f69b884dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:06.944348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.929981 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-podinfo" pod "40b88c7d-08e9-4203-9ebd-2c9f69b884dc" (UID: "40b88c7d-08e9-4203-9ebd-2c9f69b884dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:06.944348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.930702 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m"] Nov 03 20:33:06.944348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.931199 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp"] Nov 03 20:33:06.944348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.931249 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk" event={"ID":"8b648c6b-1f7d-42ca-9c3f-a435494bad03","Type":"ContainerDied","Data":"45c262ea7a7351d836fc55259b94b8ec172b3cdad20f1f7dcfa206941ac825c0"} Nov 03 20:33:06.944348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.937315 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m" Nov 03 20:33:06.944348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.939793 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp" Nov 03 20:33:06.944348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.944289 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-credential-socket\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.947209 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.947185 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfncw\" (UniqueName: \"kubernetes.io/projected/b1573256-23a1-4a26-9972-87bc9b27f2c1-kube-api-access-kfncw\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:06.947683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.947647 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-74dfc6\"" Nov 03 20:33:06.947857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.947838 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-podinfo" pod "c8683a65-9898-4b4f-a692-f2da325a93b4" (UID: "c8683a65-9898-4b4f-a692-f2da325a93b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:06.948565 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.948542 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-74dfc6-dockercfg-9ddm6\"" Nov 03 20:33:06.949300 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.949184 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44" podUID="26375297-81f4-427f-94c3-d56f7a6a11fb" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.244:50052\" within 1s: context deadline exceeded" Nov 03 20:33:06.949727 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:33:06.949704 5059 logging.go:55] [core] [Channel #6809 SubChannel #6810]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.244:50052", ServerName: "10.132.0.244:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.244:50052: operation was canceled" Nov 03 20:33:06.951693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.951654 5059 scope.go:122] "RemoveContainer" containerID="6dc7ad3073c40aab49d043739b212390e9623e46a6f995b1c1ca2fa55cf79644" Nov 03 20:33:06.954628 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.954110 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd5rh\" (UniqueName: \"kubernetes.io/projected/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-kube-api-access-wd5rh\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.954628 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.954480 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-istio-envoy\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.957770 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.954856 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-istio-data\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.957770 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.955083 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-podinfo\") pod \"e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:06.957770 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.955931 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abce531c-32e5-4fee-b6fa-7b489da29704-istiod-ca-cert\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.957770 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.956353 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-workload-socket\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:06.988307 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:06.988217 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:33:07.028480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.021745 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpvhm\" (UniqueName: \"kubernetes.io/projected/de4efb53-8b59-49f1-894f-ef366f00732a-kube-api-access-cpvhm\") pod \"payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m\" (UID: \"de4efb53-8b59-49f1-894f-ef366f00732a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m" Nov 03 20:33:07.028480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.021886 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-workload-certs\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:07.028480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.022016 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-istio-envoy\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:07.028480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.022048 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-istio-data\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:07.028480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.022147 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9069949b-78ef-42cc-99b0-1feffc84b679-istiod-ca-cert\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:07.028480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.022248 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b40731-f01a-4209-af40-de342324d721-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp\" (UID: \"a9b40731-f01a-4209-af40-de342324d721\") " pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp" Nov 03 20:33:07.028480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.022347 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9069949b-78ef-42cc-99b0-1feffc84b679-istio-token\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:07.028480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.022409 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-workload-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:07.028480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.022449 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-credential-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:07.028480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.022515 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhlgt\" (UniqueName: \"kubernetes.io/projected/9069949b-78ef-42cc-99b0-1feffc84b679-kube-api-access-mhlgt\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:07.028480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.022609 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5twj\" (UniqueName: \"kubernetes.io/projected/a9b40731-f01a-4209-af40-de342324d721-kube-api-access-g5twj\") pod \"payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp\" (UID: \"a9b40731-f01a-4209-af40-de342324d721\") " pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp" Nov 03 20:33:07.028480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.022677 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9069949b-78ef-42cc-99b0-1feffc84b679-istio-podinfo\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:07.028480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.022800 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de4efb53-8b59-49f1-894f-ef366f00732a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m\" (UID: \"de4efb53-8b59-49f1-894f-ef366f00732a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m" Nov 03 20:33:07.028480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.022942 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8683a65-9898-4b4f-a692-f2da325a93b4-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.028480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.022961 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.028480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.022978 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b88c7d-08e9-4203-9ebd-2c9f69b884dc-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.028480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.025569 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-workload-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:07.028480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.028237 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-istio-envoy\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:07.036159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.031209 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-credential-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:07.036159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.032778 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-workload-certs\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:07.036159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.033543 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-istio-data\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:07.036159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.033887 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abce531c-32e5-4fee-b6fa-7b489da29704-istio-podinfo\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:07.037093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.037043 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" event={"ID":"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd","Type":"ContainerStarted","Data":"0e34ccd56022844f4709995fa385f307c970109d38e0aa73908019a3605706a5"} Nov 03 20:33:07.037154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.037095 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" event={"ID":"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd","Type":"ContainerStarted","Data":"2d0da3958f8a206492b090761f8d10fc84d3560053575f8560f7d743955337a2"} Nov 03 20:33:07.038490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.038421 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9069949b-78ef-42cc-99b0-1feffc84b679-istiod-ca-cert\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:07.038844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.038760 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9069949b-78ef-42cc-99b0-1feffc84b679-istio-podinfo\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:07.042595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.042105 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abce531c-32e5-4fee-b6fa-7b489da29704-istio-token\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:07.042595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.042522 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxmp4\" (UniqueName: \"kubernetes.io/projected/abce531c-32e5-4fee-b6fa-7b489da29704-kube-api-access-bxmp4\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:07.043002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.042949 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhlgt\" (UniqueName: \"kubernetes.io/projected/9069949b-78ef-42cc-99b0-1feffc84b679-kube-api-access-mhlgt\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:07.045577 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.044729 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" event={"ID":"171c7288-3157-4fe0-80ee-94cf5b2ca2dd","Type":"ContainerDied","Data":"5356f35de0393003597c8867655dc2f00fd424a96cbcb72e79de3ca84e22e177"} Nov 03 20:33:07.045577 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.044776 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5356f35de0393003597c8867655dc2f00fd424a96cbcb72e79de3ca84e22e177" Nov 03 20:33:07.050751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.050726 5059 scope.go:122] "RemoveContainer" containerID="49fc76b3559a7fdac7c4de24b68b9e183d25cec8fc53c075c32d7308be58c0fa" Nov 03 20:33:07.052160 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.052137 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv"] Nov 03 20:33:07.056201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.053458 5059 cpu_manager.go:414] "containerMap: removing container" podUID="14a06bdf-e15d-406c-bb76-47ae5dd06b2d" containerName="istio-proxy" Nov 03 20:33:07.056201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.053482 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a06bdf-e15d-406c-bb76-47ae5dd06b2d" containerName="istio-proxy" Nov 03 20:33:07.056201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.053750 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a06bdf-e15d-406c-bb76-47ae5dd06b2d" containerName="istio-proxy" Nov 03 20:33:07.056201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.053773 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a06bdf-e15d-406c-bb76-47ae5dd06b2d" containerName="istio-proxy" Nov 03 20:33:07.056201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.053985 5059 cpu_manager.go:414] "containerMap: removing container" podUID="14a06bdf-e15d-406c-bb76-47ae5dd06b2d" containerName="istio-proxy" Nov 03 20:33:07.056201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.053997 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a06bdf-e15d-406c-bb76-47ae5dd06b2d" containerName="istio-proxy" Nov 03 20:33:07.108774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.108693 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4" event={"ID":"dc10f92e-e902-48c4-9cd5-4f768c4611be","Type":"ContainerStarted","Data":"192cded1132012fbc6e17035522a365a76221127162bbd64c7c3d1d27b94d412"} Nov 03 20:33:07.109056 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.109035 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" Nov 03 20:33:07.110541 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.110514 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9069949b-78ef-42cc-99b0-1feffc84b679-istio-token\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:07.114839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.114813 5059 scope.go:122] "RemoveContainer" containerID="89b4df7aa727c470a64d52c6160fcb987049e7d0bf5bc45f6489c13773b39202" Nov 03 20:33:07.152034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.123872 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 20:33:07.152034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.123996 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-847xh\"" Nov 03 20:33:07.152034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.124080 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 20:33:07.152034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.124130 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 20:33:07.152034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.124215 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 20:33:07.152034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.124295 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 20:33:07.152034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.124371 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-kube-api-access-pvlmp\" (UniqueName: \"kubernetes.io/projected/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-kube-api-access-pvlmp\") pod \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " Nov 03 20:33:07.152034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.124466 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-data\") pod \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " Nov 03 20:33:07.152034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.124682 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-workload-socket\") pod \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " Nov 03 20:33:07.152034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.124729 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-token\" (UniqueName: \"kubernetes.io/projected/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-token\") pod \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " Nov 03 20:33:07.152034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.124761 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istiod-ca-cert\") pod \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " Nov 03 20:33:07.152034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.124907 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-workload-certs\") pod \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " Nov 03 20:33:07.152034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.124972 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-podinfo\") pod \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " Nov 03 20:33:07.152034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.125028 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-envoy\") pod \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " Nov 03 20:33:07.152034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.125146 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-credential-socket\") pod \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\" (UID: \"14a06bdf-e15d-406c-bb76-47ae5dd06b2d\") " Nov 03 20:33:07.152034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.125536 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de4efb53-8b59-49f1-894f-ef366f00732a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m\" (UID: \"de4efb53-8b59-49f1-894f-ef366f00732a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m" Nov 03 20:33:07.152034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.125598 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpvhm\" (UniqueName: \"kubernetes.io/projected/de4efb53-8b59-49f1-894f-ef366f00732a-kube-api-access-cpvhm\") pod \"payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m\" (UID: \"de4efb53-8b59-49f1-894f-ef366f00732a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m" Nov 03 20:33:07.152034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.126032 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b40731-f01a-4209-af40-de342324d721-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp\" (UID: \"a9b40731-f01a-4209-af40-de342324d721\") " pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp" Nov 03 20:33:07.152034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.127175 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b40731-f01a-4209-af40-de342324d721-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp\" (UID: \"a9b40731-f01a-4209-af40-de342324d721\") " pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp" Nov 03 20:33:07.152942 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.152321 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-mnmjc"] Nov 03 20:33:07.153490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.153460 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5twj\" (UniqueName: \"kubernetes.io/projected/a9b40731-f01a-4209-af40-de342324d721-kube-api-access-g5twj\") pod \"payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp\" (UID: \"a9b40731-f01a-4209-af40-de342324d721\") " pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp" Nov 03 20:33:07.170543 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.157268 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body= Nov 03 20:33:07.170543 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.157311 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" podUID="d931dc13-4399-48dc-80e0-b0b304c5dda0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" Nov 03 20:33:07.213923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.187883 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-kube-api-access-pvlmp" pod "14a06bdf-e15d-406c-bb76-47ae5dd06b2d" (UID: "14a06bdf-e15d-406c-bb76-47ae5dd06b2d"). InnerVolumeSpecName "kube-api-access-pvlmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:07.213923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.194809 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de4efb53-8b59-49f1-894f-ef366f00732a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m\" (UID: \"de4efb53-8b59-49f1-894f-ef366f00732a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m" Nov 03 20:33:07.213923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.213884 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-credential-socket" pod "14a06bdf-e15d-406c-bb76-47ae5dd06b2d" (UID: "14a06bdf-e15d-406c-bb76-47ae5dd06b2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.214383 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.214349 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-workload-certs" pod "14a06bdf-e15d-406c-bb76-47ae5dd06b2d" (UID: "14a06bdf-e15d-406c-bb76-47ae5dd06b2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.223598 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.222457 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-envoy" pod "14a06bdf-e15d-406c-bb76-47ae5dd06b2d" (UID: "14a06bdf-e15d-406c-bb76-47ae5dd06b2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.223598 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.222905 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istiod-ca-cert" pod "14a06bdf-e15d-406c-bb76-47ae5dd06b2d" (UID: "14a06bdf-e15d-406c-bb76-47ae5dd06b2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:07.226202 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.226171 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-podinfo" pod "14a06bdf-e15d-406c-bb76-47ae5dd06b2d" (UID: "14a06bdf-e15d-406c-bb76-47ae5dd06b2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:07.226549 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.226528 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-workload-socket" pod "14a06bdf-e15d-406c-bb76-47ae5dd06b2d" (UID: "14a06bdf-e15d-406c-bb76-47ae5dd06b2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.227829 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.226708 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-token" pod "14a06bdf-e15d-406c-bb76-47ae5dd06b2d" (UID: "14a06bdf-e15d-406c-bb76-47ae5dd06b2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:07.227829 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.227468 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5twj\" (UniqueName: \"kubernetes.io/projected/a9b40731-f01a-4209-af40-de342324d721-kube-api-access-g5twj\") pod \"payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp\" (UID: \"a9b40731-f01a-4209-af40-de342324d721\") " pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp" Nov 03 20:33:07.237582 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.229918 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-data" pod "14a06bdf-e15d-406c-bb76-47ae5dd06b2d" (UID: "14a06bdf-e15d-406c-bb76-47ae5dd06b2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.237582 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.230022 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" Nov 03 20:33:07.237582 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.235945 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp" Nov 03 20:33:07.242079 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.241885 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv"] Nov 03 20:33:07.242079 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.241937 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-mnmjc"] Nov 03 20:33:07.242079 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.241968 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w" event={"ID":"cef4417c-3d04-4d65-a5bc-6ec1af6f09bb","Type":"ContainerDied","Data":"701216e74fc88586037ac5f2aabf13be12a48852e8dc32da7bea458c18dba20f"} Nov 03 20:33:07.242079 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.242071 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" Nov 03 20:33:07.251579 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.246805 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpvhm\" (UniqueName: \"kubernetes.io/projected/de4efb53-8b59-49f1-894f-ef366f00732a-kube-api-access-cpvhm\") pod \"payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m\" (UID: \"de4efb53-8b59-49f1-894f-ef366f00732a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m" Nov 03 20:33:07.251579 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.248797 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" event={"ID":"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c","Type":"ContainerDied","Data":"4de41ea151f3956a8493e5e18a1b5d1167462b47993743ad2f7cd53a11edc886"} Nov 03 20:33:07.251579 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.248842 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4de41ea151f3956a8493e5e18a1b5d1167462b47993743ad2f7cd53a11edc886" Nov 03 20:33:07.251579 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.249581 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 20:33:07.251579 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.250577 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:33:07.253687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.253227 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44" event={"ID":"26375297-81f4-427f-94c3-d56f7a6a11fb","Type":"ContainerDied","Data":"e99e44e38a56617ba12741f6caaea0dcc706de1443380b8c96103d96c6112496"} Nov 03 20:33:07.253687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.253274 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e99e44e38a56617ba12741f6caaea0dcc706de1443380b8c96103d96c6112496" Nov 03 20:33:07.267566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.255275 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/32deeada-f159-4266-8026-9b0a9ffa5eb1-manifests\") pod \"opendatahub-operator-controller-manager-8464c976fd-dgdfv\" (UID: \"32deeada-f159-4266-8026-9b0a9ffa5eb1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" Nov 03 20:33:07.267566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.255403 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8dw7\" (UniqueName: \"kubernetes.io/projected/32deeada-f159-4266-8026-9b0a9ffa5eb1-kube-api-access-v8dw7\") pod \"opendatahub-operator-controller-manager-8464c976fd-dgdfv\" (UID: \"32deeada-f159-4266-8026-9b0a9ffa5eb1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" Nov 03 20:33:07.267566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.255462 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32deeada-f159-4266-8026-9b0a9ffa5eb1-tmp\") pod \"opendatahub-operator-controller-manager-8464c976fd-dgdfv\" (UID: \"32deeada-f159-4266-8026-9b0a9ffa5eb1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" Nov 03 20:33:07.267566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.255520 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32deeada-f159-4266-8026-9b0a9ffa5eb1-webhook-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-dgdfv\" (UID: \"32deeada-f159-4266-8026-9b0a9ffa5eb1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" Nov 03 20:33:07.267566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.255559 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/32deeada-f159-4266-8026-9b0a9ffa5eb1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-dgdfv\" (UID: \"32deeada-f159-4266-8026-9b0a9ffa5eb1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" Nov 03 20:33:07.267566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.255617 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32deeada-f159-4266-8026-9b0a9ffa5eb1-metrics-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-dgdfv\" (UID: \"32deeada-f159-4266-8026-9b0a9ffa5eb1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" Nov 03 20:33:07.267566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.255626 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7" event={"ID":"2c363281-0226-420f-a963-5922d5ca609e","Type":"ContainerDied","Data":"0790100bd49a069e5ae6fe894ddc55096ee6648b0ba9da367a971377af75ae84"} Nov 03 20:33:07.267566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.255698 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0790100bd49a069e5ae6fe894ddc55096ee6648b0ba9da367a971377af75ae84" Nov 03 20:33:07.267566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.255752 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.267566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.255764 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.267566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.255774 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.267566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.255784 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.267566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.255792 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvlmp\" (UniqueName: \"kubernetes.io/projected/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-kube-api-access-pvlmp\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.267566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.255801 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.267566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.255809 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.267566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.255818 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.267566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.255826 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a06bdf-e15d-406c-bb76-47ae5dd06b2d-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.324393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.315724 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf"] Nov 03 20:33:07.324393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.317207 5059 cpu_manager.go:414] "containerMap: removing container" podUID="ab8320f7-2354-433a-a8a7-4d91cc5d37d2" containerName="istio-proxy" Nov 03 20:33:07.324393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.317233 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab8320f7-2354-433a-a8a7-4d91cc5d37d2" containerName="istio-proxy" Nov 03 20:33:07.324393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.317442 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab8320f7-2354-433a-a8a7-4d91cc5d37d2" containerName="istio-proxy" Nov 03 20:33:07.329882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.329790 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 20:33:07.330576 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.330545 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" event={"ID":"4946e3aa-a961-415a-b71d-95164d5fbca3","Type":"ContainerDied","Data":"3ced3533cde36871565d3f5b342e68e1183b5249dab2a5f6aecb26a3d24f2a80"} Nov 03 20:33:07.330711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.330592 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ced3533cde36871565d3f5b342e68e1183b5249dab2a5f6aecb26a3d24f2a80" Nov 03 20:33:07.330768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.330741 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:07.333473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.333455 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf"] Nov 03 20:33:07.341453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.341423 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" event={"ID":"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193","Type":"ContainerDied","Data":"f7f451b988823a9abe607e4d43baa7cfb31cca43dcac0c1eb59db7b68d0ee813"} Nov 03 20:33:07.341560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.341473 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7f451b988823a9abe607e4d43baa7cfb31cca43dcac0c1eb59db7b68d0ee813" Nov 03 20:33:07.345063 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.344825 5059 scope.go:122] "RemoveContainer" containerID="aafac57bb6098e1fd609b713edc7da1b412af807beeafa6e9063b066a1c006b1" Nov 03 20:33:07.348858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.348645 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:33:07.350353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.350326 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn" event={"ID":"25ceab3d-6344-4512-a471-d91aec0cb48c","Type":"ContainerDied","Data":"78cef308b00fc9217bde938ee3c21f496c6ebd2c6184bb774beaa70f0faf95db"} Nov 03 20:33:07.350437 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.350373 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78cef308b00fc9217bde938ee3c21f496c6ebd2c6184bb774beaa70f0faf95db" Nov 03 20:33:07.351057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.350970 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:33:07.356059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.356035 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:33:07.356144 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.356117 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg"] Nov 03 20:33:07.357201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.357176 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-token\" (UniqueName: \"kubernetes.io/projected/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-token\") pod \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " Nov 03 20:33:07.433835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.433797 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istiod-ca-cert\") pod \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " Nov 03 20:33:07.433986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.433886 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-envoy\") pod \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " Nov 03 20:33:07.433986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.433962 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-kube-api-access-fbmnk\" (UniqueName: \"kubernetes.io/projected/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-kube-api-access-fbmnk\") pod \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " Nov 03 20:33:07.434095 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.434003 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-podinfo\") pod \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " Nov 03 20:33:07.434338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.434312 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-workload-socket\") pod \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " Nov 03 20:33:07.434424 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.434361 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-data\") pod \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " Nov 03 20:33:07.434424 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.434411 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-workload-certs\") pod \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.434533 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-credential-socket\") pod \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\" (UID: \"ab8320f7-2354-433a-a8a7-4d91cc5d37d2\") " Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.434964 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32deeada-f159-4266-8026-9b0a9ffa5eb1-tmp\") pod \"opendatahub-operator-controller-manager-8464c976fd-dgdfv\" (UID: \"32deeada-f159-4266-8026-9b0a9ffa5eb1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.435030 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-mnmjc\" (UID: \"89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.435081 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5-home\") pod \"echo-simulator-kserve-64b6f8f96c-mnmjc\" (UID: \"89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.435176 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32deeada-f159-4266-8026-9b0a9ffa5eb1-webhook-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-dgdfv\" (UID: \"32deeada-f159-4266-8026-9b0a9ffa5eb1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.435247 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-mnmjc\" (UID: \"89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.435286 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/32deeada-f159-4266-8026-9b0a9ffa5eb1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-dgdfv\" (UID: \"32deeada-f159-4266-8026-9b0a9ffa5eb1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.435384 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-mnmjc\" (UID: \"89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.435460 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32deeada-f159-4266-8026-9b0a9ffa5eb1-metrics-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-dgdfv\" (UID: \"32deeada-f159-4266-8026-9b0a9ffa5eb1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.435510 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-mnmjc\" (UID: \"89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.435805 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/32deeada-f159-4266-8026-9b0a9ffa5eb1-manifests\") pod \"opendatahub-operator-controller-manager-8464c976fd-dgdfv\" (UID: \"32deeada-f159-4266-8026-9b0a9ffa5eb1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.435985 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lxg9\" (UniqueName: \"kubernetes.io/projected/89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5-kube-api-access-8lxg9\") pod \"echo-simulator-kserve-64b6f8f96c-mnmjc\" (UID: \"89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.436128 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8dw7\" (UniqueName: \"kubernetes.io/projected/32deeada-f159-4266-8026-9b0a9ffa5eb1-kube-api-access-v8dw7\") pod \"opendatahub-operator-controller-manager-8464c976fd-dgdfv\" (UID: \"32deeada-f159-4266-8026-9b0a9ffa5eb1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.357247 5059 cpu_manager.go:414] "containerMap: removing container" podUID="f0d3c590-f523-4b7e-9086-60df5fc585e8" containerName="istio-proxy" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.436594 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d3c590-f523-4b7e-9086-60df5fc585e8" containerName="istio-proxy" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.436638 5059 cpu_manager.go:414] "containerMap: removing container" podUID="08b3a993-1a64-4fe5-9577-b0f11a36779e" containerName="istio-proxy" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.436648 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08b3a993-1a64-4fe5-9577-b0f11a36779e" containerName="istio-proxy" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.436686 5059 cpu_manager.go:414] "containerMap: removing container" podUID="f0d3c590-f523-4b7e-9086-60df5fc585e8" containerName="istio-proxy" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.436694 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d3c590-f523-4b7e-9086-60df5fc585e8" containerName="istio-proxy" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.436721 5059 cpu_manager.go:414] "containerMap: removing container" podUID="171c7288-3157-4fe0-80ee-94cf5b2ca2dd" containerName="istio-proxy" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.436727 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="171c7288-3157-4fe0-80ee-94cf5b2ca2dd" containerName="istio-proxy" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.436755 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-token" pod "ab8320f7-2354-433a-a8a7-4d91cc5d37d2" (UID: "ab8320f7-2354-433a-a8a7-4d91cc5d37d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.436767 5059 cpu_manager.go:414] "containerMap: removing container" podUID="34880975-a897-4ad4-af59-18617a93cdad" containerName="istio-proxy" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.436774 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34880975-a897-4ad4-af59-18617a93cdad" containerName="istio-proxy" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.437026 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d3c590-f523-4b7e-9086-60df5fc585e8" containerName="istio-proxy" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.437043 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="08b3a993-1a64-4fe5-9577-b0f11a36779e" containerName="istio-proxy" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.437059 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d3c590-f523-4b7e-9086-60df5fc585e8" containerName="istio-proxy" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.437073 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="171c7288-3157-4fe0-80ee-94cf5b2ca2dd" containerName="istio-proxy" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.437085 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="08b3a993-1a64-4fe5-9577-b0f11a36779e" containerName="istio-proxy" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.437100 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="34880975-a897-4ad4-af59-18617a93cdad" containerName="istio-proxy" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.437820 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istiod-ca-cert" pod "ab8320f7-2354-433a-a8a7-4d91cc5d37d2" (UID: "ab8320f7-2354-433a-a8a7-4d91cc5d37d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.426207 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.439808 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-data" pod "ab8320f7-2354-433a-a8a7-4d91cc5d37d2" (UID: "ab8320f7-2354-433a-a8a7-4d91cc5d37d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.439912 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-credential-socket" pod "ab8320f7-2354-433a-a8a7-4d91cc5d37d2" (UID: "ab8320f7-2354-433a-a8a7-4d91cc5d37d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.440078 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-workload-socket" pod "ab8320f7-2354-433a-a8a7-4d91cc5d37d2" (UID: "ab8320f7-2354-433a-a8a7-4d91cc5d37d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.440250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.440113 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-workload-certs" pod "ab8320f7-2354-433a-a8a7-4d91cc5d37d2" (UID: "ab8320f7-2354-433a-a8a7-4d91cc5d37d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.443698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.441761 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/32deeada-f159-4266-8026-9b0a9ffa5eb1-manifests\") pod \"opendatahub-operator-controller-manager-8464c976fd-dgdfv\" (UID: \"32deeada-f159-4266-8026-9b0a9ffa5eb1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" Nov 03 20:33:07.443698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.442997 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq" event={"ID":"0d960e4e-9539-4a53-9df7-1c520ab7d708","Type":"ContainerDied","Data":"a06ee8f1bbe5022a2a8de8dbe428f1689ba38c88a172963d6ef8b6f18ad320ef"} Nov 03 20:33:07.443869 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.443831 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a06ee8f1bbe5022a2a8de8dbe428f1689ba38c88a172963d6ef8b6f18ad320ef" Nov 03 20:33:07.443981 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.443947 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg" Nov 03 20:33:07.487686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.474571 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:33:07.546527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.489197 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32deeada-f159-4266-8026-9b0a9ffa5eb1-metrics-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-dgdfv\" (UID: \"32deeada-f159-4266-8026-9b0a9ffa5eb1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" Nov 03 20:33:07.546527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.490359 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m" Nov 03 20:33:07.546527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.522574 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2" podUID="cfcb49c5-4381-411e-9d64-d9dcd6181239" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.74:50052\" within 1s: context deadline exceeded" Nov 03 20:33:07.546527 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:33:07.524338 5059 logging.go:55] [core] [Channel #6814 SubChannel #6815]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.74:50052", ServerName: "10.132.0.74:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.74:50052: operation was canceled" Nov 03 20:33:07.546527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.524582 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-envoy" pod "ab8320f7-2354-433a-a8a7-4d91cc5d37d2" (UID: "ab8320f7-2354-433a-a8a7-4d91cc5d37d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.546527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.531146 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-942q2\"" Nov 03 20:33:07.546527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.531296 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-74dfc6-serving-cert\"" Nov 03 20:33:07.546527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.532233 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq" podUID="0d960e4e-9539-4a53-9df7-1c520ab7d708" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.72:50052\" within 1s: context deadline exceeded" Nov 03 20:33:07.546527 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:33:07.532640 5059 logging.go:55] [core] [Channel #6816 SubChannel #6817]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.72:50052", ServerName: "10.132.0.72:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.72:50052: operation was canceled" Nov 03 20:33:07.546527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.533001 5059 scope.go:122] "RemoveContainer" containerID="f2e801a6eee34b07dcc874c7303b6ec332d7470d1b006afc0f98c9a4ae0ba486" Nov 03 20:33:07.546527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.535054 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt" podUID="c92b1be0-0890-4db6-ac75-968b6e5a3434" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.73:50052\" within 1s: context deadline exceeded" Nov 03 20:33:07.546527 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:33:07.535175 5059 logging.go:55] [core] [Channel #6813 SubChannel #6818]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.73:50052", ServerName: "10.132.0.73:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.73:50052: operation was canceled" Nov 03 20:33:07.546527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.537301 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p" podUID="ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.75:50052\" within 1s: context deadline exceeded" Nov 03 20:33:07.546527 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:33:07.537424 5059 logging.go:55] [core] [Channel #6811 SubChannel #6812]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.75:50052", ServerName: "10.132.0.75:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.75:50052: operation was canceled" Nov 03 20:33:07.546527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.538207 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg"] Nov 03 20:33:07.551923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.547383 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32deeada-f159-4266-8026-9b0a9ffa5eb1-webhook-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-dgdfv\" (UID: \"32deeada-f159-4266-8026-9b0a9ffa5eb1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" Nov 03 20:33:07.551923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.547483 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-podinfo" pod "ab8320f7-2354-433a-a8a7-4d91cc5d37d2" (UID: "ab8320f7-2354-433a-a8a7-4d91cc5d37d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:07.551923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.547612 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-kube-api-access-fbmnk" pod "ab8320f7-2354-433a-a8a7-4d91cc5d37d2" (UID: "ab8320f7-2354-433a-a8a7-4d91cc5d37d2"). InnerVolumeSpecName "kube-api-access-fbmnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:07.551923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.547812 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32deeada-f159-4266-8026-9b0a9ffa5eb1-tmp\") pod \"opendatahub-operator-controller-manager-8464c976fd-dgdfv\" (UID: \"32deeada-f159-4266-8026-9b0a9ffa5eb1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" Nov 03 20:33:07.553798 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.552737 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" event={"ID":"ab8320f7-2354-433a-a8a7-4d91cc5d37d2","Type":"ContainerDied","Data":"5c734102d818e68d05b3ceb8796ce96b37ab13c4b474fd2808bd7c4bb32e2de8"} Nov 03 20:33:07.553798 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.553222 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5" Nov 03 20:33:07.555599 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.555575 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/32deeada-f159-4266-8026-9b0a9ffa5eb1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-dgdfv\" (UID: \"32deeada-f159-4266-8026-9b0a9ffa5eb1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" Nov 03 20:33:07.589069 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.557131 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:07.592748 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.591914 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:33:07.635242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.621324 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Nov 03 20:33:07.635242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.621377 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" podUID="bbd92cdc-e2bd-4f72-8c63-1221069ca3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Nov 03 20:33:07.635242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.624225 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:33:07.635242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.633005 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" event={"ID":"d098ef77-addb-4077-b6d4-f0ae477409b8","Type":"ContainerStarted","Data":"2d6ccc3fde38d6b105feaac697775af9f843f350e85decc50c0dec055353e2fe"} Nov 03 20:33:07.644116 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.643706 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8dw7\" (UniqueName: \"kubernetes.io/projected/32deeada-f159-4266-8026-9b0a9ffa5eb1-kube-api-access-v8dw7\") pod \"opendatahub-operator-controller-manager-8464c976fd-dgdfv\" (UID: \"32deeada-f159-4266-8026-9b0a9ffa5eb1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" Nov 03 20:33:07.644116 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.643886 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" Nov 03 20:33:07.652339 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.652313 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-mnmjc\" (UID: \"89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" Nov 03 20:33:07.667103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.657113 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5-home\") pod \"echo-simulator-kserve-64b6f8f96c-mnmjc\" (UID: \"89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" Nov 03 20:33:07.667103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.657169 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eda42a5-5b61-4b34-8447-8767ed097200-istiod-ca-cert\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:07.667103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.657257 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-mnmjc\" (UID: \"89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" Nov 03 20:33:07.667103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.657342 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-mnmjc\" (UID: \"89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" Nov 03 20:33:07.667103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.657395 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eda42a5-5b61-4b34-8447-8767ed097200-istio-podinfo\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:07.667103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.657440 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb32457-81a5-4505-b189-2a98f5fda7da-openshift-service-ca\") pod \"maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg\" (UID: \"aeb32457-81a5-4505-b189-2a98f5fda7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg" Nov 03 20:33:07.667103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.657469 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-mnmjc\" (UID: \"89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" Nov 03 20:33:07.667103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.657501 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7slk\" (UniqueName: \"kubernetes.io/projected/3eda42a5-5b61-4b34-8447-8767ed097200-kube-api-access-n7slk\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:07.667103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.657703 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeb32457-81a5-4505-b189-2a98f5fda7da-maas-api-tls\") pod \"maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg\" (UID: \"aeb32457-81a5-4505-b189-2a98f5fda7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg" Nov 03 20:33:07.667103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.657740 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxj5l\" (UniqueName: \"kubernetes.io/projected/aeb32457-81a5-4505-b189-2a98f5fda7da-kube-api-access-kxj5l\") pod \"maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg\" (UID: \"aeb32457-81a5-4505-b189-2a98f5fda7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg" Nov 03 20:33:07.667103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.657774 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-istio-envoy\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:07.667103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.657804 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-credential-socket\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:07.667103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.657859 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeb32457-81a5-4505-b189-2a98f5fda7da-metrics-certs\") pod \"maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg\" (UID: \"aeb32457-81a5-4505-b189-2a98f5fda7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg" Nov 03 20:33:07.667103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.657908 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eda42a5-5b61-4b34-8447-8767ed097200-istio-token\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:07.667103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.657987 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-workload-socket\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:07.667103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.658024 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lxg9\" (UniqueName: \"kubernetes.io/projected/89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5-kube-api-access-8lxg9\") pod \"echo-simulator-kserve-64b6f8f96c-mnmjc\" (UID: \"89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" Nov 03 20:33:07.667103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.658073 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-istio-data\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:07.667103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.658150 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-workload-certs\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:07.667103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.658405 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.667103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.653239 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 20:33:07.667103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.658902 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5-home\") pod \"echo-simulator-kserve-64b6f8f96c-mnmjc\" (UID: \"89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" Nov 03 20:33:07.667103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.658947 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.667103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.658982 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.736255 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.733003 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.736255 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.733700 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-mnmjc\" (UID: \"89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" Nov 03 20:33:07.736255 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.733962 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-mnmjc\" (UID: \"89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" Nov 03 20:33:07.736255 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.734834 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.744986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.744608 5059 scope.go:122] "RemoveContainer" containerID="f40938fdfd8d12fe6529e191274b54750315e326f7aeec32fac2e699b64c43a1" Nov 03 20:33:07.754683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.746404 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-mnmjc\" (UID: \"89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" Nov 03 20:33:07.754683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.752720 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-mnmjc\" (UID: \"89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" Nov 03 20:33:07.754683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.752791 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.754683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.752879 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.754683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.753004 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbmnk\" (UniqueName: \"kubernetes.io/projected/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-kube-api-access-fbmnk\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.754683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.753035 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab8320f7-2354-433a-a8a7-4d91cc5d37d2-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.754683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.753350 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lxg9\" (UniqueName: \"kubernetes.io/projected/89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5-kube-api-access-8lxg9\") pod \"echo-simulator-kserve-64b6f8f96c-mnmjc\" (UID: \"89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" Nov 03 20:33:07.808005 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.755191 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl" event={"ID":"9c3d72b6-73c4-402f-8308-ee5d87a9dd83","Type":"ContainerDied","Data":"79ebb628096fb75f5b9ad65ace257111a2c6249c36bb44d175dca428fee2c503"} Nov 03 20:33:07.808005 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.755249 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79ebb628096fb75f5b9ad65ace257111a2c6249c36bb44d175dca428fee2c503" Nov 03 20:33:07.808005 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.767854 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:33:07.809457 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.809421 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p" event={"ID":"ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51","Type":"ContainerDied","Data":"c178def3a32d1f877a58690af646328cbb2be8a3529de23564ab32855263ec87"} Nov 03 20:33:07.841290 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.834651 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" Nov 03 20:33:07.841533 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.840152 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj" Nov 03 20:33:07.841652 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.841636 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c178def3a32d1f877a58690af646328cbb2be8a3529de23564ab32855263ec87" Nov 03 20:33:07.845490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.845457 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" event={"ID":"4c609469-d7d7-4530-8d74-e34aa56dda8f","Type":"ContainerDied","Data":"e1c46745a7ea7a6fa41f53b3c9f518f32c768160d6591f0ccc7d8b4fc14c110d"} Nov 03 20:33:07.847889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.846057 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1c46745a7ea7a6fa41f53b3c9f518f32c768160d6591f0ccc7d8b4fc14c110d" Nov 03 20:33:07.847889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.846523 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt" event={"ID":"c92b1be0-0890-4db6-ac75-968b6e5a3434","Type":"ContainerDied","Data":"41ae12074f52cc4f3189ae21cd41ddc55c92d901d330aef39e9eebafc0c4a8aa"} Nov 03 20:33:07.847889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.846992 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41ae12074f52cc4f3189ae21cd41ddc55c92d901d330aef39e9eebafc0c4a8aa" Nov 03 20:33:07.847889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.847011 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g" event={"ID":"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0","Type":"ContainerDied","Data":"25f8aee821c0a81ef57b7376c1dcec476f8f49ac04b243baa0808bf0af58ce9c"} Nov 03 20:33:07.848113 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.847960 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" event={"ID":"1731953b-1850-40d1-9ba4-cb08dc4ebc5d","Type":"ContainerDied","Data":"a86c61ebb787ebdf61871133a5429b258ffa023ca9d00cf03137a978d0bb7408"} Nov 03 20:33:07.848229 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.848214 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a86c61ebb787ebdf61871133a5429b258ffa023ca9d00cf03137a978d0bb7408" Nov 03 20:33:07.848330 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.848315 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj" event={"ID":"06b78cd0-94df-40f9-a1ed-87c1751675d3","Type":"ContainerDied","Data":"79ba33873a545827d72cc34d9ead20855333ee4a7c7053d21cb8dfd4094a631f"} Nov 03 20:33:07.869877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.869837 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-envoy\") pod \"f0d3c590-f523-4b7e-9086-60df5fc585e8\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " Nov 03 20:33:07.870112 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.870089 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-token\" (UniqueName: \"kubernetes.io/projected/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-token\") pod \"4946e3aa-a961-415a-b71d-95164d5fbca3\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " Nov 03 20:33:07.870246 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.870230 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istiod-ca-cert\") pod \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " Nov 03 20:33:07.870361 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.870345 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-credential-socket\") pod \"08b3a993-1a64-4fe5-9577-b0f11a36779e\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " Nov 03 20:33:07.870478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.870463 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0309a608-f6c4-45ca-991b-33ba73221a6c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0309a608-f6c4-45ca-991b-33ba73221a6c-provider-credentials\") pod \"0309a608-f6c4-45ca-991b-33ba73221a6c\" (UID: \"0309a608-f6c4-45ca-991b-33ba73221a6c\") " Nov 03 20:33:07.870615 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.870598 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0309a608-f6c4-45ca-991b-33ba73221a6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0309a608-f6c4-45ca-991b-33ba73221a6c-plugins-config-volume\") pod \"0309a608-f6c4-45ca-991b-33ba73221a6c\" (UID: \"0309a608-f6c4-45ca-991b-33ba73221a6c\") " Nov 03 20:33:07.870755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.870738 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-kube-api-access-bjdzj\" (UniqueName: \"kubernetes.io/projected/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-kube-api-access-bjdzj\") pod \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " Nov 03 20:33:07.870917 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.870900 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-podinfo\") pod \"4946e3aa-a961-415a-b71d-95164d5fbca3\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " Nov 03 20:33:07.871050 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.871035 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-podinfo\") pod \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " Nov 03 20:33:07.871201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.871186 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-podinfo\") pod \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " Nov 03 20:33:07.871319 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.871305 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-credential-socket\") pod \"34880975-a897-4ad4-af59-18617a93cdad\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " Nov 03 20:33:07.871441 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.871426 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-credential-socket\") pod \"f0d3c590-f523-4b7e-9086-60df5fc585e8\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " Nov 03 20:33:07.871549 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.871535 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34880975-a897-4ad4-af59-18617a93cdad-kube-api-access-5cjvf\" (UniqueName: \"kubernetes.io/projected/34880975-a897-4ad4-af59-18617a93cdad-kube-api-access-5cjvf\") pod \"34880975-a897-4ad4-af59-18617a93cdad\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " Nov 03 20:33:07.871688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.871653 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-token\" (UniqueName: \"kubernetes.io/projected/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-token\") pod \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " Nov 03 20:33:07.871828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.871813 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-token\" (UniqueName: \"kubernetes.io/projected/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-token\") pod \"08b3a993-1a64-4fe5-9577-b0f11a36779e\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " Nov 03 20:33:07.871985 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.871971 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34880975-a897-4ad4-af59-18617a93cdad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34880975-a897-4ad4-af59-18617a93cdad-istiod-ca-cert\") pod \"34880975-a897-4ad4-af59-18617a93cdad\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " Nov 03 20:33:07.872116 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.872102 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4946e3aa-a961-415a-b71d-95164d5fbca3-kube-api-access-wltn5\" (UniqueName: \"kubernetes.io/projected/4946e3aa-a961-415a-b71d-95164d5fbca3-kube-api-access-wltn5\") pod \"4946e3aa-a961-415a-b71d-95164d5fbca3\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " Nov 03 20:33:07.872261 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.872247 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-credential-socket\") pod \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " Nov 03 20:33:07.872384 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.872369 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-podinfo\") pod \"08b3a993-1a64-4fe5-9577-b0f11a36779e\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " Nov 03 20:33:07.872493 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.872478 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-data\") pod \"f0d3c590-f523-4b7e-9086-60df5fc585e8\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " Nov 03 20:33:07.872601 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.872587 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istiod-ca-cert\") pod \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " Nov 03 20:33:07.872730 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.872714 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-workload-socket\") pod \"08b3a993-1a64-4fe5-9577-b0f11a36779e\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " Nov 03 20:33:07.872888 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.872872 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-workload-socket\") pod \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " Nov 03 20:33:07.873040 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.873024 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-workload-certs\") pod \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " Nov 03 20:33:07.873169 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.873155 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-data\") pod \"4946e3aa-a961-415a-b71d-95164d5fbca3\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " Nov 03 20:33:07.873309 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.873295 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-workload-socket\") pod \"34880975-a897-4ad4-af59-18617a93cdad\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " Nov 03 20:33:07.873413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.873400 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-envoy\") pod \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " Nov 03 20:33:07.873518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.873502 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0309a608-f6c4-45ca-991b-33ba73221a6c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0309a608-f6c4-45ca-991b-33ba73221a6c-routing-overlay-volume\") pod \"0309a608-f6c4-45ca-991b-33ba73221a6c\" (UID: \"0309a608-f6c4-45ca-991b-33ba73221a6c\") " Nov 03 20:33:07.873640 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.873626 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-podinfo\") pod \"f0d3c590-f523-4b7e-9086-60df5fc585e8\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " Nov 03 20:33:07.878828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.878805 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-credential-socket\") pod \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " Nov 03 20:33:07.879278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.879258 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-token\" (UniqueName: \"kubernetes.io/projected/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-token\") pod \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " Nov 03 20:33:07.879404 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.879390 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-token\" (UniqueName: \"kubernetes.io/projected/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-token\") pod \"f0d3c590-f523-4b7e-9086-60df5fc585e8\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " Nov 03 20:33:07.879514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.879492 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d3c590-f523-4b7e-9086-60df5fc585e8-kube-api-access-mf55b\" (UniqueName: \"kubernetes.io/projected/f0d3c590-f523-4b7e-9086-60df5fc585e8-kube-api-access-mf55b\") pod \"f0d3c590-f523-4b7e-9086-60df5fc585e8\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " Nov 03 20:33:07.879646 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.879630 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-kube-api-access-jzmbv\" (UniqueName: \"kubernetes.io/projected/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-kube-api-access-jzmbv\") pod \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " Nov 03 20:33:07.879809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.879789 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-kube-api-access-qgszk\" (UniqueName: \"kubernetes.io/projected/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-kube-api-access-qgszk\") pod \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " Nov 03 20:33:07.879959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.879942 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-envoy\") pod \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " Nov 03 20:33:07.880079 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.880064 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-workload-certs\") pod \"4946e3aa-a961-415a-b71d-95164d5fbca3\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " Nov 03 20:33:07.880209 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.880192 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-workload-certs\") pod \"08b3a993-1a64-4fe5-9577-b0f11a36779e\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " Nov 03 20:33:07.880393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.880378 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-envoy\") pod \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " Nov 03 20:33:07.880493 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.880479 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-envoy\") pod \"4946e3aa-a961-415a-b71d-95164d5fbca3\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " Nov 03 20:33:07.880649 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.880629 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-workload-socket\") pod \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " Nov 03 20:33:07.880801 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.880785 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-data\") pod \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " Nov 03 20:33:07.880948 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.880930 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istiod-ca-cert\") pod \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " Nov 03 20:33:07.881076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.881061 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4946e3aa-a961-415a-b71d-95164d5fbca3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4946e3aa-a961-415a-b71d-95164d5fbca3-istiod-ca-cert\") pod \"4946e3aa-a961-415a-b71d-95164d5fbca3\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " Nov 03 20:33:07.881191 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.881176 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-credential-socket\") pod \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.882093 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istiod-ca-cert" pod "a594e899-889e-4f7f-a8ba-b7e7a389ba8a" (UID: "a594e899-889e-4f7f-a8ba-b7e7a389ba8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.882139 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-workload-certs\") pod \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.882203 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-token\" (UniqueName: \"kubernetes.io/projected/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-token\") pod \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.882254 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-podinfo\") pod \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.882298 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-workload-socket\") pod \"4946e3aa-a961-415a-b71d-95164d5fbca3\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.882371 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08b3a993-1a64-4fe5-9577-b0f11a36779e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b3a993-1a64-4fe5-9577-b0f11a36779e-istiod-ca-cert\") pod \"08b3a993-1a64-4fe5-9577-b0f11a36779e\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.882412 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0d3c590-f523-4b7e-9086-60df5fc585e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d3c590-f523-4b7e-9086-60df5fc585e8-istiod-ca-cert\") pod \"f0d3c590-f523-4b7e-9086-60df5fc585e8\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.882463 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-workload-socket\") pod \"f0d3c590-f523-4b7e-9086-60df5fc585e8\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.882520 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-istio-data\") pod \"34880975-a897-4ad4-af59-18617a93cdad\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.882559 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-envoy\") pod \"08b3a993-1a64-4fe5-9577-b0f11a36779e\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.882604 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-workload-certs\") pod \"34880975-a897-4ad4-af59-18617a93cdad\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.882673 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-credential-socket\") pod \"4946e3aa-a961-415a-b71d-95164d5fbca3\" (UID: \"4946e3aa-a961-415a-b71d-95164d5fbca3\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.882747 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34880975-a897-4ad4-af59-18617a93cdad-istio-token\" (UniqueName: \"kubernetes.io/projected/34880975-a897-4ad4-af59-18617a93cdad-istio-token\") pod \"34880975-a897-4ad4-af59-18617a93cdad\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.882841 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-workload-socket\") pod \"7909b42c-1572-4fe2-816a-447020594fe4\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.882894 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34880975-a897-4ad4-af59-18617a93cdad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34880975-a897-4ad4-af59-18617a93cdad-istio-podinfo\") pod \"34880975-a897-4ad4-af59-18617a93cdad\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.882937 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-data\") pod \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\" (UID: \"5ea38154-8a1f-4e1f-8fa0-50481ba4fec0\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.882985 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-data\") pod \"08b3a993-1a64-4fe5-9577-b0f11a36779e\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.883023 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-workload-certs\") pod \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\" (UID: \"171c7288-3157-4fe0-80ee-94cf5b2ca2dd\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.883095 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0309a608-f6c4-45ca-991b-33ba73221a6c-kube-api-access-mtp2k\" (UniqueName: \"kubernetes.io/projected/0309a608-f6c4-45ca-991b-33ba73221a6c-kube-api-access-mtp2k\") pod \"0309a608-f6c4-45ca-991b-33ba73221a6c\" (UID: \"0309a608-f6c4-45ca-991b-33ba73221a6c\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.883149 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-workload-socket\") pod \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.883198 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-istio-envoy\") pod \"34880975-a897-4ad4-af59-18617a93cdad\" (UID: \"34880975-a897-4ad4-af59-18617a93cdad\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.883246 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7909b42c-1572-4fe2-816a-447020594fe4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7909b42c-1572-4fe2-816a-447020594fe4-istiod-ca-cert\") pod \"7909b42c-1572-4fe2-816a-447020594fe4\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.883284 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-workload-certs\") pod \"f0d3c590-f523-4b7e-9086-60df5fc585e8\" (UID: \"f0d3c590-f523-4b7e-9086-60df5fc585e8\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.883331 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08b3a993-1a64-4fe5-9577-b0f11a36779e-kube-api-access-d2779\" (UniqueName: \"kubernetes.io/projected/08b3a993-1a64-4fe5-9577-b0f11a36779e-kube-api-access-d2779\") pod \"08b3a993-1a64-4fe5-9577-b0f11a36779e\" (UID: \"08b3a993-1a64-4fe5-9577-b0f11a36779e\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.883413 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-data\") pod \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\" (UID: \"a594e899-889e-4f7f-a8ba-b7e7a389ba8a\") " Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.884092 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7slk\" (UniqueName: \"kubernetes.io/projected/3eda42a5-5b61-4b34-8447-8767ed097200-kube-api-access-n7slk\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.884351 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeb32457-81a5-4505-b189-2a98f5fda7da-maas-api-tls\") pod \"maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg\" (UID: \"aeb32457-81a5-4505-b189-2a98f5fda7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg" Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.884408 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxj5l\" (UniqueName: \"kubernetes.io/projected/aeb32457-81a5-4505-b189-2a98f5fda7da-kube-api-access-kxj5l\") pod \"maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg\" (UID: \"aeb32457-81a5-4505-b189-2a98f5fda7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg" Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.884455 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-istio-envoy\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.884497 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-credential-socket\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:07.888727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.884607 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeb32457-81a5-4505-b189-2a98f5fda7da-metrics-certs\") pod \"maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg\" (UID: \"aeb32457-81a5-4505-b189-2a98f5fda7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg" Nov 03 20:33:07.890331 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.890301 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-workload-certs" pod "5ea38154-8a1f-4e1f-8fa0-50481ba4fec0" (UID: "5ea38154-8a1f-4e1f-8fa0-50481ba4fec0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.893760 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.893735 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-workload-socket" pod "08b3a993-1a64-4fe5-9577-b0f11a36779e" (UID: "08b3a993-1a64-4fe5-9577-b0f11a36779e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.894114 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.894094 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-workload-socket" pod "171c7288-3157-4fe0-80ee-94cf5b2ca2dd" (UID: "171c7288-3157-4fe0-80ee-94cf5b2ca2dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.894486 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.894465 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-workload-certs" pod "a594e899-889e-4f7f-a8ba-b7e7a389ba8a" (UID: "a594e899-889e-4f7f-a8ba-b7e7a389ba8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.898766 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.898740 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-data" pod "4946e3aa-a961-415a-b71d-95164d5fbca3" (UID: "4946e3aa-a961-415a-b71d-95164d5fbca3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.899111 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.899088 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-workload-socket" pod "34880975-a897-4ad4-af59-18617a93cdad" (UID: "34880975-a897-4ad4-af59-18617a93cdad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.910929 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.910583 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0309a608-f6c4-45ca-991b-33ba73221a6c-routing-overlay-volume" pod "0309a608-f6c4-45ca-991b-33ba73221a6c" (UID: "0309a608-f6c4-45ca-991b-33ba73221a6c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:07.911923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.911819 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-workload-certs" pod "4946e3aa-a961-415a-b71d-95164d5fbca3" (UID: "4946e3aa-a961-415a-b71d-95164d5fbca3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.922897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.922732 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-workload-certs" pod "08b3a993-1a64-4fe5-9577-b0f11a36779e" (UID: "08b3a993-1a64-4fe5-9577-b0f11a36779e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.923550 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.923060 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eda42a5-5b61-4b34-8447-8767ed097200-istio-token\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:07.923550 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.923265 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-workload-socket\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:07.923550 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.923347 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-credential-socket" pod "a594e899-889e-4f7f-a8ba-b7e7a389ba8a" (UID: "a594e899-889e-4f7f-a8ba-b7e7a389ba8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.926053 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.925729 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-workload-socket\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:07.926384 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.926259 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-credential-socket" pod "08b3a993-1a64-4fe5-9577-b0f11a36779e" (UID: "08b3a993-1a64-4fe5-9577-b0f11a36779e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.926933 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.926910 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-workload-socket" pod "4946e3aa-a961-415a-b71d-95164d5fbca3" (UID: "4946e3aa-a961-415a-b71d-95164d5fbca3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.927263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.927163 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b3a993-1a64-4fe5-9577-b0f11a36779e-istiod-ca-cert" pod "08b3a993-1a64-4fe5-9577-b0f11a36779e" (UID: "08b3a993-1a64-4fe5-9577-b0f11a36779e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:07.927444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.927428 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d3c590-f523-4b7e-9086-60df5fc585e8-istiod-ca-cert" pod "f0d3c590-f523-4b7e-9086-60df5fc585e8" (UID: "f0d3c590-f523-4b7e-9086-60df5fc585e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:07.927682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.927616 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-workload-socket" pod "f0d3c590-f523-4b7e-9086-60df5fc585e8" (UID: "f0d3c590-f523-4b7e-9086-60df5fc585e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.927807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.927793 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-istio-data" pod "34880975-a897-4ad4-af59-18617a93cdad" (UID: "34880975-a897-4ad4-af59-18617a93cdad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.928357 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.928147 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-workload-socket" pod "5ea38154-8a1f-4e1f-8fa0-50481ba4fec0" (UID: "5ea38154-8a1f-4e1f-8fa0-50481ba4fec0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.928550 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.928471 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-data" pod "171c7288-3157-4fe0-80ee-94cf5b2ca2dd" (UID: "171c7288-3157-4fe0-80ee-94cf5b2ca2dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.929750 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.929480 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-istio-data\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:07.929750 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.929625 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-workload-certs\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:07.930234 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.929960 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eda42a5-5b61-4b34-8447-8767ed097200-istiod-ca-cert\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:07.930596 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.930332 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eda42a5-5b61-4b34-8447-8767ed097200-istio-podinfo\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:07.930596 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.930415 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb32457-81a5-4505-b189-2a98f5fda7da-openshift-service-ca\") pod \"maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg\" (UID: \"aeb32457-81a5-4505-b189-2a98f5fda7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg" Nov 03 20:33:07.930734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.930680 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.930734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.930706 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.930734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.930731 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.930923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.930764 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.930923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.930783 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.930923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.930798 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.930923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.930834 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.930923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.930852 5059 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0309a608-f6c4-45ca-991b-33ba73221a6c-routing-overlay-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.930923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.930869 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.930923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.930890 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.930923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.930921 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.931304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.930938 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.931304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.930955 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.931304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.930990 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.931304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.931009 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.931304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.931026 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b3a993-1a64-4fe5-9577-b0f11a36779e-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.931304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.931043 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d3c590-f523-4b7e-9086-60df5fc585e8-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.931304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.931079 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.931304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.931095 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.932391 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.932119 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-istio-data\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:07.932391 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.932307 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-workload-certs\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:07.934241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.932850 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eda42a5-5b61-4b34-8447-8767ed097200-istiod-ca-cert\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:07.934241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.933273 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:33:07.934241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.933644 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istiod-ca-cert" pod "5ea38154-8a1f-4e1f-8fa0-50481ba4fec0" (UID: "5ea38154-8a1f-4e1f-8fa0-50481ba4fec0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:07.934241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.933931 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-podinfo" pod "f0d3c590-f523-4b7e-9086-60df5fc585e8" (UID: "f0d3c590-f523-4b7e-9086-60df5fc585e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:07.934241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.934021 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-envoy" pod "a594e899-889e-4f7f-a8ba-b7e7a389ba8a" (UID: "a594e899-889e-4f7f-a8ba-b7e7a389ba8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.934592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.934000 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-credential-socket" pod "5ea38154-8a1f-4e1f-8fa0-50481ba4fec0" (UID: "5ea38154-8a1f-4e1f-8fa0-50481ba4fec0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.934592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.934401 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb32457-81a5-4505-b189-2a98f5fda7da-openshift-service-ca\") pod \"maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg\" (UID: \"aeb32457-81a5-4505-b189-2a98f5fda7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg" Nov 03 20:33:07.934815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.934776 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeb32457-81a5-4505-b189-2a98f5fda7da-metrics-certs\") pod \"maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg\" (UID: \"aeb32457-81a5-4505-b189-2a98f5fda7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg" Nov 03 20:33:07.934973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.934933 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0309a608-f6c4-45ca-991b-33ba73221a6c-plugins-config-volume" pod "0309a608-f6c4-45ca-991b-33ba73221a6c" (UID: "0309a608-f6c4-45ca-991b-33ba73221a6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:07.935477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.935350 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-token" pod "5ea38154-8a1f-4e1f-8fa0-50481ba4fec0" (UID: "5ea38154-8a1f-4e1f-8fa0-50481ba4fec0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:07.937376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.935974 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-workload-socket" pod "7909b42c-1572-4fe2-816a-447020594fe4" (UID: "7909b42c-1572-4fe2-816a-447020594fe4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.937376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.936225 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-workload-certs" pod "34880975-a897-4ad4-af59-18617a93cdad" (UID: "34880975-a897-4ad4-af59-18617a93cdad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.937376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.936467 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-credential-socket" pod "4946e3aa-a961-415a-b71d-95164d5fbca3" (UID: "4946e3aa-a961-415a-b71d-95164d5fbca3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.937376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.936580 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4946e3aa-a961-415a-b71d-95164d5fbca3-istiod-ca-cert" pod "4946e3aa-a961-415a-b71d-95164d5fbca3" (UID: "4946e3aa-a961-415a-b71d-95164d5fbca3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:07.937376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.937031 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-workload-certs" pod "f0d3c590-f523-4b7e-9086-60df5fc585e8" (UID: "f0d3c590-f523-4b7e-9086-60df5fc585e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.937376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.937249 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-data" pod "5ea38154-8a1f-4e1f-8fa0-50481ba4fec0" (UID: "5ea38154-8a1f-4e1f-8fa0-50481ba4fec0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.937780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.937444 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-data" pod "08b3a993-1a64-4fe5-9577-b0f11a36779e" (UID: "08b3a993-1a64-4fe5-9577-b0f11a36779e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.937833 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.937759 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-workload-certs" pod "171c7288-3157-4fe0-80ee-94cf5b2ca2dd" (UID: "171c7288-3157-4fe0-80ee-94cf5b2ca2dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.940527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:07.939275 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-data" pod "a594e899-889e-4f7f-a8ba-b7e7a389ba8a" (UID: "a594e899-889e-4f7f-a8ba-b7e7a389ba8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.038971 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.038699 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-token" pod "171c7288-3157-4fe0-80ee-94cf5b2ca2dd" (UID: "171c7288-3157-4fe0-80ee-94cf5b2ca2dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.049175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.044337 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-workload-socket" pod "a594e899-889e-4f7f-a8ba-b7e7a389ba8a" (UID: "a594e899-889e-4f7f-a8ba-b7e7a389ba8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.049175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.044476 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4946e3aa-a961-415a-b71d-95164d5fbca3-kube-api-access-wltn5" pod "4946e3aa-a961-415a-b71d-95164d5fbca3" (UID: "4946e3aa-a961-415a-b71d-95164d5fbca3"). InnerVolumeSpecName "kube-api-access-wltn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.049175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.044890 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34880975-a897-4ad4-af59-18617a93cdad-istiod-ca-cert" pod "34880975-a897-4ad4-af59-18617a93cdad" (UID: "34880975-a897-4ad4-af59-18617a93cdad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:08.049175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.044986 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-credential-socket" pod "171c7288-3157-4fe0-80ee-94cf5b2ca2dd" (UID: "171c7288-3157-4fe0-80ee-94cf5b2ca2dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.049175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.044281 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-istio-envoy\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:08.049175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.047402 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-credential-socket" pod "f0d3c590-f523-4b7e-9086-60df5fc585e8" (UID: "f0d3c590-f523-4b7e-9086-60df5fc585e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.049175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.047606 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-credential-socket" pod "34880975-a897-4ad4-af59-18617a93cdad" (UID: "34880975-a897-4ad4-af59-18617a93cdad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.050904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.050839 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-credential-socket\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:08.051698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.051320 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-istio-envoy" pod "34880975-a897-4ad4-af59-18617a93cdad" (UID: "34880975-a897-4ad4-af59-18617a93cdad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.051698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.051480 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istiod-ca-cert" pod "171c7288-3157-4fe0-80ee-94cf5b2ca2dd" (UID: "171c7288-3157-4fe0-80ee-94cf5b2ca2dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:08.051698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.051516 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-envoy" pod "4946e3aa-a961-415a-b71d-95164d5fbca3" (UID: "4946e3aa-a961-415a-b71d-95164d5fbca3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.051698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.051530 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-envoy" pod "5ea38154-8a1f-4e1f-8fa0-50481ba4fec0" (UID: "5ea38154-8a1f-4e1f-8fa0-50481ba4fec0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.051698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.051570 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-envoy" pod "08b3a993-1a64-4fe5-9577-b0f11a36779e" (UID: "08b3a993-1a64-4fe5-9577-b0f11a36779e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.051698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.051694 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-token" pod "a594e899-889e-4f7f-a8ba-b7e7a389ba8a" (UID: "a594e899-889e-4f7f-a8ba-b7e7a389ba8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.052029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.051776 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-token" pod "f0d3c590-f523-4b7e-9086-60df5fc585e8" (UID: "f0d3c590-f523-4b7e-9086-60df5fc585e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.052029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.051830 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-envoy" pod "171c7288-3157-4fe0-80ee-94cf5b2ca2dd" (UID: "171c7288-3157-4fe0-80ee-94cf5b2ca2dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.099687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.057590 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-token" pod "4946e3aa-a961-415a-b71d-95164d5fbca3" (UID: "4946e3aa-a961-415a-b71d-95164d5fbca3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.099687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.057730 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b3a993-1a64-4fe5-9577-b0f11a36779e-kube-api-access-d2779" pod "08b3a993-1a64-4fe5-9577-b0f11a36779e" (UID: "08b3a993-1a64-4fe5-9577-b0f11a36779e"). InnerVolumeSpecName "kube-api-access-d2779". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.099687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.057805 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d3c590-f523-4b7e-9086-60df5fc585e8-kube-api-access-mf55b" pod "f0d3c590-f523-4b7e-9086-60df5fc585e8" (UID: "f0d3c590-f523-4b7e-9086-60df5fc585e8"). InnerVolumeSpecName "kube-api-access-mf55b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.099687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.080260 5059 scope.go:122] "RemoveContainer" containerID="56b80e220cc2df35563d8697c1e14db7387aa8d2faaa80e6b681d5be3a5c172e" Nov 03 20:33:08.099687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.092173 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7909b42c-1572-4fe2-816a-447020594fe4-istiod-ca-cert" pod "7909b42c-1572-4fe2-816a-447020594fe4" (UID: "7909b42c-1572-4fe2-816a-447020594fe4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:08.129209 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.122298 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34880975-a897-4ad4-af59-18617a93cdad-kube-api-access-5cjvf" pod "34880975-a897-4ad4-af59-18617a93cdad" (UID: "34880975-a897-4ad4-af59-18617a93cdad"). InnerVolumeSpecName "kube-api-access-5cjvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.129209 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.124119 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-podinfo" pod "a594e899-889e-4f7f-a8ba-b7e7a389ba8a" (UID: "a594e899-889e-4f7f-a8ba-b7e7a389ba8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:08.129209 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.124214 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-kube-api-access-qgszk" pod "171c7288-3157-4fe0-80ee-94cf5b2ca2dd" (UID: "171c7288-3157-4fe0-80ee-94cf5b2ca2dd"). InnerVolumeSpecName "kube-api-access-qgszk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.129209 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.124258 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-kube-api-access-bjdzj" pod "a594e899-889e-4f7f-a8ba-b7e7a389ba8a" (UID: "a594e899-889e-4f7f-a8ba-b7e7a389ba8a"). InnerVolumeSpecName "kube-api-access-bjdzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.129209 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.124300 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-podinfo" pod "5ea38154-8a1f-4e1f-8fa0-50481ba4fec0" (UID: "5ea38154-8a1f-4e1f-8fa0-50481ba4fec0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:08.129209 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.124341 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34880975-a897-4ad4-af59-18617a93cdad-istio-token" pod "34880975-a897-4ad4-af59-18617a93cdad" (UID: "34880975-a897-4ad4-af59-18617a93cdad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.129209 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.124849 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-kube-api-access-jzmbv" pod "5ea38154-8a1f-4e1f-8fa0-50481ba4fec0" (UID: "5ea38154-8a1f-4e1f-8fa0-50481ba4fec0"). InnerVolumeSpecName "kube-api-access-jzmbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.129209 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.125326 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eda42a5-5b61-4b34-8447-8767ed097200-istio-podinfo\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:08.129209 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.125416 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34880975-a897-4ad4-af59-18617a93cdad-istio-podinfo" pod "34880975-a897-4ad4-af59-18617a93cdad" (UID: "34880975-a897-4ad4-af59-18617a93cdad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:08.129209 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.125504 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0309a608-f6c4-45ca-991b-33ba73221a6c-provider-credentials" pod "0309a608-f6c4-45ca-991b-33ba73221a6c" (UID: "0309a608-f6c4-45ca-991b-33ba73221a6c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.129209 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.125552 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-envoy" pod "f0d3c590-f523-4b7e-9086-60df5fc585e8" (UID: "f0d3c590-f523-4b7e-9086-60df5fc585e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.129209 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.127609 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeb32457-81a5-4505-b189-2a98f5fda7da-maas-api-tls\") pod \"maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg\" (UID: \"aeb32457-81a5-4505-b189-2a98f5fda7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg" Nov 03 20:33:08.129209 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.128610 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-podinfo" pod "4946e3aa-a961-415a-b71d-95164d5fbca3" (UID: "4946e3aa-a961-415a-b71d-95164d5fbca3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:08.138136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.132410 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-credential-socket\") pod \"c9721811-d27d-4e13-a53d-4ae81e16fb96\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " Nov 03 20:33:08.138136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.132513 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-token\" (UniqueName: \"kubernetes.io/projected/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-token\") pod \"c9721811-d27d-4e13-a53d-4ae81e16fb96\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " Nov 03 20:33:08.138136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.132589 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7909b42c-1572-4fe2-816a-447020594fe4-kube-api-access-zksbz\" (UniqueName: \"kubernetes.io/projected/7909b42c-1572-4fe2-816a-447020594fe4-kube-api-access-zksbz\") pod \"7909b42c-1572-4fe2-816a-447020594fe4\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " Nov 03 20:33:08.138136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.132655 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-workload-certs\") pod \"c9721811-d27d-4e13-a53d-4ae81e16fb96\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " Nov 03 20:33:08.138136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.132717 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-istio-envoy\") pod \"7909b42c-1572-4fe2-816a-447020594fe4\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " Nov 03 20:33:08.138136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.132870 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-workload-certs\") pod \"7909b42c-1572-4fe2-816a-447020594fe4\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " Nov 03 20:33:08.138136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.132901 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-data\") pod \"c9721811-d27d-4e13-a53d-4ae81e16fb96\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " Nov 03 20:33:08.138136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.132935 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-podinfo\") pod \"c9721811-d27d-4e13-a53d-4ae81e16fb96\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " Nov 03 20:33:08.138136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.133026 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7909b42c-1572-4fe2-816a-447020594fe4-istio-token\" (UniqueName: \"kubernetes.io/projected/7909b42c-1572-4fe2-816a-447020594fe4-istio-token\") pod \"7909b42c-1572-4fe2-816a-447020594fe4\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " Nov 03 20:33:08.138136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.133143 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7909b42c-1572-4fe2-816a-447020594fe4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7909b42c-1572-4fe2-816a-447020594fe4-istio-podinfo\") pod \"7909b42c-1572-4fe2-816a-447020594fe4\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " Nov 03 20:33:08.138136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.133174 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9721811-d27d-4e13-a53d-4ae81e16fb96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9721811-d27d-4e13-a53d-4ae81e16fb96-istiod-ca-cert\") pod \"c9721811-d27d-4e13-a53d-4ae81e16fb96\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " Nov 03 20:33:08.138136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.133232 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-envoy\") pod \"c9721811-d27d-4e13-a53d-4ae81e16fb96\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " Nov 03 20:33:08.138136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.133283 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-workload-socket\") pod \"c9721811-d27d-4e13-a53d-4ae81e16fb96\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " Nov 03 20:33:08.138136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.133362 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-credential-socket\") pod \"7909b42c-1572-4fe2-816a-447020594fe4\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " Nov 03 20:33:08.138136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.133390 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9721811-d27d-4e13-a53d-4ae81e16fb96-kube-api-access-d6qbw\" (UniqueName: \"kubernetes.io/projected/c9721811-d27d-4e13-a53d-4ae81e16fb96-kube-api-access-d6qbw\") pod \"c9721811-d27d-4e13-a53d-4ae81e16fb96\" (UID: \"c9721811-d27d-4e13-a53d-4ae81e16fb96\") " Nov 03 20:33:08.138136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.133484 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-istio-data\") pod \"7909b42c-1572-4fe2-816a-447020594fe4\" (UID: \"7909b42c-1572-4fe2-816a-447020594fe4\") " Nov 03 20:33:08.138136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.134673 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-token" pod "08b3a993-1a64-4fe5-9577-b0f11a36779e" (UID: "08b3a993-1a64-4fe5-9577-b0f11a36779e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.138136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.137515 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eda42a5-5b61-4b34-8447-8767ed097200-istio-token\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:08.142876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.142848 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0309a608-f6c4-45ca-991b-33ba73221a6c-kube-api-access-mtp2k" pod "0309a608-f6c4-45ca-991b-33ba73221a6c" (UID: "0309a608-f6c4-45ca-991b-33ba73221a6c"). InnerVolumeSpecName "kube-api-access-mtp2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.143143 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.143120 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-podinfo" pod "171c7288-3157-4fe0-80ee-94cf5b2ca2dd" (UID: "171c7288-3157-4fe0-80ee-94cf5b2ca2dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:08.143969 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.143948 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxj5l\" (UniqueName: \"kubernetes.io/projected/aeb32457-81a5-4505-b189-2a98f5fda7da-kube-api-access-kxj5l\") pod \"maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg\" (UID: \"aeb32457-81a5-4505-b189-2a98f5fda7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg" Nov 03 20:33:08.144145 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.144125 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body= Nov 03 20:33:08.144680 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.144639 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" podUID="d931dc13-4399-48dc-80e0-b0b304c5dda0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" Nov 03 20:33:08.145123 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.145087 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-data" pod "f0d3c590-f523-4b7e-9086-60df5fc585e8" (UID: "f0d3c590-f523-4b7e-9086-60df5fc585e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.145752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.145730 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-istio-data" pod "7909b42c-1572-4fe2-816a-447020594fe4" (UID: "7909b42c-1572-4fe2-816a-447020594fe4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.149420 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.149356 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-workload-socket" pod "c9721811-d27d-4e13-a53d-4ae81e16fb96" (UID: "c9721811-d27d-4e13-a53d-4ae81e16fb96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.151792 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.151599 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9721811-d27d-4e13-a53d-4ae81e16fb96-istiod-ca-cert" pod "c9721811-d27d-4e13-a53d-4ae81e16fb96" (UID: "c9721811-d27d-4e13-a53d-4ae81e16fb96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:08.151910 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.151879 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-credential-socket" pod "7909b42c-1572-4fe2-816a-447020594fe4" (UID: "7909b42c-1572-4fe2-816a-447020594fe4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.155253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.155205 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-credential-socket" pod "c9721811-d27d-4e13-a53d-4ae81e16fb96" (UID: "c9721811-d27d-4e13-a53d-4ae81e16fb96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.157002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.156193 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-workload-certs" pod "c9721811-d27d-4e13-a53d-4ae81e16fb96" (UID: "c9721811-d27d-4e13-a53d-4ae81e16fb96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.165950 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.165976 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.165991 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166004 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166019 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166033 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf55b\" (UniqueName: \"kubernetes.io/projected/f0d3c590-f523-4b7e-9086-60df5fc585e8-kube-api-access-mf55b\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166048 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzmbv\" (UniqueName: \"kubernetes.io/projected/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-kube-api-access-jzmbv\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166061 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgszk\" (UniqueName: \"kubernetes.io/projected/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-kube-api-access-qgszk\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166075 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166091 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166105 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166149 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166170 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4946e3aa-a961-415a-b71d-95164d5fbca3-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166186 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166200 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166215 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166230 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166245 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166281 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4946e3aa-a961-415a-b71d-95164d5fbca3-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166296 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34880975-a897-4ad4-af59-18617a93cdad-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166312 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166327 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34880975-a897-4ad4-af59-18617a93cdad-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166342 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166356 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166371 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166387 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtp2k\" (UniqueName: \"kubernetes.io/projected/0309a608-f6c4-45ca-991b-33ba73221a6c-kube-api-access-mtp2k\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166402 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166415 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166428 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7909b42c-1572-4fe2-816a-447020594fe4-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166443 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166460 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2779\" (UniqueName: \"kubernetes.io/projected/08b3a993-1a64-4fe5-9577-b0f11a36779e-kube-api-access-d2779\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166474 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166489 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9721811-d27d-4e13-a53d-4ae81e16fb96-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166504 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166527 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166542 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166557 5059 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0309a608-f6c4-45ca-991b-33ba73221a6c-provider-credentials\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166572 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0309a608-f6c4-45ca-991b-33ba73221a6c-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166586 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjdzj\" (UniqueName: \"kubernetes.io/projected/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-kube-api-access-bjdzj\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166600 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166615 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4946e3aa-a961-415a-b71d-95164d5fbca3-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166629 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a594e899-889e-4f7f-a8ba-b7e7a389ba8a-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166644 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166672 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166687 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34880975-a897-4ad4-af59-18617a93cdad-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166699 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166714 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cjvf\" (UniqueName: \"kubernetes.io/projected/34880975-a897-4ad4-af59-18617a93cdad-kube-api-access-5cjvf\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166727 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166738 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166750 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166765 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34880975-a897-4ad4-af59-18617a93cdad-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166778 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wltn5\" (UniqueName: \"kubernetes.io/projected/4946e3aa-a961-415a-b71d-95164d5fbca3-kube-api-access-wltn5\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.172159 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166792 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.174607 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166806 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171c7288-3157-4fe0-80ee-94cf5b2ca2dd-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.174607 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166820 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d3c590-f523-4b7e-9086-60df5fc585e8-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.174607 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.166985 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn" Nov 03 20:33:08.174607 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.167295 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9721811-d27d-4e13-a53d-4ae81e16fb96-kube-api-access-d6qbw" pod "c9721811-d27d-4e13-a53d-4ae81e16fb96" (UID: "c9721811-d27d-4e13-a53d-4ae81e16fb96"). InnerVolumeSpecName "kube-api-access-d6qbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.226683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.192525 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:33:08.226683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.192597 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" podUID="c2d80995-5f1c-46db-8d71-a130cb92e279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:08.226683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.222990 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8" podStartSLOduration=66.222975299 podStartE2EDuration="1m6.222975299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:07.295231035 +0000 UTC m=+3381.553028293" watchObservedRunningTime="2026-11-03 20:33:08.222975299 +0000 UTC m=+3382.480772556" Nov 03 20:33:08.226683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.223160 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-data" pod "c9721811-d27d-4e13-a53d-4ae81e16fb96" (UID: "c9721811-d27d-4e13-a53d-4ae81e16fb96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.226683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.223386 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-workload-certs" pod "7909b42c-1572-4fe2-816a-447020594fe4" (UID: "7909b42c-1572-4fe2-816a-447020594fe4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.244815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.244784 5059 scope.go:122] "RemoveContainer" containerID="b5a63189fd2f2d22d3e6f03c32065113a3ccf85221d07eb70f923a647b5b8d38" Nov 03 20:33:08.252873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.252843 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:33:08.253038 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.253014 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" podUID="08b3a993-1a64-4fe5-9577-b0f11a36779e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:08.319444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.319403 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-podinfo" pod "08b3a993-1a64-4fe5-9577-b0f11a36779e" (UID: "08b3a993-1a64-4fe5-9577-b0f11a36779e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:08.332715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.321357 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg"] Nov 03 20:33:08.332715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.332601 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c9721811-d27d-4e13-a53d-4ae81e16fb96" containerName="istio-proxy" Nov 03 20:33:08.332715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.332623 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9721811-d27d-4e13-a53d-4ae81e16fb96" containerName="istio-proxy" Nov 03 20:33:08.332715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.332638 5059 cpu_manager.go:414] "containerMap: removing container" podUID="25ceab3d-6344-4512-a471-d91aec0cb48c" containerName="payload-processing" Nov 03 20:33:08.332715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.332645 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ceab3d-6344-4512-a471-d91aec0cb48c" containerName="payload-processing" Nov 03 20:33:08.332715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.332675 5059 cpu_manager.go:414] "containerMap: removing container" podUID="0309a608-f6c4-45ca-991b-33ba73221a6c" containerName="payload-processing" Nov 03 20:33:08.332715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.332683 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0309a608-f6c4-45ca-991b-33ba73221a6c" containerName="payload-processing" Nov 03 20:33:08.332715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.332699 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c9721811-d27d-4e13-a53d-4ae81e16fb96" containerName="istio-proxy" Nov 03 20:33:08.333569 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333425 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9721811-d27d-4e13-a53d-4ae81e16fb96" containerName="istio-proxy" Nov 03 20:33:08.333569 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333450 5059 cpu_manager.go:414] "containerMap: removing container" podUID="5ea38154-8a1f-4e1f-8fa0-50481ba4fec0" containerName="istio-proxy" Nov 03 20:33:08.333569 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333458 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea38154-8a1f-4e1f-8fa0-50481ba4fec0" containerName="istio-proxy" Nov 03 20:33:08.333569 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333473 5059 cpu_manager.go:414] "containerMap: removing container" podUID="7909b42c-1572-4fe2-816a-447020594fe4" containerName="istio-proxy" Nov 03 20:33:08.333569 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333480 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7909b42c-1572-4fe2-816a-447020594fe4" containerName="istio-proxy" Nov 03 20:33:08.333569 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333496 5059 cpu_manager.go:414] "containerMap: removing container" podUID="08b3a993-1a64-4fe5-9577-b0f11a36779e" containerName="istio-proxy" Nov 03 20:33:08.333569 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333503 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08b3a993-1a64-4fe5-9577-b0f11a36779e" containerName="istio-proxy" Nov 03 20:33:08.333569 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333525 5059 cpu_manager.go:414] "containerMap: removing container" podUID="7909b42c-1572-4fe2-816a-447020594fe4" containerName="istio-proxy" Nov 03 20:33:08.333569 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333533 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7909b42c-1572-4fe2-816a-447020594fe4" containerName="istio-proxy" Nov 03 20:33:08.333569 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333548 5059 cpu_manager.go:414] "containerMap: removing container" podUID="a594e899-889e-4f7f-a8ba-b7e7a389ba8a" containerName="istio-proxy" Nov 03 20:33:08.333569 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333555 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a594e899-889e-4f7f-a8ba-b7e7a389ba8a" containerName="istio-proxy" Nov 03 20:33:08.334117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333610 5059 cpu_manager.go:414] "containerMap: removing container" podUID="4946e3aa-a961-415a-b71d-95164d5fbca3" containerName="istio-proxy" Nov 03 20:33:08.334117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333617 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4946e3aa-a961-415a-b71d-95164d5fbca3" containerName="istio-proxy" Nov 03 20:33:08.334117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333627 5059 cpu_manager.go:414] "containerMap: removing container" podUID="a594e899-889e-4f7f-a8ba-b7e7a389ba8a" containerName="istio-proxy" Nov 03 20:33:08.334117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333634 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a594e899-889e-4f7f-a8ba-b7e7a389ba8a" containerName="istio-proxy" Nov 03 20:33:08.334117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333676 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25ceab3d-6344-4512-a471-d91aec0cb48c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ceab3d-6344-4512-a471-d91aec0cb48c-plugins-config-volume\") pod \"25ceab3d-6344-4512-a471-d91aec0cb48c\" (UID: \"25ceab3d-6344-4512-a471-d91aec0cb48c\") " Nov 03 20:33:08.334117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333849 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4946e3aa-a961-415a-b71d-95164d5fbca3" containerName="istio-proxy" Nov 03 20:33:08.334117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333865 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a594e899-889e-4f7f-a8ba-b7e7a389ba8a" containerName="istio-proxy" Nov 03 20:33:08.334117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333876 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9721811-d27d-4e13-a53d-4ae81e16fb96" containerName="istio-proxy" Nov 03 20:33:08.334117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333885 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ceab3d-6344-4512-a471-d91aec0cb48c" containerName="payload-processing" Nov 03 20:33:08.334117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333895 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0309a608-f6c4-45ca-991b-33ba73221a6c" containerName="payload-processing" Nov 03 20:33:08.334117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333904 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9721811-d27d-4e13-a53d-4ae81e16fb96" containerName="istio-proxy" Nov 03 20:33:08.334117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333914 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ea38154-8a1f-4e1f-8fa0-50481ba4fec0" containerName="istio-proxy" Nov 03 20:33:08.334117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333921 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7909b42c-1572-4fe2-816a-447020594fe4" containerName="istio-proxy" Nov 03 20:33:08.334117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333932 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7909b42c-1572-4fe2-816a-447020594fe4" containerName="istio-proxy" Nov 03 20:33:08.334117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.334108 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25ceab3d-6344-4512-a471-d91aec0cb48c-kube-api-access-hpnqh\" (UniqueName: \"kubernetes.io/projected/25ceab3d-6344-4512-a471-d91aec0cb48c-kube-api-access-hpnqh\") pod \"25ceab3d-6344-4512-a471-d91aec0cb48c\" (UID: \"25ceab3d-6344-4512-a471-d91aec0cb48c\") " Nov 03 20:33:08.335020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.334191 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7slk\" (UniqueName: \"kubernetes.io/projected/3eda42a5-5b61-4b34-8447-8767ed097200-kube-api-access-n7slk\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:08.335020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.332818 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:33:08.335020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333390 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7909b42c-1572-4fe2-816a-447020594fe4-kube-api-access-zksbz" pod "7909b42c-1572-4fe2-816a-447020594fe4" (UID: "7909b42c-1572-4fe2-816a-447020594fe4"). InnerVolumeSpecName "kube-api-access-zksbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.335020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333410 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-token" pod "c9721811-d27d-4e13-a53d-4ae81e16fb96" (UID: "c9721811-d27d-4e13-a53d-4ae81e16fb96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.335020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333416 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7909b42c-1572-4fe2-816a-447020594fe4-istio-podinfo" pod "7909b42c-1572-4fe2-816a-447020594fe4" (UID: "7909b42c-1572-4fe2-816a-447020594fe4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:08.335020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333584 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-podinfo" pod "c9721811-d27d-4e13-a53d-4ae81e16fb96" (UID: "c9721811-d27d-4e13-a53d-4ae81e16fb96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:08.335020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333633 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-envoy" pod "c9721811-d27d-4e13-a53d-4ae81e16fb96" (UID: "c9721811-d27d-4e13-a53d-4ae81e16fb96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.335020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333728 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7909b42c-1572-4fe2-816a-447020594fe4-istio-token" pod "7909b42c-1572-4fe2-816a-447020594fe4" (UID: "7909b42c-1572-4fe2-816a-447020594fe4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.335020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.333927 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-istio-envoy" pod "7909b42c-1572-4fe2-816a-447020594fe4" (UID: "7909b42c-1572-4fe2-816a-447020594fe4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.335020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.334558 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25ceab3d-6344-4512-a471-d91aec0cb48c-plugins-config-volume" pod "25ceab3d-6344-4512-a471-d91aec0cb48c" (UID: "25ceab3d-6344-4512-a471-d91aec0cb48c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:08.335020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.334904 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.335020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.334927 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7909b42c-1572-4fe2-816a-447020594fe4-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.335020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.334942 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.335020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.334955 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.335020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.334969 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7909b42c-1572-4fe2-816a-447020594fe4-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.335020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.334982 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7909b42c-1572-4fe2-816a-447020594fe4-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.335020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.334996 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.335020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.335009 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ceab3d-6344-4512-a471-d91aec0cb48c-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.335020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.335024 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6qbw\" (UniqueName: \"kubernetes.io/projected/c9721811-d27d-4e13-a53d-4ae81e16fb96-kube-api-access-d6qbw\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.335822 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.335038 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b3a993-1a64-4fe5-9577-b0f11a36779e-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.335822 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.335052 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9721811-d27d-4e13-a53d-4ae81e16fb96-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.335822 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.335068 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zksbz\" (UniqueName: \"kubernetes.io/projected/7909b42c-1572-4fe2-816a-447020594fe4-kube-api-access-zksbz\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.337873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.337850 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ceab3d-6344-4512-a471-d91aec0cb48c-kube-api-access-hpnqh" pod "25ceab3d-6344-4512-a471-d91aec0cb48c" (UID: "25ceab3d-6344-4512-a471-d91aec0cb48c"). InnerVolumeSpecName "kube-api-access-hpnqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.338237 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.338164 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" Nov 03 20:33:08.338237 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.338200 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg"] Nov 03 20:33:08.338237 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.338231 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-fzr95" event={"ID":"191c7922-07dd-4c9c-89d2-d804f638854d","Type":"ContainerStarted","Data":"5b8b13cbc2033c173ade8b942c9ceb1488367004b85a0b7ef2da11d87503d3de"} Nov 03 20:33:08.338565 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.338266 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.338565 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.338266 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7"] Nov 03 20:33:08.339163 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.339136 5059 cpu_manager.go:414] "containerMap: removing container" podUID="4c609469-d7d7-4530-8d74-e34aa56dda8f" containerName="istio-proxy" Nov 03 20:33:08.339163 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.339164 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c609469-d7d7-4530-8d74-e34aa56dda8f" containerName="istio-proxy" Nov 03 20:33:08.339285 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.339173 5059 cpu_manager.go:414] "containerMap: removing container" podUID="4c609469-d7d7-4530-8d74-e34aa56dda8f" containerName="istio-proxy" Nov 03 20:33:08.339285 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.339178 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c609469-d7d7-4530-8d74-e34aa56dda8f" containerName="istio-proxy" Nov 03 20:33:08.339370 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.339320 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c609469-d7d7-4530-8d74-e34aa56dda8f" containerName="istio-proxy" Nov 03 20:33:08.339370 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.339327 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c609469-d7d7-4530-8d74-e34aa56dda8f" containerName="istio-proxy" Nov 03 20:33:08.339370 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.339337 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a594e899-889e-4f7f-a8ba-b7e7a389ba8a" containerName="istio-proxy" Nov 03 20:33:08.341975 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.341930 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h" podStartSLOduration=66.341912711 podStartE2EDuration="1m6.341912711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:07.373717828 +0000 UTC m=+3381.631515087" watchObservedRunningTime="2026-11-03 20:33:08.341912711 +0000 UTC m=+3382.599709970" Nov 03 20:33:08.344889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.343272 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7"] Nov 03 20:33:08.344889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.343328 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr"] Nov 03 20:33:08.344889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.343744 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.349263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.349245 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.360188 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.360078 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr"] Nov 03 20:33:08.388931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.388883 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 20:33:08.413027 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.412287 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf"] Nov 03 20:33:08.414452 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.414212 5059 cpu_manager.go:414] "containerMap: removing container" podUID="fc57130b-c950-49f2-8a98-22c47bd98288" containerName="istio-proxy" Nov 03 20:33:08.414452 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.414241 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc57130b-c950-49f2-8a98-22c47bd98288" containerName="istio-proxy" Nov 03 20:33:08.416350 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.416053 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc57130b-c950-49f2-8a98-22c47bd98288" containerName="istio-proxy" Nov 03 20:33:08.427841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.427820 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.435746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.435723 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf"] Nov 03 20:33:08.436897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.436874 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-workload-socket\") pod \"4c609469-d7d7-4530-8d74-e34aa56dda8f\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " Nov 03 20:33:08.437110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.437090 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-data\") pod \"4c609469-d7d7-4530-8d74-e34aa56dda8f\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " Nov 03 20:33:08.437233 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.437220 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c609469-d7d7-4530-8d74-e34aa56dda8f-kube-api-access-wskdx\" (UniqueName: \"kubernetes.io/projected/4c609469-d7d7-4530-8d74-e34aa56dda8f-kube-api-access-wskdx\") pod \"4c609469-d7d7-4530-8d74-e34aa56dda8f\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " Nov 03 20:33:08.437442 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.437424 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-podinfo\") pod \"4c609469-d7d7-4530-8d74-e34aa56dda8f\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " Nov 03 20:33:08.437717 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.437698 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-token\" (UniqueName: \"kubernetes.io/projected/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-token\") pod \"4c609469-d7d7-4530-8d74-e34aa56dda8f\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " Nov 03 20:33:08.437847 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.437831 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-credential-socket\") pod \"4c609469-d7d7-4530-8d74-e34aa56dda8f\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " Nov 03 20:33:08.438115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.438093 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-envoy\") pod \"4c609469-d7d7-4530-8d74-e34aa56dda8f\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " Nov 03 20:33:08.440778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.439471 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c609469-d7d7-4530-8d74-e34aa56dda8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c609469-d7d7-4530-8d74-e34aa56dda8f-istiod-ca-cert\") pod \"4c609469-d7d7-4530-8d74-e34aa56dda8f\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " Nov 03 20:33:08.440778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.439550 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-workload-certs\") pod \"4c609469-d7d7-4530-8d74-e34aa56dda8f\" (UID: \"4c609469-d7d7-4530-8d74-e34aa56dda8f\") " Nov 03 20:33:08.440778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.439893 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.440778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.439981 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.440778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.440023 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.440778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.440178 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.440778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.440203 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.440778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.440232 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.440778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.440352 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.440778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.440396 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2fe4277-3acc-4378-86d9-374b69918ec9-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.440778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.440417 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.440778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.440476 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.440778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.440529 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhmg9\" (UniqueName: \"kubernetes.io/projected/e2fe4277-3acc-4378-86d9-374b69918ec9-kube-api-access-bhmg9\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.440778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.440570 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.440778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.440628 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.442270 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.442232 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecab4d7a-4034-4b59-8500-8434baacafaf-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.442365 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.442280 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fzss\" (UniqueName: \"kubernetes.io/projected/ecab4d7a-4034-4b59-8500-8434baacafaf-kube-api-access-5fzss\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.442365 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.442307 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.442457 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.442361 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.442457 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.442393 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.442822 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.442602 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpnqh\" (UniqueName: \"kubernetes.io/projected/25ceab3d-6344-4512-a471-d91aec0cb48c-kube-api-access-hpnqh\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.450783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.437248 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-workload-socket" pod "4c609469-d7d7-4530-8d74-e34aa56dda8f" (UID: "4c609469-d7d7-4530-8d74-e34aa56dda8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.450783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.437479 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-data" pod "4c609469-d7d7-4530-8d74-e34aa56dda8f" (UID: "4c609469-d7d7-4530-8d74-e34aa56dda8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.450783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.442974 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c609469-d7d7-4530-8d74-e34aa56dda8f-istiod-ca-cert" pod "4c609469-d7d7-4530-8d74-e34aa56dda8f" (UID: "4c609469-d7d7-4530-8d74-e34aa56dda8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:08.450783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.443227 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-workload-certs" pod "4c609469-d7d7-4530-8d74-e34aa56dda8f" (UID: "4c609469-d7d7-4530-8d74-e34aa56dda8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.450783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.449493 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-credential-socket" pod "4c609469-d7d7-4530-8d74-e34aa56dda8f" (UID: "4c609469-d7d7-4530-8d74-e34aa56dda8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.456772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.456746 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn"] Nov 03 20:33:08.472839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.472687 5059 scope.go:122] "RemoveContainer" containerID="f40938fdfd8d12fe6529e191274b54750315e326f7aeec32fac2e699b64c43a1" Nov 03 20:33:08.483788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.483754 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.497097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.496780 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:33:08.497097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.496842 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" podUID="ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:08.507136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.507096 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn"] Nov 03 20:33:08.509851 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.509820 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-token" pod "4c609469-d7d7-4530-8d74-e34aa56dda8f" (UID: "4c609469-d7d7-4530-8d74-e34aa56dda8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.510026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.510006 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c609469-d7d7-4530-8d74-e34aa56dda8f-kube-api-access-wskdx" pod "4c609469-d7d7-4530-8d74-e34aa56dda8f" (UID: "4c609469-d7d7-4530-8d74-e34aa56dda8f"). InnerVolumeSpecName "kube-api-access-wskdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.526002 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:08.525806 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f40938fdfd8d12fe6529e191274b54750315e326f7aeec32fac2e699b64c43a1\": container with ID starting with f40938fdfd8d12fe6529e191274b54750315e326f7aeec32fac2e699b64c43a1 not found: ID does not exist" containerID="f40938fdfd8d12fe6529e191274b54750315e326f7aeec32fac2e699b64c43a1" Nov 03 20:33:08.526002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.525849 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-envoy" pod "4c609469-d7d7-4530-8d74-e34aa56dda8f" (UID: "4c609469-d7d7-4530-8d74-e34aa56dda8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.526002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.525851 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f40938fdfd8d12fe6529e191274b54750315e326f7aeec32fac2e699b64c43a1"} err="failed to get container status \"f40938fdfd8d12fe6529e191274b54750315e326f7aeec32fac2e699b64c43a1\": rpc error: code = NotFound desc = could not find container \"f40938fdfd8d12fe6529e191274b54750315e326f7aeec32fac2e699b64c43a1\": container with ID starting with f40938fdfd8d12fe6529e191274b54750315e326f7aeec32fac2e699b64c43a1 not found: ID does not exist" Nov 03 20:33:08.526002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.525876 5059 scope.go:122] "RemoveContainer" containerID="80563e166a6378b34d09a54f08d0ed8f1e665107abe95933f0fa76b31fc439e7" Nov 03 20:33:08.533476 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.533450 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-podinfo" pod "4c609469-d7d7-4530-8d74-e34aa56dda8f" (UID: "4c609469-d7d7-4530-8d74-e34aa56dda8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:08.543632 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.543606 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc57130b-c950-49f2-8a98-22c47bd98288-kube-api-access-p9shs\" (UniqueName: \"kubernetes.io/projected/fc57130b-c950-49f2-8a98-22c47bd98288-kube-api-access-p9shs\") pod \"fc57130b-c950-49f2-8a98-22c47bd98288\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " Nov 03 20:33:08.543874 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.543851 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc57130b-c950-49f2-8a98-22c47bd98288-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc57130b-c950-49f2-8a98-22c47bd98288-istio-podinfo\") pod \"fc57130b-c950-49f2-8a98-22c47bd98288\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " Nov 03 20:33:08.544020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.544007 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc57130b-c950-49f2-8a98-22c47bd98288-istio-token\" (UniqueName: \"kubernetes.io/projected/fc57130b-c950-49f2-8a98-22c47bd98288-istio-token\") pod \"fc57130b-c950-49f2-8a98-22c47bd98288\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " Nov 03 20:33:08.544202 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.544189 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc57130b-c950-49f2-8a98-22c47bd98288-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc57130b-c950-49f2-8a98-22c47bd98288-istiod-ca-cert\") pod \"fc57130b-c950-49f2-8a98-22c47bd98288\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " Nov 03 20:33:08.544297 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.544285 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-workload-socket\") pod \"fc57130b-c950-49f2-8a98-22c47bd98288\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " Nov 03 20:33:08.544393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.544381 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-istio-data\") pod \"fc57130b-c950-49f2-8a98-22c47bd98288\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " Nov 03 20:33:08.544501 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.544487 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-istio-envoy\") pod \"fc57130b-c950-49f2-8a98-22c47bd98288\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " Nov 03 20:33:08.544730 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.544714 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-workload-certs\") pod \"fc57130b-c950-49f2-8a98-22c47bd98288\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " Nov 03 20:33:08.545012 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.544997 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-credential-socket\") pod \"fc57130b-c950-49f2-8a98-22c47bd98288\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " Nov 03 20:33:08.545448 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.545426 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.545645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.545628 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.545841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.545824 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.545934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.545921 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.546039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.546026 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.546149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.546136 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.546248 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.546233 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcptm\" (UniqueName: \"kubernetes.io/projected/82894901-9901-4dc4-948a-321782ef8288-kube-api-access-jcptm\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.546380 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.546355 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.546563 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.546547 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.546673 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.546642 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82894901-9901-4dc4-948a-321782ef8288-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.546854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.546839 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.546942 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.546931 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.547038 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.547026 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.547127 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.547116 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.547209 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.547198 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.547291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.547279 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82894901-9901-4dc4-948a-321782ef8288-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.547454 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.547440 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.547592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.547567 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-credential-socket" pod "fc57130b-c950-49f2-8a98-22c47bd98288" (UID: "fc57130b-c950-49f2-8a98-22c47bd98288"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.547646 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.547576 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.547709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.547642 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.547758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.547711 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2fe4277-3acc-4378-86d9-374b69918ec9-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.547758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.547740 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.547861 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.547776 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.547861 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.547824 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82894901-9901-4dc4-948a-321782ef8288-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.547951 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.547866 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.547951 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.547895 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrhl7\" (UniqueName: \"kubernetes.io/projected/d4fdd390-7fe2-406d-832a-7ecc545d97f4-kube-api-access-mrhl7\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.548043 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.547954 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhmg9\" (UniqueName: \"kubernetes.io/projected/e2fe4277-3acc-4378-86d9-374b69918ec9-kube-api-access-bhmg9\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.548043 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.547981 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.548043 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.548026 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.548175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.548097 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.548226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.548175 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecab4d7a-4034-4b59-8500-8434baacafaf-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.548226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.548202 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fzss\" (UniqueName: \"kubernetes.io/projected/ecab4d7a-4034-4b59-8500-8434baacafaf-kube-api-access-5fzss\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.548315 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.548235 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.548315 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.548293 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.548405 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.548325 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.548405 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.548369 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.548503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.548456 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.548690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.548653 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w"] Nov 03 20:33:08.554149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.551730 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.554149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.551767 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.554149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.551789 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.554149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.551814 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wskdx\" (UniqueName: \"kubernetes.io/projected/4c609469-d7d7-4530-8d74-e34aa56dda8f-kube-api-access-wskdx\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.554149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.551832 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.554149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.551848 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.554149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.551864 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.554149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.551885 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.554149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.551902 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c609469-d7d7-4530-8d74-e34aa56dda8f-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.554149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.551918 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c609469-d7d7-4530-8d74-e34aa56dda8f-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.554149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.552303 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-workload-socket" pod "fc57130b-c950-49f2-8a98-22c47bd98288" (UID: "fc57130b-c950-49f2-8a98-22c47bd98288"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.554149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.552519 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-istio-data" pod "fc57130b-c950-49f2-8a98-22c47bd98288" (UID: "fc57130b-c950-49f2-8a98-22c47bd98288"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.554149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.553293 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.554149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.546071 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:33:08.554149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.553755 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.554149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.554090 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.555244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.555222 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc57130b-c950-49f2-8a98-22c47bd98288-istiod-ca-cert" pod "fc57130b-c950-49f2-8a98-22c47bd98288" (UID: "fc57130b-c950-49f2-8a98-22c47bd98288"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:08.555780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.555755 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.558958 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.558925 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecab4d7a-4034-4b59-8500-8434baacafaf-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.562577 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.561885 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.565574 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.565189 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.565574 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.565468 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.568821 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.567763 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc57130b-c950-49f2-8a98-22c47bd98288-istio-podinfo" pod "fc57130b-c950-49f2-8a98-22c47bd98288" (UID: "fc57130b-c950-49f2-8a98-22c47bd98288"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:08.568821 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.568497 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2fe4277-3acc-4378-86d9-374b69918ec9-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.568950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.568847 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.615697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.602909 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.615697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.605189 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.615697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.607382 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.615697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.607913 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Nov 03 20:33:08.615697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.607964 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" podUID="bbd92cdc-e2bd-4f72-8c63-1221069ca3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Nov 03 20:33:08.615697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.609607 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.615697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.613188 5059 cpu_manager.go:414] "containerMap: removing container" podUID="9d60eb56-9852-4c25-ae50-8a932e752793" containerName="istio-proxy" Nov 03 20:33:08.615697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.613211 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d60eb56-9852-4c25-ae50-8a932e752793" containerName="istio-proxy" Nov 03 20:33:08.615697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.613250 5059 cpu_manager.go:414] "containerMap: removing container" podUID="9d60eb56-9852-4c25-ae50-8a932e752793" containerName="istio-proxy" Nov 03 20:33:08.615697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.613259 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d60eb56-9852-4c25-ae50-8a932e752793" containerName="istio-proxy" Nov 03 20:33:08.615697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.613983 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d60eb56-9852-4c25-ae50-8a932e752793" containerName="istio-proxy" Nov 03 20:33:08.615697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.614006 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d60eb56-9852-4c25-ae50-8a932e752793" containerName="istio-proxy" Nov 03 20:33:08.635030 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.631346 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-workload-certs" pod "fc57130b-c950-49f2-8a98-22c47bd98288" (UID: "fc57130b-c950-49f2-8a98-22c47bd98288"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.639806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.637721 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.655698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.654690 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d60eb56-9852-4c25-ae50-8a932e752793-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d60eb56-9852-4c25-ae50-8a932e752793-istio-podinfo\") pod \"9d60eb56-9852-4c25-ae50-8a932e752793\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " Nov 03 20:33:08.655698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.654936 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-credential-socket\") pod \"9d60eb56-9852-4c25-ae50-8a932e752793\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " Nov 03 20:33:08.655698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.655097 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-istio-envoy\") pod \"9d60eb56-9852-4c25-ae50-8a932e752793\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " Nov 03 20:33:08.655698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.655140 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d60eb56-9852-4c25-ae50-8a932e752793-istio-token\" (UniqueName: \"kubernetes.io/projected/9d60eb56-9852-4c25-ae50-8a932e752793-istio-token\") pod \"9d60eb56-9852-4c25-ae50-8a932e752793\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " Nov 03 20:33:08.655698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.655190 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d60eb56-9852-4c25-ae50-8a932e752793-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d60eb56-9852-4c25-ae50-8a932e752793-istiod-ca-cert\") pod \"9d60eb56-9852-4c25-ae50-8a932e752793\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " Nov 03 20:33:08.655698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.655307 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-istio-data\") pod \"9d60eb56-9852-4c25-ae50-8a932e752793\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " Nov 03 20:33:08.655698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.655350 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-workload-certs\") pod \"9d60eb56-9852-4c25-ae50-8a932e752793\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " Nov 03 20:33:08.655698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.655394 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d60eb56-9852-4c25-ae50-8a932e752793-kube-api-access-q79lh\" (UniqueName: \"kubernetes.io/projected/9d60eb56-9852-4c25-ae50-8a932e752793-kube-api-access-q79lh\") pod \"9d60eb56-9852-4c25-ae50-8a932e752793\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " Nov 03 20:33:08.655698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.655458 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-workload-socket\") pod \"9d60eb56-9852-4c25-ae50-8a932e752793\" (UID: \"9d60eb56-9852-4c25-ae50-8a932e752793\") " Nov 03 20:33:08.666845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.656179 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.666845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.656282 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.666845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.656340 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.666845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.656381 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.666845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.656478 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.666845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.656514 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.666845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.656589 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.666845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.656632 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.666845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.656701 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.666845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.656749 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.666845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.656792 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.666845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.656826 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.666845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.656896 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcptm\" (UniqueName: \"kubernetes.io/projected/82894901-9901-4dc4-948a-321782ef8288-kube-api-access-jcptm\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.666845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.656938 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.666845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.657006 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.666845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.657048 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.666845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.657086 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82894901-9901-4dc4-948a-321782ef8288-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.666845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.657253 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.666845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.657298 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.666845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.657334 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82894901-9901-4dc4-948a-321782ef8288-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.666845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.657410 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.666845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.657521 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.666845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.657615 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.666845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.657703 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.674111 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.672068 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc57130b-c950-49f2-8a98-22c47bd98288-kube-api-access-p9shs" pod "fc57130b-c950-49f2-8a98-22c47bd98288" (UID: "fc57130b-c950-49f2-8a98-22c47bd98288"). InnerVolumeSpecName "kube-api-access-p9shs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.692710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.685795 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8rw6\" (UniqueName: \"kubernetes.io/projected/d75a00ce-f97c-431f-be9e-aac200bcd9d4-kube-api-access-f8rw6\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.692710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.685888 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82894901-9901-4dc4-948a-321782ef8288-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.692710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.685964 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrhl7\" (UniqueName: \"kubernetes.io/projected/d4fdd390-7fe2-406d-832a-7ecc545d97f4-kube-api-access-mrhl7\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.692710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.686260 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc57130b-c950-49f2-8a98-22c47bd98288-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.692710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.686291 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.692710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.686309 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.692710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.686334 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.692710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.686356 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9shs\" (UniqueName: \"kubernetes.io/projected/fc57130b-c950-49f2-8a98-22c47bd98288-kube-api-access-p9shs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.692710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.686377 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc57130b-c950-49f2-8a98-22c47bd98288-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.734130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.733719 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-credential-socket" pod "9d60eb56-9852-4c25-ae50-8a932e752793" (UID: "9d60eb56-9852-4c25-ae50-8a932e752793"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.748595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.747856 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.748595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.748166 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.753689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.752332 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.753689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.752984 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.753689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.753279 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.789291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.789255 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.789800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.789777 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.790645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.790620 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82894901-9901-4dc4-948a-321782ef8288-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.826364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.806142 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.826364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.811861 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:33:08.826364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.816108 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" podUID="c9721811-d27d-4e13-a53d-4ae81e16fb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:08.826364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.813677 5059 scope.go:122] "RemoveContainer" containerID="3513472a5d02bb21de815edd2ec4ca696e8e46e17be5d839916c093e17bd9499" Nov 03 20:33:08.826364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.813829 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82894901-9901-4dc4-948a-321782ef8288-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.826364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.813216 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fzss\" (UniqueName: \"kubernetes.io/projected/ecab4d7a-4034-4b59-8500-8434baacafaf-kube-api-access-5fzss\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:08.826364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.817054 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.826364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.818784 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7" Nov 03 20:33:08.826364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.819554 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc57130b-c950-49f2-8a98-22c47bd98288-istio-token" pod "fc57130b-c950-49f2-8a98-22c47bd98288" (UID: "fc57130b-c950-49f2-8a98-22c47bd98288"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.826364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.820054 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.826364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.825938 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w"] Nov 03 20:33:08.826364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.826073 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:08.833652 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.833623 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.833940 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.833904 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-istio-data" pod "9d60eb56-9852-4c25-ae50-8a932e752793" (UID: "9d60eb56-9852-4c25-ae50-8a932e752793"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.834164 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.834134 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-workload-certs" pod "9d60eb56-9852-4c25-ae50-8a932e752793" (UID: "9d60eb56-9852-4c25-ae50-8a932e752793"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.837442 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.837418 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d60eb56-9852-4c25-ae50-8a932e752793-istiod-ca-cert" pod "9d60eb56-9852-4c25-ae50-8a932e752793" (UID: "9d60eb56-9852-4c25-ae50-8a932e752793"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:08.837639 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.837613 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d60eb56-9852-4c25-ae50-8a932e752793-istio-podinfo" pod "9d60eb56-9852-4c25-ae50-8a932e752793" (UID: "9d60eb56-9852-4c25-ae50-8a932e752793"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:08.837802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.837786 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-istio-envoy" pod "fc57130b-c950-49f2-8a98-22c47bd98288" (UID: "fc57130b-c950-49f2-8a98-22c47bd98288"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.839526 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.839495 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.840218 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.840197 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.840387 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.840356 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcptm\" (UniqueName: \"kubernetes.io/projected/82894901-9901-4dc4-948a-321782ef8288-kube-api-access-jcptm\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.840923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.840890 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrhl7\" (UniqueName: \"kubernetes.io/projected/d4fdd390-7fe2-406d-832a-7ecc545d97f4-kube-api-access-mrhl7\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.841256 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.841234 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-workload-socket" pod "9d60eb56-9852-4c25-ae50-8a932e752793" (UID: "9d60eb56-9852-4c25-ae50-8a932e752793"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.861844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.844944 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhmg9\" (UniqueName: \"kubernetes.io/projected/e2fe4277-3acc-4378-86d9-374b69918ec9-kube-api-access-bhmg9\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:08.861844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.860340 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c363281-0226-420f-a963-5922d5ca609e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c363281-0226-420f-a963-5922d5ca609e-plugins-config-volume\") pod \"2c363281-0226-420f-a963-5922d5ca609e\" (UID: \"2c363281-0226-420f-a963-5922d5ca609e\") " Nov 03 20:33:08.861844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.860403 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c363281-0226-420f-a963-5922d5ca609e-kube-api-access-qltdw\" (UniqueName: \"kubernetes.io/projected/2c363281-0226-420f-a963-5922d5ca609e-kube-api-access-qltdw\") pod \"2c363281-0226-420f-a963-5922d5ca609e\" (UID: \"2c363281-0226-420f-a963-5922d5ca609e\") " Nov 03 20:33:08.869428 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.869401 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c363281-0226-420f-a963-5922d5ca609e-plugins-config-volume" pod "2c363281-0226-420f-a963-5922d5ca609e" (UID: "2c363281-0226-420f-a963-5922d5ca609e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:08.871212 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.871173 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:08.879498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.879461 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c363281-0226-420f-a963-5922d5ca609e-kube-api-access-qltdw" pod "2c363281-0226-420f-a963-5922d5ca609e" (UID: "2c363281-0226-420f-a963-5922d5ca609e"). InnerVolumeSpecName "kube-api-access-qltdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.882459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.881960 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-istio-envoy\") pod \"fc57130b-c950-49f2-8a98-22c47bd98288\" (UID: \"fc57130b-c950-49f2-8a98-22c47bd98288\") " Nov 03 20:33:08.882871 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:33:08.882848 5059 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/fc57130b-c950-49f2-8a98-22c47bd98288/volumes/kubernetes.io~empty-dir/istio-envoy Nov 03 20:33:08.882964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.882874 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-istio-envoy" pod "fc57130b-c950-49f2-8a98-22c47bd98288" (UID: "fc57130b-c950-49f2-8a98-22c47bd98288"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.892240 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.892200 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:08.892729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.892694 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:08.892943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.892904 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:08.903397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.900150 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:08.903397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.902856 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:08.903397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.902908 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-istio-envoy" pod "9d60eb56-9852-4c25-ae50-8a932e752793" (UID: "9d60eb56-9852-4c25-ae50-8a932e752793"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:08.903397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.902957 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:08.903616 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.903553 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82894901-9901-4dc4-948a-321782ef8288-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:08.905949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.903690 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.905949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.903756 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.905949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.903797 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.905949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.903875 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.905949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.903941 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.905949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.903985 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.905949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.904076 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.905949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.904257 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.905949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.904378 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8rw6\" (UniqueName: \"kubernetes.io/projected/d75a00ce-f97c-431f-be9e-aac200bcd9d4-kube-api-access-f8rw6\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.905949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.904577 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d60eb56-9852-4c25-ae50-8a932e752793-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.905949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.904609 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c363281-0226-420f-a963-5922d5ca609e-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.905949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.904627 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qltdw\" (UniqueName: \"kubernetes.io/projected/2c363281-0226-420f-a963-5922d5ca609e-kube-api-access-qltdw\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.905949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.904644 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.905949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.904680 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.905949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.904708 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc57130b-c950-49f2-8a98-22c47bd98288-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.905949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.905432 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.912017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.908455 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.912017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.905532 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.912017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.908576 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:08.912017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.908748 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:08.912017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.908874 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:08.912017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.909779 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d60eb56-9852-4c25-ae50-8a932e752793-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.912017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.910606 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.912017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.910823 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.912017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.911009 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.912017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.911798 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.912017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.911825 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc57130b-c950-49f2-8a98-22c47bd98288-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.913524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.913108 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.913524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.913129 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d60eb56-9852-4c25-ae50-8a932e752793-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.914478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.914271 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:08.914478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.914383 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:08.914478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.914469 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:08.925368 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.925231 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d60eb56-9852-4c25-ae50-8a932e752793-kube-api-access-q79lh" pod "9d60eb56-9852-4c25-ae50-8a932e752793" (UID: "9d60eb56-9852-4c25-ae50-8a932e752793"). InnerVolumeSpecName "kube-api-access-q79lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.926685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.926521 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.926847 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.926821 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8rw6\" (UniqueName: \"kubernetes.io/projected/d75a00ce-f97c-431f-be9e-aac200bcd9d4-kube-api-access-f8rw6\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.930700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.928168 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:08.930959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.930816 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d60eb56-9852-4c25-ae50-8a932e752793-istio-token" pod "9d60eb56-9852-4c25-ae50-8a932e752793" (UID: "9d60eb56-9852-4c25-ae50-8a932e752793"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:08.937853 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.936813 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" podStartSLOduration=66.936795685 podStartE2EDuration="1m6.936795685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:07.810505419 +0000 UTC m=+3382.068302679" watchObservedRunningTime="2026-11-03 20:33:08.936795685 +0000 UTC m=+3383.194592944" Nov 03 20:33:08.937853 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.937252 5059 generic.go:356] "Generic (PLEG): container finished" podID="1326708b-4642-4992-91ea-8a590863a703" containerID="8cc39ee98f0eb5245e5d5488f9855a4beaa057aa6bbd7dc9d1116291e5c04e35" exitCode=22 Nov 03 20:33:08.937853 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.937351 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4" event={"ID":"1326708b-4642-4992-91ea-8a590863a703","Type":"ContainerDied","Data":"8cc39ee98f0eb5245e5d5488f9855a4beaa057aa6bbd7dc9d1116291e5c04e35"} Nov 03 20:33:08.937853 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.937848 5059 scope.go:122] "RemoveContainer" containerID="8cc39ee98f0eb5245e5d5488f9855a4beaa057aa6bbd7dc9d1116291e5c04e35" Nov 03 20:33:08.938225 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:08.938192 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-4bdd0b59-9wxx4_odh-ai-gateway-infra(1326708b-4642-4992-91ea-8a590863a703)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4" podUID="1326708b-4642-4992-91ea-8a590863a703" Nov 03 20:33:08.952557 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.949055 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w"] Nov 03 20:33:08.955112 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.954183 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9" event={"ID":"fd9d29fb-d764-4eac-a744-dbdecde12ff0","Type":"ContainerStarted","Data":"6fea7c3735106edfc3123cf029de6ac4573a06a4c349292f2bbcb66df7a37986"} Nov 03 20:33:08.955482 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.955425 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-nlv6w"] Nov 03 20:33:09.001260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.957337 5059 generic.go:356] "Generic (PLEG): container finished" podID="dc10f92e-e902-48c4-9cd5-4f768c4611be" containerID="49fc76b3559a7fdac7c4de24b68b9e183d25cec8fc53c075c32d7308be58c0fa" exitCode=22 Nov 03 20:33:09.001260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.957425 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4" event={"ID":"dc10f92e-e902-48c4-9cd5-4f768c4611be","Type":"ContainerStarted","Data":"4d94405c1bd7e9ab077c0ca4e40445d24510310ab3bd698c9d3fc466bbff887d"} Nov 03 20:33:09.001260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.957464 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4" event={"ID":"dc10f92e-e902-48c4-9cd5-4f768c4611be","Type":"ContainerDied","Data":"49fc76b3559a7fdac7c4de24b68b9e183d25cec8fc53c075c32d7308be58c0fa"} Nov 03 20:33:09.001260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.958078 5059 scope.go:122] "RemoveContainer" containerID="4d94405c1bd7e9ab077c0ca4e40445d24510310ab3bd698c9d3fc466bbff887d" Nov 03 20:33:09.001260 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:08.958470 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-r5fd4_odh-ai-gateway-infra(dc10f92e-e902-48c4-9cd5-4f768c4611be)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4" podUID="dc10f92e-e902-48c4-9cd5-4f768c4611be" Nov 03 20:33:09.001260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:08.975881 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8" Nov 03 20:33:09.016943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.016695 5059 scope.go:122] "RemoveContainer" containerID="bab3644b9638d4ac499f2df75a736f3f63d2b12e17964b2ccc440b6a34499ce5" Nov 03 20:33:09.022133 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.022096 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.022290 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.022268 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.022387 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.022359 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.022447 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.022413 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b615028-a2db-4ad0-8a57-53efc040d6a6-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.022495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.022465 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.022495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.022489 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvx8m\" (UniqueName: \"kubernetes.io/projected/6b615028-a2db-4ad0-8a57-53efc040d6a6-kube-api-access-xvx8m\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.022732 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.022710 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.022822 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.022804 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.022874 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.022864 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.023007 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.022989 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d60eb56-9852-4c25-ae50-8a932e752793-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.023073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.023015 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q79lh\" (UniqueName: \"kubernetes.io/projected/9d60eb56-9852-4c25-ae50-8a932e752793-kube-api-access-q79lh\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.029201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.028824 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:09.038534 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.038512 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc"] Nov 03 20:33:09.041915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.039959 5059 cpu_manager.go:414] "containerMap: removing container" podUID="705dea2b-ad85-4b1c-a3c6-a9f705ac9039" containerName="payload-processing" Nov 03 20:33:09.041915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.039981 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="705dea2b-ad85-4b1c-a3c6-a9f705ac9039" containerName="payload-processing" Nov 03 20:33:09.041915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.040007 5059 cpu_manager.go:414] "containerMap: removing container" podUID="2c363281-0226-420f-a963-5922d5ca609e" containerName="payload-pre-processing" Nov 03 20:33:09.041915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.040017 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c363281-0226-420f-a963-5922d5ca609e" containerName="payload-pre-processing" Nov 03 20:33:09.041915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.040244 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="705dea2b-ad85-4b1c-a3c6-a9f705ac9039" containerName="payload-processing" Nov 03 20:33:09.041915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.040259 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c363281-0226-420f-a963-5922d5ca609e" containerName="payload-pre-processing" Nov 03 20:33:09.046885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.046864 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc"] Nov 03 20:33:09.046988 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.046973 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.054237 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.052420 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:33:09.054237 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.052519 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9"] Nov 03 20:33:09.101029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.100985 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-8fbn9"] Nov 03 20:33:09.113070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.113036 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7"] Nov 03 20:33:09.115350 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.115326 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:09.116780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.116760 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" event={"ID":"d098ef77-addb-4077-b6d4-f0ae477409b8","Type":"ContainerStarted","Data":"fc11b42681fecc377fcf8f010c85cfa139f2739d4ef02581bc1cce8cd2af1475"} Nov 03 20:33:09.121624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.121606 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:33:09.140115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.137974 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/705dea2b-ad85-4b1c-a3c6-a9f705ac9039-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705dea2b-ad85-4b1c-a3c6-a9f705ac9039-plugins-config-volume\") pod \"705dea2b-ad85-4b1c-a3c6-a9f705ac9039\" (UID: \"705dea2b-ad85-4b1c-a3c6-a9f705ac9039\") " Nov 03 20:33:09.140115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.138029 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/705dea2b-ad85-4b1c-a3c6-a9f705ac9039-kube-api-access-p57rb\" (UniqueName: \"kubernetes.io/projected/705dea2b-ad85-4b1c-a3c6-a9f705ac9039-kube-api-access-p57rb\") pod \"705dea2b-ad85-4b1c-a3c6-a9f705ac9039\" (UID: \"705dea2b-ad85-4b1c-a3c6-a9f705ac9039\") " Nov 03 20:33:09.140115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.138471 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.140115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.138705 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.140115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.138800 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.140115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.138843 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b615028-a2db-4ad0-8a57-53efc040d6a6-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.140115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.138892 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.140115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.138928 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvx8m\" (UniqueName: \"kubernetes.io/projected/6b615028-a2db-4ad0-8a57-53efc040d6a6-kube-api-access-xvx8m\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.140115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.139153 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.140115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.139235 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.140115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.139301 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.140115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.139871 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.144749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.144711 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/705dea2b-ad85-4b1c-a3c6-a9f705ac9039-plugins-config-volume" pod "705dea2b-ad85-4b1c-a3c6-a9f705ac9039" (UID: "705dea2b-ad85-4b1c-a3c6-a9f705ac9039"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:09.145486 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.145463 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b615028-a2db-4ad0-8a57-53efc040d6a6-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.148468 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.148049 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.149750 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.149597 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.203931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.156651 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.204094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.171601 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:33:09.209337 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.209302 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvx8m\" (UniqueName: \"kubernetes.io/projected/6b615028-a2db-4ad0-8a57-53efc040d6a6-kube-api-access-xvx8m\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.219069 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.219035 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.222188 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.222150 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/705dea2b-ad85-4b1c-a3c6-a9f705ac9039-kube-api-access-p57rb" pod "705dea2b-ad85-4b1c-a3c6-a9f705ac9039" (UID: "705dea2b-ad85-4b1c-a3c6-a9f705ac9039"). InnerVolumeSpecName "kube-api-access-p57rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:09.229731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.229460 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.230863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.230816 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-464r7"] Nov 03 20:33:09.231487 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.231459 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:09.241574 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.241157 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8" Nov 03 20:33:09.242113 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.241998 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-token\" (UniqueName: \"kubernetes.io/projected/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-token\") pod \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " Nov 03 20:33:09.242113 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.242057 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-credential-socket\") pod \"0f32ecf7-06f7-4098-9793-acf761a30c09\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " Nov 03 20:33:09.242251 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.242162 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istiod-ca-cert\") pod \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " Nov 03 20:33:09.242251 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.242207 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-podinfo\") pod \"0f32ecf7-06f7-4098-9793-acf761a30c09\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " Nov 03 20:33:09.242344 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.242266 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-credential-socket\") pod \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " Nov 03 20:33:09.242344 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.242312 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-podinfo\") pod \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " Nov 03 20:33:09.242438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.242382 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-envoy\") pod \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " Nov 03 20:33:09.242486 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.242458 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-workload-certs\") pod \"0f32ecf7-06f7-4098-9793-acf761a30c09\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " Nov 03 20:33:09.242535 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.242492 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-token\" (UniqueName: \"kubernetes.io/projected/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-token\") pod \"0f32ecf7-06f7-4098-9793-acf761a30c09\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " Nov 03 20:33:09.242585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.242553 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-envoy\") pod \"0f32ecf7-06f7-4098-9793-acf761a30c09\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " Nov 03 20:33:09.242642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.242600 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f32ecf7-06f7-4098-9793-acf761a30c09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f32ecf7-06f7-4098-9793-acf761a30c09-istiod-ca-cert\") pod \"0f32ecf7-06f7-4098-9793-acf761a30c09\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " Nov 03 20:33:09.244919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.242893 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-workload-socket\") pod \"0f32ecf7-06f7-4098-9793-acf761a30c09\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " Nov 03 20:33:09.244919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.242941 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-workload-socket\") pod \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " Nov 03 20:33:09.244919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.242975 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-data\") pod \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " Nov 03 20:33:09.244919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.243016 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f32ecf7-06f7-4098-9793-acf761a30c09-kube-api-access-kztkj\" (UniqueName: \"kubernetes.io/projected/0f32ecf7-06f7-4098-9793-acf761a30c09-kube-api-access-kztkj\") pod \"0f32ecf7-06f7-4098-9793-acf761a30c09\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " Nov 03 20:33:09.244919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.243098 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-workload-certs\") pod \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " Nov 03 20:33:09.244919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.243168 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-data\") pod \"0f32ecf7-06f7-4098-9793-acf761a30c09\" (UID: \"0f32ecf7-06f7-4098-9793-acf761a30c09\") " Nov 03 20:33:09.244919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.243218 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-kube-api-access-vwcjn\" (UniqueName: \"kubernetes.io/projected/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-kube-api-access-vwcjn\") pod \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\" (UID: \"ade5b9f4-d822-4418-a31b-f0c0a21ddb0c\") " Nov 03 20:33:09.244919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.243556 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.244919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.243654 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.244919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.243821 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7101a242-11ce-4b8d-9a74-dae3baf6444b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.244919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.243947 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.244919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.244082 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsnvd\" (UniqueName: \"kubernetes.io/projected/7101a242-11ce-4b8d-9a74-dae3baf6444b-kube-api-access-nsnvd\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.244919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.244192 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.244919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.244269 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.244919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.244351 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.244919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.244392 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.244919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.244568 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705dea2b-ad85-4b1c-a3c6-a9f705ac9039-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.244919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.244590 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p57rb\" (UniqueName: \"kubernetes.io/projected/705dea2b-ad85-4b1c-a3c6-a9f705ac9039-kube-api-access-p57rb\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.246223 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.245979 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-data" pod "0f32ecf7-06f7-4098-9793-acf761a30c09" (UID: "0f32ecf7-06f7-4098-9793-acf761a30c09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:09.246223 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.246027 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-workload-certs" pod "0f32ecf7-06f7-4098-9793-acf761a30c09" (UID: "0f32ecf7-06f7-4098-9793-acf761a30c09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:09.248318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.246593 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-workload-certs" pod "ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" (UID: "ade5b9f4-d822-4418-a31b-f0c0a21ddb0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:09.248318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.247986 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk"] Nov 03 20:33:09.252491 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.249709 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-workload-socket" pod "ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" (UID: "ade5b9f4-d822-4418-a31b-f0c0a21ddb0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:09.252491 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.249899 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-credential-socket" pod "0f32ecf7-06f7-4098-9793-acf761a30c09" (UID: "0f32ecf7-06f7-4098-9793-acf761a30c09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:09.252491 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.250161 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f32ecf7-06f7-4098-9793-acf761a30c09-istiod-ca-cert" pod "0f32ecf7-06f7-4098-9793-acf761a30c09" (UID: "0f32ecf7-06f7-4098-9793-acf761a30c09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:09.252491 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.251470 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-workload-socket" pod "0f32ecf7-06f7-4098-9793-acf761a30c09" (UID: "0f32ecf7-06f7-4098-9793-acf761a30c09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:09.252491 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.251646 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-credential-socket" pod "ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" (UID: "ade5b9f4-d822-4418-a31b-f0c0a21ddb0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:09.252491 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.251956 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-data" pod "ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" (UID: "ade5b9f4-d822-4418-a31b-f0c0a21ddb0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:09.252491 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.252227 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istiod-ca-cert" pod "ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" (UID: "ade5b9f4-d822-4418-a31b-f0c0a21ddb0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:09.295768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.281885 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 20:33:09.295768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.282843 5059 scope.go:122] "RemoveContainer" containerID="d36cbc617302444ca97907469c12558d0fc4a6b46d78b847f1d5ccbc843b16e6" Nov 03 20:33:09.295940 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.295852 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-envoy" pod "0f32ecf7-06f7-4098-9793-acf761a30c09" (UID: "0f32ecf7-06f7-4098-9793-acf761a30c09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:09.297174 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.297010 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-token" pod "0f32ecf7-06f7-4098-9793-acf761a30c09" (UID: "0f32ecf7-06f7-4098-9793-acf761a30c09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:09.297174 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.297117 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-kube-api-access-vwcjn" pod "ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" (UID: "ade5b9f4-d822-4418-a31b-f0c0a21ddb0c"). InnerVolumeSpecName "kube-api-access-vwcjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:09.301304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.300649 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-envoy" pod "ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" (UID: "ade5b9f4-d822-4418-a31b-f0c0a21ddb0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:09.303558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.303477 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2" Nov 03 20:33:09.305111 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.304986 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-token" pod "ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" (UID: "ade5b9f4-d822-4418-a31b-f0c0a21ddb0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:09.305111 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.305063 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-podinfo" pod "ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" (UID: "ade5b9f4-d822-4418-a31b-f0c0a21ddb0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:09.305111 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.305086 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f32ecf7-06f7-4098-9793-acf761a30c09-kube-api-access-kztkj" pod "0f32ecf7-06f7-4098-9793-acf761a30c09" (UID: "0f32ecf7-06f7-4098-9793-acf761a30c09"). InnerVolumeSpecName "kube-api-access-kztkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:09.305398 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.305368 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-podinfo" pod "0f32ecf7-06f7-4098-9793-acf761a30c09" (UID: "0f32ecf7-06f7-4098-9793-acf761a30c09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:09.306545 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.306515 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" event={"ID":"60b2ec36-7e99-46e0-9671-1c500c402e45","Type":"ContainerStarted","Data":"5c778c28cc39de4888cd4d58e5dea06a380e13b8ea4866c0bb58c2f25f68cd63"} Nov 03 20:33:09.309282 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.309211 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" event={"ID":"4b43d3ba-a346-45d6-8077-8a52b42de081","Type":"ContainerStarted","Data":"076de4cff2c0552482a581fd5c826904e8d033b83f6fa9bd083ed79471c950bc"} Nov 03 20:33:09.311450 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.311398 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-pf4gk"] Nov 03 20:33:09.314689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.313006 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj" Nov 03 20:33:09.323277 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.323253 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc"] Nov 03 20:33:09.327021 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.326185 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd" Nov 03 20:33:09.333948 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.333888 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" event={"ID":"4b3627ae-66c6-44a4-aa60-480bd54008f0","Type":"ContainerStarted","Data":"56675e05c55dfbccd6b9e31541db45c3dccf7f2cfe09e69dcaadea37cd39f9bc"} Nov 03 20:33:09.338676 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.338418 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6" Nov 03 20:33:09.346475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.345474 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-credential-socket\") pod \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " Nov 03 20:33:09.346475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.345547 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-workload-certs\") pod \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " Nov 03 20:33:09.346475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.345591 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eac98813-7b4c-4c4a-8bf9-444bba2fa644-kube-api-access-q5jnb\" (UniqueName: \"kubernetes.io/projected/eac98813-7b4c-4c4a-8bf9-444bba2fa644-kube-api-access-q5jnb\") pod \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " Nov 03 20:33:09.346475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.345725 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-workload-socket\") pod \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " Nov 03 20:33:09.346475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.345756 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-token\" (UniqueName: \"kubernetes.io/projected/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-token\") pod \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " Nov 03 20:33:09.346475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.345799 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfcb49c5-4381-411e-9d64-d9dcd6181239-kube-api-access-d75d7\" (UniqueName: \"kubernetes.io/projected/cfcb49c5-4381-411e-9d64-d9dcd6181239-kube-api-access-d75d7\") pod \"cfcb49c5-4381-411e-9d64-d9dcd6181239\" (UID: \"cfcb49c5-4381-411e-9d64-d9dcd6181239\") " Nov 03 20:33:09.346475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.345904 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-data\") pod \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " Nov 03 20:33:09.346475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.345928 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-podinfo\") pod \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " Nov 03 20:33:09.346475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.346005 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfcb49c5-4381-411e-9d64-d9dcd6181239-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfcb49c5-4381-411e-9d64-d9dcd6181239-plugins-config-volume\") pod \"cfcb49c5-4381-411e-9d64-d9dcd6181239\" (UID: \"cfcb49c5-4381-411e-9d64-d9dcd6181239\") " Nov 03 20:33:09.346475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.346143 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istiod-ca-cert\") pod \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " Nov 03 20:33:09.346475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.346197 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-envoy\") pod \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\" (UID: \"eac98813-7b4c-4c4a-8bf9-444bba2fa644\") " Nov 03 20:33:09.346475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.346438 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.347402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.346502 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.347402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.346634 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7101a242-11ce-4b8d-9a74-dae3baf6444b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.347402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.346760 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.347402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.346870 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsnvd\" (UniqueName: \"kubernetes.io/projected/7101a242-11ce-4b8d-9a74-dae3baf6444b-kube-api-access-nsnvd\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.347402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.346965 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.347402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.347036 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.347402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.347097 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.347402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.347139 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.347402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.347268 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.347402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.347287 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.347402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.347301 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.347402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.347319 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kztkj\" (UniqueName: \"kubernetes.io/projected/0f32ecf7-06f7-4098-9793-acf761a30c09-kube-api-access-kztkj\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.347402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.347333 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.347402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.347348 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.347402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.347365 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwcjn\" (UniqueName: \"kubernetes.io/projected/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-kube-api-access-vwcjn\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.347402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.347381 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.347402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.347396 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.347402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.347412 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.348243 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.347428 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.348243 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.347445 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.348243 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.347459 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.348243 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.347472 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.348243 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.347486 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.348243 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.347500 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.348243 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.347514 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f32ecf7-06f7-4098-9793-acf761a30c09-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.348243 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.347528 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f32ecf7-06f7-4098-9793-acf761a30c09-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.348243 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.347838 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.351914 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.349166 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.351914 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.349406 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.351914 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.349604 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7101a242-11ce-4b8d-9a74-dae3baf6444b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.414721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.359975 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.414721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.376605 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" event={"ID":"9bb4f711-0bcc-46f4-86b9-795c66613832","Type":"ContainerStarted","Data":"89548cf27e0936cd588dd9c9835fdc9577a746519416c028395390c730ad8fd5"} Nov 03 20:33:09.414721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.346868 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-credential-socket" pod "eac98813-7b4c-4c4a-8bf9-444bba2fa644" (UID: "eac98813-7b4c-4c4a-8bf9-444bba2fa644"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:09.414721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.347140 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-workload-certs" pod "eac98813-7b4c-4c4a-8bf9-444bba2fa644" (UID: "eac98813-7b4c-4c4a-8bf9-444bba2fa644"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:09.414721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.348105 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-workload-socket" pod "eac98813-7b4c-4c4a-8bf9-444bba2fa644" (UID: "eac98813-7b4c-4c4a-8bf9-444bba2fa644"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:09.414721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.348924 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eac98813-7b4c-4c4a-8bf9-444bba2fa644-kube-api-access-q5jnb" pod "eac98813-7b4c-4c4a-8bf9-444bba2fa644" (UID: "eac98813-7b4c-4c4a-8bf9-444bba2fa644"). InnerVolumeSpecName "kube-api-access-q5jnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:09.414721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.375585 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfcb49c5-4381-411e-9d64-d9dcd6181239-plugins-config-volume" pod "cfcb49c5-4381-411e-9d64-d9dcd6181239" (UID: "cfcb49c5-4381-411e-9d64-d9dcd6181239"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:09.424240 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.375986 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istiod-ca-cert" pod "eac98813-7b4c-4c4a-8bf9-444bba2fa644" (UID: "eac98813-7b4c-4c4a-8bf9-444bba2fa644"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:09.424867 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.424844 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.432218 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.432193 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-data" pod "eac98813-7b4c-4c4a-8bf9-444bba2fa644" (UID: "eac98813-7b4c-4c4a-8bf9-444bba2fa644"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:09.437398 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.436231 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lqrkc"] Nov 03 20:33:09.455071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.449135 5059 scope.go:122] "RemoveContainer" containerID="fcb62eadee37b9a5aa9bdc20f77cd13d99c001430b049cbc4d5006aa5cc74354" Nov 03 20:33:09.455071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.452359 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.455071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.452389 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfcb49c5-4381-411e-9d64-d9dcd6181239-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.455071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.452404 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.455071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.452424 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.455071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.452438 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.455071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.452454 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5jnb\" (UniqueName: \"kubernetes.io/projected/eac98813-7b4c-4c4a-8bf9-444bba2fa644-kube-api-access-q5jnb\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.455071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.452469 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.500039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.498169 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk"] Nov 03 20:33:09.500039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.499381 5059 cpu_manager.go:414] "containerMap: removing container" podUID="ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" containerName="istio-proxy" Nov 03 20:33:09.500039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.499401 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" containerName="istio-proxy" Nov 03 20:33:09.500039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.499429 5059 cpu_manager.go:414] "containerMap: removing container" podUID="cfcb49c5-4381-411e-9d64-d9dcd6181239" containerName="payload-pre-processing" Nov 03 20:33:09.500039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.499438 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfcb49c5-4381-411e-9d64-d9dcd6181239" containerName="payload-pre-processing" Nov 03 20:33:09.500039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.499463 5059 cpu_manager.go:414] "containerMap: removing container" podUID="0f32ecf7-06f7-4098-9793-acf761a30c09" containerName="istio-proxy" Nov 03 20:33:09.500039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.499471 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f32ecf7-06f7-4098-9793-acf761a30c09" containerName="istio-proxy" Nov 03 20:33:09.500039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.499484 5059 cpu_manager.go:414] "containerMap: removing container" podUID="ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" containerName="istio-proxy" Nov 03 20:33:09.500039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.499491 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" containerName="istio-proxy" Nov 03 20:33:09.500039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.499511 5059 cpu_manager.go:414] "containerMap: removing container" podUID="eac98813-7b4c-4c4a-8bf9-444bba2fa644" containerName="istio-proxy" Nov 03 20:33:09.500039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.499518 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eac98813-7b4c-4c4a-8bf9-444bba2fa644" containerName="istio-proxy" Nov 03 20:33:09.500039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.499531 5059 cpu_manager.go:414] "containerMap: removing container" podUID="eac98813-7b4c-4c4a-8bf9-444bba2fa644" containerName="istio-proxy" Nov 03 20:33:09.500039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.499538 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eac98813-7b4c-4c4a-8bf9-444bba2fa644" containerName="istio-proxy" Nov 03 20:33:09.500039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.499716 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" containerName="istio-proxy" Nov 03 20:33:09.500039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.499731 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfcb49c5-4381-411e-9d64-d9dcd6181239" containerName="payload-pre-processing" Nov 03 20:33:09.500039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.499743 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f32ecf7-06f7-4098-9793-acf761a30c09" containerName="istio-proxy" Nov 03 20:33:09.500039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.499753 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" containerName="istio-proxy" Nov 03 20:33:09.500039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.499764 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="eac98813-7b4c-4c4a-8bf9-444bba2fa644" containerName="istio-proxy" Nov 03 20:33:09.500039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.499773 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="eac98813-7b4c-4c4a-8bf9-444bba2fa644" containerName="istio-proxy" Nov 03 20:33:09.504525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.504328 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.521007 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.519036 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl" Nov 03 20:33:09.527863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.525433 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-envoy" pod "eac98813-7b4c-4c4a-8bf9-444bba2fa644" (UID: "eac98813-7b4c-4c4a-8bf9-444bba2fa644"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:09.527863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.525614 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfcb49c5-4381-411e-9d64-d9dcd6181239-kube-api-access-d75d7" pod "cfcb49c5-4381-411e-9d64-d9dcd6181239" (UID: "cfcb49c5-4381-411e-9d64-d9dcd6181239"). InnerVolumeSpecName "kube-api-access-d75d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:09.527863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.525845 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-token" pod "eac98813-7b4c-4c4a-8bf9-444bba2fa644" (UID: "eac98813-7b4c-4c4a-8bf9-444bba2fa644"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:09.527863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.525885 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-podinfo" pod "eac98813-7b4c-4c4a-8bf9-444bba2fa644" (UID: "eac98813-7b4c-4c4a-8bf9-444bba2fa644"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:09.527863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.526200 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.531248 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.529553 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsnvd\" (UniqueName: \"kubernetes.io/projected/7101a242-11ce-4b8d-9a74-dae3baf6444b-kube-api-access-nsnvd\") pod \"maas-default-gateway-openshift-default-855f9574cf-7dmwc\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:09.531248 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.531013 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p" Nov 03 20:33:09.546501 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.542456 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk"] Nov 03 20:33:09.546501 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.542607 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.552589 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.552550 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:09.552698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.552637 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:09.552761 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.552695 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:09.600967 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.576886 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.600967 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.576920 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.600967 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.576936 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d75d7\" (UniqueName: \"kubernetes.io/projected/cfcb49c5-4381-411e-9d64-d9dcd6181239-kube-api-access-d75d7\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.600967 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.576951 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eac98813-7b4c-4c4a-8bf9-444bba2fa644-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.602927 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.602766 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Nov 03 20:33:09.602927 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.602821 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" podUID="bbd92cdc-e2bd-4f72-8c63-1221069ca3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Nov 03 20:33:09.606818 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:33:09.606259 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9aefe5_8450_4eb9_8d8b_df89be81ebb3.slice/crio-6c0c462e53b0f3e601530d1f63a4aa75eb98bbedf81934f22d8ce6d5eb1793b7 WatchSource:0}: Error finding container 6c0c462e53b0f3e601530d1f63a4aa75eb98bbedf81934f22d8ce6d5eb1793b7: Status 404 returned error can't find the container with id 6c0c462e53b0f3e601530d1f63a4aa75eb98bbedf81934f22d8ce6d5eb1793b7 Nov 03 20:33:09.607379 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.606924 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x" Nov 03 20:33:09.709313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.661513 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg"] Nov 03 20:33:09.709313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.682957 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51-kube-api-access-tvmxs\" (UniqueName: \"kubernetes.io/projected/ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51-kube-api-access-tvmxs\") pod \"ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51\" (UID: \"ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51\") " Nov 03 20:33:09.709313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.683039 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51-plugins-config-volume\") pod \"ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51\" (UID: \"ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51\") " Nov 03 20:33:09.709313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.683428 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-workload-socket\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.709313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.683467 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-workload-certs\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.709313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.683542 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-envoy\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.709313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.683695 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-token\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.709313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.683743 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-credential-socket\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.709313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.683769 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istiod-ca-cert\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.709313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.683920 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-podinfo\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.709313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.683994 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-data\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.709313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.684025 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl7tt\" (UniqueName: \"kubernetes.io/projected/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-kube-api-access-cl7tt\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.709313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.690199 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lz2v2" event={"ID":"b4d89c5d-1730-4077-a442-d3924153133f","Type":"ContainerStarted","Data":"2c4b42bad7f07ffb5392a5643526e456df40c9d25a11f78596f3b6937a3813c8"} Nov 03 20:33:09.709313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.705451 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51-plugins-config-volume" pod "ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51" (UID: "ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:09.718424 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.718401 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44" Nov 03 20:33:09.725124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.725047 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-nqclg"] Nov 03 20:33:09.730024 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.728507 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lz2v2" Nov 03 20:33:09.808150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.754106 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51-kube-api-access-tvmxs" pod "ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51" (UID: "ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51"). InnerVolumeSpecName "kube-api-access-tvmxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:09.808150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.757074 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r"] Nov 03 20:33:09.808150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.790031 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26375297-81f4-427f-94c3-d56f7a6a11fb-kube-api-access-pp2bg\" (UniqueName: \"kubernetes.io/projected/26375297-81f4-427f-94c3-d56f7a6a11fb-kube-api-access-pp2bg\") pod \"26375297-81f4-427f-94c3-d56f7a6a11fb\" (UID: \"26375297-81f4-427f-94c3-d56f7a6a11fb\") " Nov 03 20:33:09.808150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.790125 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26375297-81f4-427f-94c3-d56f7a6a11fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26375297-81f4-427f-94c3-d56f7a6a11fb-plugins-config-volume\") pod \"26375297-81f4-427f-94c3-d56f7a6a11fb\" (UID: \"26375297-81f4-427f-94c3-d56f7a6a11fb\") " Nov 03 20:33:09.808150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.790407 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-workload-socket\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.808150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.790439 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-workload-certs\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.808150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.790515 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-envoy\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.808150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.790649 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-token\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.808150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.790717 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-credential-socket\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.808150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.790741 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istiod-ca-cert\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.808150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.790889 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-podinfo\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.808150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.790965 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-data\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.808150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.790997 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl7tt\" (UniqueName: \"kubernetes.io/projected/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-kube-api-access-cl7tt\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.808150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.791193 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvmxs\" (UniqueName: \"kubernetes.io/projected/ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51-kube-api-access-tvmxs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.808150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.791215 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:09.808150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.802508 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-workload-socket\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.808150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.806853 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istiod-ca-cert\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.808150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.807331 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-credential-socket\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.809223 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.808590 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26375297-81f4-427f-94c3-d56f7a6a11fb-plugins-config-volume" pod "26375297-81f4-427f-94c3-d56f7a6a11fb" (UID: "26375297-81f4-427f-94c3-d56f7a6a11fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:09.815184 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.809973 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-workload-certs\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.815184 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:33:09.810758 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5585d8d_d156_418c_ae87_012781bc0e8c.slice/crio-1de9be30a5b178d039d2f261dcb78f436d56d55ddd9f813a89d82ed40779e573 WatchSource:0}: Error finding container 1de9be30a5b178d039d2f261dcb78f436d56d55ddd9f813a89d82ed40779e573: Status 404 returned error can't find the container with id 1de9be30a5b178d039d2f261dcb78f436d56d55ddd9f813a89d82ed40779e573 Nov 03 20:33:09.815184 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.812485 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl7tt\" (UniqueName: \"kubernetes.io/projected/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-kube-api-access-cl7tt\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.815184 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.813749 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-data\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:09.815184 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:09.813794 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297"] Nov 03 20:33:10.020282 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.016431 5059 cpu_manager.go:414] "containerMap: removing container" podUID="26375297-81f4-427f-94c3-d56f7a6a11fb" containerName="payload-pre-processing" Nov 03 20:33:10.020282 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.016461 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26375297-81f4-427f-94c3-d56f7a6a11fb" containerName="payload-pre-processing" Nov 03 20:33:10.020282 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.016511 5059 cpu_manager.go:414] "containerMap: removing container" podUID="ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51" containerName="payload-processing" Nov 03 20:33:10.020282 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.016520 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51" containerName="payload-processing" Nov 03 20:33:10.020282 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.016857 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="26375297-81f4-427f-94c3-d56f7a6a11fb" containerName="payload-pre-processing" Nov 03 20:33:10.020282 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.016881 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51" containerName="payload-processing" Nov 03 20:33:10.028743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.023321 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs"] Nov 03 20:33:10.028743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.023405 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26375297-81f4-427f-94c3-d56f7a6a11fb-kube-api-access-pp2bg" pod "26375297-81f4-427f-94c3-d56f7a6a11fb" (UID: "26375297-81f4-427f-94c3-d56f7a6a11fb"). InnerVolumeSpecName "kube-api-access-pp2bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:10.030426 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.028909 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.038684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.037505 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff" Nov 03 20:33:10.038684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.038302 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26375297-81f4-427f-94c3-d56f7a6a11fb-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.044581 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.041158 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:10.044581 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.041323 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:10.044581 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.041448 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:10.050120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.050097 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-envoy\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:10.112973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.104529 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-podinfo\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:10.112973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.104938 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-token\") pod \"e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:10.115684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.115053 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq" Nov 03 20:33:10.122986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.122952 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f" Nov 03 20:33:10.146973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.141956 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.146973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.142024 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.146973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.142127 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.146973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.142162 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.146973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.142214 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.146973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.142264 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.146973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.142321 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.146973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.142563 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m4fg\" (UniqueName: \"kubernetes.io/projected/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-kube-api-access-2m4fg\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.146973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.142607 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.146973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.142836 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp2bg\" (UniqueName: \"kubernetes.io/projected/26375297-81f4-427f-94c3-d56f7a6a11fb-kube-api-access-pp2bg\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.233744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.159359 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:10.233744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.159510 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:10.233744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.159621 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:10.244379 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.244345 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d960e4e-9539-4a53-9df7-1c520ab7d708-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d960e4e-9539-4a53-9df7-1c520ab7d708-plugins-config-volume\") pod \"0d960e4e-9539-4a53-9df7-1c520ab7d708\" (UID: \"0d960e4e-9539-4a53-9df7-1c520ab7d708\") " Nov 03 20:33:10.244645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.244624 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d960e4e-9539-4a53-9df7-1c520ab7d708-kube-api-access-q7dv8\" (UniqueName: \"kubernetes.io/projected/0d960e4e-9539-4a53-9df7-1c520ab7d708-kube-api-access-q7dv8\") pod \"0d960e4e-9539-4a53-9df7-1c520ab7d708\" (UID: \"0d960e4e-9539-4a53-9df7-1c520ab7d708\") " Nov 03 20:33:10.245805 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.245013 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d960e4e-9539-4a53-9df7-1c520ab7d708-plugins-config-volume" pod "0d960e4e-9539-4a53-9df7-1c520ab7d708" (UID: "0d960e4e-9539-4a53-9df7-1c520ab7d708"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:10.249242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.249212 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m4fg\" (UniqueName: \"kubernetes.io/projected/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-kube-api-access-2m4fg\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.249432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.249410 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.249610 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.249590 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.252791 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.252763 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.263934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.263894 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.289166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.289135 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.296868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.289340 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.322685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.297093 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.331759 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.321963 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.332126 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.332102 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.333138 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.333113 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d960e4e-9539-4a53-9df7-1c520ab7d708-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.358680 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.337771 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.362766 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.338766 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.362911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.339032 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.366778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.341315 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.366917 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.341325 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.367040 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.342109 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.385683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.356726 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m4fg\" (UniqueName: \"kubernetes.io/projected/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-kube-api-access-2m4fg\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.385906 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.356837 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-nt297\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:10.527488 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.521940 5059 scope.go:122] "RemoveContainer" containerID="1fe212cabed0fc214cbab7c5d208f7f12d710cdc5a5c86dc76e26a0430ec5af5" Nov 03 20:33:10.527488 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.524194 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="474b2e1d-832b-4edd-8765-31e0bc77cc2a" path="/var/lib/kubelet/pods/474b2e1d-832b-4edd-8765-31e0bc77cc2a/volumes" Nov 03 20:33:10.532578 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.531916 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:33:10.534743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.533884 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d960e4e-9539-4a53-9df7-1c520ab7d708-kube-api-access-q7dv8" pod "0d960e4e-9539-4a53-9df7-1c520ab7d708" (UID: "0d960e4e-9539-4a53-9df7-1c520ab7d708"). InnerVolumeSpecName "kube-api-access-q7dv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:10.544167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.538734 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="518169ac-2f8d-4645-ae47-210b67ca080a" path="/var/lib/kubelet/pods/518169ac-2f8d-4645-ae47-210b67ca080a/volumes" Nov 03 20:33:10.544167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.539450 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="890f130b-cdeb-43da-acd4-1654673271cc" path="/var/lib/kubelet/pods/890f130b-cdeb-43da-acd4-1654673271cc/volumes" Nov 03 20:33:10.544167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.540234 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b648c6b-1f7d-42ca-9c3f-a435494bad03" path="/var/lib/kubelet/pods/8b648c6b-1f7d-42ca-9c3f-a435494bad03/volumes" Nov 03 20:33:10.544167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.542813 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8683a65-9898-4b4f-a692-f2da325a93b4" path="/var/lib/kubelet/pods/c8683a65-9898-4b4f-a692-f2da325a93b4/volumes" Nov 03 20:33:10.544167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.542860 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps" Nov 03 20:33:10.544167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.543762 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cef4417c-3d04-4d65-a5bc-6ec1af6f09bb" path="/var/lib/kubelet/pods/cef4417c-3d04-4d65-a5bc-6ec1af6f09bb/volumes" Nov 03 20:33:10.556181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.548755 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l"] Nov 03 20:33:10.556181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.548830 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297"] Nov 03 20:33:10.556181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.548871 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9"] Nov 03 20:33:10.556181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.548911 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95"] Nov 03 20:33:10.556181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.550127 5059 cpu_manager.go:414] "containerMap: removing container" podUID="2faa14fa-ec86-4633-a968-c3f761d13443" containerName="payload-processing" Nov 03 20:33:10.556181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.550148 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2faa14fa-ec86-4633-a968-c3f761d13443" containerName="payload-processing" Nov 03 20:33:10.556181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.550176 5059 cpu_manager.go:414] "containerMap: removing container" podUID="2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" containerName="istio-proxy" Nov 03 20:33:10.556181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.550188 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" containerName="istio-proxy" Nov 03 20:33:10.556181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.550204 5059 cpu_manager.go:414] "containerMap: removing container" podUID="0d960e4e-9539-4a53-9df7-1c520ab7d708" containerName="payload-pre-processing" Nov 03 20:33:10.556181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.550215 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d960e4e-9539-4a53-9df7-1c520ab7d708" containerName="payload-pre-processing" Nov 03 20:33:10.556181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.550481 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2faa14fa-ec86-4633-a968-c3f761d13443" containerName="payload-processing" Nov 03 20:33:10.556181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.550502 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" containerName="istio-proxy" Nov 03 20:33:10.556181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.550513 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d960e4e-9539-4a53-9df7-1c520ab7d708" containerName="payload-pre-processing" Nov 03 20:33:10.556181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.550532 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" containerName="istio-proxy" Nov 03 20:33:10.556181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.552969 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7dv8\" (UniqueName: \"kubernetes.io/projected/0d960e4e-9539-4a53-9df7-1c520ab7d708-kube-api-access-q7dv8\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.562703 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.557049 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:10.562703 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.557132 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:10.562703 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.557188 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:10.582885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.582843 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:10.582981 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.582938 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:10.583050 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.582985 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:10.604713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.604624 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:10.604806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.604737 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:10.604806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.604789 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:10.608994 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.607421 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Nov 03 20:33:10.608994 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.607465 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" podUID="bbd92cdc-e2bd-4f72-8c63-1221069ca3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Nov 03 20:33:10.626737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.625395 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:10.626737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.625527 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:10.626737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.625590 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:10.642165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.642118 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:10.642269 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.642239 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:10.642348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.642291 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:10.646632 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.644586 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl" Nov 03 20:33:10.647577 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.647357 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 20:33:10.651108 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.651080 5059 generic.go:356] "Generic (PLEG): container finished" podID="6b9428a2-ab50-4e44-a725-e0da980bdb93" containerID="2d957762bcd8fed6693d101c9e9584a4ee5948a1b3f069a82b1b80826caa7347" exitCode=22 Nov 03 20:33:10.651243 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.651225 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf" Nov 03 20:33:10.651862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.651844 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt" Nov 03 20:33:10.652346 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.652329 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7" Nov 03 20:33:10.653375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.652947 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd" Nov 03 20:33:10.654846 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.654293 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4" Nov 03 20:33:10.655081 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.655064 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5" Nov 03 20:33:10.686382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.655706 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2" Nov 03 20:33:10.686382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.656200 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn" Nov 03 20:33:10.686382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.656394 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95"] Nov 03 20:33:10.686382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.656541 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb" event={"ID":"26576b40-fd9c-4298-9129-5b341d2c0b3d","Type":"ContainerStarted","Data":"843a76e69bdcc8b626cf0ba45546fd6598ae7eb974cbddb9144d57af0fad2b12"} Nov 03 20:33:10.686382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.656579 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb"] Nov 03 20:33:10.686382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.656611 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5" event={"ID":"6b9428a2-ab50-4e44-a725-e0da980bdb93","Type":"ContainerDied","Data":"2d957762bcd8fed6693d101c9e9584a4ee5948a1b3f069a82b1b80826caa7347"} Nov 03 20:33:10.686382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.656641 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz"] Nov 03 20:33:10.686382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.656929 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-workload-certs\") pod \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " Nov 03 20:33:10.686382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.656979 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-credential-socket\") pod \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " Nov 03 20:33:10.686382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.657078 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-data\") pod \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " Nov 03 20:33:10.686382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.657132 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-workload-socket\") pod \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " Nov 03 20:33:10.686382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.657168 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-envoy\") pod \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " Nov 03 20:33:10.686382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.657289 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istiod-ca-cert\") pod \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " Nov 03 20:33:10.686382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.657371 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-token\" (UniqueName: \"kubernetes.io/projected/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-token\") pod \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " Nov 03 20:33:10.686382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.657408 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2faa14fa-ec86-4633-a968-c3f761d13443-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2faa14fa-ec86-4633-a968-c3f761d13443-plugins-config-volume\") pod \"2faa14fa-ec86-4633-a968-c3f761d13443\" (UID: \"2faa14fa-ec86-4633-a968-c3f761d13443\") " Nov 03 20:33:10.686382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.657521 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-kube-api-access-hdbbr\" (UniqueName: \"kubernetes.io/projected/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-kube-api-access-hdbbr\") pod \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " Nov 03 20:33:10.686382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.657560 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-podinfo\") pod \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\" (UID: \"2ed21c5c-ef42-4fb0-bf41-34e84b9cf193\") " Nov 03 20:33:10.686382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.657620 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2faa14fa-ec86-4633-a968-c3f761d13443-kube-api-access-g8n84\" (UniqueName: \"kubernetes.io/projected/2faa14fa-ec86-4633-a968-c3f761d13443-kube-api-access-g8n84\") pod \"2faa14fa-ec86-4633-a968-c3f761d13443\" (UID: \"2faa14fa-ec86-4633-a968-c3f761d13443\") " Nov 03 20:33:10.686382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.668623 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l" Nov 03 20:33:10.707693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.704805 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm" Nov 03 20:33:10.707693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.706177 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d" Nov 03 20:33:10.707693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.706846 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn" Nov 03 20:33:10.707693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.706870 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h" Nov 03 20:33:10.709743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.706176 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-podinfo\") pod \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " Nov 03 20:33:10.709743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.708006 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c3d72b6-73c4-402f-8308-ee5d87a9dd83-kube-api-access-8g2ll\" (UniqueName: \"kubernetes.io/projected/9c3d72b6-73c4-402f-8308-ee5d87a9dd83-kube-api-access-8g2ll\") pod \"9c3d72b6-73c4-402f-8308-ee5d87a9dd83\" (UID: \"9c3d72b6-73c4-402f-8308-ee5d87a9dd83\") " Nov 03 20:33:10.709743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.708067 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istiod-ca-cert\") pod \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " Nov 03 20:33:10.709743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.708129 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-workload-certs\") pod \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " Nov 03 20:33:10.709743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.708157 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-credential-socket\") pod \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " Nov 03 20:33:10.709743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.708653 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-data" pod "2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" (UID: "2ed21c5c-ef42-4fb0-bf41-34e84b9cf193"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:10.709743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.708732 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0298aa52-aec0-481a-b7d1-12964e8137c8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95\" (UID: \"0298aa52-aec0-481a-b7d1-12964e8137c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95" Nov 03 20:33:10.709743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.708985 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6glj\" (UniqueName: \"kubernetes.io/projected/0298aa52-aec0-481a-b7d1-12964e8137c8-kube-api-access-w6glj\") pod \"maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95\" (UID: \"0298aa52-aec0-481a-b7d1-12964e8137c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95" Nov 03 20:33:10.709743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.709028 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-workload-certs" pod "2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" (UID: "2ed21c5c-ef42-4fb0-bf41-34e84b9cf193"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:10.709743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.709108 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0298aa52-aec0-481a-b7d1-12964e8137c8-metrics-certs\") pod \"maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95\" (UID: \"0298aa52-aec0-481a-b7d1-12964e8137c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95" Nov 03 20:33:10.709743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.709143 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0298aa52-aec0-481a-b7d1-12964e8137c8-maas-api-tls\") pod \"maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95\" (UID: \"0298aa52-aec0-481a-b7d1-12964e8137c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95" Nov 03 20:33:10.709743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.709454 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.709743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.709476 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.710776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.710619 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istiod-ca-cert" pod "2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" (UID: "2ed21c5c-ef42-4fb0-bf41-34e84b9cf193"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:10.710872 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.710803 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-workload-socket" pod "2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" (UID: "2ed21c5c-ef42-4fb0-bf41-34e84b9cf193"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:10.710872 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.710854 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-credential-socket" pod "2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" (UID: "2ed21c5c-ef42-4fb0-bf41-34e84b9cf193"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:10.711212 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.711191 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-token" pod "2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" (UID: "2ed21c5c-ef42-4fb0-bf41-34e84b9cf193"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:10.711910 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.711638 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2faa14fa-ec86-4633-a968-c3f761d13443-plugins-config-volume" pod "2faa14fa-ec86-4633-a968-c3f761d13443" (UID: "2faa14fa-ec86-4633-a968-c3f761d13443"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:10.712700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.712431 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istiod-ca-cert" pod "d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" (UID: "d3de18b3-c3d1-4ec3-961f-d95ca3656f4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:10.712700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.712482 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-workload-certs" pod "d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" (UID: "d3de18b3-c3d1-4ec3-961f-d95ca3656f4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:10.712700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.712560 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4"] Nov 03 20:33:10.712944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.712922 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3d72b6-73c4-402f-8308-ee5d87a9dd83-kube-api-access-8g2ll" pod "9c3d72b6-73c4-402f-8308-ee5d87a9dd83" (UID: "9c3d72b6-73c4-402f-8308-ee5d87a9dd83"). InnerVolumeSpecName "kube-api-access-8g2ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:10.713013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.712965 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-credential-socket" pod "d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" (UID: "d3de18b3-c3d1-4ec3-961f-d95ca3656f4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:10.714842 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.713305 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm" Nov 03 20:33:10.721742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.716584 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r" Nov 03 20:33:10.721742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.720988 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42" Nov 03 20:33:10.721742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.721149 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95" Nov 03 20:33:10.728588 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.723408 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-envoy" pod "2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" (UID: "2ed21c5c-ef42-4fb0-bf41-34e84b9cf193"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:10.728588 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.725302 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-podinfo" pod "d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" (UID: "d3de18b3-c3d1-4ec3-961f-d95ca3656f4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:10.728588 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.725534 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm"] Nov 03 20:33:10.728588 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.725588 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk" Nov 03 20:33:10.728588 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.725824 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-podinfo" pod "2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" (UID: "2ed21c5c-ef42-4fb0-bf41-34e84b9cf193"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:10.734797 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.729431 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2faa14fa-ec86-4633-a968-c3f761d13443-kube-api-access-g8n84" pod "2faa14fa-ec86-4633-a968-c3f761d13443" (UID: "2faa14fa-ec86-4633-a968-c3f761d13443"). InnerVolumeSpecName "kube-api-access-g8n84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:10.734797 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.731821 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:33:10.734797 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.732201 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e7b70d-serving-cert\"" Nov 03 20:33:10.734797 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.733609 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-kube-api-access-hdbbr" pod "2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" (UID: "2ed21c5c-ef42-4fb0-bf41-34e84b9cf193"). InnerVolumeSpecName "kube-api-access-hdbbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:10.734797 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.734740 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:33:10.736072 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.735984 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8" Nov 03 20:33:10.737246 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.736962 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls" Nov 03 20:33:10.737685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.737621 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h" Nov 03 20:33:10.749103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.744559 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62"] Nov 03 20:33:10.749103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.745796 5059 cpu_manager.go:414] "containerMap: removing container" podUID="2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" containerName="istio-proxy" Nov 03 20:33:10.749103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.745820 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" containerName="istio-proxy" Nov 03 20:33:10.749103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.745856 5059 cpu_manager.go:414] "containerMap: removing container" podUID="d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" containerName="istio-proxy" Nov 03 20:33:10.749103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.745864 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" containerName="istio-proxy" Nov 03 20:33:10.749103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.745877 5059 cpu_manager.go:414] "containerMap: removing container" podUID="d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" containerName="istio-proxy" Nov 03 20:33:10.749103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.745885 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" containerName="istio-proxy" Nov 03 20:33:10.749103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.745907 5059 cpu_manager.go:414] "containerMap: removing container" podUID="9c3d72b6-73c4-402f-8308-ee5d87a9dd83" containerName="payload-pre-processing" Nov 03 20:33:10.749103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.745915 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c3d72b6-73c4-402f-8308-ee5d87a9dd83" containerName="payload-pre-processing" Nov 03 20:33:10.749103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.746102 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" containerName="istio-proxy" Nov 03 20:33:10.749103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.746124 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c3d72b6-73c4-402f-8308-ee5d87a9dd83" containerName="payload-pre-processing" Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.826557 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-envoy\") pod \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.826801 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-token\" (UniqueName: \"kubernetes.io/projected/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-token\") pod \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.826897 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-workload-socket\") pod \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.827015 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c3d72b6-73c4-402f-8308-ee5d87a9dd83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3d72b6-73c4-402f-8308-ee5d87a9dd83-plugins-config-volume\") pod \"9c3d72b6-73c4-402f-8308-ee5d87a9dd83\" (UID: \"9c3d72b6-73c4-402f-8308-ee5d87a9dd83\") " Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.827136 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-kube-api-access-bnp2q\" (UniqueName: \"kubernetes.io/projected/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-kube-api-access-bnp2q\") pod \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.827199 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-data\") pod \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\" (UID: \"d3de18b3-c3d1-4ec3-961f-d95ca3656f4b\") " Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.828024 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0298aa52-aec0-481a-b7d1-12964e8137c8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95\" (UID: \"0298aa52-aec0-481a-b7d1-12964e8137c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95" Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.828265 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6glj\" (UniqueName: \"kubernetes.io/projected/0298aa52-aec0-481a-b7d1-12964e8137c8-kube-api-access-w6glj\") pod \"maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95\" (UID: \"0298aa52-aec0-481a-b7d1-12964e8137c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95" Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.828334 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0298aa52-aec0-481a-b7d1-12964e8137c8-metrics-certs\") pod \"maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95\" (UID: \"0298aa52-aec0-481a-b7d1-12964e8137c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95" Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.828366 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0298aa52-aec0-481a-b7d1-12964e8137c8-maas-api-tls\") pod \"maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95\" (UID: \"0298aa52-aec0-481a-b7d1-12964e8137c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95" Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.828622 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.828642 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g2ll\" (UniqueName: \"kubernetes.io/projected/9c3d72b6-73c4-402f-8308-ee5d87a9dd83-kube-api-access-8g2ll\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.828678 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.828692 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.828706 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.828719 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.828734 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.828748 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.828759 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.828773 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.828787 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2faa14fa-ec86-4633-a968-c3f761d13443-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.828799 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdbbr\" (UniqueName: \"kubernetes.io/projected/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-kube-api-access-hdbbr\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.828812 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.828828 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8n84\" (UniqueName: \"kubernetes.io/projected/2faa14fa-ec86-4633-a968-c3f761d13443-kube-api-access-g8n84\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.838631 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0298aa52-aec0-481a-b7d1-12964e8137c8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95\" (UID: \"0298aa52-aec0-481a-b7d1-12964e8137c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95" Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.839760 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-data" pod "d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" (UID: "d3de18b3-c3d1-4ec3-961f-d95ca3656f4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.839975 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-workload-socket" pod "d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" (UID: "d3de18b3-c3d1-4ec3-961f-d95ca3656f4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.840494 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c3d72b6-73c4-402f-8308-ee5d87a9dd83-plugins-config-volume" pod "9c3d72b6-73c4-402f-8308-ee5d87a9dd83" (UID: "9c3d72b6-73c4-402f-8308-ee5d87a9dd83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.840903 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62"] Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.840944 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk"] Nov 03 20:33:10.842022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.841031 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.075517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.849859 5059 scope.go:122] "RemoveContainer" containerID="2d957762bcd8fed6693d101c9e9584a4ee5948a1b3f069a82b1b80826caa7347" Nov 03 20:33:11.075517 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:10.850229 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-266d0609-xqmv5_odh-ai-gateway-infra(6b9428a2-ab50-4e44-a725-e0da980bdb93)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5" podUID="6b9428a2-ab50-4e44-a725-e0da980bdb93" Nov 03 20:33:11.075517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.853469 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-8pmsk"] Nov 03 20:33:11.075517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.854251 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" Nov 03 20:33:11.075517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.932568 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.075517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.932606 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.075517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.932623 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3d72b6-73c4-402f-8308-ee5d87a9dd83-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.075517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:10.942355 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" Nov 03 20:33:11.075517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.035505 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-workload-socket\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.075517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.035629 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75efd589-b943-46c2-add0-d44ccdaff537-istiod-ca-cert\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.075517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.035731 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75efd589-b943-46c2-add0-d44ccdaff537-istio-podinfo\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.075517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.035773 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75efd589-b943-46c2-add0-d44ccdaff537-istio-token\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.075517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.035889 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-workload-certs\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.075517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.036026 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-credential-socket\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.075517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.036062 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-istio-data\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.075517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.036116 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp7pg\" (UniqueName: \"kubernetes.io/projected/75efd589-b943-46c2-add0-d44ccdaff537-kube-api-access-zp7pg\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.075517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.036199 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-istio-envoy\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.147686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.146077 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75efd589-b943-46c2-add0-d44ccdaff537-istiod-ca-cert\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.147686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.146176 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75efd589-b943-46c2-add0-d44ccdaff537-istio-podinfo\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.147686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.146222 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75efd589-b943-46c2-add0-d44ccdaff537-istio-token\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.147686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.146289 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-workload-certs\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.147686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.146385 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-credential-socket\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.147686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.146419 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-istio-data\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.147686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.146469 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp7pg\" (UniqueName: \"kubernetes.io/projected/75efd589-b943-46c2-add0-d44ccdaff537-kube-api-access-zp7pg\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.147686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.146533 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-istio-envoy\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.147686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.146767 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-workload-socket\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.147686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.147419 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-workload-socket\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.152708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.148529 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75efd589-b943-46c2-add0-d44ccdaff537-istiod-ca-cert\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.173358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.159070 5059 patch_prober.go:29] interesting pod/router-default-f7b87d695-cwlfq container/router namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]backend-proxy-http ok Nov 03 20:33:11.173358 ip-10-0-132-83 kubenswrapper[5059]: [+]has-synced ok Nov 03 20:33:11.173358 ip-10-0-132-83 kubenswrapper[5059]: [-]process-running failed: reason withheld Nov 03 20:33:11.173358 ip-10-0-132-83 kubenswrapper[5059]: healthz check failed Nov 03 20:33:11.173358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.159125 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-f7b87d695-cwlfq" podUID="603868b0-8628-4227-8471-1894e2e1de50" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Nov 03 20:33:11.173358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.160338 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-credential-socket\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.173358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.160980 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-workload-certs\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.173358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.161211 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-istio-data\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.181991 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.181924 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75efd589-b943-46c2-add0-d44ccdaff537-istio-token\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:11.762846 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.762648 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" containerID="cri-o://1317df3f930488af045a4ea5f2ffd5b2d0b6bac1972a73208461be0692b6e2b2" gracePeriod=45 Nov 03 20:33:11.763740 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:11.763712 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:33:11.763957 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.763935 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" containerID="cri-o://5b05929b67ea95a6b627445e0a3fa63ef0a0d7ef033293a27edabc525d4bd1cd" gracePeriod=45 Nov 03 20:33:11.764184 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.764163 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" containerID="cri-o://238b88770b80cec7d339089cb9d69cc1f7ef5a1fdcefa3eb790154cc7f3289f1" gracePeriod=45 Nov 03 20:33:11.764393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.764374 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" podUID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" containerName="main" containerID="cri-o://90b3ff7ea089036c1955493d6ad07b652eb65d8c482b1abd7f0dc8c58bea2751" gracePeriod=45 Nov 03 20:33:11.764620 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.764601 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" containerID="cri-o://b8d82ca6d1d4fd46ae40cb0b5685d96dc2c2f3703f2b90b33d10896376edb8cc" gracePeriod=45 Nov 03 20:33:11.801720 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.790726 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Nov 03 20:33:11.801720 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.790794 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" podUID="bbd92cdc-e2bd-4f72-8c63-1221069ca3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Nov 03 20:33:11.852698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.846499 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" containerID="cri-o://1c69adcd1fc6db5a42ddeb00863f24a144f0cf528222cad083ac5a885e389943" gracePeriod=45 Nov 03 20:33:11.852698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.847017 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" containerID="cri-o://44d16c3b122733147748e39e219fb00f390ff6e37a49c47ed3e51ba2960a62c1" gracePeriod=45 Nov 03 20:33:12.016981 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:11.980821 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:12.188524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.153727 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp7pg\" (UniqueName: \"kubernetes.io/projected/75efd589-b943-46c2-add0-d44ccdaff537-kube-api-access-zp7pg\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:12.188524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.154039 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-istio-envoy\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:12.265878 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.227633 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-kube-api-access-bnp2q" pod "d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" (UID: "d3de18b3-c3d1-4ec3-961f-d95ca3656f4b"). InnerVolumeSpecName "kube-api-access-bnp2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:12.265878 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.227997 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": EOF" Nov 03 20:33:12.265878 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.228515 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-token" pod "d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" (UID: "d3de18b3-c3d1-4ec3-961f-d95ca3656f4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:12.265878 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.229470 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-envoy" pod "d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" (UID: "d3de18b3-c3d1-4ec3-961f-d95ca3656f4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:12.265878 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.230513 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75efd589-b943-46c2-add0-d44ccdaff537-istio-podinfo\") pod \"e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:12.265878 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.244867 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.265878 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.244904 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.265878 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.244922 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnp2q\" (UniqueName: \"kubernetes.io/projected/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b-kube-api-access-bnp2q\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.296141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.272305 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0298aa52-aec0-481a-b7d1-12964e8137c8-metrics-certs\") pod \"maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95\" (UID: \"0298aa52-aec0-481a-b7d1-12964e8137c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95" Nov 03 20:33:12.389297 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.389260 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6glj\" (UniqueName: \"kubernetes.io/projected/0298aa52-aec0-481a-b7d1-12964e8137c8-kube-api-access-w6glj\") pod \"maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95\" (UID: \"0298aa52-aec0-481a-b7d1-12964e8137c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95" Nov 03 20:33:12.479693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.469447 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" podUID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": EOF" Nov 03 20:33:12.479693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.469780 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": read tcp 10.132.0.2:34696->10.132.0.94:8000: read: connection reset by peer" Nov 03 20:33:12.515932 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.484353 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": EOF" Nov 03 20:33:12.608705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.601057 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": EOF" Nov 03 20:33:12.747694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.741910 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": EOF" Nov 03 20:33:12.761748 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.752517 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": dial tcp 10.132.0.94:8000: connect: connection refused" Nov 03 20:33:12.761748 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.758718 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" podUID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": dial tcp 10.132.0.49:8000: connect: connection refused" Nov 03 20:33:12.778009 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.769363 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": read tcp 10.132.0.2:51774->10.132.0.78:8000: read: connection reset by peer" Nov 03 20:33:12.778009 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.770494 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": dial tcp 10.132.0.51:8000: connect: connection refused" Nov 03 20:33:12.833689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.823372 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": dial tcp 10.132.0.48:8000: connect: connection refused" Nov 03 20:33:12.833689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.828785 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Nov 03 20:33:12.833689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.828851 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" podUID="bbd92cdc-e2bd-4f72-8c63-1221069ca3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Nov 03 20:33:12.857698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.855413 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": dial tcp 10.132.0.77:8000: connect: connection refused" Nov 03 20:33:12.909702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.895461 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0298aa52-aec0-481a-b7d1-12964e8137c8-maas-api-tls\") pod \"maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95\" (UID: \"0298aa52-aec0-481a-b7d1-12964e8137c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95" Nov 03 20:33:12.921519 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:12.920861 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": dial tcp 10.132.0.76:8000: connect: connection refused" Nov 03 20:33:13.137686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:13.122725 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": dial tcp 10.132.0.78:8000: connect: connection refused" Nov 03 20:33:13.581689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:13.458524 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:13.600535 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:13.600483 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:13.600867 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:13.600824 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:13.622874 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:13.530185 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": dial tcp 10.132.0.94:8000: connect: connection refused" Nov 03 20:33:13.622874 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:13.558148 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:33:13.622874 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:13.621475 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": dial tcp 10.132.0.48:8000: connect: connection refused" Nov 03 20:33:13.628916 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:13.580252 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Nov 03 20:33:13.640863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:13.632725 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" podUID="bbd92cdc-e2bd-4f72-8c63-1221069ca3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Nov 03 20:33:13.688770 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:13.673122 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": dial tcp 10.132.0.77:8000: connect: connection refused" Nov 03 20:33:13.688770 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:13.673506 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" podUID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": dial tcp 10.132.0.49:8000: connect: connection refused" Nov 03 20:33:13.701443 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:13.701360 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt" Nov 03 20:33:13.834925 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:13.834881 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edcfb7ab-b14f-4d5c-b752-5b43a6eee163" path="/var/lib/kubelet/pods/edcfb7ab-b14f-4d5c-b752-5b43a6eee163/volumes" Nov 03 20:33:14.099689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.074751 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c92b1be0-0890-4db6-ac75-968b6e5a3434-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92b1be0-0890-4db6-ac75-968b6e5a3434-plugins-config-volume\") pod \"c92b1be0-0890-4db6-ac75-968b6e5a3434\" (UID: \"c92b1be0-0890-4db6-ac75-968b6e5a3434\") " Nov 03 20:33:14.100004 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.099973 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-workload-certs\") pod \"8153e018-7f9a-4645-b102-06ff3913f682\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " Nov 03 20:33:14.103685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.100256 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c92b1be0-0890-4db6-ac75-968b6e5a3434-kube-api-access-b4m8l\" (UniqueName: \"kubernetes.io/projected/c92b1be0-0890-4db6-ac75-968b6e5a3434-kube-api-access-b4m8l\") pod \"c92b1be0-0890-4db6-ac75-968b6e5a3434\" (UID: \"c92b1be0-0890-4db6-ac75-968b6e5a3434\") " Nov 03 20:33:14.116906 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.116868 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8153e018-7f9a-4645-b102-06ff3913f682-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8153e018-7f9a-4645-b102-06ff3913f682-istio-podinfo\") pod \"8153e018-7f9a-4645-b102-06ff3913f682\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " Nov 03 20:33:14.162879 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.118946 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-workload-certs" pod "8153e018-7f9a-4645-b102-06ff3913f682" (UID: "8153e018-7f9a-4645-b102-06ff3913f682"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:14.163076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.142068 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c92b1be0-0890-4db6-ac75-968b6e5a3434-plugins-config-volume" pod "c92b1be0-0890-4db6-ac75-968b6e5a3434" (UID: "c92b1be0-0890-4db6-ac75-968b6e5a3434"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:14.163178 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.119282 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": dial tcp 10.132.0.51:8000: connect: connection refused" Nov 03 20:33:14.173944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.173888 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": dial tcp 10.132.0.76:8000: connect: connection refused" Nov 03 20:33:14.234687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.211120 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8153e018-7f9a-4645-b102-06ff3913f682-kube-api-access-55dfj\" (UniqueName: \"kubernetes.io/projected/8153e018-7f9a-4645-b102-06ff3913f682-kube-api-access-55dfj\") pod \"8153e018-7f9a-4645-b102-06ff3913f682\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " Nov 03 20:33:14.313183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.313152 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8153e018-7f9a-4645-b102-06ff3913f682-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8153e018-7f9a-4645-b102-06ff3913f682-istiod-ca-cert\") pod \"8153e018-7f9a-4645-b102-06ff3913f682\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " Nov 03 20:33:14.313409 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.313392 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-credential-socket\") pod \"8153e018-7f9a-4645-b102-06ff3913f682\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " Nov 03 20:33:14.313523 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.313508 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8153e018-7f9a-4645-b102-06ff3913f682-istio-token\" (UniqueName: \"kubernetes.io/projected/8153e018-7f9a-4645-b102-06ff3913f682-istio-token\") pod \"8153e018-7f9a-4645-b102-06ff3913f682\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " Nov 03 20:33:14.313636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.313624 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-istio-envoy\") pod \"8153e018-7f9a-4645-b102-06ff3913f682\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " Nov 03 20:33:14.313781 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.313767 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-istio-data\") pod \"8153e018-7f9a-4645-b102-06ff3913f682\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " Nov 03 20:33:14.313918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.313905 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-workload-socket\") pod \"8153e018-7f9a-4645-b102-06ff3913f682\" (UID: \"8153e018-7f9a-4645-b102-06ff3913f682\") " Nov 03 20:33:14.350598 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.333058 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8153e018-7f9a-4645-b102-06ff3913f682-istiod-ca-cert" pod "8153e018-7f9a-4645-b102-06ff3913f682" (UID: "8153e018-7f9a-4645-b102-06ff3913f682"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:14.424814 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.337303 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92b1be0-0890-4db6-ac75-968b6e5a3434-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:14.424814 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.412463 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-credential-socket" pod "8153e018-7f9a-4645-b102-06ff3913f682" (UID: "8153e018-7f9a-4645-b102-06ff3913f682"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:14.448836 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.448776 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-workload-socket" pod "8153e018-7f9a-4645-b102-06ff3913f682" (UID: "8153e018-7f9a-4645-b102-06ff3913f682"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:14.527692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.499109 5059 scope.go:122] "RemoveContainer" containerID="c56ee037a40c322b89e375002cf24e08b31d3e1b30e1b41b0f4819b338d40ec8" Nov 03 20:33:14.527692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.499798 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": dial tcp 10.132.0.78:8000: connect: connection refused" Nov 03 20:33:14.527692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.499818 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:14.572716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.572633 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:33:14.624691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.608342 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": dial tcp 10.132.0.94:8000: connect: connection refused" Nov 03 20:33:14.668828 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:14.654781 5059 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.782s" Nov 03 20:33:14.668828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.661790 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-istio-data" pod "8153e018-7f9a-4645-b102-06ff3913f682" (UID: "8153e018-7f9a-4645-b102-06ff3913f682"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:14.668828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.662089 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": dial tcp 10.132.0.77:8000: connect: connection refused" Nov 03 20:33:14.713688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.712106 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-token\" (UniqueName: \"kubernetes.io/projected/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-token\") pod \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " Nov 03 20:33:14.713688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.712231 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-data\") pod \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " Nov 03 20:33:14.713688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.712295 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-kube-api-access-czj55\" (UniqueName: \"kubernetes.io/projected/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-kube-api-access-czj55\") pod \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " Nov 03 20:33:14.713688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.712422 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-workload-socket\") pod \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " Nov 03 20:33:14.713688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.712476 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-credential-socket\") pod \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " Nov 03 20:33:14.713688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.712554 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-workload-certs\") pod \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " Nov 03 20:33:14.713688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.712638 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-podinfo\") pod \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " Nov 03 20:33:14.713688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.712904 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-envoy\") pod \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " Nov 03 20:33:14.713688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.712979 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istiod-ca-cert\") pod \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\" (UID: \"1731953b-1850-40d1-9ba4-cb08dc4ebc5d\") " Nov 03 20:33:14.722688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.720038 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istiod-ca-cert" pod "1731953b-1850-40d1-9ba4-cb08dc4ebc5d" (UID: "1731953b-1850-40d1-9ba4-cb08dc4ebc5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:14.858686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.855187 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": dial tcp 10.132.0.51:8000: connect: connection refused" Nov 03 20:33:14.867684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.861074 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": dial tcp 10.132.0.48:8000: connect: connection refused" Nov 03 20:33:14.917687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.893389 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-data" pod "1731953b-1850-40d1-9ba4-cb08dc4ebc5d" (UID: "1731953b-1850-40d1-9ba4-cb08dc4ebc5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:14.992348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.921121 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8153e018-7f9a-4645-b102-06ff3913f682-istio-podinfo" pod "8153e018-7f9a-4645-b102-06ff3913f682" (UID: "8153e018-7f9a-4645-b102-06ff3913f682"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:14.992348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.941855 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8153e018-7f9a-4645-b102-06ff3913f682-kube-api-access-55dfj" pod "8153e018-7f9a-4645-b102-06ff3913f682" (UID: "8153e018-7f9a-4645-b102-06ff3913f682"). InnerVolumeSpecName "kube-api-access-55dfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:14.992348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.893812 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": dial tcp 10.132.0.76:8000: connect: connection refused" Nov 03 20:33:14.992348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:14.992307 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-istio-envoy" pod "8153e018-7f9a-4645-b102-06ff3913f682" (UID: "8153e018-7f9a-4645-b102-06ff3913f682"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:15.140854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.052629 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-workload-socket" pod "1731953b-1850-40d1-9ba4-cb08dc4ebc5d" (UID: "1731953b-1850-40d1-9ba4-cb08dc4ebc5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:15.140854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.052946 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-token" pod "1731953b-1850-40d1-9ba4-cb08dc4ebc5d" (UID: "1731953b-1850-40d1-9ba4-cb08dc4ebc5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:15.140854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.055370 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-credential-socket" pod "1731953b-1850-40d1-9ba4-cb08dc4ebc5d" (UID: "1731953b-1850-40d1-9ba4-cb08dc4ebc5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:15.140854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.067090 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" podUID="d098ef77-addb-4077-b6d4-f0ae477409b8" containerName="main" probeResult="failure" output="Get \"https://10.132.1.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:15.140854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.072640 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" podUID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": dial tcp 10.132.0.49:8000: connect: connection refused" Nov 03 20:33:15.140854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.121097 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-workload-certs" pod "1731953b-1850-40d1-9ba4-cb08dc4ebc5d" (UID: "1731953b-1850-40d1-9ba4-cb08dc4ebc5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:15.141966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.141740 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8153e018-7f9a-4645-b102-06ff3913f682-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:15.141966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.141777 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:15.141966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.141793 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:15.141966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.141809 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:15.141966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.141822 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:15.141966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.141835 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8153e018-7f9a-4645-b102-06ff3913f682-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:15.141966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.141849 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:15.141966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.141862 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:15.141966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.141875 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8153e018-7f9a-4645-b102-06ff3913f682-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:15.141966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.141890 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:15.141966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.141904 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:15.141966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.141920 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55dfj\" (UniqueName: \"kubernetes.io/projected/8153e018-7f9a-4645-b102-06ff3913f682-kube-api-access-55dfj\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:15.178438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.177562 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": dial tcp 10.132.0.78:8000: connect: connection refused" Nov 03 20:33:15.365697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.353596 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:15.450596 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.435466 5059 generic.go:356] "Generic (PLEG): container finished" podID="dc10f92e-e902-48c4-9cd5-4f768c4611be" containerID="4d94405c1bd7e9ab077c0ca4e40445d24510310ab3bd698c9d3fc466bbff887d" exitCode=22 Nov 03 20:33:15.548689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.542615 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": dial tcp 10.132.0.94:8000: connect: connection refused" Nov 03 20:33:15.576861 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.576804 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c92b1be0-0890-4db6-ac75-968b6e5a3434-kube-api-access-b4m8l" pod "c92b1be0-0890-4db6-ac75-968b6e5a3434" (UID: "c92b1be0-0890-4db6-ac75-968b6e5a3434"). InnerVolumeSpecName "kube-api-access-b4m8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:15.626293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.626241 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4m8l\" (UniqueName: \"kubernetes.io/projected/c92b1be0-0890-4db6-ac75-968b6e5a3434-kube-api-access-b4m8l\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:15.633732 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.628912 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": dial tcp 10.132.0.48:8000: connect: connection refused" Nov 03 20:33:15.648901 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.648482 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": dial tcp 10.132.0.77:8000: connect: connection refused" Nov 03 20:33:15.649015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.648985 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-kube-api-access-czj55" pod "1731953b-1850-40d1-9ba4-cb08dc4ebc5d" (UID: "1731953b-1850-40d1-9ba4-cb08dc4ebc5d"). InnerVolumeSpecName "kube-api-access-czj55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:15.663789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.653337 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" podUID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": dial tcp 10.132.0.49:8000: connect: connection refused" Nov 03 20:33:15.710224 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.710130 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-podinfo" pod "1731953b-1850-40d1-9ba4-cb08dc4ebc5d" (UID: "1731953b-1850-40d1-9ba4-cb08dc4ebc5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:15.758810 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.756394 5059 scope.go:122] "RemoveContainer" containerID="4d94405c1bd7e9ab077c0ca4e40445d24510310ab3bd698c9d3fc466bbff887d" Nov 03 20:33:15.758810 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:15.756907 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-r5fd4_odh-ai-gateway-infra(dc10f92e-e902-48c4-9cd5-4f768c4611be)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4" podUID="dc10f92e-e902-48c4-9cd5-4f768c4611be" Nov 03 20:33:15.758810 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.757105 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8153e018-7f9a-4645-b102-06ff3913f682-istio-token" pod "8153e018-7f9a-4645-b102-06ff3913f682" (UID: "8153e018-7f9a-4645-b102-06ff3913f682"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:15.775687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.760518 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8153e018-7f9a-4645-b102-06ff3913f682-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:15.817692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.797739 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czj55\" (UniqueName: \"kubernetes.io/projected/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-kube-api-access-czj55\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:15.817692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.797780 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:15.821690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.821281 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:33:15.821690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.821349 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" podUID="bbd92cdc-e2bd-4f72-8c63-1221069ca3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:15.850761 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.844120 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-envoy" pod "1731953b-1850-40d1-9ba4-cb08dc4ebc5d" (UID: "1731953b-1850-40d1-9ba4-cb08dc4ebc5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:15.850761 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.849761 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": dial tcp 10.132.0.51:8000: connect: connection refused" Nov 03 20:33:15.859217 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.858795 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": dial tcp 10.132.0.76:8000: connect: connection refused" Nov 03 20:33:15.943734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:15.923603 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1731953b-1850-40d1-9ba4-cb08dc4ebc5d-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:16.249690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:16.227836 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": dial tcp 10.132.0.78:8000: connect: connection refused" Nov 03 20:33:16.594742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:16.575434 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": dial tcp 10.132.0.94:8000: connect: connection refused" Nov 03 20:33:16.644687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:16.640055 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": dial tcp 10.132.0.48:8000: connect: connection refused" Nov 03 20:33:16.644687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:16.640681 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" podUID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": dial tcp 10.132.0.49:8000: connect: connection refused" Nov 03 20:33:16.644687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:16.641049 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": dial tcp 10.132.0.77:8000: connect: connection refused" Nov 03 20:33:16.736930 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:16.735074 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": dial tcp 10.132.0.51:8000: connect: connection refused" Nov 03 20:33:17.122699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:17.122350 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": dial tcp 10.132.0.78:8000: connect: connection refused" Nov 03 20:33:17.208787 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:17.188218 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": dial tcp 10.132.0.76:8000: connect: connection refused" Nov 03 20:33:17.579760 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:17.577344 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": dial tcp 10.132.0.94:8000: connect: connection refused" Nov 03 20:33:17.695927 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:17.693341 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": dial tcp 10.132.0.48:8000: connect: connection refused" Nov 03 20:33:17.955296 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:17.955250 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": dial tcp 10.132.0.76:8000: connect: connection refused" Nov 03 20:33:18.180537 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:17.956076 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": dial tcp 10.132.0.77:8000: connect: connection refused" Nov 03 20:33:18.180537 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:17.956135 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" podUID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": dial tcp 10.132.0.49:8000: connect: connection refused" Nov 03 20:33:18.180537 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:18.124745 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": dial tcp 10.132.0.78:8000: connect: connection refused" Nov 03 20:33:18.255693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:18.233060 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused" start-of-body= Nov 03 20:33:18.317854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:18.313745 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" podUID="6752fea6-4adc-47e6-858c-101dcb6d6a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused" Nov 03 20:33:18.600688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:18.544059 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": dial tcp 10.132.0.94:8000: connect: connection refused" Nov 03 20:33:18.761820 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:18.756937 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 20:33:18.961689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:18.961467 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": dial tcp 10.132.0.76:8000: connect: connection refused" Nov 03 20:33:19.153688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:19.152834 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": dial tcp 10.132.0.78:8000: connect: connection refused" Nov 03 20:33:19.258699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:19.255795 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused" start-of-body= Nov 03 20:33:19.258699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:19.255863 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" podUID="6752fea6-4adc-47e6-858c-101dcb6d6a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused" Nov 03 20:33:19.303688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:19.298882 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body= Nov 03 20:33:19.303688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:19.298941 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" podUID="4b43d3ba-a346-45d6-8077-8a52b42de081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" Nov 03 20:33:19.582857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:19.581104 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body= Nov 03 20:33:19.582857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:19.581184 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" podUID="41778c8b-1334-4a0a-afbc-2d377d514e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" Nov 03 20:33:19.925707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:19.882257 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:19.925707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:19.882574 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" podUID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Nov 03 20:33:19.925707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:19.882792 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:33:19.925707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:19.902461 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:20.237721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:20.236176 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body= Nov 03 20:33:20.237721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:20.236351 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" podUID="4b3627ae-66c6-44a4-aa60-480bd54008f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" Nov 03 20:33:20.237721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:20.237094 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body= Nov 03 20:33:20.237721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:20.237135 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" podUID="9bb4f711-0bcc-46f4-86b9-795c66613832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" Nov 03 20:33:20.237721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:20.237236 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused" start-of-body= Nov 03 20:33:20.237721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:20.237259 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" podUID="6752fea6-4adc-47e6-858c-101dcb6d6a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused" Nov 03 20:33:20.398766 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:20.398282 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body= Nov 03 20:33:20.398766 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:20.398343 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" podUID="4b43d3ba-a346-45d6-8077-8a52b42de081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" Nov 03 20:33:20.538765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:20.174626 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" podStartSLOduration=77.588148274 podStartE2EDuration="1m18.174602477s" totalImagesPullingTime="586.454203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:09.518305126 +0000 UTC m=+3383.776102383" watchObservedRunningTime="2026-11-03 20:33:20.174602477 +0000 UTC m=+3394.432399734" Nov 03 20:33:20.614879 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:20.609760 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:20.755642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:20.698904 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body= Nov 03 20:33:20.755642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:20.698972 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" podUID="41778c8b-1334-4a0a-afbc-2d377d514e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" Nov 03 20:33:20.912687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:20.901261 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:20.912934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:20.912798 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:21.205710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.201038 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:33:21.340945 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.340392 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body= Nov 03 20:33:21.340945 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.340467 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" podUID="4b43d3ba-a346-45d6-8077-8a52b42de081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" Nov 03 20:33:21.363696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.351776 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 20:33:21.363696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.352075 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" podUID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Nov 03 20:33:21.363696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.352294 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:21.431711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.426885 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-78xj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" start-of-body= Nov 03 20:33:21.431711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.426980 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" podUID="09d624de-215a-40be-9b38-0ba5a08743ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" Nov 03 20:33:21.431711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.427335 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused" start-of-body= Nov 03 20:33:21.431711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.427387 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" podUID="6752fea6-4adc-47e6-858c-101dcb6d6a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.88:15021/healthz/ready\": dial tcp 10.132.1.88:15021: connect: connection refused" Nov 03 20:33:21.431711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.427905 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body= Nov 03 20:33:21.431711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.427944 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" podUID="4b3627ae-66c6-44a4-aa60-480bd54008f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" Nov 03 20:33:21.431711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.428165 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body= Nov 03 20:33:21.431711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.428203 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" podUID="9bb4f711-0bcc-46f4-86b9-795c66613832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" Nov 03 20:33:21.517112 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.479997 5059 patch_prober.go:29] interesting pod/router-default-f7b87d695-cwlfq container/router namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]backend-proxy-http ok Nov 03 20:33:21.517112 ip-10-0-132-83 kubenswrapper[5059]: [+]has-synced ok Nov 03 20:33:21.517112 ip-10-0-132-83 kubenswrapper[5059]: [-]process-running failed: reason withheld Nov 03 20:33:21.517112 ip-10-0-132-83 kubenswrapper[5059]: healthz check failed Nov 03 20:33:21.517112 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.480053 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-f7b87d695-cwlfq" podUID="603868b0-8628-4227-8471-1894e2e1de50" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Nov 03 20:33:21.641737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.614693 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Nov 03 20:33:21.659516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.658751 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body= Nov 03 20:33:21.659516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.658816 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" podUID="41778c8b-1334-4a0a-afbc-2d377d514e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" Nov 03 20:33:21.659516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.659193 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-78xj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" start-of-body= Nov 03 20:33:21.659516 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.659236 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" podUID="09d624de-215a-40be-9b38-0ba5a08743ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" Nov 03 20:33:21.906003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.901532 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:21.929706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.926899 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded" Nov 03 20:33:21.933242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.933197 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6c57db574-fmm6v" podUID="3265bd58-d654-467c-9b2a-14cedc27b754" containerName="registry" containerID="cri-o://c00733af03a11c26991be300ebced8bc887b057aa45430c49b4b704481238514" gracePeriod=30 Nov 03 20:33:21.996688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:21.991262 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p" Nov 03 20:33:22.053873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.000247 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq" Nov 03 20:33:22.053873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.001474 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps" Nov 03 20:33:22.053873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.004023 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8" Nov 03 20:33:22.053873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.012779 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2" Nov 03 20:33:22.053873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.020744 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj" Nov 03 20:33:22.053873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.034707 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r" Nov 03 20:33:22.053873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.035406 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7" Nov 03 20:33:22.053873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.036701 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx" Nov 03 20:33:22.053873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.038705 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl" Nov 03 20:33:22.053873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.041071 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44" Nov 03 20:33:22.202743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.201747 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:22.238947 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.230813 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body= Nov 03 20:33:22.238947 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.230871 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" podUID="9bb4f711-0bcc-46f4-86b9-795c66613832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" Nov 03 20:33:22.239776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.239134 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body= Nov 03 20:33:22.239776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.239181 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" podUID="4b43d3ba-a346-45d6-8077-8a52b42de081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" Nov 03 20:33:22.379706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.373021 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:33:22.379706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.373315 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 20:33:22.379706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.373507 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" podUID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Nov 03 20:33:22.508741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.507952 5059 scope.go:122] "RemoveContainer" containerID="8cc39ee98f0eb5245e5d5488f9855a4beaa057aa6bbd7dc9d1116291e5c04e35" Nov 03 20:33:22.647637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.618797 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:22.647637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.621202 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-78xj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" start-of-body= Nov 03 20:33:22.647637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.621247 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" podUID="09d624de-215a-40be-9b38-0ba5a08743ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" Nov 03 20:33:22.647637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.621893 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body= Nov 03 20:33:22.647637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.621932 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" podUID="41778c8b-1334-4a0a-afbc-2d377d514e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" Nov 03 20:33:22.953257 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.952199 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:22.953257 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:22.952608 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:23.171690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:23.152927 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:33:23.171690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:23.152996 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" podUID="c2d80995-5f1c-46db-8d71-a130cb92e279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:23.265688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:23.254713 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body= Nov 03 20:33:23.265688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:23.254779 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" podUID="4b43d3ba-a346-45d6-8077-8a52b42de081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" Nov 03 20:33:23.265688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:23.255185 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:33:23.265688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:23.255616 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body= Nov 03 20:33:23.265688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:23.255649 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" podUID="9bb4f711-0bcc-46f4-86b9-795c66613832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" Nov 03 20:33:23.384639 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:23.383746 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" podUID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:23.384639 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:23.384058 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:23.384639 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:23.384250 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:23.417366 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:23.417162 5059 generic.go:356] "Generic (PLEG): container finished" podID="3265bd58-d654-467c-9b2a-14cedc27b754" containerID="c00733af03a11c26991be300ebced8bc887b057aa45430c49b4b704481238514" exitCode=0 Nov 03 20:33:23.423892 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:23.419655 5059 generic.go:356] "Generic (PLEG): container finished" podID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerID="238b88770b80cec7d339089cb9d69cc1f7ef5a1fdcefa3eb790154cc7f3289f1" exitCode=0 Nov 03 20:33:23.433218 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:23.433106 5059 generic.go:356] "Generic (PLEG): container finished" podID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerID="b8d82ca6d1d4fd46ae40cb0b5685d96dc2c2f3703f2b90b33d10896376edb8cc" exitCode=0 Nov 03 20:33:23.549309 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:23.548811 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body= Nov 03 20:33:23.549309 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:23.548878 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" podUID="41778c8b-1334-4a0a-afbc-2d377d514e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" Nov 03 20:33:23.603465 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:23.603008 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-78xj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" start-of-body= Nov 03 20:33:23.603465 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:23.603070 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" podUID="09d624de-215a-40be-9b38-0ba5a08743ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" Nov 03 20:33:23.622039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:23.621739 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:23.723913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:23.723748 5059 generic.go:356] "Generic (PLEG): container finished" podID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerID="1317df3f930488af045a4ea5f2ffd5b2d0b6bac1972a73208461be0692b6e2b2" exitCode=0 Nov 03 20:33:23.917687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:23.913693 5059 generic.go:356] "Generic (PLEG): container finished" podID="f14445d5-52ed-487d-972c-4499966065fa" containerID="44d16c3b122733147748e39e219fb00f390ff6e37a49c47ed3e51ba2960a62c1" exitCode=0 Nov 03 20:33:23.961844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:23.957631 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:23.961844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:23.961176 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded" Nov 03 20:33:24.034765 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:24.025996 5059 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="9.371s" Nov 03 20:33:24.034765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:24.026047 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" Nov 03 20:33:24.034765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:24.026094 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:24.034765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:24.026124 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:24.275767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:24.249917 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body= Nov 03 20:33:24.275767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:24.249996 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" podUID="4b43d3ba-a346-45d6-8077-8a52b42de081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" Nov 03 20:33:24.275767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:24.257770 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:33:24.275767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:24.268274 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Nov 03 20:33:24.275767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:24.268329 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" podUID="cca72a7d-a69d-4e05-ac4d-da20e15a0243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Nov 03 20:33:24.405910 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:24.398968 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:24.439683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:24.439024 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" podUID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:24.439683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:24.439299 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:33:24.562108 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:24.538237 5059 scope.go:122] "RemoveContainer" containerID="2d957762bcd8fed6693d101c9e9584a4ee5948a1b3f069a82b1b80826caa7347" Nov 03 20:33:24.640706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:24.624844 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:24.816959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:24.705675 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body= Nov 03 20:33:24.816959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:24.705741 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" podUID="41778c8b-1334-4a0a-afbc-2d377d514e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" Nov 03 20:33:24.898995 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:24.898802 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-5sk9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused" start-of-body= Nov 03 20:33:24.898995 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:24.898863 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" podUID="81fa410e-25d7-4800-a22b-9dea6ccca3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused" Nov 03 20:33:24.927175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:24.914952 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-78xj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" start-of-body= Nov 03 20:33:24.927175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:24.915020 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" podUID="09d624de-215a-40be-9b38-0ba5a08743ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" Nov 03 20:33:25.297642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:25.263833 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:25.340695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:25.339252 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Nov 03 20:33:25.340695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:25.339317 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" podUID="cca72a7d-a69d-4e05-ac4d-da20e15a0243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Nov 03 20:33:25.345780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:25.345739 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:25.405266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:25.404917 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 03 20:33:25.405917 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:25.405541 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:33:25.425267 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:25.424815 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:33:25.425267 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:25.424891 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" podUID="9bb4f711-0bcc-46f4-86b9-795c66613832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:25.473743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:25.473440 5059 generic.go:356] "Generic (PLEG): container finished" podID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerID="1c69adcd1fc6db5a42ddeb00863f24a144f0cf528222cad083ac5a885e389943" exitCode=0 Nov 03 20:33:25.478845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:25.477203 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:25.543249 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:25.542963 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" podUID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Nov 03 20:33:25.640812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:25.621132 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body= Nov 03 20:33:25.640812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:25.621190 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" podUID="41778c8b-1334-4a0a-afbc-2d377d514e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" Nov 03 20:33:25.640812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:25.625258 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:25.684676 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:25.684269 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-78xj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" start-of-body= Nov 03 20:33:25.684676 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:25.684330 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" podUID="09d624de-215a-40be-9b38-0ba5a08743ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" Nov 03 20:33:25.696686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:25.693892 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-5sk9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused" start-of-body= Nov 03 20:33:25.696686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:25.693958 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" podUID="81fa410e-25d7-4800-a22b-9dea6ccca3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused" Nov 03 20:33:25.708793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:25.707977 5059 generic.go:356] "Generic (PLEG): container finished" podID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerID="5b05929b67ea95a6b627445e0a3fa63ef0a0d7ef033293a27edabc525d4bd1cd" exitCode=0 Nov 03 20:33:25.944687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:25.944496 5059 generic.go:356] "Generic (PLEG): container finished" podID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" containerID="90b3ff7ea089036c1955493d6ad07b652eb65d8c482b1abd7f0dc8c58bea2751" exitCode=0 Nov 03 20:33:26.153073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.150785 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w" Nov 03 20:33:26.153073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.152058 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt" Nov 03 20:33:26.153073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.152622 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn" Nov 03 20:33:26.223840 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.221891 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Nov 03 20:33:26.223840 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.221954 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" podUID="cca72a7d-a69d-4e05-ac4d-da20e15a0243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Nov 03 20:33:26.241567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.241533 5059 scope.go:122] "RemoveContainer" containerID="4d94405c1bd7e9ab077c0ca4e40445d24510310ab3bd698c9d3fc466bbff887d" Nov 03 20:33:26.321495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.306092 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:26.342919 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:26.342760 5059 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.316s" Nov 03 20:33:26.342919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.342838 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 20:33:26.343223 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.343155 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:26.343290 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.343259 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:33:26.353745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.353716 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded" Nov 03 20:33:26.411250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.411206 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:26.411955 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.411881 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:26.511813 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.511759 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:26.516122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.516093 5059 scope.go:122] "RemoveContainer" containerID="91f3fb2bb610fbeea4466047557a60f91aee6d1477483a4dc37e84c19840f464" Nov 03 20:33:26.546755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.546716 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" podUID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:26.570685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.558477 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:26.584883 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.584860 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:26.585115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.585098 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:26.585316 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.585302 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 20:33:26.585435 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.585416 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" event={"ID":"cca72a7d-a69d-4e05-ac4d-da20e15a0243","Type":"ContainerStarted","Data":"f004ee6b7be2108660e25a6868948ec67f63eae85339a25ac11570bd56bd1c93"} Nov 03 20:33:26.585586 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.585571 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 20:33:26.585722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.585705 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47" event={"ID":"6c29d53f-b738-49ec-9b15-1c88e5950008","Type":"ContainerStarted","Data":"c0d6aef855ff12a4c072c15a072a0bfa3716b31ab4083ee5af5263844754bb97"} Nov 03 20:33:26.585838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.585824 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9" event={"ID":"fd9d29fb-d764-4eac-a744-dbdecde12ff0","Type":"ContainerStarted","Data":"69579c0e40da217f8ae4fdef8b052b316aff66db158b9fbdeef2495a9ac79ec8"} Nov 03 20:33:26.585998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.585984 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 20:33:26.586101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.586088 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" event={"ID":"a2c58648-4022-4bfb-be9f-170a6222dfd8","Type":"ContainerStarted","Data":"cb8f0b2617626011a18289e66bb16c024dc66fe6709c48401aea2b0c398d900c"} Nov 03 20:33:26.586224 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.586210 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:26.586324 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.586313 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl"] Nov 03 20:33:26.590684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.587490 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1731953b-1850-40d1-9ba4-cb08dc4ebc5d" containerName="istio-proxy" Nov 03 20:33:26.591981 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.591955 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1731953b-1850-40d1-9ba4-cb08dc4ebc5d" containerName="istio-proxy" Nov 03 20:33:26.592084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.591989 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c92b1be0-0890-4db6-ac75-968b6e5a3434" containerName="payload-processing" Nov 03 20:33:26.592084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.592000 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c92b1be0-0890-4db6-ac75-968b6e5a3434" containerName="payload-processing" Nov 03 20:33:26.592084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.592035 5059 cpu_manager.go:414] "containerMap: removing container" podUID="8153e018-7f9a-4645-b102-06ff3913f682" containerName="istio-proxy" Nov 03 20:33:26.592084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.592044 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8153e018-7f9a-4645-b102-06ff3913f682" containerName="istio-proxy" Nov 03 20:33:26.592274 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.592249 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" containerName="istio-proxy" Nov 03 20:33:26.592321 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.592278 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c92b1be0-0890-4db6-ac75-968b6e5a3434" containerName="payload-processing" Nov 03 20:33:26.592321 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.592289 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1731953b-1850-40d1-9ba4-cb08dc4ebc5d" containerName="istio-proxy" Nov 03 20:33:26.592321 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.592301 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8153e018-7f9a-4645-b102-06ff3913f682" containerName="istio-proxy" Nov 03 20:33:26.599000 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.598979 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 20:33:26.599092 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.599016 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q" event={"ID":"8e9aefe5-8450-4eb9-8d8b-df89be81ebb3","Type":"ContainerStarted","Data":"6c0c462e53b0f3e601530d1f63a4aa75eb98bbedf81934f22d8ce6d5eb1793b7"} Nov 03 20:33:26.600138 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.600102 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t" Nov 03 20:33:26.600241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.600171 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl"] Nov 03 20:33:26.600241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.600208 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:26.600352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.600252 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" event={"ID":"81fa410e-25d7-4800-a22b-9dea6ccca3ea","Type":"ContainerStarted","Data":"51c703129ce1a1ea1f456f57a42dec573640dddd718987494f89381adc463ee3"} Nov 03 20:33:26.600352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.600280 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4" event={"ID":"dc10f92e-e902-48c4-9cd5-4f768c4611be","Type":"ContainerDied","Data":"4d94405c1bd7e9ab077c0ca4e40445d24510310ab3bd698c9d3fc466bbff887d"} Nov 03 20:33:26.600651 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.600603 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcrh4" event={"ID":"f44d81a0-55a2-4021-b34f-3400deb30c82","Type":"ContainerStarted","Data":"a49bfd925f72a4267b619ce6573bfcac727c17dc9bc0d14f96986ea565df743f"} Nov 03 20:33:26.601983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.601959 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 20:33:26.602074 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.602004 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" Nov 03 20:33:26.602130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.602076 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9" Nov 03 20:33:26.602187 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.602140 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" Nov 03 20:33:26.602187 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.602169 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp" event={"ID":"a9b40731-f01a-4209-af40-de342324d721","Type":"ContainerStarted","Data":"35fb4ca7ab225127f28dc977cdce50d38860e87ed78e7cbbb6e2dc8c57830d53"} Nov 03 20:33:26.602263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.602202 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk"] Nov 03 20:33:26.603424 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.603400 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1731953b-1850-40d1-9ba4-cb08dc4ebc5d" containerName="istio-proxy" Nov 03 20:33:26.603498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.603427 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1731953b-1850-40d1-9ba4-cb08dc4ebc5d" containerName="istio-proxy" Nov 03 20:33:26.603708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.603689 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1731953b-1850-40d1-9ba4-cb08dc4ebc5d" containerName="istio-proxy" Nov 03 20:33:26.603952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.603929 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-78xj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" start-of-body= Nov 03 20:33:26.604020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.603976 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" podUID="09d624de-215a-40be-9b38-0ba5a08743ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" Nov 03 20:33:26.608279 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.608097 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk"] Nov 03 20:33:26.608279 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.608133 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:26.608279 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.608155 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:26.608279 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.608171 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252"] Nov 03 20:33:26.608495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.608409 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:26.617768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.615860 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9" Nov 03 20:33:26.617768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.615897 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:26.617768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.615923 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252"] Nov 03 20:33:26.617768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.615986 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:26.617768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.616009 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" event={"ID":"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08","Type":"ContainerStarted","Data":"1f259cdff7b8ebb19f3ed7d0557be47aa2ea41bd1b4a0f614d46d58523416d81"} Nov 03 20:33:26.617768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.616037 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj"] Nov 03 20:33:26.620900 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.620874 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:26.622097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.621219 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj"] Nov 03 20:33:26.622097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.621260 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst" Nov 03 20:33:26.622097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.621286 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:26.622097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.621312 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst" event={"ID":"977cec40-92df-410a-a3b6-68520b98aaf2","Type":"ContainerStarted","Data":"22d805127f7df6029c93afae2159725cef5c3fda386692204c7376c78cf1f788"} Nov 03 20:33:26.622097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.621366 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst" Nov 03 20:33:26.622097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.621389 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst" event={"ID":"977cec40-92df-410a-a3b6-68520b98aaf2","Type":"ContainerStarted","Data":"0e8dd4143aa1ca3e3e68053f35530077b364020b2c76a67ec8b8b8920aa8be5a"} Nov 03 20:33:26.622097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.621456 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:26.622097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.621515 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:26.622097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.621538 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:26.622097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.621560 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" event={"ID":"6752fea6-4adc-47e6-858c-101dcb6d6a84","Type":"ContainerStarted","Data":"aa92feed8292653704478e650971480d864ee8d63ab7879692555ed5a5ac1b97"} Nov 03 20:33:26.622097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.621583 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" event={"ID":"6752fea6-4adc-47e6-858c-101dcb6d6a84","Type":"ContainerStarted","Data":"19ab4abf491c7bd264adde0c19aaf80256930441969660b13f973717c3f387f6"} Nov 03 20:33:26.622097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.621612 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" event={"ID":"60b2ec36-7e99-46e0-9671-1c500c402e45","Type":"ContainerStarted","Data":"35786554f09b31f2d61a5260f85d6c7a85abe0ba3cede27c9355d295fcf1bcc6"} Nov 03 20:33:26.622097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.621634 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" event={"ID":"4b43d3ba-a346-45d6-8077-8a52b42de081","Type":"ContainerStarted","Data":"1b5c2d5036db0940f7c2add6187517d59bd2c5c0560de10c0ec0d49cbf6bb032"} Nov 03 20:33:26.622097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.621677 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" Nov 03 20:33:26.622097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.621704 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c" Nov 03 20:33:26.622097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.621732 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" event={"ID":"41778c8b-1334-4a0a-afbc-2d377d514e8d","Type":"ContainerStarted","Data":"14ceb520f6d4105401a7b46c0723e09fd8ad248fcbc4030ad7aa7c5f0ef82185"} Nov 03 20:33:26.622097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.621754 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" event={"ID":"4b3627ae-66c6-44a4-aa60-480bd54008f0","Type":"ContainerStarted","Data":"0a4e5fbcc2f971059f484ad26dce5294c486b94025b6ad0f0353b5190dfe9a11"} Nov 03 20:33:26.622097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.621787 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x"] Nov 03 20:33:26.623102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.622657 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:26.632877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.632826 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:26.635633 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.634545 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-81a5ae-openshift-default-dockercfg-pvl2f\"" Nov 03 20:33:26.635633 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.635556 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0e4ce7-openshift-default-dockercfg-nppzh\"" Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.642859 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x"] Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.642910 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c" Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.642938 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" event={"ID":"e5585d8d-d156-418c-ae87-012781bc0e8c","Type":"ContainerStarted","Data":"1de9be30a5b178d039d2f261dcb78f436d56d55ddd9f813a89d82ed40779e573"} Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.642969 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c"] Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.642997 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" event={"ID":"426d1445-9e48-4516-a920-fca04d918f22","Type":"ContainerStarted","Data":"449a5120f0d2ba152b610add4e451094682460cd3022ae0152f8fdb0df868c9a"} Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643023 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643046 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c" event={"ID":"ece17ed8-afcd-41ab-85e0-adb35875ecd6","Type":"ContainerStarted","Data":"1fb871bbe1a064bd6c076f10538e03887172e9b7c42b6bc26be22afe9e84cc0b"} Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643074 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2"] Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643107 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" event={"ID":"9bb4f711-0bcc-46f4-86b9-795c66613832","Type":"ContainerStarted","Data":"15e2cc84c3e225d2ad2495aa2c827011b729fdaac93a952c40e3cc3bde3f078d"} Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643134 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst"] Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643165 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn" Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643187 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m" event={"ID":"de4efb53-8b59-49f1-894f-ef366f00732a","Type":"ContainerStarted","Data":"10af7362d56af3e4af6be86d228fac82334d40308666b2e0622589438ccc3e50"} Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643212 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" event={"ID":"09d624de-215a-40be-9b38-0ba5a08743ef","Type":"ContainerStarted","Data":"e333907c379f464c17e6a1b2bdafaba5c2ec787715543737b32901885a0ca269"} Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643240 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcrh4"] Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643268 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp"] Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643326 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb" Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643378 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643397 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn" event={"ID":"c9ead1ac-0ace-4761-9fde-b7f97a9dfa31","Type":"ContainerStarted","Data":"2944eaaae47115096c908bfd9d08b4061d4dafe6b809a6b937a0b15f266b2edf"} Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643421 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb" event={"ID":"26576b40-fd9c-4298-9129-5b341d2c0b3d","Type":"ContainerStarted","Data":"99a0d9d2e14726b927ef1738be56c6ac25f24a2999b1b04a05311141ddb6c181"} Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643446 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q"] Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643472 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm" event={"ID":"7dc48f99-393e-4af4-b942-1ed63909b588","Type":"ContainerStarted","Data":"9cda458c63e04efc7e5496fc503f63daafb5eed92eca9ae85fff848227a806aa"} Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643494 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm" event={"ID":"7dc48f99-393e-4af4-b942-1ed63909b588","Type":"ContainerStarted","Data":"5b5c31d092c9ba53079cea073aa8897c3126023324d7fbd6bbe70935eae34a3b"} Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643546 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb" Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643569 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w"] Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643596 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47"] Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643622 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q"] Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643647 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn"] Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643689 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" event={"ID":"77396fff-1778-4f09-9e81-05e715fc015e","Type":"ContainerStarted","Data":"a768bd48720a7e153a5bf47fdd7017557daff7e1edb6146eddc2b95d1e9dc6e9"} Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643717 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2"] Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643747 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643792 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn" Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643818 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm" Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643847 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5"] Nov 03 20:33:26.645258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.643875 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws"] Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.649104 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.651720 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws"] Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.651779 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m"] Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.651719 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-5sk9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused" start-of-body= Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.651814 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" podUID="81fa410e-25d7-4800-a22b-9dea6ccca3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused" Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.651834 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.651858 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5"] Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.651878 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lz2v2" Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.651901 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.651924 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.651949 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn"] Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.651975 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.651997 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp" event={"ID":"a9b40731-f01a-4209-af40-de342324d721","Type":"ContainerStarted","Data":"19bd246d98f0088e731c3a855629700c012e09eaf4097b51292b87ae09515fee"} Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.652022 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c57db574-fmm6v" event={"ID":"3265bd58-d654-467c-9b2a-14cedc27b754","Type":"ContainerDied","Data":"c00733af03a11c26991be300ebced8bc887b057aa45430c49b4b704481238514"} Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.652044 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" event={"ID":"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02","Type":"ContainerDied","Data":"238b88770b80cec7d339089cb9d69cc1f7ef5a1fdcefa3eb790154cc7f3289f1"} Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.652064 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.651781 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.652092 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz"] Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.652113 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" event={"ID":"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02","Type":"ContainerDied","Data":"04f4c299d03d7084481887555642aa0b3229a752661fb38bc15ee6075eb3e8c8"} Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.652132 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04f4c299d03d7084481887555642aa0b3229a752661fb38bc15ee6075eb3e8c8" Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.652142 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" event={"ID":"1a57d851-12b3-4535-b12c-b7d2c579d0fb","Type":"ContainerDied","Data":"b8d82ca6d1d4fd46ae40cb0b5685d96dc2c2f3703f2b90b33d10896376edb8cc"} Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.652158 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" event={"ID":"1a57d851-12b3-4535-b12c-b7d2c579d0fb","Type":"ContainerDied","Data":"0ecf7d084b4aa7d821db382650cbb5f46133e201ecbfd77d88b048f1a09a26c3"} Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.652172 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ecf7d084b4aa7d821db382650cbb5f46133e201ecbfd77d88b048f1a09a26c3" Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.652180 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" event={"ID":"41778c8b-1334-4a0a-afbc-2d377d514e8d","Type":"ContainerStarted","Data":"6c9c6f5fc6b9469ca4ba0efeed4e557ba33c9e08b55f4b2d7e630a958c2f8963"} Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.652214 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.652242 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.652258 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-gn7v5"] Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.652280 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp" Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.652295 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c" event={"ID":"ece17ed8-afcd-41ab-85e0-adb35875ecd6","Type":"ContainerStarted","Data":"2f439e80b9652e884e5ddcb17983cc536b4f3f4964293cac6b93378706a542b3"} Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.652310 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj"] Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.652323 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-gwvkj"] Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.652336 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" event={"ID":"09d624de-215a-40be-9b38-0ba5a08743ef","Type":"ContainerStarted","Data":"2b4e52afd7a499fdc801448c104ffdb86585ae2b2371d359f5cc52f6255a59a8"} Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.652363 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm" Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.652381 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn" event={"ID":"c9ead1ac-0ace-4761-9fde-b7f97a9dfa31","Type":"ContainerStarted","Data":"9c318b2e852863d226308b67f4a9c795472db2505655e98f81679d2b9182e1e4"} Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.652395 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" event={"ID":"cde430c2-2518-49a0-9851-42356d3f2fe9","Type":"ContainerDied","Data":"1317df3f930488af045a4ea5f2ffd5b2d0b6bac1972a73208461be0692b6e2b2"} Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.652430 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:33:26.652785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.652445 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-v5xhc"] Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.657164 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-v5xhc"] Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.657236 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.657266 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" event={"ID":"cde430c2-2518-49a0-9851-42356d3f2fe9","Type":"ContainerDied","Data":"4c35e8e3486d38738c9026d17e0fc6245f178ee71991f27f201b46ee79442abe"} Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.657295 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c35e8e3486d38738c9026d17e0fc6245f178ee71991f27f201b46ee79442abe" Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.657334 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.657361 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7"] Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.657409 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp" Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.657434 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" event={"ID":"cca72a7d-a69d-4e05-ac4d-da20e15a0243","Type":"ContainerStarted","Data":"f7ee1b1fba03037ed075aa90d678d3be16c71ebef4f5221a532f1d7a2e72e552"} Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.657459 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" event={"ID":"f14445d5-52ed-487d-972c-4499966065fa","Type":"ContainerDied","Data":"44d16c3b122733147748e39e219fb00f390ff6e37a49c47ed3e51ba2960a62c1"} Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.657479 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g"] Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.657497 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" event={"ID":"f14445d5-52ed-487d-972c-4499966065fa","Type":"ContainerDied","Data":"a2b18e67c031d89367ba22383c2672c4ea5db2ef498701a9b608b8cf3af93bb9"} Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.657512 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2b18e67c031d89367ba22383c2672c4ea5db2ef498701a9b608b8cf3af93bb9" Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.657524 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" event={"ID":"81fa410e-25d7-4800-a22b-9dea6ccca3ea","Type":"ContainerStarted","Data":"514688f85488dcbdb4fdedf393ca4f059f4c0876a90f03bed4d0f402479011c4"} Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.657550 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-f2v7g"] Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.657572 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" event={"ID":"541bc63e-66f1-42ca-a946-aedc34beb62f","Type":"ContainerDied","Data":"1c69adcd1fc6db5a42ddeb00863f24a144f0cf528222cad083ac5a885e389943"} Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.657637 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs"] Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.657708 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" event={"ID":"541bc63e-66f1-42ca-a946-aedc34beb62f","Type":"ContainerDied","Data":"1c219679b70309c8c555a19601b051b1cb61e2cff65c58137d12b1f924d60147"} Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.657726 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c219679b70309c8c555a19601b051b1cb61e2cff65c58137d12b1f924d60147" Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.657739 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" event={"ID":"20690fd2-7022-4657-8c8c-bd738493ffc0","Type":"ContainerDied","Data":"5b05929b67ea95a6b627445e0a3fa63ef0a0d7ef033293a27edabc525d4bd1cd"} Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.657806 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-v5xhc" Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.657911 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" event={"ID":"20690fd2-7022-4657-8c8c-bd738493ffc0","Type":"ContainerDied","Data":"74e01caa8834dc2b10b8f8ebf75574792274d55c381887f50b657f8472821284"} Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.657936 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74e01caa8834dc2b10b8f8ebf75574792274d55c381887f50b657f8472821284" Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.657949 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcrh4" event={"ID":"f44d81a0-55a2-4021-b34f-3400deb30c82","Type":"ContainerStarted","Data":"f246ec3d53e94ed7532f7ce0cd597f3a203daa47e283a2e2ac4cad53d41fdef2"} Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.657977 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" event={"ID":"e7cf1267-7d06-459b-9df7-fc7c477d36e1","Type":"ContainerDied","Data":"90b3ff7ea089036c1955493d6ad07b652eb65d8c482b1abd7f0dc8c58bea2751"} Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.658008 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" event={"ID":"e7cf1267-7d06-459b-9df7-fc7c477d36e1","Type":"ContainerDied","Data":"6ce4ab0a96f501ffbbc5268b102246760445565e5b43e450a568003ae42adf0c"} Nov 03 20:33:26.663677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.658031 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ce4ab0a96f501ffbbc5268b102246760445565e5b43e450a568003ae42adf0c" Nov 03 20:33:26.724293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.722811 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-data\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:26.724293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.722901 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:26.727389 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.726896 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-dq8cl\"" Nov 03 20:33:26.727389 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.727090 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8"] Nov 03 20:33:26.731812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.731466 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-credential-socket\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:26.731812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.731545 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:26.731812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.731578 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:26.731812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.731618 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-token\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:26.731812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.731698 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-kpmj8"] Nov 03 20:33:26.731812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.731714 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:26.733456 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.732260 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:26.733456 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.732299 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:26.733456 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.732482 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bff4885-a4dc-4b17-b596-e745e86fbccd-istiod-ca-cert\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:26.733456 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.732564 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-workload-socket\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:26.733456 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.732593 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjh84\" (UniqueName: \"kubernetes.io/projected/9bff4885-a4dc-4b17-b596-e745e86fbccd-kube-api-access-vjh84\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:26.733456 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.732653 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h9kv\" (UniqueName: \"kubernetes.io/projected/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-kube-api-access-4h9kv\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:26.733456 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.732801 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-podinfo\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:26.733456 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.732836 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-envoy\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:26.733456 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.732893 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:26.733456 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.733310 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:26.733456 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.733399 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:26.733456 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.733428 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-workload-certs\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:26.734086 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.733486 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:26.734086 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.733529 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:26.734086 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.733580 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:26.734086 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.733997 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:26.734630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.734367 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:26.734630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.734437 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:26.734630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.734553 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569b2103-147f-4ea1-9b55-0e859b1c5e52-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:26.734630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.734601 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hzr5\" (UniqueName: \"kubernetes.io/projected/569b2103-147f-4ea1-9b55-0e859b1c5e52-kube-api-access-5hzr5\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:26.837693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.834306 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d9f95b69-x7n2b"] Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.918289 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.918356 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.918392 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.918442 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68bc094a-84e2-41c2-9fb5-081682413f27-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.918482 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.918522 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.918564 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.918605 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.918642 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.918697 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.918791 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569b2103-147f-4ea1-9b55-0e859b1c5e52-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.918820 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.918860 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hzr5\" (UniqueName: \"kubernetes.io/projected/569b2103-147f-4ea1-9b55-0e859b1c5e52-kube-api-access-5hzr5\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.918904 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f438514-df13-4871-9c2a-ef8e56a570c9-istiod-ca-cert\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.918952 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-token\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.918985 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68bc094a-84e2-41c2-9fb5-081682413f27-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.919017 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.919054 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-data\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.919094 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.919135 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.919200 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.919242 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.919295 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-workload-socket\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.919334 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-credential-socket\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.919371 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.919403 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.919434 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.919469 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-token\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.919535 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.919591 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:26.920877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.919623 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:26.923963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.922719 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68bc094a-84e2-41c2-9fb5-081682413f27-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:26.923963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.922788 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-podinfo\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:26.923963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.922836 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bff4885-a4dc-4b17-b596-e745e86fbccd-istiod-ca-cert\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:26.923963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.922884 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-workload-socket\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:26.923963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.922921 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:26.923963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.922953 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjh84\" (UniqueName: \"kubernetes.io/projected/9bff4885-a4dc-4b17-b596-e745e86fbccd-kube-api-access-vjh84\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:26.923963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.922996 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:26.923963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.923058 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-data\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:26.923963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.923106 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2j7f\" (UniqueName: \"kubernetes.io/projected/7f438514-df13-4871-9c2a-ef8e56a570c9-kube-api-access-t2j7f\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:26.923963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.923153 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h9kv\" (UniqueName: \"kubernetes.io/projected/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-kube-api-access-4h9kv\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:26.923963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.923233 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-podinfo\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:26.923963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.923280 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-envoy\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:26.923963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.923324 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:26.923963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.923427 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-workload-certs\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:26.923963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.923470 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:26.923963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.923543 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxwwd\" (UniqueName: \"kubernetes.io/projected/d50891bc-16c2-497d-8e00-b79ef492ab6e-kube-api-access-dxwwd\") pod \"authorino-operator-7b7bb4bc77-v5xhc\" (UID: \"d50891bc-16c2-497d-8e00-b79ef492ab6e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-v5xhc" Nov 03 20:33:26.923963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.923615 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:26.923963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.923651 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-envoy\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:26.923963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.923704 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-workload-certs\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:26.923963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.923742 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-credential-socket\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:26.923963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.923809 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:26.923963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.923860 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj2v6\" (UniqueName: \"kubernetes.io/projected/19ab8a5f-fa5f-4548-bf5f-0465281d8677-kube-api-access-xj2v6\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:26.927256 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.925149 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:26.927256 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.925232 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwfgt\" (UniqueName: \"kubernetes.io/projected/68bc094a-84e2-41c2-9fb5-081682413f27-kube-api-access-wwfgt\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:26.927256 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.926420 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:26.940080 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.939888 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-x7n2b" Nov 03 20:33:26.949566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.949519 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 20:33:26.950230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.950204 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 20:33:26.950914 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.950887 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-data\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:26.951945 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.951920 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:26.952270 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.952248 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-credential-socket\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:26.952588 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.952538 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:26.952929 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.952887 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:26.953822 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.953797 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:26.954124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.954100 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:27.011704 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.954468 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bff4885-a4dc-4b17-b596-e745e86fbccd-istiod-ca-cert\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:27.011704 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.954798 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-workload-socket\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:27.011704 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:26.964405 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569b2103-147f-4ea1-9b55-0e859b1c5e52-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:27.015015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.014984 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-k5vf6\"" Nov 03 20:33:27.015144 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.015092 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 20:33:27.024743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.022049 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:33:27.024743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.022980 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:27.024743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.023477 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 20:33:27.027263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.027237 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff"] Nov 03 20:33:27.038728 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.036466 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-token\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:27.038728 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.036892 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:27.039736 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.039005 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-workload-certs\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:27.047689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.046718 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:27.047689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.047160 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:27.047689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.047269 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-rb8ff"] Nov 03 20:33:27.047689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.047461 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:27.047689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.047557 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:27.050772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.049120 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:27.050772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.049197 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:27.050772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.049618 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:27.050772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.049943 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:27.050772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.049995 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:27.050772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.050146 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:27.050772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.050211 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f438514-df13-4871-9c2a-ef8e56a570c9-istiod-ca-cert\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:27.050772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.050271 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-token\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:27.050772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.050308 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68bc094a-84e2-41c2-9fb5-081682413f27-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:27.051751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.051728 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:27.051841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.051794 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:27.051911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.051843 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:27.051968 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.051911 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:27.052022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.051966 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-workload-socket\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:27.052022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.052015 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:27.052154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.052137 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68bc094a-84e2-41c2-9fb5-081682413f27-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:27.052225 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.052188 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-podinfo\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:27.052290 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.052240 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:27.052290 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.052270 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:27.052290 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.052280 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:27.052462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.052326 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-data\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:27.052462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.052402 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2j7f\" (UniqueName: \"kubernetes.io/projected/7f438514-df13-4871-9c2a-ef8e56a570c9-kube-api-access-t2j7f\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:27.052605 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.052587 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-envoy\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:27.052739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.052691 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj2v6\" (UniqueName: \"kubernetes.io/projected/19ab8a5f-fa5f-4548-bf5f-0465281d8677-kube-api-access-xj2v6\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:27.052803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.052748 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwfgt\" (UniqueName: \"kubernetes.io/projected/68bc094a-84e2-41c2-9fb5-081682413f27-kube-api-access-wwfgt\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:27.052868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.052806 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:27.052868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.052843 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:27.052974 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.052922 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68bc094a-84e2-41c2-9fb5-081682413f27-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:27.074166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.073090 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68bc094a-84e2-41c2-9fb5-081682413f27-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:27.212803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.212769 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68bc094a-84e2-41c2-9fb5-081682413f27-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:27.214944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.214911 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:27.215073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.215050 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:27.216567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.215372 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:27.216567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.215691 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-data\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:27.216567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.215790 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-podinfo\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:27.216567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.215965 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h9kv\" (UniqueName: \"kubernetes.io/projected/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-kube-api-access-4h9kv\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:27.217338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.217002 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjh84\" (UniqueName: \"kubernetes.io/projected/9bff4885-a4dc-4b17-b596-e745e86fbccd-kube-api-access-vjh84\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:27.221247 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.221189 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-podinfo\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:27.222620 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.222566 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-token\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:27.222620 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.222592 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hzr5\" (UniqueName: \"kubernetes.io/projected/569b2103-147f-4ea1-9b55-0e859b1c5e52-kube-api-access-5hzr5\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:27.223555 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.223512 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-envoy\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:27.224000 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.223973 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68bc094a-84e2-41c2-9fb5-081682413f27-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:27.228351 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.228172 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-envoy\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:27.231202 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.229949 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f438514-df13-4871-9c2a-ef8e56a570c9-istiod-ca-cert\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:27.231202 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.230216 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:27.231202 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.230844 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:27.231202 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.231178 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9f37244-d91a-480d-baa1-b1d59335804c-metrics-certs\") pod \"jobset-controller-manager-5d9f95b69-x7n2b\" (UID: \"d9f37244-d91a-480d-baa1-b1d59335804c\") " pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-x7n2b" Nov 03 20:33:27.233362 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.231844 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9f37244-d91a-480d-baa1-b1d59335804c-cert\") pod \"jobset-controller-manager-5d9f95b69-x7n2b\" (UID: \"d9f37244-d91a-480d-baa1-b1d59335804c\") " pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-x7n2b" Nov 03 20:33:27.233362 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.232052 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d9f37244-d91a-480d-baa1-b1d59335804c-manager-config\") pod \"jobset-controller-manager-5d9f95b69-x7n2b\" (UID: \"d9f37244-d91a-480d-baa1-b1d59335804c\") " pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-x7n2b" Nov 03 20:33:27.233362 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.232272 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rtqg\" (UniqueName: \"kubernetes.io/projected/d9f37244-d91a-480d-baa1-b1d59335804c-kube-api-access-8rtqg\") pod \"jobset-controller-manager-5d9f95b69-x7n2b\" (UID: \"d9f37244-d91a-480d-baa1-b1d59335804c\") " pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-x7n2b" Nov 03 20:33:27.233362 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.232312 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-workload-certs\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:27.233362 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.232366 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxwwd\" (UniqueName: \"kubernetes.io/projected/d50891bc-16c2-497d-8e00-b79ef492ab6e-kube-api-access-dxwwd\") pod \"authorino-operator-7b7bb4bc77-v5xhc\" (UID: \"d50891bc-16c2-497d-8e00-b79ef492ab6e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-v5xhc" Nov 03 20:33:27.233362 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.232404 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-credential-socket\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:27.233362 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.232853 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-credential-socket\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:27.233362 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.231853 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:27.233362 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.233121 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-workload-certs\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:27.236448 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.234928 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:27.236448 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.235146 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:27.236448 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.235149 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Nov 03 20:33:27.236448 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.235206 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" podUID="cca72a7d-a69d-4e05-ac4d-da20e15a0243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Nov 03 20:33:27.236448 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.235684 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:27.236448 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.236195 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-workload-socket\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:27.236448 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.236410 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:27.240187 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.239645 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d9f95b69-x7n2b"] Nov 03 20:33:27.240187 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.239938 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-jt54m_e2155512-2116-44f6-be5e-77cef72f9e14/connection-checker/0.log" Nov 03 20:33:27.240187 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.239975 5059 generic.go:356] "Generic (PLEG): container finished" podID="e2155512-2116-44f6-be5e-77cef72f9e14" containerID="27b2092e2b1ae773885e5c47b4223f33ad58ad3523d44d0fdfca69cbbee17e22" exitCode=137 Nov 03 20:33:27.240187 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.240095 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jt54m" event={"ID":"e2155512-2116-44f6-be5e-77cef72f9e14","Type":"ContainerDied","Data":"27b2092e2b1ae773885e5c47b4223f33ad58ad3523d44d0fdfca69cbbee17e22"} Nov 03 20:33:27.241287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.241262 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:27.244113 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.244088 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6hx8t_f9d935b2-aa51-474c-836d-a74f91efab07/connection-checker/0.log" Nov 03 20:33:27.244177 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.244130 5059 generic.go:356] "Generic (PLEG): container finished" podID="f9d935b2-aa51-474c-836d-a74f91efab07" containerID="e7fcbceef75077181869de0f40c073966ec17006f2685b45f17586d4fcf3eeb3" exitCode=137 Nov 03 20:33:27.244222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.244196 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6hx8t" event={"ID":"f9d935b2-aa51-474c-836d-a74f91efab07","Type":"ContainerDied","Data":"e7fcbceef75077181869de0f40c073966ec17006f2685b45f17586d4fcf3eeb3"} Nov 03 20:33:27.247619 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.247342 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-bd8f8_0fc518fc-6302-4a1a-9ff9-f1e036a2199b/connection-checker/0.log" Nov 03 20:33:27.247619 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.247380 5059 generic.go:356] "Generic (PLEG): container finished" podID="0fc518fc-6302-4a1a-9ff9-f1e036a2199b" containerID="13a83c66d1721d1a4304250d96ba6c4a613cfb0ba59048a8e63f219d8a6f74c8" exitCode=137 Nov 03 20:33:27.247619 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.247582 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bd8f8" event={"ID":"0fc518fc-6302-4a1a-9ff9-f1e036a2199b","Type":"ContainerDied","Data":"13a83c66d1721d1a4304250d96ba6c4a613cfb0ba59048a8e63f219d8a6f74c8"} Nov 03 20:33:27.257684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.250971 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:27.257684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.252224 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwfgt\" (UniqueName: \"kubernetes.io/projected/68bc094a-84e2-41c2-9fb5-081682413f27-kube-api-access-wwfgt\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:27.257684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.252910 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj2v6\" (UniqueName: \"kubernetes.io/projected/19ab8a5f-fa5f-4548-bf5f-0465281d8677-kube-api-access-xj2v6\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:27.257684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.252956 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2j7f\" (UniqueName: \"kubernetes.io/projected/7f438514-df13-4871-9c2a-ef8e56a570c9-kube-api-access-t2j7f\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:27.257684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.253722 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:27.280743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.275591 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x"] Nov 03 20:33:27.280743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.277737 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q" event={"ID":"8e9aefe5-8450-4eb9-8d8b-df89be81ebb3","Type":"ContainerStarted","Data":"0c8c232fc6576eb6c8950c19f0d5c165e8a2cdcac6e16524643b9275e091b3df"} Nov 03 20:33:27.297818 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.281944 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q" Nov 03 20:33:27.306710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.306240 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:27.310476 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.310448 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-b5d6x"] Nov 03 20:33:27.317435 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.317255 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c57db574-fmm6v" event={"ID":"3265bd58-d654-467c-9b2a-14cedc27b754","Type":"ContainerDied","Data":"dbf317add738c05f7cddd2b4c4bf1f3d7defd27f850f8568bdb62aa2afa1b980"} Nov 03 20:33:27.317435 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.317301 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbf317add738c05f7cddd2b4c4bf1f3d7defd27f850f8568bdb62aa2afa1b980" Nov 03 20:33:27.319929 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.319473 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcrh4" Nov 03 20:33:27.320766 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.320744 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:33:27.321210 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.321193 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcrh4" Nov 03 20:33:27.324438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.324387 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxwwd\" (UniqueName: \"kubernetes.io/projected/d50891bc-16c2-497d-8e00-b79ef492ab6e-kube-api-access-dxwwd\") pod \"authorino-operator-7b7bb4bc77-v5xhc\" (UID: \"d50891bc-16c2-497d-8e00-b79ef492ab6e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-v5xhc" Nov 03 20:33:27.329083 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.329038 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd"] Nov 03 20:33:27.335467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.333312 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rtqg\" (UniqueName: \"kubernetes.io/projected/d9f37244-d91a-480d-baa1-b1d59335804c-kube-api-access-8rtqg\") pod \"jobset-controller-manager-5d9f95b69-x7n2b\" (UID: \"d9f37244-d91a-480d-baa1-b1d59335804c\") " pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-x7n2b" Nov 03 20:33:27.335467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.333507 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9f37244-d91a-480d-baa1-b1d59335804c-metrics-certs\") pod \"jobset-controller-manager-5d9f95b69-x7n2b\" (UID: \"d9f37244-d91a-480d-baa1-b1d59335804c\") " pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-x7n2b" Nov 03 20:33:27.335467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.333806 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9f37244-d91a-480d-baa1-b1d59335804c-cert\") pod \"jobset-controller-manager-5d9f95b69-x7n2b\" (UID: \"d9f37244-d91a-480d-baa1-b1d59335804c\") " pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-x7n2b" Nov 03 20:33:27.335467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.334051 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d9f37244-d91a-480d-baa1-b1d59335804c-manager-config\") pod \"jobset-controller-manager-5d9f95b69-x7n2b\" (UID: \"d9f37244-d91a-480d-baa1-b1d59335804c\") " pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-x7n2b" Nov 03 20:33:27.335467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.334897 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-bbffd"] Nov 03 20:33:27.338222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.338199 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d9f37244-d91a-480d-baa1-b1d59335804c-manager-config\") pod \"jobset-controller-manager-5d9f95b69-x7n2b\" (UID: \"d9f37244-d91a-480d-baa1-b1d59335804c\") " pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-x7n2b" Nov 03 20:33:27.354679 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.351747 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9f37244-d91a-480d-baa1-b1d59335804c-metrics-certs\") pod \"jobset-controller-manager-5d9f95b69-x7n2b\" (UID: \"d9f37244-d91a-480d-baa1-b1d59335804c\") " pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-x7n2b" Nov 03 20:33:27.354679 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.351796 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-4vl4x"] Nov 03 20:33:27.354679 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.354169 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:27.376050 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.372441 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-4vl4x" Nov 03 20:33:27.398702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.398367 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-4vl4x"] Nov 03 20:33:27.405786 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.405727 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9f37244-d91a-480d-baa1-b1d59335804c-cert\") pod \"jobset-controller-manager-5d9f95b69-x7n2b\" (UID: \"d9f37244-d91a-480d-baa1-b1d59335804c\") " pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-x7n2b" Nov 03 20:33:27.406471 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.406441 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rtqg\" (UniqueName: \"kubernetes.io/projected/d9f37244-d91a-480d-baa1-b1d59335804c-kube-api-access-8rtqg\") pod \"jobset-controller-manager-5d9f95b69-x7n2b\" (UID: \"d9f37244-d91a-480d-baa1-b1d59335804c\") " pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-x7n2b" Nov 03 20:33:27.412222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.412175 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:27.412614 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.412196 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:27.422970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.422922 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj"] Nov 03 20:33:27.431692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.431652 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-wbnrj"] Nov 03 20:33:27.435814 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.435786 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e83c10-58b0-4bc7-9abd-3e144ba6e258-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-4vl4x\" (UID: \"94e83c10-58b0-4bc7-9abd-3e144ba6e258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-4vl4x" Nov 03 20:33:27.436312 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.436291 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nksbf\" (UniqueName: \"kubernetes.io/projected/94e83c10-58b0-4bc7-9abd-3e144ba6e258-kube-api-access-nksbf\") pod \"maas-api-revoke-sub-403cf98d-4vl4x\" (UID: \"94e83c10-58b0-4bc7-9abd-3e144ba6e258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-4vl4x" Nov 03 20:33:27.436483 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.436465 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e83c10-58b0-4bc7-9abd-3e144ba6e258-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-4vl4x\" (UID: \"94e83c10-58b0-4bc7-9abd-3e144ba6e258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-4vl4x" Nov 03 20:33:27.448201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.448168 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f"] Nov 03 20:33:27.454063 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.453643 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-ckz5f"] Nov 03 20:33:27.471493 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.464179 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lz2v2" podStartSLOduration=83.742041171 podStartE2EDuration="1m25.464157269s" totalImagesPullingTime="1.722116098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:13.420013212 +0000 UTC m=+3387.677810504" watchObservedRunningTime="2026-11-03 20:33:27.464157269 +0000 UTC m=+3401.721954531" Nov 03 20:33:27.514806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.512965 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:27.517937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.517872 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6"] Nov 03 20:33:27.526082 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.526053 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-xpkd6"] Nov 03 20:33:27.539692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.538110 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nksbf\" (UniqueName: \"kubernetes.io/projected/94e83c10-58b0-4bc7-9abd-3e144ba6e258-kube-api-access-nksbf\") pod \"maas-api-revoke-sub-403cf98d-4vl4x\" (UID: \"94e83c10-58b0-4bc7-9abd-3e144ba6e258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-4vl4x" Nov 03 20:33:27.539692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.538234 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e83c10-58b0-4bc7-9abd-3e144ba6e258-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-4vl4x\" (UID: \"94e83c10-58b0-4bc7-9abd-3e144ba6e258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-4vl4x" Nov 03 20:33:27.539692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.538371 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e83c10-58b0-4bc7-9abd-3e144ba6e258-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-4vl4x\" (UID: \"94e83c10-58b0-4bc7-9abd-3e144ba6e258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-4vl4x" Nov 03 20:33:27.539692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.539454 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e83c10-58b0-4bc7-9abd-3e144ba6e258-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-4vl4x\" (UID: \"94e83c10-58b0-4bc7-9abd-3e144ba6e258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-4vl4x" Nov 03 20:33:27.542129 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.542085 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl"] Nov 03 20:33:27.545117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.545011 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e83c10-58b0-4bc7-9abd-3e144ba6e258-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-4vl4x\" (UID: \"94e83c10-58b0-4bc7-9abd-3e144ba6e258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-4vl4x" Nov 03 20:33:27.552021 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.551131 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" podUID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:27.553601 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.553582 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-fv9cl"] Nov 03 20:33:27.571789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.557640 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nksbf\" (UniqueName: \"kubernetes.io/projected/94e83c10-58b0-4bc7-9abd-3e144ba6e258-kube-api-access-nksbf\") pod \"maas-api-revoke-sub-403cf98d-4vl4x\" (UID: \"94e83c10-58b0-4bc7-9abd-3e144ba6e258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-4vl4x" Nov 03 20:33:27.605717 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.605653 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-78xj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" start-of-body= Nov 03 20:33:27.605881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.605736 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" podUID="09d624de-215a-40be-9b38-0ba5a08743ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" Nov 03 20:33:27.616978 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.616921 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7"] Nov 03 20:33:27.628717 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.628340 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-2q8l7"] Nov 03 20:33:27.657058 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.648253 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:27.657058 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.653237 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-5sk9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused" start-of-body= Nov 03 20:33:27.657058 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.653287 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" podUID="81fa410e-25d7-4800-a22b-9dea6ccca3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused" Nov 03 20:33:27.735825 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.735651 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk"] Nov 03 20:33:27.759090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.758959 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7krsk"] Nov 03 20:33:27.771778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.768717 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd"] Nov 03 20:33:27.771778 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:27.769759 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a57d851_12b3_4535_b12c_b7d2c579d0fb.slice/crio-conmon-b8d82ca6d1d4fd46ae40cb0b5685d96dc2c2f3703f2b90b33d10896376edb8cc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfcb49c5_4381_411e_9d64_d9dcd6181239.slice/crio-6eb6c3a433e2fe458436cce8e74c7f5d996106a0fc51f6eab2da16d2f07bc977\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ceab3d_6344_4512_a471_d91aec0cb48c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c363281_0226_420f_a963_5922d5ca609e.slice/crio-0790100bd49a069e5ae6fe894ddc55096ee6648b0ba9da367a971377af75ae84\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3265bd58_d654_467c_9b2a_14cedc27b754.slice/crio-conmon-c00733af03a11c26991be300ebced8bc887b057aa45430c49b4b704481238514.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20690fd2_7022_4657_8c8c_bd738493ffc0.slice/crio-5b05929b67ea95a6b627445e0a3fa63ef0a0d7ef033293a27edabc525d4bd1cd.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b3a993_1a64_4fe5_9577_b0f11a36779e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod541bc63e_66f1_42ca_a946_aedc34beb62f.slice/crio-conmon-1c69adcd1fc6db5a42ddeb00863f24a144f0cf528222cad083ac5a885e389943.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20690fd2_7022_4657_8c8c_bd738493ffc0.slice/crio-conmon-5b05929b67ea95a6b627445e0a3fa63ef0a0d7ef033293a27edabc525d4bd1cd.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f32ecf7_06f7_4098_9793_acf761a30c09.slice/crio-58f0c433bf4f049cef0f62c8ae114894d5f9997d1c364fb2311ace0372d8fe70\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc518fc_6302_4a1a_9ff9_f1e036a2199b.slice/crio-13a83c66d1721d1a4304250d96ba6c4a613cfb0ba59048a8e63f219d8a6f74c8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b3a993_1a64_4fe5_9577_b0f11a36779e.slice/crio-945b49aab43d7640ffe6f9eabe8651bd8c94dc095e160eb5fb327ca5d696ada1\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16c463fa_8ddf_43c8_b00f_3f5f9ca33c02.slice/crio-238b88770b80cec7d339089cb9d69cc1f7ef5a1fdcefa3eb790154cc7f3289f1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ceab3d_6344_4512_a471_d91aec0cb48c.slice/crio-78cef308b00fc9217bde938ee3c21f496c6ebd2c6184bb774beaa70f0faf95db\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfcb49c5_4381_411e_9d64_d9dcd6181239.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf14445d5_52ed_487d_972c_4499966065fa.slice/crio-44d16c3b122733147748e39e219fb00f390ff6e37a49c47ed3e51ba2960a62c1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc518fc_6302_4a1a_9ff9_f1e036a2199b.slice/crio-conmon-13a83c66d1721d1a4304250d96ba6c4a613cfb0ba59048a8e63f219d8a6f74c8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a57d851_12b3_4535_b12c_b7d2c579d0fb.slice/crio-b8d82ca6d1d4fd46ae40cb0b5685d96dc2c2f3703f2b90b33d10896376edb8cc.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:33:27.771778 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:27.770522 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3265bd58_d654_467c_9b2a_14cedc27b754.slice/crio-conmon-c00733af03a11c26991be300ebced8bc887b057aa45430c49b4b704481238514.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2155512_2116_44f6_be5e_77cef72f9e14.slice/crio-27b2092e2b1ae773885e5c47b4223f33ad58ad3523d44d0fdfca69cbbee17e22.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3265bd58_d654_467c_9b2a_14cedc27b754.slice/crio-c00733af03a11c26991be300ebced8bc887b057aa45430c49b4b704481238514.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f32ecf7_06f7_4098_9793_acf761a30c09.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc518fc_6302_4a1a_9ff9_f1e036a2199b.slice/crio-conmon-13a83c66d1721d1a4304250d96ba6c4a613cfb0ba59048a8e63f219d8a6f74c8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d935b2_aa51_474c_836d_a74f91efab07.slice/crio-e7fcbceef75077181869de0f40c073966ec17006f2685b45f17586d4fcf3eeb3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f32ecf7_06f7_4098_9793_acf761a30c09.slice/crio-58f0c433bf4f049cef0f62c8ae114894d5f9997d1c364fb2311ace0372d8fe70\": RecentStats: unable to find data in memory cache]" Nov 03 20:33:27.773495 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:27.773432 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3265bd58_d654_467c_9b2a_14cedc27b754.slice/crio-conmon-c00733af03a11c26991be300ebced8bc887b057aa45430c49b4b704481238514.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3265bd58_d654_467c_9b2a_14cedc27b754.slice/crio-c00733af03a11c26991be300ebced8bc887b057aa45430c49b4b704481238514.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d935b2_aa51_474c_836d_a74f91efab07.slice/crio-e7fcbceef75077181869de0f40c073966ec17006f2685b45f17586d4fcf3eeb3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc518fc_6302_4a1a_9ff9_f1e036a2199b.slice/crio-13a83c66d1721d1a4304250d96ba6c4a613cfb0ba59048a8e63f219d8a6f74c8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d935b2_aa51_474c_836d_a74f91efab07.slice/crio-conmon-e7fcbceef75077181869de0f40c073966ec17006f2685b45f17586d4fcf3eeb3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f32ecf7_06f7_4098_9793_acf761a30c09.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f32ecf7_06f7_4098_9793_acf761a30c09.slice/crio-58f0c433bf4f049cef0f62c8ae114894d5f9997d1c364fb2311ace0372d8fe70\": RecentStats: unable to find data in memory cache]" Nov 03 20:33:27.782139 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:27.782082 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3265bd58_d654_467c_9b2a_14cedc27b754.slice/crio-c00733af03a11c26991be300ebced8bc887b057aa45430c49b4b704481238514.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3265bd58_d654_467c_9b2a_14cedc27b754.slice/crio-conmon-c00733af03a11c26991be300ebced8bc887b057aa45430c49b4b704481238514.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d935b2_aa51_474c_836d_a74f91efab07.slice/crio-conmon-e7fcbceef75077181869de0f40c073966ec17006f2685b45f17586d4fcf3eeb3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f32ecf7_06f7_4098_9793_acf761a30c09.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:33:27.783178 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.783127 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-7lvqd"] Nov 03 20:33:27.842589 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.842555 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d"] Nov 03 20:33:27.863389 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.863364 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-m6d9d"] Nov 03 20:33:27.886192 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.884889 5059 scope.go:122] "RemoveContainer" containerID="cc7a98566be4a3d14d59e43fa3c73400eb83547051245560b3c66cac856b8485" Nov 03 20:33:27.905953 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.904795 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06b78cd0-94df-40f9-a1ed-87c1751675d3" path="/var/lib/kubelet/pods/06b78cd0-94df-40f9-a1ed-87c1751675d3/volumes" Nov 03 20:33:27.910393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.908958 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14a06bdf-e15d-406c-bb76-47ae5dd06b2d" path="/var/lib/kubelet/pods/14a06bdf-e15d-406c-bb76-47ae5dd06b2d/volumes" Nov 03 20:33:27.910393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.909693 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c363281-0226-420f-a963-5922d5ca609e" path="/var/lib/kubelet/pods/2c363281-0226-420f-a963-5922d5ca609e/volumes" Nov 03 20:33:27.911024 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.911006 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b9d0847-df69-4b6a-b346-2cd206e42339" path="/var/lib/kubelet/pods/3b9d0847-df69-4b6a-b346-2cd206e42339/volumes" Nov 03 20:33:27.913169 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.913153 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40b88c7d-08e9-4203-9ebd-2c9f69b884dc" path="/var/lib/kubelet/pods/40b88c7d-08e9-4203-9ebd-2c9f69b884dc/volumes" Nov 03 20:33:27.914275 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.914244 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4946e3aa-a961-415a-b71d-95164d5fbca3" path="/var/lib/kubelet/pods/4946e3aa-a961-415a-b71d-95164d5fbca3/volumes" Nov 03 20:33:27.939811 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.939781 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c609469-d7d7-4530-8d74-e34aa56dda8f" path="/var/lib/kubelet/pods/4c609469-d7d7-4530-8d74-e34aa56dda8f/volumes" Nov 03 20:33:27.941544 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.940655 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ea38154-8a1f-4e1f-8fa0-50481ba4fec0" path="/var/lib/kubelet/pods/5ea38154-8a1f-4e1f-8fa0-50481ba4fec0/volumes" Nov 03 20:33:27.945266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.945238 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7909b42c-1572-4fe2-816a-447020594fe4" path="/var/lib/kubelet/pods/7909b42c-1572-4fe2-816a-447020594fe4/volumes" Nov 03 20:33:27.946081 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.946057 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d60eb56-9852-4c25-ae50-8a932e752793" path="/var/lib/kubelet/pods/9d60eb56-9852-4c25-ae50-8a932e752793/volumes" Nov 03 20:33:27.947567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.947546 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab8320f7-2354-433a-a8a7-4d91cc5d37d2" path="/var/lib/kubelet/pods/ab8320f7-2354-433a-a8a7-4d91cc5d37d2/volumes" Nov 03 20:33:27.949196 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.949172 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b962b15a-12b8-4b36-bc1d-ce5ed903f1d3" path="/var/lib/kubelet/pods/b962b15a-12b8-4b36-bc1d-ce5ed903f1d3/volumes" Nov 03 20:33:27.951078 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.950028 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9721811-d27d-4e13-a53d-4ae81e16fb96" path="/var/lib/kubelet/pods/c9721811-d27d-4e13-a53d-4ae81e16fb96/volumes" Nov 03 20:33:27.951078 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.950758 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eac98813-7b4c-4c4a-8bf9-444bba2fa644" path="/var/lib/kubelet/pods/eac98813-7b4c-4c4a-8bf9-444bba2fa644/volumes" Nov 03 20:33:27.952440 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.952414 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f88ae6f3-f1d6-4aa3-ad33-589490696167" path="/var/lib/kubelet/pods/f88ae6f3-f1d6-4aa3-ad33-589490696167/volumes" Nov 03 20:33:27.957204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.957181 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2"] Nov 03 20:33:27.957481 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.957462 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-6dd9gvhf2"] Nov 03 20:33:27.957629 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.957616 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm"] Nov 03 20:33:27.978960 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.978642 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ztrnm"] Nov 03 20:33:27.988918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.988799 5059 scope.go:122] "RemoveContainer" containerID="e188f63192d3371f9c88f1fcfd31fbc6a5bb4e3b477faa591137913f4bacb386" Nov 03 20:33:27.995498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:27.995441 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn"] Nov 03 20:33:28.005000 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.004838 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-z6ftn"] Nov 03 20:33:28.025822 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.019483 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l"] Nov 03 20:33:28.029308 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.029248 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-69c9l"] Nov 03 20:33:28.041259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.041231 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42"] Nov 03 20:33:28.187624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.187598 5059 scope.go:122] "RemoveContainer" containerID="9bf7e288190b11ae0ba4b37a69f007f930e0f2afcf8152bdfbdbfa5e3df689c6" Nov 03 20:33:28.260174 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.258767 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 20:33:28.270474 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.270438 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-qlt42"] Nov 03 20:33:28.278752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.278712 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-data\") pod \"c2d80995-5f1c-46db-8d71-a130cb92e279\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " Nov 03 20:33:28.279013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.278977 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-workload-certs\") pod \"c2d80995-5f1c-46db-8d71-a130cb92e279\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " Nov 03 20:33:28.279172 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.279154 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-token\" (UniqueName: \"kubernetes.io/projected/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-token\") pod \"c2d80995-5f1c-46db-8d71-a130cb92e279\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " Nov 03 20:33:28.279376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.279358 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-podinfo\") pod \"c2d80995-5f1c-46db-8d71-a130cb92e279\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " Nov 03 20:33:28.279737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.279718 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-envoy\") pod \"c2d80995-5f1c-46db-8d71-a130cb92e279\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " Nov 03 20:33:28.279994 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.279960 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d80995-5f1c-46db-8d71-a130cb92e279-kube-api-access-cqzgx\" (UniqueName: \"kubernetes.io/projected/c2d80995-5f1c-46db-8d71-a130cb92e279-kube-api-access-cqzgx\") pod \"c2d80995-5f1c-46db-8d71-a130cb92e279\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " Nov 03 20:33:28.280272 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.280241 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2d80995-5f1c-46db-8d71-a130cb92e279-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d80995-5f1c-46db-8d71-a130cb92e279-istiod-ca-cert\") pod \"c2d80995-5f1c-46db-8d71-a130cb92e279\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " Nov 03 20:33:28.280441 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.280417 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-credential-socket\") pod \"c2d80995-5f1c-46db-8d71-a130cb92e279\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " Nov 03 20:33:28.280615 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.280599 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-workload-socket\") pod \"c2d80995-5f1c-46db-8d71-a130cb92e279\" (UID: \"c2d80995-5f1c-46db-8d71-a130cb92e279\") " Nov 03 20:33:28.284570 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.284546 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-workload-socket" pod "c2d80995-5f1c-46db-8d71-a130cb92e279" (UID: "c2d80995-5f1c-46db-8d71-a130cb92e279"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:28.288601 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.288578 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d80995-5f1c-46db-8d71-a130cb92e279-istiod-ca-cert" pod "c2d80995-5f1c-46db-8d71-a130cb92e279" (UID: "c2d80995-5f1c-46db-8d71-a130cb92e279"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:28.288896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.288698 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-credential-socket" pod "c2d80995-5f1c-46db-8d71-a130cb92e279" (UID: "c2d80995-5f1c-46db-8d71-a130cb92e279"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:28.289130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.289113 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:28.291105 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.291083 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-data" pod "c2d80995-5f1c-46db-8d71-a130cb92e279" (UID: "c2d80995-5f1c-46db-8d71-a130cb92e279"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:28.291470 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.291449 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-workload-certs" pod "c2d80995-5f1c-46db-8d71-a130cb92e279" (UID: "c2d80995-5f1c-46db-8d71-a130cb92e279"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:28.319581 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.311562 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:28.339120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.339089 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r"] Nov 03 20:33:28.382081 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.366023 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:28.382566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.382361 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-8l94r"] Nov 03 20:33:28.464487 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.398719 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5"] Nov 03 20:33:28.464756 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.464733 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-gmfw5"] Nov 03 20:33:28.464885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.464874 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4"] Nov 03 20:33:28.464974 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.464966 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-swkn4"] Nov 03 20:33:28.465075 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.429780 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-envoy" pod "c2d80995-5f1c-46db-8d71-a130cb92e279" (UID: "c2d80995-5f1c-46db-8d71-a130cb92e279"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:28.465158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.450249 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d80995-5f1c-46db-8d71-a130cb92e279-kube-api-access-cqzgx" pod "c2d80995-5f1c-46db-8d71-a130cb92e279" (UID: "c2d80995-5f1c-46db-8d71-a130cb92e279"). InnerVolumeSpecName "kube-api-access-cqzgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:28.465236 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.414877 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:28.465518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.414919 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:33:28.465886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.420545 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.635724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.635333 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.635724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.635370 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.635724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.635387 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d80995-5f1c-46db-8d71-a130cb92e279-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.635724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.635402 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.635724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.604774 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-78xj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" start-of-body= Nov 03 20:33:28.635724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.635453 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" podUID="09d624de-215a-40be-9b38-0ba5a08743ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" Nov 03 20:33:28.636120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.481763 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf"] Nov 03 20:33:28.636120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.635850 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m" Nov 03 20:33:28.636120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.635890 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m" event={"ID":"de4efb53-8b59-49f1-894f-ef366f00732a","Type":"ContainerStarted","Data":"77b67d21274a3fb1e78412c9c2352788e224a8c423aaf976529b2a194a3814de"} Nov 03 20:33:28.636120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.635921 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-mcvtf"] Nov 03 20:33:28.636120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.514342 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:28.636363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.551541 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" podUID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:28.636363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.589172 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb" podStartSLOduration=85.58915234 podStartE2EDuration="1m25.58915234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:18.426465869 +0000 UTC m=+3392.684263127" watchObservedRunningTime="2026-11-03 20:33:28.58915234 +0000 UTC m=+3402.846949598" Nov 03 20:33:28.658990 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.629705 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-podinfo" pod "c2d80995-5f1c-46db-8d71-a130cb92e279" (UID: "c2d80995-5f1c-46db-8d71-a130cb92e279"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:28.673986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.667303 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-token" pod "c2d80995-5f1c-46db-8d71-a130cb92e279" (UID: "c2d80995-5f1c-46db-8d71-a130cb92e279"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:28.673986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.668890 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-5sk9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused" start-of-body= Nov 03 20:33:28.673986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.668941 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" podUID="81fa410e-25d7-4800-a22b-9dea6ccca3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused" Nov 03 20:33:28.673986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.669040 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:28.681617 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.681583 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt"] Nov 03 20:33:28.681900 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.681883 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-6lfvt"] Nov 03 20:33:28.702627 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.702516 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9" podStartSLOduration=86.70249824 podStartE2EDuration="1m26.70249824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:19.274066526 +0000 UTC m=+3393.531863785" watchObservedRunningTime="2026-11-03 20:33:28.70249824 +0000 UTC m=+3402.960295498" Nov 03 20:33:28.707583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.707519 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst" podStartSLOduration=86.70750359 podStartE2EDuration="1m26.70750359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:19.630097147 +0000 UTC m=+3393.887894406" watchObservedRunningTime="2026-11-03 20:33:28.70750359 +0000 UTC m=+3402.965300848" Nov 03 20:33:28.754849 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.754816 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6hx8t_f9d935b2-aa51-474c-836d-a74f91efab07/connection-checker/0.log" Nov 03 20:33:28.755225 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.755196 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6hx8t" event={"ID":"f9d935b2-aa51-474c-836d-a74f91efab07","Type":"ContainerDied","Data":"5bc830e6d84d3cd29bae7a5df0255f8f86285729e706443279b9042876a37e1c"} Nov 03 20:33:28.755413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.755392 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bc830e6d84d3cd29bae7a5df0255f8f86285729e706443279b9042876a37e1c" Nov 03 20:33:28.853306 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.853270 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-bd8f8_0fc518fc-6302-4a1a-9ff9-f1e036a2199b/connection-checker/0.log" Nov 03 20:33:28.853821 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.853797 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx"] Nov 03 20:33:28.864080 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.864048 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c2d80995-5f1c-46db-8d71-a130cb92e279" containerName="istio-proxy" Nov 03 20:33:28.903256 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.903138 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d80995-5f1c-46db-8d71-a130cb92e279" containerName="istio-proxy" Nov 03 20:33:28.903531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.878879 5059 scope.go:122] "RemoveContainer" containerID="49fc76b3559a7fdac7c4de24b68b9e183d25cec8fc53c075c32d7308be58c0fa" Nov 03 20:33:28.903841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.880381 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.903982 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.903966 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqzgx\" (UniqueName: \"kubernetes.io/projected/c2d80995-5f1c-46db-8d71-a130cb92e279-kube-api-access-cqzgx\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.904084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.904067 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.904250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.904233 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d80995-5f1c-46db-8d71-a130cb92e279-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.907771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.906099 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d80995-5f1c-46db-8d71-a130cb92e279" containerName="istio-proxy" Nov 03 20:33:28.969252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.956890 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bd8f8" event={"ID":"0fc518fc-6302-4a1a-9ff9-f1e036a2199b","Type":"ContainerDied","Data":"78e9081e9ae7d7fd90676dd54dbde8f3e82156ec5f28724f4b1e82150cb9f4ff"} Nov 03 20:33:28.969252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.956938 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78e9081e9ae7d7fd90676dd54dbde8f3e82156ec5f28724f4b1e82150cb9f4ff" Nov 03 20:33:28.969252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.956958 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx"] Nov 03 20:33:28.969252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.957001 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" event={"ID":"77396fff-1778-4f09-9e81-05e715fc015e","Type":"ContainerStarted","Data":"36a5a734a937a7a130b920c9f5650f94fab60ad77cf3785d0738d15cea0c92e1"} Nov 03 20:33:28.969252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.957035 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" event={"ID":"a2c58648-4022-4bfb-be9f-170a6222dfd8","Type":"ContainerStarted","Data":"fb118c660710eb7d73eb1decfc3f34428b1d8bf365ce4f36a506649974ba8750"} Nov 03 20:33:28.969252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.957148 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx" Nov 03 20:33:28.975466 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.975409 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" podStartSLOduration=86.975381153 podStartE2EDuration="1m26.975381153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:20.628969169 +0000 UTC m=+3394.886766443" watchObservedRunningTime="2026-11-03 20:33:28.975381153 +0000 UTC m=+3403.233178412" Nov 03 20:33:28.986306 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.986248 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" podStartSLOduration=86.535519013 podStartE2EDuration="1m26.986220908s" totalImagesPullingTime="450.701895ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:20.968201369 +0000 UTC m=+3395.225998649" watchObservedRunningTime="2026-11-03 20:33:28.986220908 +0000 UTC m=+3403.244018169" Nov 03 20:33:28.988389 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:28.988309 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" podStartSLOduration=86.988290995 podStartE2EDuration="1m26.988290995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:21.353260196 +0000 UTC m=+3395.611057489" watchObservedRunningTime="2026-11-03 20:33:28.988290995 +0000 UTC m=+3403.246088253" Nov 03 20:33:29.001699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.000170 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" event={"ID":"a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08","Type":"ContainerStarted","Data":"fa11f2fb407088c72f590514ec6549e1227c50e23e49cedbd3737ad709e6ca75"} Nov 03 20:33:29.012911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.012886 5059 scope.go:122] "RemoveContainer" containerID="b6f8babda48abb99f0cb3d851b8caae4d8cd12ad7eda8750e60f7605358c12ba" Nov 03 20:33:29.031179 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.031125 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" podStartSLOduration=87.031105943 podStartE2EDuration="1m27.031105943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:21.682430998 +0000 UTC m=+3395.940228296" watchObservedRunningTime="2026-11-03 20:33:29.031105943 +0000 UTC m=+3403.288903202" Nov 03 20:33:29.036630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.036604 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cmtzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.104:15021/healthz/ready\": dial tcp 10.132.1.104:15021: connect: connection refused" start-of-body= Nov 03 20:33:29.036984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.036954 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" podUID="a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.104:15021/healthz/ready\": dial tcp 10.132.1.104:15021: connect: connection refused" Nov 03 20:33:29.037718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.037049 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:29.044060 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.044041 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-jt54m_e2155512-2116-44f6-be5e-77cef72f9e14/connection-checker/0.log" Nov 03 20:33:29.044281 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.044266 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95" Nov 03 20:33:29.045261 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.045207 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" podStartSLOduration=87.045192979 podStartE2EDuration="1m27.045192979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:21.811207772 +0000 UTC m=+3396.069005087" watchObservedRunningTime="2026-11-03 20:33:29.045192979 +0000 UTC m=+3403.302990236" Nov 03 20:33:29.050348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.050321 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jt54m" event={"ID":"e2155512-2116-44f6-be5e-77cef72f9e14","Type":"ContainerDied","Data":"e781524536bca1b272b961d24e61957bd19573364a9d4afb8efb7d5734c4b324"} Nov 03 20:33:29.071755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.071720 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 20:33:29.071993 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.071719 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e781524536bca1b272b961d24e61957bd19573364a9d4afb8efb7d5734c4b324" Nov 03 20:33:29.072125 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.072112 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:29.072245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.051153 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" podStartSLOduration=87.051138271 podStartE2EDuration="1m27.051138271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:21.87813169 +0000 UTC m=+3396.135928957" watchObservedRunningTime="2026-11-03 20:33:29.051138271 +0000 UTC m=+3403.308935529" Nov 03 20:33:29.078394 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.078324 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 20:33:29.079419 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.079364 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:33:29.098051 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.097384 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 20:33:29.136687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.115159 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/847e7b38-8fb0-4c19-8943-f43e364f989b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14db2bd-2nlxx\" (UID: \"847e7b38-8fb0-4c19-8943-f43e364f989b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx" Nov 03 20:33:29.136687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.131393 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/847e7b38-8fb0-4c19-8943-f43e364f989b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14db2bd-2nlxx\" (UID: \"847e7b38-8fb0-4c19-8943-f43e364f989b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx" Nov 03 20:33:29.136687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.131456 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:29.136687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.131800 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlhpl\" (UniqueName: \"kubernetes.io/projected/847e7b38-8fb0-4c19-8943-f43e364f989b-kube-api-access-nlhpl\") pod \"maas-api-revoke-sub-f14db2bd-2nlxx\" (UID: \"847e7b38-8fb0-4c19-8943-f43e364f989b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx" Nov 03 20:33:29.136687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.134421 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" start-of-body= Nov 03 20:33:29.136687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.134468 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" podUID="a2c58648-4022-4bfb-be9f-170a6222dfd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" Nov 03 20:33:29.181688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.178152 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn" podStartSLOduration=87.178131117 podStartE2EDuration="1m27.178131117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:21.991027581 +0000 UTC m=+3396.248824836" watchObservedRunningTime="2026-11-03 20:33:29.178131117 +0000 UTC m=+3403.435928375" Nov 03 20:33:29.184961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.184905 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c" podStartSLOduration=86.184885901 podStartE2EDuration="1m26.184885901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:22.10570969 +0000 UTC m=+3396.363506948" watchObservedRunningTime="2026-11-03 20:33:29.184885901 +0000 UTC m=+3403.442683160" Nov 03 20:33:29.234774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.233331 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20690fd2-7022-4657-8c8c-bd738493ffc0-kube-api-access-jrhrx\" (UniqueName: \"kubernetes.io/projected/20690fd2-7022-4657-8c8c-bd738493ffc0-kube-api-access-jrhrx\") pod \"20690fd2-7022-4657-8c8c-bd738493ffc0\" (UID: \"20690fd2-7022-4657-8c8c-bd738493ffc0\") " Nov 03 20:33:29.234774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.233423 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/541bc63e-66f1-42ca-a946-aedc34beb62f-kube-api-access-g9lk5\" (UniqueName: \"kubernetes.io/projected/541bc63e-66f1-42ca-a946-aedc34beb62f-kube-api-access-g9lk5\") pod \"541bc63e-66f1-42ca-a946-aedc34beb62f\" (UID: \"541bc63e-66f1-42ca-a946-aedc34beb62f\") " Nov 03 20:33:29.234774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.233492 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-home\" (UniqueName: \"kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-home\") pod \"541bc63e-66f1-42ca-a946-aedc34beb62f\" (UID: \"541bc63e-66f1-42ca-a946-aedc34beb62f\") " Nov 03 20:33:29.234774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.233554 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7cf1267-7d06-459b-9df7-fc7c477d36e1-kube-api-access-rkvf4\" (UniqueName: \"kubernetes.io/projected/e7cf1267-7d06-459b-9df7-fc7c477d36e1-kube-api-access-rkvf4\") pod \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\" (UID: \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\") " Nov 03 20:33:29.234774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.233590 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/541bc63e-66f1-42ca-a946-aedc34beb62f-tls-certs\" (UniqueName: \"kubernetes.io/secret/541bc63e-66f1-42ca-a946-aedc34beb62f-tls-certs\") pod \"541bc63e-66f1-42ca-a946-aedc34beb62f\" (UID: \"541bc63e-66f1-42ca-a946-aedc34beb62f\") " Nov 03 20:33:29.234774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.233648 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-dshm\") pod \"20690fd2-7022-4657-8c8c-bd738493ffc0\" (UID: \"20690fd2-7022-4657-8c8c-bd738493ffc0\") " Nov 03 20:33:29.234774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.233702 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-home\" (UniqueName: \"kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-home\") pod \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\" (UID: \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\") " Nov 03 20:33:29.234774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.233771 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-model-cache\") pod \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\" (UID: \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\") " Nov 03 20:33:29.234774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.233867 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-home\" (UniqueName: \"kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-home\") pod \"20690fd2-7022-4657-8c8c-bd738493ffc0\" (UID: \"20690fd2-7022-4657-8c8c-bd738493ffc0\") " Nov 03 20:33:29.234774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.233929 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-dshm\") pod \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\" (UID: \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\") " Nov 03 20:33:29.234774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.233964 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-dshm\") pod \"541bc63e-66f1-42ca-a946-aedc34beb62f\" (UID: \"541bc63e-66f1-42ca-a946-aedc34beb62f\") " Nov 03 20:33:29.234774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.234044 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7cf1267-7d06-459b-9df7-fc7c477d36e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7cf1267-7d06-459b-9df7-fc7c477d36e1-tls-certs\") pod \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\" (UID: \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\") " Nov 03 20:33:29.234774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.234172 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20690fd2-7022-4657-8c8c-bd738493ffc0-tls-certs\" (UniqueName: \"kubernetes.io/secret/20690fd2-7022-4657-8c8c-bd738493ffc0-tls-certs\") pod \"20690fd2-7022-4657-8c8c-bd738493ffc0\" (UID: \"20690fd2-7022-4657-8c8c-bd738493ffc0\") " Nov 03 20:33:29.234774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.234232 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-model-cache\") pod \"541bc63e-66f1-42ca-a946-aedc34beb62f\" (UID: \"541bc63e-66f1-42ca-a946-aedc34beb62f\") " Nov 03 20:33:29.234774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.234281 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-tmp-dir\") pod \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\" (UID: \"e7cf1267-7d06-459b-9df7-fc7c477d36e1\") " Nov 03 20:33:29.234774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.234330 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-model-cache\") pod \"20690fd2-7022-4657-8c8c-bd738493ffc0\" (UID: \"20690fd2-7022-4657-8c8c-bd738493ffc0\") " Nov 03 20:33:29.234774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.234364 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-tmp-dir\") pod \"541bc63e-66f1-42ca-a946-aedc34beb62f\" (UID: \"541bc63e-66f1-42ca-a946-aedc34beb62f\") " Nov 03 20:33:29.234774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.234390 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-tmp-dir\") pod \"20690fd2-7022-4657-8c8c-bd738493ffc0\" (UID: \"20690fd2-7022-4657-8c8c-bd738493ffc0\") " Nov 03 20:33:29.234774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.234649 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/847e7b38-8fb0-4c19-8943-f43e364f989b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14db2bd-2nlxx\" (UID: \"847e7b38-8fb0-4c19-8943-f43e364f989b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx" Nov 03 20:33:29.304006 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.248484 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-model-cache" pod "541bc63e-66f1-42ca-a946-aedc34beb62f" (UID: "541bc63e-66f1-42ca-a946-aedc34beb62f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:29.304006 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.249282 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/847e7b38-8fb0-4c19-8943-f43e364f989b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14db2bd-2nlxx\" (UID: \"847e7b38-8fb0-4c19-8943-f43e364f989b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx" Nov 03 20:33:29.304006 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.250006 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlhpl\" (UniqueName: \"kubernetes.io/projected/847e7b38-8fb0-4c19-8943-f43e364f989b-kube-api-access-nlhpl\") pod \"maas-api-revoke-sub-f14db2bd-2nlxx\" (UID: \"847e7b38-8fb0-4c19-8943-f43e364f989b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx" Nov 03 20:33:29.304006 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.250903 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:29.304006 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.251429 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-tmp-dir" pod "e7cf1267-7d06-459b-9df7-fc7c477d36e1" (UID: "e7cf1267-7d06-459b-9df7-fc7c477d36e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:29.304006 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.266645 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-tmp-dir" pod "541bc63e-66f1-42ca-a946-aedc34beb62f" (UID: "541bc63e-66f1-42ca-a946-aedc34beb62f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:29.304006 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.280140 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-home" pod "541bc63e-66f1-42ca-a946-aedc34beb62f" (UID: "541bc63e-66f1-42ca-a946-aedc34beb62f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:29.304006 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.300195 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/847e7b38-8fb0-4c19-8943-f43e364f989b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14db2bd-2nlxx\" (UID: \"847e7b38-8fb0-4c19-8943-f43e364f989b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx" Nov 03 20:33:29.306678 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.306601 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" podStartSLOduration=86.306582556 podStartE2EDuration="1m26.306582556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:22.173176857 +0000 UTC m=+3396.430974138" watchObservedRunningTime="2026-11-03 20:33:29.306582556 +0000 UTC m=+3403.564379811" Nov 03 20:33:29.311032 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.310995 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-home" pod "20690fd2-7022-4657-8c8c-bd738493ffc0" (UID: "20690fd2-7022-4657-8c8c-bd738493ffc0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:29.314798 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.313870 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-model-cache" pod "20690fd2-7022-4657-8c8c-bd738493ffc0" (UID: "20690fd2-7022-4657-8c8c-bd738493ffc0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:29.318708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.316910 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-tmp-dir" pod "20690fd2-7022-4657-8c8c-bd738493ffc0" (UID: "20690fd2-7022-4657-8c8c-bd738493ffc0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:29.424564 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.354738 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:29.644850 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.644808 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:29.716685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.413157 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/847e7b38-8fb0-4c19-8943-f43e364f989b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14db2bd-2nlxx\" (UID: \"847e7b38-8fb0-4c19-8943-f43e364f989b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx" Nov 03 20:33:29.716685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.413233 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 20:33:29.716685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.415482 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:29.716937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.443352 5059 scope.go:122] "RemoveContainer" containerID="49fc76b3559a7fdac7c4de24b68b9e183d25cec8fc53c075c32d7308be58c0fa" Nov 03 20:33:29.750852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.444236 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-model-cache" pod "e7cf1267-7d06-459b-9df7-fc7c477d36e1" (UID: "e7cf1267-7d06-459b-9df7-fc7c477d36e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:29.750852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.454199 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:29.750852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.546550 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:29.750852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.546587 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:29.750852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.658445 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" podUID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Nov 03 20:33:29.750852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.658501 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Nov 03 20:33:29.750852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.392438 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:33:29.750852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.686315 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm" podStartSLOduration=87.686291202 podStartE2EDuration="1m27.686291202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:22.254005502 +0000 UTC m=+3396.511802760" watchObservedRunningTime="2026-11-03 20:33:29.686291202 +0000 UTC m=+3403.944088460" Nov 03 20:33:29.750852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.719384 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:29.750852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.719416 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:29.750852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.719444 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:29.750852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.719459 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:29.750852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.720166 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:29.750852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.686402 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:29.841673 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:29.755159 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49fc76b3559a7fdac7c4de24b68b9e183d25cec8fc53c075c32d7308be58c0fa\": container with ID starting with 49fc76b3559a7fdac7c4de24b68b9e183d25cec8fc53c075c32d7308be58c0fa not found: ID does not exist" containerID="49fc76b3559a7fdac7c4de24b68b9e183d25cec8fc53c075c32d7308be58c0fa" Nov 03 20:33:29.841673 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.755209 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49fc76b3559a7fdac7c4de24b68b9e183d25cec8fc53c075c32d7308be58c0fa"} err="failed to get container status \"49fc76b3559a7fdac7c4de24b68b9e183d25cec8fc53c075c32d7308be58c0fa\": rpc error: code = NotFound desc = could not find container \"49fc76b3559a7fdac7c4de24b68b9e183d25cec8fc53c075c32d7308be58c0fa\": container with ID starting with 49fc76b3559a7fdac7c4de24b68b9e183d25cec8fc53c075c32d7308be58c0fa not found: ID does not exist" Nov 03 20:33:29.841673 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.780293 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-home" pod "e7cf1267-7d06-459b-9df7-fc7c477d36e1" (UID: "e7cf1267-7d06-459b-9df7-fc7c477d36e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:29.841673 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.789835 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:33:29.841673 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.826362 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-model-cache\") pod \"cde430c2-2518-49a0-9851-42356d3f2fe9\" (UID: \"cde430c2-2518-49a0-9851-42356d3f2fe9\") " Nov 03 20:33:29.841673 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.826593 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cde430c2-2518-49a0-9851-42356d3f2fe9-tls-certs\" (UniqueName: \"kubernetes.io/secret/cde430c2-2518-49a0-9851-42356d3f2fe9-tls-certs\") pod \"cde430c2-2518-49a0-9851-42356d3f2fe9\" (UID: \"cde430c2-2518-49a0-9851-42356d3f2fe9\") " Nov 03 20:33:29.841673 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.826650 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-tmp-dir\") pod \"cde430c2-2518-49a0-9851-42356d3f2fe9\" (UID: \"cde430c2-2518-49a0-9851-42356d3f2fe9\") " Nov 03 20:33:29.841673 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.826741 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-home\" (UniqueName: \"kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-home\") pod \"cde430c2-2518-49a0-9851-42356d3f2fe9\" (UID: \"cde430c2-2518-49a0-9851-42356d3f2fe9\") " Nov 03 20:33:29.841673 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.826981 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cde430c2-2518-49a0-9851-42356d3f2fe9-kube-api-access-wfms2\" (UniqueName: \"kubernetes.io/projected/cde430c2-2518-49a0-9851-42356d3f2fe9-kube-api-access-wfms2\") pod \"cde430c2-2518-49a0-9851-42356d3f2fe9\" (UID: \"cde430c2-2518-49a0-9851-42356d3f2fe9\") " Nov 03 20:33:29.841673 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.827138 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-dshm\") pod \"cde430c2-2518-49a0-9851-42356d3f2fe9\" (UID: \"cde430c2-2518-49a0-9851-42356d3f2fe9\") " Nov 03 20:33:29.841673 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.828252 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:29.841673 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.828283 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:29.947788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.929746 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-model-cache" pod "cde430c2-2518-49a0-9851-42356d3f2fe9" (UID: "cde430c2-2518-49a0-9851-42356d3f2fe9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:29.962414 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.950124 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-tmp-dir" pod "cde430c2-2518-49a0-9851-42356d3f2fe9" (UID: "cde430c2-2518-49a0-9851-42356d3f2fe9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:30.133147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.985427 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:30.133147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:29.985468 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:30.133147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.072492 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cmtzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.104:15021/healthz/ready\": dial tcp 10.132.1.104:15021: connect: connection refused" start-of-body= Nov 03 20:33:30.133147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.072585 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" podUID="a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.104:15021/healthz/ready\": dial tcp 10.132.1.104:15021: connect: connection refused" Nov 03 20:33:30.133147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.073370 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-home" pod "cde430c2-2518-49a0-9851-42356d3f2fe9" (UID: "cde430c2-2518-49a0-9851-42356d3f2fe9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:30.133147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.105512 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:30.239839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.220891 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" start-of-body= Nov 03 20:33:30.239839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.220958 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" podUID="a2c58648-4022-4bfb-be9f-170a6222dfd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" Nov 03 20:33:30.239839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.221560 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/541bc63e-66f1-42ca-a946-aedc34beb62f-kube-api-access-g9lk5" pod "541bc63e-66f1-42ca-a946-aedc34beb62f" (UID: "541bc63e-66f1-42ca-a946-aedc34beb62f"). InnerVolumeSpecName "kube-api-access-g9lk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:30.239839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.223606 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7cf1267-7d06-459b-9df7-fc7c477d36e1-tls-certs" pod "e7cf1267-7d06-459b-9df7-fc7c477d36e1" (UID: "e7cf1267-7d06-459b-9df7-fc7c477d36e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:30.239839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.226936 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0309a608-f6c4-45ca-991b-33ba73221a6c" path="/var/lib/kubelet/pods/0309a608-f6c4-45ca-991b-33ba73221a6c/volumes" Nov 03 20:33:30.241354 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.241192 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a57d851-12b3-4535-b12c-b7d2c579d0fb-kube-api-access-9nn9g\" (UniqueName: \"kubernetes.io/projected/1a57d851-12b3-4535-b12c-b7d2c579d0fb-kube-api-access-9nn9g\") pod \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\" (UID: \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\") " Nov 03 20:33:30.241738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.241620 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-tmp-dir\") pod \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\" (UID: \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\") " Nov 03 20:33:30.249769 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.241921 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-dshm\") pod \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\" (UID: \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\") " Nov 03 20:33:30.249769 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.242049 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-home\" (UniqueName: \"kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-home\") pod \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\" (UID: \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\") " Nov 03 20:33:30.254756 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.250145 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-model-cache\") pod \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\" (UID: \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\") " Nov 03 20:33:30.254756 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.250284 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a57d851-12b3-4535-b12c-b7d2c579d0fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a57d851-12b3-4535-b12c-b7d2c579d0fb-tls-certs\") pod \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\" (UID: \"1a57d851-12b3-4535-b12c-b7d2c579d0fb\") " Nov 03 20:33:30.286247 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.286163 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7cf1267-7d06-459b-9df7-fc7c477d36e1-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:30.286247 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.286208 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9lk5\" (UniqueName: \"kubernetes.io/projected/541bc63e-66f1-42ca-a946-aedc34beb62f-kube-api-access-g9lk5\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:30.398600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.386291 5059 patch_prober.go:29] interesting pod/image-registry-6c57db574-fmm6v container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.132.0.30:5000/healthz\": context deadline exceeded" start-of-body= Nov 03 20:33:30.398600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.386343 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6c57db574-fmm6v" podUID="3265bd58-d654-467c-9b2a-14cedc27b754" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.30:5000/healthz\": context deadline exceeded" Nov 03 20:33:30.412224 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.412184 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05fceb9c-4a04-470f-a94a-055121319f82" path="/var/lib/kubelet/pods/05fceb9c-4a04-470f-a94a-055121319f82/volumes" Nov 03 20:33:30.427687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.424010 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20690fd2-7022-4657-8c8c-bd738493ffc0-tls-certs" pod "20690fd2-7022-4657-8c8c-bd738493ffc0" (UID: "20690fd2-7022-4657-8c8c-bd738493ffc0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:30.427687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.424444 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-tmp-dir" pod "1a57d851-12b3-4535-b12c-b7d2c579d0fb" (UID: "1a57d851-12b3-4535-b12c-b7d2c579d0fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:30.427687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.424800 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-home" pod "1a57d851-12b3-4535-b12c-b7d2c579d0fb" (UID: "1a57d851-12b3-4535-b12c-b7d2c579d0fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:30.427687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.424872 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-dshm" pod "541bc63e-66f1-42ca-a946-aedc34beb62f" (UID: "541bc63e-66f1-42ca-a946-aedc34beb62f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:30.427687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.425045 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-model-cache" pod "1a57d851-12b3-4535-b12c-b7d2c579d0fb" (UID: "1a57d851-12b3-4535-b12c-b7d2c579d0fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:30.427687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.425391 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7cf1267-7d06-459b-9df7-fc7c477d36e1-kube-api-access-rkvf4" pod "e7cf1267-7d06-459b-9df7-fc7c477d36e1" (UID: "e7cf1267-7d06-459b-9df7-fc7c477d36e1"). InnerVolumeSpecName "kube-api-access-rkvf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:30.427687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.426318 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlhpl\" (UniqueName: \"kubernetes.io/projected/847e7b38-8fb0-4c19-8943-f43e364f989b-kube-api-access-nlhpl\") pod \"maas-api-revoke-sub-f14db2bd-2nlxx\" (UID: \"847e7b38-8fb0-4c19-8943-f43e364f989b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx" Nov 03 20:33:30.427687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.426398 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cde430c2-2518-49a0-9851-42356d3f2fe9-tls-certs" pod "cde430c2-2518-49a0-9851-42356d3f2fe9" (UID: "cde430c2-2518-49a0-9851-42356d3f2fe9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:30.427687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.426447 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-dshm" pod "cde430c2-2518-49a0-9851-42356d3f2fe9" (UID: "cde430c2-2518-49a0-9851-42356d3f2fe9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:30.438718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.434795 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20690fd2-7022-4657-8c8c-bd738493ffc0-kube-api-access-jrhrx" pod "20690fd2-7022-4657-8c8c-bd738493ffc0" (UID: "20690fd2-7022-4657-8c8c-bd738493ffc0"). InnerVolumeSpecName "kube-api-access-jrhrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:30.438718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.435181 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-dshm" pod "e7cf1267-7d06-459b-9df7-fc7c477d36e1" (UID: "e7cf1267-7d06-459b-9df7-fc7c477d36e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:30.467956 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.467250 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 20:33:30.501897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.501627 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 20:33:30.539688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.524337 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20690fd2-7022-4657-8c8c-bd738493ffc0-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:30.578841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.578787 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrhrx\" (UniqueName: \"kubernetes.io/projected/20690fd2-7022-4657-8c8c-bd738493ffc0-kube-api-access-jrhrx\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:30.579036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.579019 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde430c2-2518-49a0-9851-42356d3f2fe9-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:30.579189 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.579177 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkvf4\" (UniqueName: \"kubernetes.io/projected/e7cf1267-7d06-459b-9df7-fc7c477d36e1-kube-api-access-rkvf4\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:30.579310 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.579298 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:30.579425 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.579414 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7cf1267-7d06-459b-9df7-fc7c477d36e1-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:30.579538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.579527 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:30.579648 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.579637 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/541bc63e-66f1-42ca-a946-aedc34beb62f-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:30.579769 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.579758 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde430c2-2518-49a0-9851-42356d3f2fe9-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:30.579898 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.579886 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:30.580004 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.527874 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" podStartSLOduration=88.527852655 podStartE2EDuration="1m28.527852655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:23.797727419 +0000 UTC m=+3398.055524676" watchObservedRunningTime="2026-11-03 20:33:30.527852655 +0000 UTC m=+3404.785649913" Nov 03 20:33:30.580165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.534698 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 20:33:30.580745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.558352 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-dshm" pod "1a57d851-12b3-4535-b12c-b7d2c579d0fb" (UID: "1a57d851-12b3-4535-b12c-b7d2c579d0fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:30.580745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.560523 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/541bc63e-66f1-42ca-a946-aedc34beb62f-tls-certs" pod "541bc63e-66f1-42ca-a946-aedc34beb62f" (UID: "541bc63e-66f1-42ca-a946-aedc34beb62f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:30.580745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.560579 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a57d851-12b3-4535-b12c-b7d2c579d0fb-tls-certs" pod "1a57d851-12b3-4535-b12c-b7d2c579d0fb" (UID: "1a57d851-12b3-4535-b12c-b7d2c579d0fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:30.580745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.560706 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a57d851-12b3-4535-b12c-b7d2c579d0fb-kube-api-access-9nn9g" pod "1a57d851-12b3-4535-b12c-b7d2c579d0fb" (UID: "1a57d851-12b3-4535-b12c-b7d2c579d0fb"). InnerVolumeSpecName "kube-api-access-9nn9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:30.580745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.560760 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-dshm" pod "20690fd2-7022-4657-8c8c-bd738493ffc0" (UID: "20690fd2-7022-4657-8c8c-bd738493ffc0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:30.611297 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.602093 5059 generic.go:356] "Generic (PLEG): container finished" podID="1326708b-4642-4992-91ea-8a590863a703" containerID="a350ab9162cb33074fa8120cd07c0f5b1f85de58b5831dbb73b25fac3f1122a6" exitCode=22 Nov 03 20:33:30.611297 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.602716 5059 scope.go:122] "RemoveContainer" containerID="a350ab9162cb33074fa8120cd07c0f5b1f85de58b5831dbb73b25fac3f1122a6" Nov 03 20:33:30.611297 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:30.603199 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-4bdd0b59-9wxx4_odh-ai-gateway-infra(1326708b-4642-4992-91ea-8a590863a703)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4" podUID="1326708b-4642-4992-91ea-8a590863a703" Nov 03 20:33:30.623630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.622627 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body= Nov 03 20:33:30.623630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.622702 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" podUID="426d1445-9e48-4516-a920-fca04d918f22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" Nov 03 20:33:30.703029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.685832 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-kube-api-access-7zjtd\" (UniqueName: \"kubernetes.io/projected/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-kube-api-access-7zjtd\") pod \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\" (UID: \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\") " Nov 03 20:33:30.703029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.685898 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3265bd58-d654-467c-9b2a-14cedc27b754-kube-api-access-n55dd\" (UniqueName: \"kubernetes.io/projected/3265bd58-d654-467c-9b2a-14cedc27b754-kube-api-access-n55dd\") pod \"3265bd58-d654-467c-9b2a-14cedc27b754\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " Nov 03 20:33:30.703029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.685941 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3265bd58-d654-467c-9b2a-14cedc27b754-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3265bd58-d654-467c-9b2a-14cedc27b754-installation-pull-secrets\") pod \"3265bd58-d654-467c-9b2a-14cedc27b754\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " Nov 03 20:33:30.703029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.686014 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f14445d5-52ed-487d-972c-4499966065fa-tls-certs\" (UniqueName: \"kubernetes.io/secret/f14445d5-52ed-487d-972c-4499966065fa-tls-certs\") pod \"f14445d5-52ed-487d-972c-4499966065fa\" (UID: \"f14445d5-52ed-487d-972c-4499966065fa\") " Nov 03 20:33:30.703029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.686054 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-model-cache\") pod \"f14445d5-52ed-487d-972c-4499966065fa\" (UID: \"f14445d5-52ed-487d-972c-4499966065fa\") " Nov 03 20:33:30.703029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.686165 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3265bd58-d654-467c-9b2a-14cedc27b754-registry-tls\" (UniqueName: \"kubernetes.io/projected/3265bd58-d654-467c-9b2a-14cedc27b754-registry-tls\") pod \"3265bd58-d654-467c-9b2a-14cedc27b754\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " Nov 03 20:33:30.703029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.686207 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f14445d5-52ed-487d-972c-4499966065fa-kube-api-access-g9s47\" (UniqueName: \"kubernetes.io/projected/f14445d5-52ed-487d-972c-4499966065fa-kube-api-access-g9s47\") pod \"f14445d5-52ed-487d-972c-4499966065fa\" (UID: \"f14445d5-52ed-487d-972c-4499966065fa\") " Nov 03 20:33:30.703029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.686259 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-dshm\" (UniqueName: \"kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-dshm\") pod \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\" (UID: \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\") " Nov 03 20:33:30.703029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.686308 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-model-cache\") pod \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\" (UID: \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\") " Nov 03 20:33:30.703029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.686378 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-dshm\") pod \"f14445d5-52ed-487d-972c-4499966065fa\" (UID: \"f14445d5-52ed-487d-972c-4499966065fa\") " Nov 03 20:33:30.703029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.686416 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3265bd58-d654-467c-9b2a-14cedc27b754-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3265bd58-d654-467c-9b2a-14cedc27b754-bound-sa-token\") pod \"3265bd58-d654-467c-9b2a-14cedc27b754\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " Nov 03 20:33:30.703029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.686469 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3265bd58-d654-467c-9b2a-14cedc27b754-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3265bd58-d654-467c-9b2a-14cedc27b754-registry-certificates\") pod \"3265bd58-d654-467c-9b2a-14cedc27b754\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " Nov 03 20:33:30.703029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.686502 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-home\" (UniqueName: \"kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-home\") pod \"f14445d5-52ed-487d-972c-4499966065fa\" (UID: \"f14445d5-52ed-487d-972c-4499966065fa\") " Nov 03 20:33:30.703029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.686553 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-tmp-dir\") pod \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\" (UID: \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\") " Nov 03 20:33:30.703029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.686610 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3265bd58-d654-467c-9b2a-14cedc27b754-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3265bd58-d654-467c-9b2a-14cedc27b754-ca-trust-extracted\") pod \"3265bd58-d654-467c-9b2a-14cedc27b754\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " Nov 03 20:33:30.705734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.704379 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3265bd58-d654-467c-9b2a-14cedc27b754-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3265bd58-d654-467c-9b2a-14cedc27b754-trusted-ca\") pod \"3265bd58-d654-467c-9b2a-14cedc27b754\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " Nov 03 20:33:30.705734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.704426 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-tls-certs\" (UniqueName: \"kubernetes.io/secret/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-tls-certs\") pod \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\" (UID: \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\") " Nov 03 20:33:30.705734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.704466 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3265bd58-d654-467c-9b2a-14cedc27b754-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3265bd58-d654-467c-9b2a-14cedc27b754-image-registry-private-configuration\") pod \"3265bd58-d654-467c-9b2a-14cedc27b754\" (UID: \"3265bd58-d654-467c-9b2a-14cedc27b754\") " Nov 03 20:33:30.705734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.704616 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-tmp-dir\") pod \"f14445d5-52ed-487d-972c-4499966065fa\" (UID: \"f14445d5-52ed-487d-972c-4499966065fa\") " Nov 03 20:33:30.705734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.704691 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-home\" (UniqueName: \"kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-home\") pod \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\" (UID: \"16c463fa-8ddf-43c8-b00f-3f5f9ca33c02\") " Nov 03 20:33:30.706639 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.706584 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/541bc63e-66f1-42ca-a946-aedc34beb62f-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:30.706844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.706620 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20690fd2-7022-4657-8c8c-bd738493ffc0-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:30.706844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.706793 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nn9g\" (UniqueName: \"kubernetes.io/projected/1a57d851-12b3-4535-b12c-b7d2c579d0fb-kube-api-access-9nn9g\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:30.706844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.706813 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a57d851-12b3-4535-b12c-b7d2c579d0fb-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:30.731702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.729547 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a57d851-12b3-4535-b12c-b7d2c579d0fb-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:30.748281 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.731075 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-78xj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:33:30.809685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.781957 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" podUID="09d624de-215a-40be-9b38-0ba5a08743ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:30.841729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.834091 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25ceab3d-6344-4512-a471-d91aec0cb48c" path="/var/lib/kubelet/pods/25ceab3d-6344-4512-a471-d91aec0cb48c/volumes" Nov 03 20:33:30.841729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.746927 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 20:33:30.846690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.806340 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-model-cache" pod "f14445d5-52ed-487d-972c-4499966065fa" (UID: "f14445d5-52ed-487d-972c-4499966065fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:30.846690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.836289 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde430c2-2518-49a0-9851-42356d3f2fe9-kube-api-access-wfms2" pod "cde430c2-2518-49a0-9851-42356d3f2fe9" (UID: "cde430c2-2518-49a0-9851-42356d3f2fe9"). InnerVolumeSpecName "kube-api-access-wfms2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:30.858701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.857918 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34880975-a897-4ad4-af59-18617a93cdad" path="/var/lib/kubelet/pods/34880975-a897-4ad4-af59-18617a93cdad/volumes" Nov 03 20:33:30.863676 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.862494 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37e90ab0-e9c0-420c-a0f4-c7923aa00297" path="/var/lib/kubelet/pods/37e90ab0-e9c0-420c-a0f4-c7923aa00297/volumes" Nov 03 20:33:30.863676 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.863461 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70d45aa9-bfa8-43c6-8674-6f8f14a3097e" path="/var/lib/kubelet/pods/70d45aa9-bfa8-43c6-8674-6f8f14a3097e/volumes" Nov 03 20:33:30.868698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.746889 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 20:33:30.890598 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.878058 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" podStartSLOduration=87.878037084 podStartE2EDuration="1m27.878037084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:25.205481188 +0000 UTC m=+3399.463278448" watchObservedRunningTime="2026-11-03 20:33:30.878037084 +0000 UTC m=+3405.135834342" Nov 03 20:33:31.044363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.892059 5059 scope.go:122] "RemoveContainer" containerID="642fce6d48241f7a656bbb89d094312c435e6bbe1d1fd5edeacf6d1e6baccd05" Nov 03 20:33:31.044363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.920121 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3265bd58-d654-467c-9b2a-14cedc27b754-trusted-ca" pod "3265bd58-d654-467c-9b2a-14cedc27b754" (UID: "3265bd58-d654-467c-9b2a-14cedc27b754"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:31.044363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.925952 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3265bd58-d654-467c-9b2a-14cedc27b754-ca-trust-extracted" pod "3265bd58-d654-467c-9b2a-14cedc27b754" (UID: "3265bd58-d654-467c-9b2a-14cedc27b754"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:31.044363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.926639 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3265bd58-d654-467c-9b2a-14cedc27b754-registry-certificates" pod "3265bd58-d654-467c-9b2a-14cedc27b754" (UID: "3265bd58-d654-467c-9b2a-14cedc27b754"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:31.044363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.929217 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8adff4a5-5376-43ec-9b53-888705ae8236" path="/var/lib/kubelet/pods/8adff4a5-5376-43ec-9b53-888705ae8236/volumes" Nov 03 20:33:31.044363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.946975 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3265bd58-d654-467c-9b2a-14cedc27b754-bound-sa-token" pod "3265bd58-d654-467c-9b2a-14cedc27b754" (UID: "3265bd58-d654-467c-9b2a-14cedc27b754"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:31.044363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.957768 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a594e899-889e-4f7f-a8ba-b7e7a389ba8a" path="/var/lib/kubelet/pods/a594e899-889e-4f7f-a8ba-b7e7a389ba8a/volumes" Nov 03 20:33:31.044363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.958697 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d912f1fd-b76b-4bb6-a5d9-ccc8b4536840" path="/var/lib/kubelet/pods/d912f1fd-b76b-4bb6-a5d9-ccc8b4536840/volumes" Nov 03 20:33:31.044363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.958774 5059 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3265bd58-d654-467c-9b2a-14cedc27b754-bound-sa-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:31.044363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.958806 5059 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3265bd58-d654-467c-9b2a-14cedc27b754-registry-certificates\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:31.044363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.958824 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfms2\" (UniqueName: \"kubernetes.io/projected/cde430c2-2518-49a0-9851-42356d3f2fe9-kube-api-access-wfms2\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:31.044363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.958840 5059 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3265bd58-d654-467c-9b2a-14cedc27b754-ca-trust-extracted\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:31.044363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.958856 5059 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3265bd58-d654-467c-9b2a-14cedc27b754-trusted-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:31.044363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.958870 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:31.044363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.983391 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0d3c590-f523-4b7e-9086-60df5fc585e8" path="/var/lib/kubelet/pods/f0d3c590-f523-4b7e-9086-60df5fc585e8/volumes" Nov 03 20:33:31.044363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.989421 5059 generic.go:356] "Generic (PLEG): container finished" podID="dc10f92e-e902-48c4-9cd5-4f768c4611be" containerID="3944dcf9452420ea8a9d89d52b744632d0da3464e6e4103dbf789969612761cf" exitCode=22 Nov 03 20:33:31.044363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.989708 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp" Nov 03 20:33:31.044363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.989799 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3265bd58-d654-467c-9b2a-14cedc27b754-image-registry-private-configuration" pod "3265bd58-d654-467c-9b2a-14cedc27b754" (UID: "3265bd58-d654-467c-9b2a-14cedc27b754"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:31.044363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:30.989938 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3265bd58-d654-467c-9b2a-14cedc27b754-kube-api-access-n55dd" pod "3265bd58-d654-467c-9b2a-14cedc27b754" (UID: "3265bd58-d654-467c-9b2a-14cedc27b754"). InnerVolumeSpecName "kube-api-access-n55dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:31.044363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.028207 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5" Nov 03 20:33:31.054240 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.053044 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-home" pod "16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" (UID: "16c463fa-8ddf-43c8-b00f-3f5f9ca33c02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:31.054240 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.053740 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-model-cache" pod "16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" (UID: "16c463fa-8ddf-43c8-b00f-3f5f9ca33c02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:31.054240 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.053930 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-tmp-dir" pod "16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" (UID: "16c463fa-8ddf-43c8-b00f-3f5f9ca33c02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:31.054792 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.054766 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-home" pod "f14445d5-52ed-487d-972c-4499966065fa" (UID: "f14445d5-52ed-487d-972c-4499966065fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:31.071688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.068019 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-tls-certs" pod "16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" (UID: "16c463fa-8ddf-43c8-b00f-3f5f9ca33c02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:31.169685 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:31.163949 5059 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.253s" Nov 03 20:33:31.169685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.164132 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" event={"ID":"e5585d8d-d156-418c-ae87-012781bc0e8c","Type":"ContainerStarted","Data":"93f36aec4480abf1e55cb1d0288047d7a72dafc88b347454eb80df7a40b6b065"} Nov 03 20:33:31.169685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.164192 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" event={"ID":"426d1445-9e48-4516-a920-fca04d918f22","Type":"ContainerStarted","Data":"3f775196318978b57c4c708d101e83f38ee80fd8c5d6c5eee6f5841b39c93535"} Nov 03 20:33:31.169685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.164242 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:31.169685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.164362 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:31.169685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.164395 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:31.169685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.164427 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:31.169685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.164531 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:31.169685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.164609 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" Nov 03 20:33:31.169685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.164726 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:31.169685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.164780 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5" event={"ID":"6b9428a2-ab50-4e44-a725-e0da980bdb93","Type":"ContainerStarted","Data":"ccbe96e26db6f56ce351a7e5d4b629564431bc79e018b93b657c373a6b88a2e6"} Nov 03 20:33:31.169685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.164847 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4" event={"ID":"1326708b-4642-4992-91ea-8a590863a703","Type":"ContainerDied","Data":"a350ab9162cb33074fa8120cd07c0f5b1f85de58b5831dbb73b25fac3f1122a6"} Nov 03 20:33:31.169685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.164895 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47" event={"ID":"6c29d53f-b738-49ec-9b15-1c88e5950008","Type":"ContainerStarted","Data":"642fce6d48241f7a656bbb89d094312c435e6bbe1d1fd5edeacf6d1e6baccd05"} Nov 03 20:33:31.169685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.164937 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4" event={"ID":"dc10f92e-e902-48c4-9cd5-4f768c4611be","Type":"ContainerDied","Data":"3944dcf9452420ea8a9d89d52b744632d0da3464e6e4103dbf789969612761cf"} Nov 03 20:33:31.169685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.165561 5059 scope.go:122] "RemoveContainer" containerID="3944dcf9452420ea8a9d89d52b744632d0da3464e6e4103dbf789969612761cf" Nov 03 20:33:31.211686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.192522 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cmtzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.104:15021/healthz/ready\": dial tcp 10.132.1.104:15021: connect: connection refused" start-of-body= Nov 03 20:33:31.211686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.192603 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" podUID="a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.104:15021/healthz/ready\": dial tcp 10.132.1.104:15021: connect: connection refused" Nov 03 20:33:31.223715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.222139 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-tmp-dir" pod "f14445d5-52ed-487d-972c-4499966065fa" (UID: "f14445d5-52ed-487d-972c-4499966065fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:31.243747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.242435 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-dshm" pod "f14445d5-52ed-487d-972c-4499966065fa" (UID: "f14445d5-52ed-487d-972c-4499966065fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:31.243747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.242497 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f14445d5-52ed-487d-972c-4499966065fa-tls-certs" pod "f14445d5-52ed-487d-972c-4499966065fa" (UID: "f14445d5-52ed-487d-972c-4499966065fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:31.243747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.242764 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-dshm" pod "16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" (UID: "16c463fa-8ddf-43c8-b00f-3f5f9ca33c02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:31.243747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.242870 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3265bd58-d654-467c-9b2a-14cedc27b754-installation-pull-secrets" pod "3265bd58-d654-467c-9b2a-14cedc27b754" (UID: "3265bd58-d654-467c-9b2a-14cedc27b754"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:31.243747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.243069 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-kube-api-access-7zjtd" pod "16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" (UID: "16c463fa-8ddf-43c8-b00f-3f5f9ca33c02"). InnerVolumeSpecName "kube-api-access-7zjtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:31.243747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.243108 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f14445d5-52ed-487d-972c-4499966065fa-kube-api-access-g9s47" pod "f14445d5-52ed-487d-972c-4499966065fa" (UID: "f14445d5-52ed-487d-972c-4499966065fa"). InnerVolumeSpecName "kube-api-access-g9s47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:31.243747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.243552 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3265bd58-d654-467c-9b2a-14cedc27b754-registry-tls" pod "3265bd58-d654-467c-9b2a-14cedc27b754" (UID: "3265bd58-d654-467c-9b2a-14cedc27b754"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:31.246597 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:31.246556 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-r5fd4_odh-ai-gateway-infra(dc10f92e-e902-48c4-9cd5-4f768c4611be)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4" podUID="dc10f92e-e902-48c4-9cd5-4f768c4611be" Nov 03 20:33:31.246920 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.246900 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:31.256557 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.256487 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp" podStartSLOduration=88.256469755 podStartE2EDuration="1m28.256469755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:27.275187825 +0000 UTC m=+3401.532985083" watchObservedRunningTime="2026-11-03 20:33:31.256469755 +0000 UTC m=+3405.514267013" Nov 03 20:33:31.257131 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.257034 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" start-of-body= Nov 03 20:33:31.257131 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.257102 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" podUID="a2c58648-4022-4bfb-be9f-170a6222dfd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" Nov 03 20:33:31.258759 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.258401 5059 scope.go:122] "RemoveContainer" containerID="8cc39ee98f0eb5245e5d5488f9855a4beaa057aa6bbd7dc9d1116291e5c04e35" Nov 03 20:33:31.341692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.336213 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:31.342220 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.342190 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:33:31.342895 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.342862 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l" Nov 03 20:33:31.343445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.343018 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:31.343445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.343036 5059 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3265bd58-d654-467c-9b2a-14cedc27b754-image-registry-private-configuration\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:31.343445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.343049 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:31.343445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.343076 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:31.343445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.343086 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zjtd\" (UniqueName: \"kubernetes.io/projected/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-kube-api-access-7zjtd\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:31.343445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.343100 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n55dd\" (UniqueName: \"kubernetes.io/projected/3265bd58-d654-467c-9b2a-14cedc27b754-kube-api-access-n55dd\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:31.343445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.343110 5059 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3265bd58-d654-467c-9b2a-14cedc27b754-installation-pull-secrets\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:31.343445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.343120 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f14445d5-52ed-487d-972c-4499966065fa-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:31.343445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.343129 5059 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3265bd58-d654-467c-9b2a-14cedc27b754-registry-tls\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:31.343445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.343142 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9s47\" (UniqueName: \"kubernetes.io/projected/f14445d5-52ed-487d-972c-4499966065fa-kube-api-access-g9s47\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:31.343445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.343152 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:31.343445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.343161 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:31.343445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.343169 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:31.343445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.343182 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f14445d5-52ed-487d-972c-4499966065fa-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:31.343445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.343191 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:31.344535 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.343994 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w" Nov 03 20:33:31.346724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.346526 5059 patch_prober.go:29] interesting pod/router-default-f7b87d695-cwlfq container/router namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]backend-proxy-http ok Nov 03 20:33:31.346724 ip-10-0-132-83 kubenswrapper[5059]: [+]has-synced ok Nov 03 20:33:31.346724 ip-10-0-132-83 kubenswrapper[5059]: [-]process-running failed: reason withheld Nov 03 20:33:31.346724 ip-10-0-132-83 kubenswrapper[5059]: healthz check failed Nov 03 20:33:31.346724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.346581 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-f7b87d695-cwlfq" podUID="603868b0-8628-4227-8471-1894e2e1de50" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Nov 03 20:33:31.346724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.346701 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4" Nov 03 20:33:31.347045 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.346941 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" Nov 03 20:33:31.347582 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.347557 5059 scope.go:122] "RemoveContainer" containerID="ccbe96e26db6f56ce351a7e5d4b629564431bc79e018b93b657c373a6b88a2e6" Nov 03 20:33:31.347943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.347927 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-bd8f8_0fc518fc-6302-4a1a-9ff9-f1e036a2199b/connection-checker/0.log" Nov 03 20:33:31.348184 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:31.348046 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-266d0609-xqmv5_odh-ai-gateway-infra(6b9428a2-ab50-4e44-a725-e0da980bdb93)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5" podUID="6b9428a2-ab50-4e44-a725-e0da980bdb93" Nov 03 20:33:31.351643 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.351617 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bd8f8" Nov 03 20:33:31.613737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.588915 5059 scope.go:122] "RemoveContainer" containerID="4d94405c1bd7e9ab077c0ca4e40445d24510310ab3bd698c9d3fc466bbff887d" Nov 03 20:33:31.613737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.597511 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc518fc-6302-4a1a-9ff9-f1e036a2199b-kube-api-access-25cd8\" (UniqueName: \"kubernetes.io/projected/0fc518fc-6302-4a1a-9ff9-f1e036a2199b-kube-api-access-25cd8\") pod \"0fc518fc-6302-4a1a-9ff9-f1e036a2199b\" (UID: \"0fc518fc-6302-4a1a-9ff9-f1e036a2199b\") " Nov 03 20:33:31.638871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.623480 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body= Nov 03 20:33:31.638871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.623541 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" podUID="426d1445-9e48-4516-a920-fca04d918f22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" Nov 03 20:33:31.657974 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.657909 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc518fc-6302-4a1a-9ff9-f1e036a2199b-kube-api-access-25cd8" pod "0fc518fc-6302-4a1a-9ff9-f1e036a2199b" (UID: "0fc518fc-6302-4a1a-9ff9-f1e036a2199b"). InnerVolumeSpecName "kube-api-access-25cd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:31.794917 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:31.736132 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25cd8\" (UniqueName: \"kubernetes.io/projected/0fc518fc-6302-4a1a-9ff9-f1e036a2199b-kube-api-access-25cd8\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:32.042851 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.042810 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cmtzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.104:15021/healthz/ready\": dial tcp 10.132.1.104:15021: connect: connection refused" start-of-body= Nov 03 20:33:32.043088 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.043059 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" podUID="a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.104:15021/healthz/ready\": dial tcp 10.132.1.104:15021: connect: connection refused" Nov 03 20:33:32.124334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.123455 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" start-of-body= Nov 03 20:33:32.124334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.123521 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" podUID="a2c58648-4022-4bfb-be9f-170a6222dfd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" Nov 03 20:33:32.252599 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.251593 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6hx8t_f9d935b2-aa51-474c-836d-a74f91efab07/connection-checker/0.log" Nov 03 20:33:32.319082 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.255725 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6hx8t" Nov 03 20:33:32.352009 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.351973 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9d935b2-aa51-474c-836d-a74f91efab07-kube-api-access-7c5zw\" (UniqueName: \"kubernetes.io/projected/f9d935b2-aa51-474c-836d-a74f91efab07-kube-api-access-7c5zw\") pod \"f9d935b2-aa51-474c-836d-a74f91efab07\" (UID: \"f9d935b2-aa51-474c-836d-a74f91efab07\") " Nov 03 20:33:32.355129 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.355106 5059 scope.go:122] "RemoveContainer" containerID="a350ab9162cb33074fa8120cd07c0f5b1f85de58b5831dbb73b25fac3f1122a6" Nov 03 20:33:32.355485 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:32.355455 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-4bdd0b59-9wxx4_odh-ai-gateway-infra(1326708b-4642-4992-91ea-8a590863a703)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4" podUID="1326708b-4642-4992-91ea-8a590863a703" Nov 03 20:33:32.428418 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.427202 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:32.428418 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.427309 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:32.428418 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.427358 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:32.434370 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.433883 5059 generic.go:356] "Generic (PLEG): container finished" podID="6c29d53f-b738-49ec-9b15-1c88e5950008" containerID="642fce6d48241f7a656bbb89d094312c435e6bbe1d1fd5edeacf6d1e6baccd05" exitCode=22 Nov 03 20:33:32.434370 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.434291 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47" event={"ID":"6c29d53f-b738-49ec-9b15-1c88e5950008","Type":"ContainerDied","Data":"642fce6d48241f7a656bbb89d094312c435e6bbe1d1fd5edeacf6d1e6baccd05"} Nov 03 20:33:32.451095 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.451072 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d935b2-aa51-474c-836d-a74f91efab07-kube-api-access-7c5zw" pod "f9d935b2-aa51-474c-836d-a74f91efab07" (UID: "f9d935b2-aa51-474c-836d-a74f91efab07"). InnerVolumeSpecName "kube-api-access-7c5zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:32.457315 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.454649 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c5zw\" (UniqueName: \"kubernetes.io/projected/f9d935b2-aa51-474c-836d-a74f91efab07-kube-api-access-7c5zw\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:32.457315 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.454867 5059 generic.go:356] "Generic (PLEG): container finished" podID="6b9428a2-ab50-4e44-a725-e0da980bdb93" containerID="ccbe96e26db6f56ce351a7e5d4b629564431bc79e018b93b657c373a6b88a2e6" exitCode=22 Nov 03 20:33:32.457315 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.455496 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c57db574-fmm6v" Nov 03 20:33:32.457315 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.455986 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5" event={"ID":"6b9428a2-ab50-4e44-a725-e0da980bdb93","Type":"ContainerDied","Data":"ccbe96e26db6f56ce351a7e5d4b629564431bc79e018b93b657c373a6b88a2e6"} Nov 03 20:33:32.457315 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.456043 5059 scope.go:122] "RemoveContainer" containerID="2d957762bcd8fed6693d101c9e9584a4ee5948a1b3f069a82b1b80826caa7347" Nov 03 20:33:32.457315 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.456740 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55" Nov 03 20:33:32.465102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.458212 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22" Nov 03 20:33:32.492692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.486473 5059 scope.go:122] "RemoveContainer" containerID="ccbe96e26db6f56ce351a7e5d4b629564431bc79e018b93b657c373a6b88a2e6" Nov 03 20:33:32.493751 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:32.493698 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-266d0609-xqmv5_odh-ai-gateway-infra(6b9428a2-ab50-4e44-a725-e0da980bdb93)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5" podUID="6b9428a2-ab50-4e44-a725-e0da980bdb93" Nov 03 20:33:32.553287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.552771 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-jt54m_e2155512-2116-44f6-be5e-77cef72f9e14/connection-checker/0.log" Nov 03 20:33:32.557240 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.554921 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:32.597461 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.597431 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bd8f8" Nov 03 20:33:32.645195 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.625197 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" Nov 03 20:33:32.747817 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.625498 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body= Nov 03 20:33:32.747949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.747846 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" podUID="426d1445-9e48-4516-a920-fca04d918f22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" Nov 03 20:33:32.747949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.668828 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jt54m" Nov 03 20:33:32.820617 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.820577 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2155512-2116-44f6-be5e-77cef72f9e14-kube-api-access-cjhsn\" (UniqueName: \"kubernetes.io/projected/e2155512-2116-44f6-be5e-77cef72f9e14-kube-api-access-cjhsn\") pod \"e2155512-2116-44f6-be5e-77cef72f9e14\" (UID: \"e2155512-2116-44f6-be5e-77cef72f9e14\") " Nov 03 20:33:32.846838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.845857 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2155512-2116-44f6-be5e-77cef72f9e14-kube-api-access-cjhsn" pod "e2155512-2116-44f6-be5e-77cef72f9e14" (UID: "e2155512-2116-44f6-be5e-77cef72f9e14"). InnerVolumeSpecName "kube-api-access-cjhsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:32.948721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.939333 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjhsn\" (UniqueName: \"kubernetes.io/projected/e2155512-2116-44f6-be5e-77cef72f9e14-kube-api-access-cjhsn\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:32.953938 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:32.953100 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcrh4" podStartSLOduration=90.953062251 podStartE2EDuration="1m30.953062251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:27.923241979 +0000 UTC m=+3402.181039238" watchObservedRunningTime="2026-11-03 20:33:32.953062251 +0000 UTC m=+3407.210859509" Nov 03 20:33:33.049790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:33.043430 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cmtzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.104:15021/healthz/ready\": dial tcp 10.132.1.104:15021: connect: connection refused" start-of-body= Nov 03 20:33:33.049790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:33.043494 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" podUID="a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.104:15021/healthz/ready\": dial tcp 10.132.1.104:15021: connect: connection refused" Nov 03 20:33:33.147970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:33.144943 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" start-of-body= Nov 03 20:33:33.147970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:33.145005 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" podUID="a2c58648-4022-4bfb-be9f-170a6222dfd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" Nov 03 20:33:33.474704 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:33.469051 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx" Nov 03 20:33:33.517286 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:33.517202 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:33.610695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:33.609041 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body= Nov 03 20:33:33.610695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:33.609110 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" podUID="426d1445-9e48-4516-a920-fca04d918f22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" Nov 03 20:33:33.630448 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:33.616157 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h" Nov 03 20:33:33.670378 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:33.637029 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-x7n2b" Nov 03 20:33:33.670378 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:33.658180 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:33.727723 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:33.710740 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" Nov 03 20:33:33.727723 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:33.713407 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:33.733688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:33.732111 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:33.755891 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:33.755858 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:33.756207 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:33.756175 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" event={"ID":"e997605a-f2cc-4ffe-be15-d03219b3b25b","Type":"ContainerStarted","Data":"fd322771dcdddadbf22cfe8ba57e5548a7171a87db9d178ad5e8b09c84395e9b"} Nov 03 20:33:33.756292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:33.756275 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" event={"ID":"e997605a-f2cc-4ffe-be15-d03219b3b25b","Type":"ContainerStarted","Data":"cb376944400b85358eb91951cc698aa63ba76455bb67c197f937eec674cf802c"} Nov 03 20:33:33.792729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:33.790180 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:33.849296 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:33.849182 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" Nov 03 20:33:33.894064 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:33.890030 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg" Nov 03 20:33:34.055718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:34.053526 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q" podStartSLOduration=91.053505491 podStartE2EDuration="1m31.053505491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:28.475525664 +0000 UTC m=+3402.733322922" watchObservedRunningTime="2026-11-03 20:33:34.053505491 +0000 UTC m=+3408.311302750" Nov 03 20:33:34.105978 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:34.056846 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cmtzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.104:15021/healthz/ready\": dial tcp 10.132.1.104:15021: connect: connection refused" start-of-body= Nov 03 20:33:34.106223 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:34.106190 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" podUID="a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.104:15021/healthz/ready\": dial tcp 10.132.1.104:15021: connect: connection refused" Nov 03 20:33:34.136690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:34.128247 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" start-of-body= Nov 03 20:33:34.149737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:34.135380 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:34.150182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:34.150132 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" podUID="a2c58648-4022-4bfb-be9f-170a6222dfd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" Nov 03 20:33:34.443702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:34.401953 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:34.443702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:34.402328 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Nov 03 20:33:34.443702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:34.402398 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" podUID="e997605a-f2cc-4ffe-be15-d03219b3b25b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Nov 03 20:33:34.475006 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:34.474959 5059 generic.go:356] "Generic (PLEG): container finished" podID="6c29d53f-b738-49ec-9b15-1c88e5950008" containerID="ca6370d9da34484b1b31be792a765188027ebdb4619fdaee7682eef377ae36b3" exitCode=22 Nov 03 20:33:34.507785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:34.507750 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:33:34.539688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:34.511509 5059 scope.go:122] "RemoveContainer" containerID="ca6370d9da34484b1b31be792a765188027ebdb4619fdaee7682eef377ae36b3" Nov 03 20:33:34.539931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:34.523200 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47" event={"ID":"6c29d53f-b738-49ec-9b15-1c88e5950008","Type":"ContainerDied","Data":"ca6370d9da34484b1b31be792a765188027ebdb4619fdaee7682eef377ae36b3"} Nov 03 20:33:34.540321 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:34.540297 5059 scope.go:122] "RemoveContainer" containerID="642fce6d48241f7a656bbb89d094312c435e6bbe1d1fd5edeacf6d1e6baccd05" Nov 03 20:33:34.578820 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:34.578767 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-gqz47_odh-ai-gateway-infra(6c29d53f-b738-49ec-9b15-1c88e5950008)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47" podUID="6c29d53f-b738-49ec-9b15-1c88e5950008" Nov 03 20:33:34.610063 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:34.587017 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:33:34.625683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:34.615572 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body= Nov 03 20:33:34.625683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:34.615632 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" podUID="426d1445-9e48-4516-a920-fca04d918f22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" Nov 03 20:33:34.625683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:34.594202 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jt54m" Nov 03 20:33:34.625683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:34.594077 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6hx8t" Nov 03 20:33:35.057450 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:35.027922 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:33:35.182407 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:35.177427 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cmtzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.104:15021/healthz/ready\": dial tcp 10.132.1.104:15021: connect: connection refused" start-of-body= Nov 03 20:33:35.182407 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:35.177488 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" podUID="a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.104:15021/healthz/ready\": dial tcp 10.132.1.104:15021: connect: connection refused" Nov 03 20:33:35.182407 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:35.177965 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" start-of-body= Nov 03 20:33:35.182407 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:35.178018 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" podUID="a2c58648-4022-4bfb-be9f-170a6222dfd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" Nov 03 20:33:35.315737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:35.298911 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Nov 03 20:33:35.364686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:35.351024 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" podUID="e997605a-f2cc-4ffe-be15-d03219b3b25b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Nov 03 20:33:35.586632 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:35.586411 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m" podStartSLOduration=92.586388321 podStartE2EDuration="1m32.586388321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:28.754089384 +0000 UTC m=+3403.011886642" watchObservedRunningTime="2026-11-03 20:33:35.586388321 +0000 UTC m=+3409.844185570" Nov 03 20:33:35.643701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:35.638780 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:35.715073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:35.715012 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body= Nov 03 20:33:35.715314 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:35.715283 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" podUID="426d1445-9e48-4516-a920-fca04d918f22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" Nov 03 20:33:35.850259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:35.850213 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:33:35.979969 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:35.967572 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:33:36.040646 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.038595 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-v5xhc" Nov 03 20:33:36.048166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.045466 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:36.048166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.046839 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cmtzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.104:15021/healthz/ready\": dial tcp 10.132.1.104:15021: connect: connection refused" start-of-body= Nov 03 20:33:36.048166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.046902 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" podUID="a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.104:15021/healthz/ready\": dial tcp 10.132.1.104:15021: connect: connection refused" Nov 03 20:33:36.049651 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.049626 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-4vl4x" Nov 03 20:33:36.049966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.049911 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn" podStartSLOduration=94.049894869 podStartE2EDuration="1m34.049894869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:28.917484062 +0000 UTC m=+3403.175281331" watchObservedRunningTime="2026-11-03 20:33:36.049894869 +0000 UTC m=+3410.307692126" Nov 03 20:33:36.091344 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.091312 5059 scope.go:122] "RemoveContainer" containerID="ca6370d9da34484b1b31be792a765188027ebdb4619fdaee7682eef377ae36b3" Nov 03 20:33:36.137610 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.137582 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:36.143451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.141490 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" podStartSLOduration=93.141469466 podStartE2EDuration="1m33.141469466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:28.969780009 +0000 UTC m=+3403.227577280" watchObservedRunningTime="2026-11-03 20:33:36.141469466 +0000 UTC m=+3410.399266726" Nov 03 20:33:36.148723 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:36.146476 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-gqz47_odh-ai-gateway-infra(6c29d53f-b738-49ec-9b15-1c88e5950008)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47" podUID="6c29d53f-b738-49ec-9b15-1c88e5950008" Nov 03 20:33:36.148723 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.146748 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:33:36.154772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.154581 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:36.167697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.158091 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4" podStartSLOduration=54.158072028 podStartE2EDuration="54.158072028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:29.036564897 +0000 UTC m=+3403.294362169" watchObservedRunningTime="2026-11-03 20:33:36.158072028 +0000 UTC m=+3410.415869286" Nov 03 20:33:36.215134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.192517 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:33:36.215134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.196035 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:33:36.215134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.212298 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:36.231399 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.228332 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:33:36.231399 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.228909 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:36.234761 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.232016 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" podStartSLOduration=93.231999147 podStartE2EDuration="1m33.231999147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:29.090306197 +0000 UTC m=+3403.348103467" watchObservedRunningTime="2026-11-03 20:33:36.231999147 +0000 UTC m=+3410.489796405" Nov 03 20:33:36.234761 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.234585 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:33:36.246545 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.246338 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95" Nov 03 20:33:36.251404 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.251355 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Nov 03 20:33:36.251578 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.251413 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" podUID="e997605a-f2cc-4ffe-be15-d03219b3b25b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Nov 03 20:33:36.322115 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:33:36.302132 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28034528_e342_417a_a9df_050d7ea91453.slice/crio-7553fcdab810b85acc004e15950eef060706caf88e5df5ba2f9e43e9ce29c977 WatchSource:0}: Error finding container 7553fcdab810b85acc004e15950eef060706caf88e5df5ba2f9e43e9ce29c977: Status 404 returned error can't find the container with id 7553fcdab810b85acc004e15950eef060706caf88e5df5ba2f9e43e9ce29c977 Nov 03 20:33:36.359106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.342395 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:33:36.359106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.344943 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:33:36.359106 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:33:36.351703 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1573256_23a1_4a26_9972_87bc9b27f2c1.slice/crio-e295e3f39ed26ad84473c631398dfe509800dfda3f929613c9ca1432d6bb0a75 WatchSource:0}: Error finding container e295e3f39ed26ad84473c631398dfe509800dfda3f929613c9ca1432d6bb0a75: Status 404 returned error can't find the container with id e295e3f39ed26ad84473c631398dfe509800dfda3f929613c9ca1432d6bb0a75 Nov 03 20:33:36.416490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.415962 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4"] Nov 03 20:33:36.447591 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.447535 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557dng2q" podStartSLOduration=93.447515905 podStartE2EDuration="1m33.447515905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:30.418895081 +0000 UTC m=+3404.676692339" watchObservedRunningTime="2026-11-03 20:33:36.447515905 +0000 UTC m=+3410.705313164" Nov 03 20:33:36.504681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.504310 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" podStartSLOduration=93.50429083 podStartE2EDuration="1m33.50429083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:30.623393219 +0000 UTC m=+3404.881190477" watchObservedRunningTime="2026-11-03 20:33:36.50429083 +0000 UTC m=+3410.762088088" Nov 03 20:33:36.548090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.547954 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:36.548090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.548076 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:36.548299 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.548126 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:36.632332 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.631387 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:36.954764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:36.954731 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4"] Nov 03 20:33:37.033845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.033814 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cmtzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.104:15021/healthz/ready\": dial tcp 10.132.1.104:15021: connect: connection refused" start-of-body= Nov 03 20:33:37.034221 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.034004 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" podUID="a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.104:15021/healthz/ready\": dial tcp 10.132.1.104:15021: connect: connection refused" Nov 03 20:33:37.036505 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.035432 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:37.036505 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.035529 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:37.036505 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.035575 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:37.051888 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.050722 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw"] Nov 03 20:33:37.120357 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.120276 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:37.120752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.120369 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:37.120752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.120415 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:37.129036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.124699 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5"] Nov 03 20:33:37.139020 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:33:37.137305 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32deeada_f159_4266_8026_9b0a9ffa5eb1.slice/crio-2c963072abffe28925a29f6dbe75601e449ca3f03d4854604efeed2430e66a17 WatchSource:0}: Error finding container 2c963072abffe28925a29f6dbe75601e449ca3f03d4854604efeed2430e66a17: Status 404 returned error can't find the container with id 2c963072abffe28925a29f6dbe75601e449ca3f03d4854604efeed2430e66a17 Nov 03 20:33:37.249181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.249115 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c57db574-fmm6v"] Nov 03 20:33:37.307535 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.307498 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Nov 03 20:33:37.307639 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.307565 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" podUID="e997605a-f2cc-4ffe-be15-d03219b3b25b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Nov 03 20:33:37.314701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.314001 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6c57db574-fmm6v"] Nov 03 20:33:37.322583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.322498 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" event={"ID":"abce531c-32e5-4fee-b6fa-7b489da29704","Type":"ContainerStarted","Data":"1bfeee2fc9ea6eb7dc1380fb81842c27e5c42d85ed11286b3d86d298a93ef9fb"} Nov 03 20:33:37.336867 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.324727 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn"] Nov 03 20:33:37.336867 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.328049 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h" event={"ID":"28034528-e342-417a-a9df-050d7ea91453","Type":"ContainerStarted","Data":"7553fcdab810b85acc004e15950eef060706caf88e5df5ba2f9e43e9ce29c977"} Nov 03 20:33:37.340336 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.340272 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-x6pbn"] Nov 03 20:33:37.349125 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.349030 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" event={"ID":"9c4b9c59-a980-4ec8-af7b-fc2c08133f19","Type":"ContainerStarted","Data":"4bbb12ec9bb4aa5fd6ab26e02ad9fcff691d47daa518d4e66b7d56093ec2c10d"} Nov 03 20:33:37.350255 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.350201 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq"] Nov 03 20:33:37.351951 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.351762 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx" event={"ID":"847e7b38-8fb0-4c19-8943-f43e364f989b","Type":"ContainerStarted","Data":"e32f190c4a0f1b9ade21ae5829e9ac0f0a81a815dc4a906aabec877a5680692a"} Nov 03 20:33:37.414825 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.414590 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" event={"ID":"b1573256-23a1-4a26-9972-87bc9b27f2c1","Type":"ContainerStarted","Data":"e295e3f39ed26ad84473c631398dfe509800dfda3f929613c9ca1432d6bb0a75"} Nov 03 20:33:37.427771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.427700 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-lfqvq"] Nov 03 20:33:37.479687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.449727 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p"] Nov 03 20:33:37.511688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.510406 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-6487p"] Nov 03 20:33:37.515304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.514423 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:33:37.542496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.542449 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx"] Nov 03 20:33:37.607299 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.605770 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-6gnsx"] Nov 03 20:33:37.718775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.718739 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44"] Nov 03 20:33:37.734536 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.733456 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-48j44"] Nov 03 20:33:37.744540 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.744314 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl"] Nov 03 20:33:37.752975 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.752600 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4" Nov 03 20:33:37.754298 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.754269 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-n8krl"] Nov 03 20:33:37.811987 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.811956 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2"] Nov 03 20:33:37.827769 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.827742 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-4dtv2"] Nov 03 20:33:37.836480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.836432 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj"] Nov 03 20:33:37.843380 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.843318 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-rpkpj"] Nov 03 20:33:37.853942 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.851689 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w"] Nov 03 20:33:37.853942 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.852378 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc10f92e-e902-48c4-9cd5-4f768c4611be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc10f92e-e902-48c4-9cd5-4f768c4611be-maas-api-openshift-ca\") pod \"dc10f92e-e902-48c4-9cd5-4f768c4611be\" (UID: \"dc10f92e-e902-48c4-9cd5-4f768c4611be\") " Nov 03 20:33:37.853942 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.852542 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc10f92e-e902-48c4-9cd5-4f768c4611be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc10f92e-e902-48c4-9cd5-4f768c4611be-maas-api-xks-ca\") pod \"dc10f92e-e902-48c4-9cd5-4f768c4611be\" (UID: \"dc10f92e-e902-48c4-9cd5-4f768c4611be\") " Nov 03 20:33:37.853942 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.852599 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc10f92e-e902-48c4-9cd5-4f768c4611be-kube-api-access-z79fs\" (UniqueName: \"kubernetes.io/projected/dc10f92e-e902-48c4-9cd5-4f768c4611be-kube-api-access-z79fs\") pod \"dc10f92e-e902-48c4-9cd5-4f768c4611be\" (UID: \"dc10f92e-e902-48c4-9cd5-4f768c4611be\") " Nov 03 20:33:37.855465 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.855398 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc10f92e-e902-48c4-9cd5-4f768c4611be-maas-api-openshift-ca" pod "dc10f92e-e902-48c4-9cd5-4f768c4611be" (UID: "dc10f92e-e902-48c4-9cd5-4f768c4611be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:37.971098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.955322 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc10f92e-e902-48c4-9cd5-4f768c4611be-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:37.971098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.958124 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc10f92e-e902-48c4-9cd5-4f768c4611be-kube-api-access-z79fs" pod "dc10f92e-e902-48c4-9cd5-4f768c4611be" (UID: "dc10f92e-e902-48c4-9cd5-4f768c4611be"). InnerVolumeSpecName "kube-api-access-z79fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:37.971098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:37.961219 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc10f92e-e902-48c4-9cd5-4f768c4611be-maas-api-xks-ca" pod "dc10f92e-e902-48c4-9cd5-4f768c4611be" (UID: "dc10f92e-e902-48c4-9cd5-4f768c4611be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:38.036130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.034461 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cmtzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.104:15021/healthz/ready\": dial tcp 10.132.1.104:15021: connect: connection refused" start-of-body= Nov 03 20:33:38.036130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.034518 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" podUID="a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.104:15021/healthz/ready\": dial tcp 10.132.1.104:15021: connect: connection refused" Nov 03 20:33:38.067345 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.057400 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc10f92e-e902-48c4-9cd5-4f768c4611be-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:38.067345 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.057432 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z79fs\" (UniqueName: \"kubernetes.io/projected/dc10f92e-e902-48c4-9cd5-4f768c4611be-kube-api-access-z79fs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:38.108097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.103386 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:38.108097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.103509 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:38.108097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.103558 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:38.113930 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.113494 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08b3a993-1a64-4fe5-9577-b0f11a36779e" path="/var/lib/kubelet/pods/08b3a993-1a64-4fe5-9577-b0f11a36779e/volumes" Nov 03 20:33:38.114768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.114613 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d960e4e-9539-4a53-9df7-1c520ab7d708" path="/var/lib/kubelet/pods/0d960e4e-9539-4a53-9df7-1c520ab7d708/volumes" Nov 03 20:33:38.116988 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.115624 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26375297-81f4-427f-94c3-d56f7a6a11fb" path="/var/lib/kubelet/pods/26375297-81f4-427f-94c3-d56f7a6a11fb/volumes" Nov 03 20:33:38.118071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.117198 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ed21c5c-ef42-4fb0-bf41-34e84b9cf193" path="/var/lib/kubelet/pods/2ed21c5c-ef42-4fb0-bf41-34e84b9cf193/volumes" Nov 03 20:33:38.121510 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.121491 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3265bd58-d654-467c-9b2a-14cedc27b754" path="/var/lib/kubelet/pods/3265bd58-d654-467c-9b2a-14cedc27b754/volumes" Nov 03 20:33:38.122624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.122594 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c3d72b6-73c4-402f-8308-ee5d87a9dd83" path="/var/lib/kubelet/pods/9c3d72b6-73c4-402f-8308-ee5d87a9dd83/volumes" Nov 03 20:33:38.123445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.123425 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51" path="/var/lib/kubelet/pods/ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51/volumes" Nov 03 20:33:38.124161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.124140 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfcb49c5-4381-411e-9d64-d9dcd6181239" path="/var/lib/kubelet/pods/cfcb49c5-4381-411e-9d64-d9dcd6181239/volumes" Nov 03 20:33:38.124717 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.124342 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4" Nov 03 20:33:38.125656 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.125340 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3de18b3-c3d1-4ec3-961f-d95ca3656f4b" path="/var/lib/kubelet/pods/d3de18b3-c3d1-4ec3-961f-d95ca3656f4b/volumes" Nov 03 20:33:38.126213 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.126083 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d2aa3b-openshift-default-6d55565c69-7rr6w"] Nov 03 20:33:38.126213 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.126135 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h"] Nov 03 20:33:38.126213 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.126179 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-s7r4h"] Nov 03 20:33:38.126213 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.126202 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt"] Nov 03 20:33:38.126448 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.126223 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-fx8mt"] Nov 03 20:33:38.126448 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.126244 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r"] Nov 03 20:33:38.140148 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.137908 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rmz2r"] Nov 03 20:33:38.151765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.148943 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8"] Nov 03 20:33:38.176776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.172937 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-wr8l8"] Nov 03 20:33:38.301010 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.274254 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Nov 03 20:33:38.301010 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.274316 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" podUID="e997605a-f2cc-4ffe-be15-d03219b3b25b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Nov 03 20:33:38.301010 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.293505 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1326708b-4642-4992-91ea-8a590863a703-kube-api-access-9nkdn\" (UniqueName: \"kubernetes.io/projected/1326708b-4642-4992-91ea-8a590863a703-kube-api-access-9nkdn\") pod \"1326708b-4642-4992-91ea-8a590863a703\" (UID: \"1326708b-4642-4992-91ea-8a590863a703\") " Nov 03 20:33:38.301010 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.293573 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1326708b-4642-4992-91ea-8a590863a703-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1326708b-4642-4992-91ea-8a590863a703-maas-api-openshift-ca\") pod \"1326708b-4642-4992-91ea-8a590863a703\" (UID: \"1326708b-4642-4992-91ea-8a590863a703\") " Nov 03 20:33:38.301010 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.293708 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1326708b-4642-4992-91ea-8a590863a703-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1326708b-4642-4992-91ea-8a590863a703-maas-api-xks-ca\") pod \"1326708b-4642-4992-91ea-8a590863a703\" (UID: \"1326708b-4642-4992-91ea-8a590863a703\") " Nov 03 20:33:38.310425 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.310394 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm"] Nov 03 20:33:38.310919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.310886 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1326708b-4642-4992-91ea-8a590863a703-maas-api-openshift-ca" pod "1326708b-4642-4992-91ea-8a590863a703" (UID: "1326708b-4642-4992-91ea-8a590863a703"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:38.335406 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.334758 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1326708b-4642-4992-91ea-8a590863a703-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:38.337042 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.337020 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5" Nov 03 20:33:38.346960 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.346909 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vznnm"] Nov 03 20:33:38.368653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.364168 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps"] Nov 03 20:33:38.403626 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.403470 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1326708b-4642-4992-91ea-8a590863a703-kube-api-access-9nkdn" pod "1326708b-4642-4992-91ea-8a590863a703" (UID: "1326708b-4642-4992-91ea-8a590863a703"). InnerVolumeSpecName "kube-api-access-9nkdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:38.413143 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.413089 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-tchps"] Nov 03 20:33:38.416014 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:33:38.415983 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9069949b_78ef_42cc_99b0_1feffc84b679.slice/crio-eccf70e6270bc299851bddc769809e6c8081a6a29606df42984f367920b3621e WatchSource:0}: Error finding container eccf70e6270bc299851bddc769809e6c8081a6a29606df42984f367920b3621e: Status 404 returned error can't find the container with id eccf70e6270bc299851bddc769809e6c8081a6a29606df42984f367920b3621e Nov 03 20:33:38.416182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.416009 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1326708b-4642-4992-91ea-8a590863a703-maas-api-xks-ca" pod "1326708b-4642-4992-91ea-8a590863a703" (UID: "1326708b-4642-4992-91ea-8a590863a703"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:38.426528 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:33:38.426500 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50891bc_16c2_497d_8e00_b79ef492ab6e.slice/crio-ee94243f3de6f885a9faa9f81770f87a0b7b46507e5de1949692c0bcdb0f2bad WatchSource:0}: Error finding container ee94243f3de6f885a9faa9f81770f87a0b7b46507e5de1949692c0bcdb0f2bad: Status 404 returned error can't find the container with id ee94243f3de6f885a9faa9f81770f87a0b7b46507e5de1949692c0bcdb0f2bad Nov 03 20:33:38.428752 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:33:38.428722 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eda42a5_5b61_4b34_8447_8767ed097200.slice/crio-5b8c83ec24a2946aa3e8001a3b660835c9b3e7c2ec1dfc3bc8aa2c6464424b71 WatchSource:0}: Error finding container 5b8c83ec24a2946aa3e8001a3b660835c9b3e7c2ec1dfc3bc8aa2c6464424b71: Status 404 returned error can't find the container with id 5b8c83ec24a2946aa3e8001a3b660835c9b3e7c2ec1dfc3bc8aa2c6464424b71 Nov 03 20:33:38.432313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.432287 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7"] Nov 03 20:33:38.437101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.435789 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b9428a2-ab50-4e44-a725-e0da980bdb93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9428a2-ab50-4e44-a725-e0da980bdb93-maas-api-xks-ca\") pod \"6b9428a2-ab50-4e44-a725-e0da980bdb93\" (UID: \"6b9428a2-ab50-4e44-a725-e0da980bdb93\") " Nov 03 20:33:38.437101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.435853 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b9428a2-ab50-4e44-a725-e0da980bdb93-kube-api-access-dshjq\" (UniqueName: \"kubernetes.io/projected/6b9428a2-ab50-4e44-a725-e0da980bdb93-kube-api-access-dshjq\") pod \"6b9428a2-ab50-4e44-a725-e0da980bdb93\" (UID: \"6b9428a2-ab50-4e44-a725-e0da980bdb93\") " Nov 03 20:33:38.437101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.436073 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b9428a2-ab50-4e44-a725-e0da980bdb93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9428a2-ab50-4e44-a725-e0da980bdb93-maas-api-openshift-ca\") pod \"6b9428a2-ab50-4e44-a725-e0da980bdb93\" (UID: \"6b9428a2-ab50-4e44-a725-e0da980bdb93\") " Nov 03 20:33:38.437101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.436841 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nkdn\" (UniqueName: \"kubernetes.io/projected/1326708b-4642-4992-91ea-8a590863a703-kube-api-access-9nkdn\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:38.437101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.436870 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1326708b-4642-4992-91ea-8a590863a703-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:38.437870 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.437549 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b9428a2-ab50-4e44-a725-e0da980bdb93-maas-api-openshift-ca" pod "6b9428a2-ab50-4e44-a725-e0da980bdb93" (UID: "6b9428a2-ab50-4e44-a725-e0da980bdb93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:38.448741 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:33:38.448652 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569b2103_147f_4ea1_9b55_0e859b1c5e52.slice/crio-0e46c7c45209747169d06bd8d48946c023e60bcdc2467de6d71bf9b81bf509fd WatchSource:0}: Error finding container 0e46c7c45209747169d06bd8d48946c023e60bcdc2467de6d71bf9b81bf509fd: Status 404 returned error can't find the container with id 0e46c7c45209747169d06bd8d48946c023e60bcdc2467de6d71bf9b81bf509fd Nov 03 20:33:38.448951 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.448924 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b9428a2-ab50-4e44-a725-e0da980bdb93-maas-api-xks-ca" pod "6b9428a2-ab50-4e44-a725-e0da980bdb93" (UID: "6b9428a2-ab50-4e44-a725-e0da980bdb93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:38.456884 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.456855 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g6xz7"] Nov 03 20:33:38.462046 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.459427 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:38.462046 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.459545 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:38.462046 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.459593 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:38.503548 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.502905 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:38.503548 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.503022 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:38.503548 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.503072 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:38.511729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.511639 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:38.512821 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.511758 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:38.512821 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.511805 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:38.540174 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.540147 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9428a2-ab50-4e44-a725-e0da980bdb93-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:38.540312 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.540294 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9428a2-ab50-4e44-a725-e0da980bdb93-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:38.541549 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.541521 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b9428a2-ab50-4e44-a725-e0da980bdb93-kube-api-access-dshjq" pod "6b9428a2-ab50-4e44-a725-e0da980bdb93" (UID: "6b9428a2-ab50-4e44-a725-e0da980bdb93"). InnerVolumeSpecName "kube-api-access-dshjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:38.617144 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.552741 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx"] Nov 03 20:33:38.619689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.619650 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4" Nov 03 20:33:38.619891 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.619856 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4" event={"ID":"1326708b-4642-4992-91ea-8a590863a703","Type":"ContainerDied","Data":"b79ef057628756807c50f783b8a0e808e8ec00460d8b0941097f753f392a1bf3"} Nov 03 20:33:38.619951 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.619928 5059 scope.go:122] "RemoveContainer" containerID="a350ab9162cb33074fa8120cd07c0f5b1f85de58b5831dbb73b25fac3f1122a6" Nov 03 20:33:38.629923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.629900 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" event={"ID":"9bff4885-a4dc-4b17-b596-e745e86fbccd","Type":"ContainerStarted","Data":"b37f042b2b7ad71eb9819f0dbd537f0d279768df5494387b93bed653d485896b"} Nov 03 20:33:38.633947 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.633853 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" event={"ID":"89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5","Type":"ContainerStarted","Data":"c1c2d58153f7854808b6ee2a057a380278167cec0967385719b722e0f2dad7ce"} Nov 03 20:33:38.636817 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.636058 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4" event={"ID":"dc10f92e-e902-48c4-9cd5-4f768c4611be","Type":"ContainerDied","Data":"192cded1132012fbc6e17035522a365a76221127162bbd64c7c3d1d27b94d412"} Nov 03 20:33:38.636817 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.636167 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4" Nov 03 20:33:38.649411 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.649262 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jt54m"] Nov 03 20:33:38.649411 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.649301 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg" event={"ID":"aeb32457-81a5-4505-b189-2a98f5fda7da","Type":"ContainerStarted","Data":"f9e9ea28bc7e11bb18a879b11400bbe3b69a5bbc6f66b5ffe248c798052afb31"} Nov 03 20:33:38.649411 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.649336 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" event={"ID":"abce531c-32e5-4fee-b6fa-7b489da29704","Type":"ContainerStarted","Data":"b45a6bff1af76d642a374e7fb94b73d65da2201d87f7ddea070a688d25f2fb96"} Nov 03 20:33:38.649411 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.649364 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" event={"ID":"9c4b9c59-a980-4ec8-af7b-fc2c08133f19","Type":"ContainerStarted","Data":"3e8c7d9a4d245a077d7d5d91682130d61c0ca287bcd31fe821677ff75b460f32"} Nov 03 20:33:38.655142 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.655116 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dshjq\" (UniqueName: \"kubernetes.io/projected/6b9428a2-ab50-4e44-a725-e0da980bdb93-kube-api-access-dshjq\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:38.655473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.655444 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx" event={"ID":"847e7b38-8fb0-4c19-8943-f43e364f989b","Type":"ContainerStarted","Data":"6638b2c26dcbd2beddb822cddbf45294c8f85f17724b2ee963373d4a3c6be587"} Nov 03 20:33:38.674705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.670235 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:38.715117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.715066 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" event={"ID":"3eda42a5-5b61-4b34-8447-8767ed097200","Type":"ContainerStarted","Data":"5b8c83ec24a2946aa3e8001a3b660835c9b3e7c2ec1dfc3bc8aa2c6464424b71"} Nov 03 20:33:38.717711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.717647 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" event={"ID":"569b2103-147f-4ea1-9b55-0e859b1c5e52","Type":"ContainerStarted","Data":"0e46c7c45209747169d06bd8d48946c023e60bcdc2467de6d71bf9b81bf509fd"} Nov 03 20:33:38.731625 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.731594 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-4vl4x" event={"ID":"94e83c10-58b0-4bc7-9abd-3e144ba6e258","Type":"ContainerStarted","Data":"16c66826c64dd74f6d30973c00c34771034c241ce9bba4601580e67e603b88b9"} Nov 03 20:33:38.731800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.731779 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" event={"ID":"9069949b-78ef-42cc-99b0-1feffc84b679","Type":"ContainerStarted","Data":"eccf70e6270bc299851bddc769809e6c8081a6a29606df42984f367920b3621e"} Nov 03 20:33:38.732071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.732051 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" event={"ID":"b1573256-23a1-4a26-9972-87bc9b27f2c1","Type":"ContainerStarted","Data":"f9261494f566e526c65e3e2a7bea4c6b25d6474679613298eca55296a266371f"} Nov 03 20:33:38.733099 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.733073 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-v5xhc" event={"ID":"d50891bc-16c2-497d-8e00-b79ef492ab6e","Type":"ContainerStarted","Data":"ee94243f3de6f885a9faa9f81770f87a0b7b46507e5de1949692c0bcdb0f2bad"} Nov 03 20:33:38.748631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.748274 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:38.871687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.866568 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:38.945431 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.945391 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:38.953784 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.953707 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:38.954506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.953492 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5" event={"ID":"6b9428a2-ab50-4e44-a725-e0da980bdb93","Type":"ContainerDied","Data":"525f6d3424c4cc255ff7e85b227e91e40f3d49c08a0875d0fdde86eb52893da8"} Nov 03 20:33:38.954683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:38.953588 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5" Nov 03 20:33:39.020767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.020028 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-x7n2b" event={"ID":"d9f37244-d91a-480d-baa1-b1d59335804c","Type":"ContainerStarted","Data":"59f6b36aa2cdf0e0f09d1d9a0cb90ef333dc2d1db57858df366be1acafd9aeb2"} Nov 03 20:33:39.027276 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.027248 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jt54m"] Nov 03 20:33:39.031090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.030557 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:39.098300 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.098266 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w"] Nov 03 20:33:39.117367 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.117330 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-4892w"] Nov 03 20:33:39.117645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.117628 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55"] Nov 03 20:33:39.126764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.126739 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-mgx55"] Nov 03 20:33:39.141746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.140862 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn"] Nov 03 20:33:39.145556 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.142310 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" event={"ID":"32deeada-f159-4266-8026-9b0a9ffa5eb1","Type":"ContainerStarted","Data":"2c963072abffe28925a29f6dbe75601e449ca3f03d4854604efeed2430e66a17"} Nov 03 20:33:39.145689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.145645 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q" Nov 03 20:33:39.147474 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.147450 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-2tvmn"] Nov 03 20:33:39.238082 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.237993 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bd8f8"] Nov 03 20:33:39.247838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.247818 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bd8f8"] Nov 03 20:33:39.317515 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.317472 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Nov 03 20:33:39.317695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.317552 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" podUID="e997605a-f2cc-4ffe-be15-d03219b3b25b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Nov 03 20:33:39.318929 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:33:39.318895 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef66ce6_8b55_4ef9_8dce_fdfad6b1edb8.slice/crio-7d87b96d63a6638c8038b372010e4b7d308315e17a3c3fb4ebb4d870363b129a WatchSource:0}: Error finding container 7d87b96d63a6638c8038b372010e4b7d308315e17a3c3fb4ebb4d870363b129a: Status 404 returned error can't find the container with id 7d87b96d63a6638c8038b372010e4b7d308315e17a3c3fb4ebb4d870363b129a Nov 03 20:33:39.342043 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.342003 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22"] Nov 03 20:33:39.344087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.344055 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-r9p22"] Nov 03 20:33:39.402292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.391007 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:39.402292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.391152 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:39.402292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.391226 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:39.472687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.457798 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" event={"ID":"19ab8a5f-fa5f-4548-bf5f-0465281d8677","Type":"ContainerStarted","Data":"ca46816d93d3647be27f1def6097d494f92d55d75742712c07da2fe44f6fc3cd"} Nov 03 20:33:39.553968 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.549420 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:39.553968 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.553217 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Nov 03 20:33:39.553968 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.553273 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" podUID="b1573256-23a1-4a26-9972-87bc9b27f2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Nov 03 20:33:39.557436 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.557154 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:39.557436 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.557249 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:39.557436 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.557301 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:39.814925 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.799972 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body= Nov 03 20:33:39.814925 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.800033 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" podUID="9c4b9c59-a980-4ec8-af7b-fc2c08133f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" Nov 03 20:33:39.814925 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.802283 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6hx8t"] Nov 03 20:33:39.821748 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.821707 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body= Nov 03 20:33:39.821985 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:39.821952 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" podUID="abce531c-32e5-4fee-b6fa-7b489da29704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" Nov 03 20:33:40.181686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.177841 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cmtzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:33:40.181686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.177920 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" podUID="a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:40.211334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.204465 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:40.211334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.204612 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:40.211334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.204684 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:40.211334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.206001 5059 scope.go:122] "RemoveContainer" containerID="3944dcf9452420ea8a9d89d52b744632d0da3464e6e4103dbf789969612761cf" Nov 03 20:33:40.272838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.272804 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Nov 03 20:33:40.273136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.273106 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" podUID="e997605a-f2cc-4ffe-be15-d03219b3b25b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Nov 03 20:33:40.334689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.294836 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:40.334689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.294952 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:40.334689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.294997 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:40.587587 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.587550 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f32ecf7-06f7-4098-9793-acf761a30c09" path="/var/lib/kubelet/pods/0f32ecf7-06f7-4098-9793-acf761a30c09/volumes" Nov 03 20:33:40.609721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.597344 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fc518fc-6302-4a1a-9ff9-f1e036a2199b" path="/var/lib/kubelet/pods/0fc518fc-6302-4a1a-9ff9-f1e036a2199b/volumes" Nov 03 20:33:40.618693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.610891 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" path="/var/lib/kubelet/pods/16c463fa-8ddf-43c8-b00f-3f5f9ca33c02/volumes" Nov 03 20:33:40.637051 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.637023 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="171c7288-3157-4fe0-80ee-94cf5b2ca2dd" path="/var/lib/kubelet/pods/171c7288-3157-4fe0-80ee-94cf5b2ca2dd/volumes" Nov 03 20:33:40.648696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.638087 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1731953b-1850-40d1-9ba4-cb08dc4ebc5d" path="/var/lib/kubelet/pods/1731953b-1850-40d1-9ba4-cb08dc4ebc5d/volumes" Nov 03 20:33:40.656692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.649785 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" path="/var/lib/kubelet/pods/20690fd2-7022-4657-8c8c-bd738493ffc0/volumes" Nov 03 20:33:40.670691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.657783 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2faa14fa-ec86-4633-a968-c3f761d13443" path="/var/lib/kubelet/pods/2faa14fa-ec86-4633-a968-c3f761d13443/volumes" Nov 03 20:33:40.679693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.671853 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="705dea2b-ad85-4b1c-a3c6-a9f705ac9039" path="/var/lib/kubelet/pods/705dea2b-ad85-4b1c-a3c6-a9f705ac9039/volumes" Nov 03 20:33:40.690692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.680827 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8153e018-7f9a-4645-b102-06ff3913f682" path="/var/lib/kubelet/pods/8153e018-7f9a-4645-b102-06ff3913f682/volumes" Nov 03 20:33:40.703696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.691938 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" path="/var/lib/kubelet/pods/ade5b9f4-d822-4418-a31b-f0c0a21ddb0c/volumes" Nov 03 20:33:40.752687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.736920 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c92b1be0-0890-4db6-ac75-968b6e5a3434" path="/var/lib/kubelet/pods/c92b1be0-0890-4db6-ac75-968b6e5a3434/volumes" Nov 03 20:33:40.752687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.737798 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2155512-2116-44f6-be5e-77cef72f9e14" path="/var/lib/kubelet/pods/e2155512-2116-44f6-be5e-77cef72f9e14/volumes" Nov 03 20:33:40.752687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.742644 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f14445d5-52ed-487d-972c-4499966065fa" path="/var/lib/kubelet/pods/f14445d5-52ed-487d-972c-4499966065fa/volumes" Nov 03 20:33:40.752687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.743465 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc57130b-c950-49f2-8a98-22c47bd98288" path="/var/lib/kubelet/pods/fc57130b-c950-49f2-8a98-22c47bd98288/volumes" Nov 03 20:33:40.752687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.744042 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:40.752687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.744089 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6hx8t"] Nov 03 20:33:40.752687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.699092 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body= Nov 03 20:33:40.752687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.745275 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" podUID="9c4b9c59-a980-4ec8-af7b-fc2c08133f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" Nov 03 20:33:40.761931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.761892 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Nov 03 20:33:40.762190 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.762146 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" podUID="b1573256-23a1-4a26-9972-87bc9b27f2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Nov 03 20:33:40.776158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.776110 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body= Nov 03 20:33:40.792690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.779137 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" podUID="abce531c-32e5-4fee-b6fa-7b489da29704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" Nov 03 20:33:40.792690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.776371 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l"] Nov 03 20:33:40.792690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.791230 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tf75l"] Nov 03 20:33:40.962740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.936887 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:40.962740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.936988 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:40.962740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.937035 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:40.962740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.941770 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:40.962740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.941858 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:40.962740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.941904 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:40.962740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.942626 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:40.962740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.942714 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:40.962740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.942777 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:40.962740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.943516 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:40.962740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.943585 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:40.962740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.943629 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:40.962740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.944343 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:40.962740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.944433 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:40.962740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.944478 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:40.962740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.946081 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:40.962740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.946146 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:40.962740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:40.946198 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:41.196714 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:41.192409 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:33:41.274891 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:41.222778 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" event={"ID":"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8","Type":"ContainerStarted","Data":"7d87b96d63a6638c8038b372010e4b7d308315e17a3c3fb4ebb4d870363b129a"} Nov 03 20:33:41.274891 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:41.230145 5059 patch_prober.go:29] interesting pod/router-default-f7b87d695-cwlfq container/router namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]backend-proxy-http ok Nov 03 20:33:41.274891 ip-10-0-132-83 kubenswrapper[5059]: [+]has-synced ok Nov 03 20:33:41.274891 ip-10-0-132-83 kubenswrapper[5059]: [-]process-running failed: reason withheld Nov 03 20:33:41.274891 ip-10-0-132-83 kubenswrapper[5059]: healthz check failed Nov 03 20:33:41.274891 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:41.230230 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-f7b87d695-cwlfq" podUID="603868b0-8628-4227-8471-1894e2e1de50" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Nov 03 20:33:41.274891 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:41.238261 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m" Nov 03 20:33:41.274891 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:41.242461 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:41.274891 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:41.242608 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:41.274891 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:41.242741 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:41.315705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:41.298869 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth"] Nov 03 20:33:41.366585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:41.316216 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:41.366585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:41.316563 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:41.366585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:41.342122 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:41.580686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:41.574626 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Nov 03 20:33:41.580686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:41.574726 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" podUID="b1573256-23a1-4a26-9972-87bc9b27f2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Nov 03 20:33:41.647161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:41.623926 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h"] Nov 03 20:33:41.647161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:41.624097 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:41.647161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:41.624138 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" event={"ID":"ecab4d7a-4034-4b59-8500-8434baacafaf","Type":"ContainerStarted","Data":"4890a01b7c4abe7afbebae6c8bf2144e78a951720b341c57c7f308881ee37284"} Nov 03 20:33:41.654647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:41.654618 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:41.701754 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:41.691868 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:33:41.779260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:41.779173 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:41.780260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:41.780211 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:41.780586 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:41.780537 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:33:41.876046 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:41.875975 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" event={"ID":"d75a00ce-f97c-431f-be9e-aac200bcd9d4","Type":"ContainerStarted","Data":"515ce9f9064fcd016304283348824753b282aee6ca435c32656208c0c270345e"} Nov 03 20:33:42.251679 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:42.251599 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body= Nov 03 20:33:42.252325 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:42.252290 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" podUID="abce531c-32e5-4fee-b6fa-7b489da29704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" Nov 03 20:33:42.253031 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:42.253007 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body= Nov 03 20:33:42.253203 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:42.253180 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" podUID="9c4b9c59-a980-4ec8-af7b-fc2c08133f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" Nov 03 20:33:42.559287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:42.558909 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Nov 03 20:33:42.559560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:42.559523 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" podUID="b1573256-23a1-4a26-9972-87bc9b27f2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Nov 03 20:33:42.744372 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:42.725161 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body= Nov 03 20:33:42.744372 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:42.725219 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" podUID="9c4b9c59-a980-4ec8-af7b-fc2c08133f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" Nov 03 20:33:42.880690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:42.876849 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body= Nov 03 20:33:42.880926 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:42.880890 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" podUID="abce531c-32e5-4fee-b6fa-7b489da29704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" Nov 03 20:33:43.024686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.023902 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" path="/var/lib/kubelet/pods/cde430c2-2518-49a0-9851-42356d3f2fe9/volumes" Nov 03 20:33:43.024686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.024564 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9d935b2-aa51-474c-836d-a74f91efab07" path="/var/lib/kubelet/pods/f9d935b2-aa51-474c-836d-a74f91efab07/volumes" Nov 03 20:33:43.029702 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:43.025065 5059 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.122s" Nov 03 20:33:43.029702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.025104 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h" event={"ID":"28034528-e342-417a-a9df-050d7ea91453","Type":"ContainerStarted","Data":"c63223667135eb8a17f59b5174cf095c231af49811bd42e93ad4f78c3ee6ba55"} Nov 03 20:33:43.029702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.025157 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" event={"ID":"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507","Type":"ContainerStarted","Data":"34a6b94a9819202c88c3017986397f0bc71ff236ace7b870b4731c004cf9c609"} Nov 03 20:33:43.029702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.025307 5059 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" podUID="60b2ec36-7e99-46e0-9671-1c500c402e45" containerName="main" probeResult="failure" output="Get \"https://10.132.1.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:43.130052 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.129953 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4"] Nov 03 20:33:43.223156 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.223111 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" event={"ID":"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06","Type":"ContainerStarted","Data":"6f4798010a65daf71ca385fbaced28e5f1e5a941bca18ebf0548a3408aca06ed"} Nov 03 20:33:43.274841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.274172 5059 scope.go:122] "RemoveContainer" containerID="ccbe96e26db6f56ce351a7e5d4b629564431bc79e018b93b657c373a6b88a2e6" Nov 03 20:33:43.274841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.274279 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-qhvz4"] Nov 03 20:33:43.274841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.274323 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" event={"ID":"7101a242-11ce-4b8d-9a74-dae3baf6444b","Type":"ContainerStarted","Data":"2b33a04c0eadd1f9ac6712cc7a89380f25b0d7ef362dd8f03c2107365dee1e3b"} Nov 03 20:33:43.458822 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.458729 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95"] Nov 03 20:33:43.569199 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.565090 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-dhq95"] Nov 03 20:33:43.579522 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.579464 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Nov 03 20:33:43.588703 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.586728 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" podUID="b1573256-23a1-4a26-9972-87bc9b27f2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Nov 03 20:33:43.684055 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.663288 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5"] Nov 03 20:33:43.752686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.742808 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" event={"ID":"7f438514-df13-4871-9c2a-ef8e56a570c9","Type":"ContainerStarted","Data":"554b28fb563761ef01e05125a4f730f1a30ede2e8f438cb7d70a6f254e87ef31"} Nov 03 20:33:43.752686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.742879 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" event={"ID":"6b615028-a2db-4ad0-8a57-53efc040d6a6","Type":"ContainerStarted","Data":"480f3eb485ec680fca7451f936b9b0b3e05c116689a84694829e2750d5d7ca6c"} Nov 03 20:33:43.752686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.742907 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" event={"ID":"e2fe4277-3acc-4378-86d9-374b69918ec9","Type":"ContainerStarted","Data":"ea085d2ffafa8646c924f9e8777a5c0a04f6ac65639f507368d7fc95cd47e35d"} Nov 03 20:33:43.752686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.745769 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body= Nov 03 20:33:43.752686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.745825 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" podUID="abce531c-32e5-4fee-b6fa-7b489da29704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" Nov 03 20:33:43.752686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.746250 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body= Nov 03 20:33:43.752686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.746291 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" podUID="9c4b9c59-a980-4ec8-af7b-fc2c08133f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" Nov 03 20:33:43.843021 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.828488 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:43.843021 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.828569 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rhvd5"] Nov 03 20:33:43.872759 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.865088 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp"] Nov 03 20:33:43.903762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.900478 5059 scope.go:122] "RemoveContainer" containerID="ca6370d9da34484b1b31be792a765188027ebdb4619fdaee7682eef377ae36b3" Nov 03 20:33:43.956636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.956551 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Nov 03 20:33:43.956896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:43.956863 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Nov 03 20:33:44.099690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.015623 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" podStartSLOduration=101.015603344 podStartE2EDuration="1m41.015603344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:36.374145003 +0000 UTC m=+3410.631942261" watchObservedRunningTime="2026-11-03 20:33:44.015603344 +0000 UTC m=+3418.273400604" Nov 03 20:33:44.149917 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.149878 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" path="/var/lib/kubelet/pods/1a57d851-12b3-4535-b12c-b7d2c579d0fb/volumes" Nov 03 20:33:44.237352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.237268 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" path="/var/lib/kubelet/pods/541bc63e-66f1-42ca-a946-aedc34beb62f/volumes" Nov 03 20:33:44.292265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.267943 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2d80995-5f1c-46db-8d71-a130cb92e279" path="/var/lib/kubelet/pods/c2d80995-5f1c-46db-8d71-a130cb92e279/volumes" Nov 03 20:33:44.348858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.339370 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-x7n2b" Nov 03 20:33:44.348858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.339469 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" Nov 03 20:33:44.348858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.339501 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8whwkp"] Nov 03 20:33:44.348858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.339529 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-x7n2b" event={"ID":"d9f37244-d91a-480d-baa1-b1d59335804c","Type":"ContainerStarted","Data":"c1603b1f59da6f8172c4899dfac19d80eb33ccf71fe0aa9f2f940c42ea19db83"} Nov 03 20:33:44.348858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.339560 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" event={"ID":"9bff4885-a4dc-4b17-b596-e745e86fbccd","Type":"ContainerStarted","Data":"f98ae9867fc87a679d4159cab410d0f2e75a2ede6c5a0c454bba5459fd1c6a54"} Nov 03 20:33:44.348858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.339587 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" event={"ID":"75efd589-b943-46c2-add0-d44ccdaff537","Type":"ContainerStarted","Data":"7733b8a23d9b015a70236a5b6383a704b97047c69c34cafeea1d149c9885c552"} Nov 03 20:33:44.348858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.339627 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz"] Nov 03 20:33:44.530304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.529805 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Nov 03 20:33:44.530304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.529868 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" podUID="b1573256-23a1-4a26-9972-87bc9b27f2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Nov 03 20:33:44.715436 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.713204 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body= Nov 03 20:33:44.715436 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.713267 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" podUID="9c4b9c59-a980-4ec8-af7b-fc2c08133f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" Nov 03 20:33:44.740765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.740730 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:44.775334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.774795 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" start-of-body= Nov 03 20:33:44.775334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.774856 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" podUID="9bff4885-a4dc-4b17-b596-e745e86fbccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" Nov 03 20:33:44.775654 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.775631 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body= Nov 03 20:33:44.775819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.775790 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" podUID="abce531c-32e5-4fee-b6fa-7b489da29704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" Nov 03 20:33:44.805262 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.804768 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Nov 03 20:33:44.805262 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.804826 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Nov 03 20:33:44.833644 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.833589 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl"] Nov 03 20:33:44.854470 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.854431 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv"] Nov 03 20:33:44.854742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.854723 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg"] Nov 03 20:33:44.868303 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.868272 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-mnmjc"] Nov 03 20:33:44.891720 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.884402 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252"] Nov 03 20:33:44.937939 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:44.937909 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d9f95b69-x7n2b"] Nov 03 20:33:45.722990 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:45.719960 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body= Nov 03 20:33:45.722990 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:45.720036 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" podUID="9c4b9c59-a980-4ec8-af7b-fc2c08133f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" Nov 03 20:33:45.810838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:45.763128 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body= Nov 03 20:33:45.810838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:45.763193 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" podUID="abce531c-32e5-4fee-b6fa-7b489da29704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" Nov 03 20:33:45.810838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:45.764126 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" start-of-body= Nov 03 20:33:45.810838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:45.764172 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" podUID="9bff4885-a4dc-4b17-b596-e745e86fbccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" Nov 03 20:33:45.810838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:45.770058 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Nov 03 20:33:45.810838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:45.770102 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" podUID="b1573256-23a1-4a26-9972-87bc9b27f2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Nov 03 20:33:45.829719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:45.819833 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Nov 03 20:33:45.829719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:45.819898 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Nov 03 20:33:46.565061 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:46.550715 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Nov 03 20:33:46.565061 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:46.550776 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" podUID="b1573256-23a1-4a26-9972-87bc9b27f2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Nov 03 20:33:46.668804 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:46.655866 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" podUID="60b2ec36-7e99-46e0-9671-1c500c402e45" containerName="main" probeResult="failure" output="Get \"https://10.132.1.89:8000/ready\": context deadline exceeded" Nov 03 20:33:46.716979 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:46.714365 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body= Nov 03 20:33:46.716979 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:46.714442 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" podUID="9c4b9c59-a980-4ec8-af7b-fc2c08133f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" Nov 03 20:33:46.830691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:46.797932 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body= Nov 03 20:33:46.831600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:46.831557 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" podUID="abce531c-32e5-4fee-b6fa-7b489da29704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" Nov 03 20:33:46.908764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:46.905375 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Nov 03 20:33:46.908764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:46.905445 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Nov 03 20:33:46.908764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:46.905700 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" start-of-body= Nov 03 20:33:46.908764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:46.905751 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" podUID="9bff4885-a4dc-4b17-b596-e745e86fbccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" Nov 03 20:33:46.908764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:46.906259 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" podUID="d098ef77-addb-4077-b6d4-f0ae477409b8" containerName="main" probeResult="failure" output="Get \"https://10.132.1.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:47.255115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:47.255017 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" path="/var/lib/kubelet/pods/e7cf1267-7d06-459b-9df7-fc7c477d36e1/volumes" Nov 03 20:33:47.256925 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:47.256891 5059 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.374s" Nov 03 20:33:47.257161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:47.257134 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws"] Nov 03 20:33:47.257389 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:47.257373 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s"] Nov 03 20:33:47.298199 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:47.297383 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-v5xhc" event={"ID":"d50891bc-16c2-497d-8e00-b79ef492ab6e","Type":"ContainerStarted","Data":"cf2e9bbee261e54295b0e0e1e95b7d891fd414d591b58a211af539c5b266e96c"} Nov 03 20:33:47.314712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:47.300876 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-v5xhc" Nov 03 20:33:47.373927 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:47.371351 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-4vl4x"] Nov 03 20:33:47.541739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:47.534849 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Nov 03 20:33:47.541739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:47.534907 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" podUID="b1573256-23a1-4a26-9972-87bc9b27f2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Nov 03 20:33:47.541739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:47.535316 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" event={"ID":"68bc094a-84e2-41c2-9fb5-081682413f27","Type":"ContainerStarted","Data":"fee83e63ae644329d8b09b1c0537ad9f0cc6c7bd1363ada3503d23316698ca6d"} Nov 03 20:33:47.622691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:47.615353 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-v5xhc"] Nov 03 20:33:47.742955 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:47.737813 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" start-of-body= Nov 03 20:33:47.742955 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:47.737874 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" podUID="9bff4885-a4dc-4b17-b596-e745e86fbccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" Nov 03 20:33:47.751702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:47.750920 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body= Nov 03 20:33:47.756087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:47.751992 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" podUID="abce531c-32e5-4fee-b6fa-7b489da29704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" Nov 03 20:33:47.808739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:47.758796 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body= Nov 03 20:33:47.808739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:47.758842 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" podUID="9c4b9c59-a980-4ec8-af7b-fc2c08133f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" Nov 03 20:33:47.836689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:47.816006 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Nov 03 20:33:47.836689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:47.816060 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Nov 03 20:33:47.855900 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:47.853103 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf"] Nov 03 20:33:48.207692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:48.206433 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" event={"ID":"32deeada-f159-4266-8026-9b0a9ffa5eb1","Type":"ContainerStarted","Data":"6781a235f1334ac564063212a1f32f4190a943269e07f489ccd0b7a501738895"} Nov 03 20:33:48.250705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:48.250644 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" event={"ID":"19ab8a5f-fa5f-4548-bf5f-0465281d8677","Type":"ContainerStarted","Data":"a7892a98b6abfbe7e24a6a645e11e9dc24194d6039d51dac1b67f8117fb642b5"} Nov 03 20:33:48.352492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:48.352000 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk"] Nov 03 20:33:48.419717 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:48.371277 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" podStartSLOduration=105.371074784 podStartE2EDuration="1m45.371074784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:38.723718562 +0000 UTC m=+3412.981515817" watchObservedRunningTime="2026-11-03 20:33:48.371074784 +0000 UTC m=+3422.628872042" Nov 03 20:33:48.532795 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:48.524971 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg" event={"ID":"aeb32457-81a5-4505-b189-2a98f5fda7da","Type":"ContainerStarted","Data":"fab37fbb13493385737d28f94387461758e3c2831d009aad282214e9aa7cd031"} Nov 03 20:33:48.628014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:48.558894 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg" Nov 03 20:33:48.628014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:48.569105 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:48.628014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:48.569449 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Nov 03 20:33:48.628014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:48.569495 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" podUID="b1573256-23a1-4a26-9972-87bc9b27f2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Nov 03 20:33:48.670682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:48.656709 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg" Nov 03 20:33:48.689684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:48.675232 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:33:48.689684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:48.676071 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body= Nov 03 20:33:48.689684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:48.676214 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" podUID="9c4b9c59-a980-4ec8-af7b-fc2c08133f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" Nov 03 20:33:48.717687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:48.717058 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:33:49.036391 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:48.794981 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:33:49.036391 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:48.821604 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Nov 03 20:33:49.036391 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:48.821679 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Nov 03 20:33:49.036391 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:48.822189 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body= Nov 03 20:33:49.036391 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:48.822231 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" podUID="abce531c-32e5-4fee-b6fa-7b489da29704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" Nov 03 20:33:49.036391 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:48.823924 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:49.036391 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:48.824021 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" start-of-body= Nov 03 20:33:49.036391 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:48.824054 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" podUID="9bff4885-a4dc-4b17-b596-e745e86fbccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" Nov 03 20:33:49.036391 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:48.846955 5059 generic.go:356] "Generic (PLEG): container finished" podID="603868b0-8628-4227-8471-1894e2e1de50" containerID="a75a1e78b8fa43ee7161dbf1263337919ca861621fb782933da9fe9d87a6c4b1" exitCode=0 Nov 03 20:33:49.036391 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:48.847082 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-f7b87d695-cwlfq" event={"ID":"603868b0-8628-4227-8471-1894e2e1de50","Type":"ContainerDied","Data":"a75a1e78b8fa43ee7161dbf1263337919ca861621fb782933da9fe9d87a6c4b1"} Nov 03 20:33:49.142809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:49.131081 5059 generic.go:356] "Generic (PLEG): container finished" podID="28034528-e342-417a-a9df-050d7ea91453" containerID="c63223667135eb8a17f59b5174cf095c231af49811bd42e93ad4f78c3ee6ba55" exitCode=7 Nov 03 20:33:49.142809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:49.131256 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h" event={"ID":"28034528-e342-417a-a9df-050d7ea91453","Type":"ContainerDied","Data":"c63223667135eb8a17f59b5174cf095c231af49811bd42e93ad4f78c3ee6ba55"} Nov 03 20:33:49.162686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:49.161344 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:49.165853 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:49.165761 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:33:49.256690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:49.243144 5059 scope.go:122] "RemoveContainer" containerID="c63223667135eb8a17f59b5174cf095c231af49811bd42e93ad4f78c3ee6ba55" Nov 03 20:33:49.328868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:49.259979 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" start-of-body= Nov 03 20:33:49.328868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:49.260048 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" Nov 03 20:33:49.328868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:49.319325 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Nov 03 20:33:49.349992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:49.328809 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Nov 03 20:33:49.912775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:49.376001 5059 generic.go:356] "Generic (PLEG): container finished" podID="847e7b38-8fb0-4c19-8943-f43e364f989b" containerID="6638b2c26dcbd2beddb822cddbf45294c8f85f17724b2ee963373d4a3c6be587" exitCode=7 Nov 03 20:33:49.912775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:49.376144 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx" event={"ID":"847e7b38-8fb0-4c19-8943-f43e364f989b","Type":"ContainerDied","Data":"6638b2c26dcbd2beddb822cddbf45294c8f85f17724b2ee963373d4a3c6be587"} Nov 03 20:33:49.912775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:49.449867 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297"] Nov 03 20:33:49.912775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:49.622052 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Nov 03 20:33:49.912775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:49.622114 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" podUID="b1573256-23a1-4a26-9972-87bc9b27f2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Nov 03 20:33:49.912775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:49.691149 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg"] Nov 03 20:33:49.912775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:49.737970 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" start-of-body= Nov 03 20:33:49.912775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:49.738029 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" podUID="9bff4885-a4dc-4b17-b596-e745e86fbccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" Nov 03 20:33:49.912775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:49.738501 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body= Nov 03 20:33:49.912775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:49.738547 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" podUID="9c4b9c59-a980-4ec8-af7b-fc2c08133f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" Nov 03 20:33:49.912775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:49.812783 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Nov 03 20:33:49.912775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:49.812844 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Nov 03 20:33:49.912775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:49.852009 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body= Nov 03 20:33:49.912775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:49.852066 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" podUID="abce531c-32e5-4fee-b6fa-7b489da29704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" Nov 03 20:33:49.912775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:49.852744 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk"] Nov 03 20:33:50.163799 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:50.155339 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" podStartSLOduration=108.155319305 podStartE2EDuration="1m48.155319305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:39.044560332 +0000 UTC m=+3413.302357602" watchObservedRunningTime="2026-11-03 20:33:50.155319305 +0000 UTC m=+3424.413116562" Nov 03 20:33:50.188077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:50.169379 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Nov 03 20:33:50.188077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:50.169436 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Nov 03 20:33:50.197931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:50.189778 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" start-of-body= Nov 03 20:33:50.213173 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:50.213118 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" Nov 03 20:33:50.284782 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:50.258498 5059 scope.go:122] "RemoveContainer" containerID="6638b2c26dcbd2beddb822cddbf45294c8f85f17724b2ee963373d4a3c6be587" Nov 03 20:33:50.326757 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:50.326618 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 20:33:50.326757 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:50.326691 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 20:33:50.392698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:50.377030 5059 generic.go:356] "Generic (PLEG): container finished" podID="94e83c10-58b0-4bc7-9abd-3e144ba6e258" containerID="735b32748a8b907b9eed5ef21efd8284798ec21dd97653a226fd406ff5d16e6e" exitCode=22 Nov 03 20:33:50.684819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:50.679975 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body= Nov 03 20:33:50.684819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:50.680040 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" podUID="9c4b9c59-a980-4ec8-af7b-fc2c08133f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" Nov 03 20:33:50.776616 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:50.774939 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body= Nov 03 20:33:50.776616 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:50.774962 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" podStartSLOduration=107.77494325 podStartE2EDuration="1m47.77494325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:39.403547451 +0000 UTC m=+3413.661344709" watchObservedRunningTime="2026-11-03 20:33:50.77494325 +0000 UTC m=+3425.032740507" Nov 03 20:33:50.776616 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:50.775007 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" podUID="abce531c-32e5-4fee-b6fa-7b489da29704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" Nov 03 20:33:50.776616 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:50.776068 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" start-of-body= Nov 03 20:33:50.776616 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:50.776109 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" podUID="9bff4885-a4dc-4b17-b596-e745e86fbccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" Nov 03 20:33:50.859320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:50.833340 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:33:50.859320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:50.833405 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:33:50.859320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:50.834907 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Nov 03 20:33:50.859320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:50.834945 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Nov 03 20:33:50.992942 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:50.972628 5059 scope.go:122] "RemoveContainer" containerID="3ac340f939011b76b5f8415b2bef3772f5c72dd74e4301b698de740f839445e8" Nov 03 20:33:51.087707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:51.065486 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx" podStartSLOduration=31.065464798 podStartE2EDuration="31.065464798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:39.453158783 +0000 UTC m=+3413.710956040" watchObservedRunningTime="2026-11-03 20:33:51.065464798 +0000 UTC m=+3425.323262064" Nov 03 20:33:51.138085 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:51.137319 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 20:33:51.138085 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:51.137385 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 20:33:51.138819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:51.138721 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" start-of-body= Nov 03 20:33:51.138819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:51.138769 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" Nov 03 20:33:51.192499 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:51.183413 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Nov 03 20:33:51.192499 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:51.183473 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Nov 03 20:33:51.192499 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:51.187379 5059 scope.go:122] "RemoveContainer" containerID="735b32748a8b907b9eed5ef21efd8284798ec21dd97653a226fd406ff5d16e6e" Nov 03 20:33:51.705969 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:51.676340 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body= Nov 03 20:33:51.705969 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:51.676435 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" podUID="9c4b9c59-a980-4ec8-af7b-fc2c08133f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" Nov 03 20:33:51.842744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:51.815644 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Nov 03 20:33:51.842744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:51.815742 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Nov 03 20:33:51.842744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:51.817621 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:33:51.913346 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:51.875006 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:33:51.913346 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:51.876399 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body= Nov 03 20:33:51.913346 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:51.876446 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" podUID="abce531c-32e5-4fee-b6fa-7b489da29704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" Nov 03 20:33:51.913346 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:51.876541 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" start-of-body= Nov 03 20:33:51.913346 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:51.876568 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" podUID="9bff4885-a4dc-4b17-b596-e745e86fbccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" Nov 03 20:33:52.140712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:52.140591 5059 patch_prober.go:29] interesting pod/router-default-f7b87d695-cwlfq container/router namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.9:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:33:52.141952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:52.141351 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-f7b87d695-cwlfq" podUID="603868b0-8628-4227-8471-1894e2e1de50" containerName="router" probeResult="failure" output="Get \"http://10.132.0.9:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:52.159733 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:52.159697 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" start-of-body= Nov 03 20:33:52.160036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:52.159998 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" Nov 03 20:33:52.160749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:52.160247 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 20:33:52.160993 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:52.160949 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 20:33:52.161223 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:52.160606 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Nov 03 20:33:52.161382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:52.161363 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Nov 03 20:33:52.707030 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:52.698267 5059 generic.go:356] "Generic (PLEG): container finished" podID="32deeada-f159-4266-8026-9b0a9ffa5eb1" containerID="6781a235f1334ac564063212a1f32f4190a943269e07f489ccd0b7a501738895" exitCode=0 Nov 03 20:33:52.707030 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:52.705817 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body= Nov 03 20:33:52.707030 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:52.705909 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" podUID="9c4b9c59-a980-4ec8-af7b-fc2c08133f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" Nov 03 20:33:52.762699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:52.742061 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" start-of-body= Nov 03 20:33:52.762699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:52.742122 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" podUID="9bff4885-a4dc-4b17-b596-e745e86fbccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" Nov 03 20:33:52.784586 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:52.784451 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:33:52.784586 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:52.784508 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:33:52.843788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:52.824185 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Nov 03 20:33:52.843788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:52.824242 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Nov 03 20:33:53.002798 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:52.973831 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h" podStartSLOduration=49.973808546 podStartE2EDuration="49.973808546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:42.631307151 +0000 UTC m=+3416.889104408" watchObservedRunningTime="2026-11-03 20:33:52.973808546 +0000 UTC m=+3427.231605803" Nov 03 20:33:53.060226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.059819 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" start-of-body= Nov 03 20:33:53.060226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.059882 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" Nov 03 20:33:53.143729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.114451 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" podStartSLOduration=110.114431919 podStartE2EDuration="1m50.114431919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:43.779957452 +0000 UTC m=+3418.037754709" watchObservedRunningTime="2026-11-03 20:33:53.114431919 +0000 UTC m=+3427.372229177" Nov 03 20:33:53.143729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.140125 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" start-of-body= Nov 03 20:33:53.143729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.140187 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" Nov 03 20:33:53.143729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.140898 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 20:33:53.143729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.140945 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 20:33:53.181687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.159534 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Nov 03 20:33:53.181687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.159599 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Nov 03 20:33:53.181687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.159538 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Nov 03 20:33:53.181687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.159678 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Nov 03 20:33:53.209154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.209086 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-x7n2b" podStartSLOduration=110.209066828 podStartE2EDuration="1m50.209066828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:43.938020444 +0000 UTC m=+3418.195817718" watchObservedRunningTime="2026-11-03 20:33:53.209066828 +0000 UTC m=+3427.466864086" Nov 03 20:33:53.234719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.215816 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:33:53.234719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.215878 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:33:53.234719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.230815 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:33:53.234719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.230867 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:33:53.294097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.293979 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" podStartSLOduration=111.293955908 podStartE2EDuration="1m51.293955908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:44.104336524 +0000 UTC m=+3418.362133787" watchObservedRunningTime="2026-11-03 20:33:53.293955908 +0000 UTC m=+3427.551753178" Nov 03 20:33:53.384687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.367792 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Nov 03 20:33:53.384687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.367850 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Nov 03 20:33:53.384687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.370152 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Nov 03 20:33:53.384687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.370209 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Nov 03 20:33:53.440733 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.433520 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-v5xhc" podStartSLOduration=110.433496396 podStartE2EDuration="1m50.433496396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:48.142683863 +0000 UTC m=+3422.400481121" watchObservedRunningTime="2026-11-03 20:33:53.433496396 +0000 UTC m=+3427.691293655" Nov 03 20:33:53.541409 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.539650 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:33:53.541409 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.539727 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:33:53.588353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.588286 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" podStartSLOduration=111.588263008 podStartE2EDuration="1m51.588263008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:48.291130308 +0000 UTC m=+3422.548927584" watchObservedRunningTime="2026-11-03 20:33:53.588263008 +0000 UTC m=+3427.846060266" Nov 03 20:33:53.670689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.635673 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podStartSLOduration=111.635633208 podStartE2EDuration="1m51.635633208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:48.325997927 +0000 UTC m=+3422.583795187" watchObservedRunningTime="2026-11-03 20:33:53.635633208 +0000 UTC m=+3427.893430465" Nov 03 20:33:53.719279 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.718871 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body= Nov 03 20:33:53.719279 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.718932 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" podUID="9c4b9c59-a980-4ec8-af7b-fc2c08133f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" Nov 03 20:33:53.772571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.751192 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" start-of-body= Nov 03 20:33:53.772571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.751251 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" podUID="9bff4885-a4dc-4b17-b596-e745e86fbccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" Nov 03 20:33:53.772571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.772082 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:33:53.772571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.772139 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:33:53.870136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.843996 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Nov 03 20:33:53.870136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.844060 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Nov 03 20:33:53.994698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:53.992174 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg" podStartSLOduration=109.571726288 podStartE2EDuration="1m50.992152233s" totalImagesPullingTime="1.420425945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:48.645920247 +0000 UTC m=+3422.903717510" watchObservedRunningTime="2026-11-03 20:33:53.992152233 +0000 UTC m=+3428.249949491" Nov 03 20:33:54.118923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.094755 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" start-of-body= Nov 03 20:33:54.118923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.094815 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" Nov 03 20:33:54.172434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.159243 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 20:33:54.172434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.159304 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 20:33:54.172434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.159390 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" start-of-body= Nov 03 20:33:54.172434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.159416 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" Nov 03 20:33:54.172434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.162416 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" podStartSLOduration=111.162396138 podStartE2EDuration="1m51.162396138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:49.944175731 +0000 UTC m=+3424.201973036" watchObservedRunningTime="2026-11-03 20:33:54.162396138 +0000 UTC m=+3428.420193395" Nov 03 20:33:54.220771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.179060 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Nov 03 20:33:54.220771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.179121 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Nov 03 20:33:54.220771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.179317 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Nov 03 20:33:54.220771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.179348 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Nov 03 20:33:54.220771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.215899 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:33:54.220771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.215968 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:33:54.273856 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.273812 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:33:54.274116 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.274082 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:33:54.369696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.344161 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podStartSLOduration=112.344138666 podStartE2EDuration="1m52.344138666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:50.285797368 +0000 UTC m=+3424.543594639" watchObservedRunningTime="2026-11-03 20:33:54.344138666 +0000 UTC m=+3428.601935923" Nov 03 20:33:54.506745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.493044 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Nov 03 20:33:54.506745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.493111 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Nov 03 20:33:54.506745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.493202 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Nov 03 20:33:54.506745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.493231 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Nov 03 20:33:54.568650 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.568580 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podStartSLOduration=112.568559055 podStartE2EDuration="1m52.568559055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:52.338888251 +0000 UTC m=+3426.596685504" watchObservedRunningTime="2026-11-03 20:33:54.568559055 +0000 UTC m=+3428.826356313" Nov 03 20:33:54.619687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.587917 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:33:54.619687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.587977 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:33:54.632782 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.622084 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podStartSLOduration=112.622062417 podStartE2EDuration="1m52.622062417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:52.474153791 +0000 UTC m=+3426.731951048" watchObservedRunningTime="2026-11-03 20:33:54.622062417 +0000 UTC m=+3428.879859675" Nov 03 20:33:54.717993 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.690857 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podStartSLOduration=111.690838823 podStartE2EDuration="1m51.690838823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:52.559312698 +0000 UTC m=+3426.817109956" watchObservedRunningTime="2026-11-03 20:33:54.690838823 +0000 UTC m=+3428.948636082" Nov 03 20:33:54.717993 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.698908 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body= Nov 03 20:33:54.717993 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.698961 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" podUID="9c4b9c59-a980-4ec8-af7b-fc2c08133f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" Nov 03 20:33:54.746897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.742049 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" start-of-body= Nov 03 20:33:54.746897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.742108 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" podUID="9bff4885-a4dc-4b17-b596-e745e86fbccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" Nov 03 20:33:54.762841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.761759 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:33:54.762841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.761821 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:33:54.802125 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.802087 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Nov 03 20:33:54.810719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:54.802372 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Nov 03 20:33:55.040740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.040244 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podStartSLOduration=112.040220447 podStartE2EDuration="1m52.040220447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:52.639123813 +0000 UTC m=+3426.896921070" watchObservedRunningTime="2026-11-03 20:33:55.040220447 +0000 UTC m=+3429.298017712" Nov 03 20:33:55.187595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.163414 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" start-of-body= Nov 03 20:33:55.187595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.163501 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" Nov 03 20:33:55.187892 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.187764 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:33:55.187892 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.187843 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:33:55.188894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.188606 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 20:33:55.188894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.188691 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 20:33:55.244228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.214695 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" start-of-body= Nov 03 20:33:55.244228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.214765 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" Nov 03 20:33:55.244228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.215094 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Nov 03 20:33:55.244228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.215134 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Nov 03 20:33:55.244228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.217961 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:33:55.244228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.217999 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:33:55.343694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.343291 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Nov 03 20:33:55.343694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.343361 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Nov 03 20:33:55.454844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.394817 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Nov 03 20:33:55.454844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.394882 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Nov 03 20:33:55.479685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.476782 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Nov 03 20:33:55.479685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.476857 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Nov 03 20:33:55.542687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.541083 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:33:55.542687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.541160 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:33:55.743749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.739964 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" start-of-body= Nov 03 20:33:55.743749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.740030 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" podUID="9bff4885-a4dc-4b17-b596-e745e86fbccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" Nov 03 20:33:55.764203 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.764080 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:33:55.764203 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.764142 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:33:55.809228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.809088 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Nov 03 20:33:55.809228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:55.809148 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Nov 03 20:33:56.107095 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.106360 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" start-of-body= Nov 03 20:33:56.107095 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.106423 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" Nov 03 20:33:56.181345 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:56.177452 5059 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="6.325s" Nov 03 20:33:56.181345 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.177500 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5d9f95b69-x7n2b" Nov 03 20:33:56.181345 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.177545 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" event={"ID":"3eda42a5-5b61-4b34-8447-8767ed097200","Type":"ContainerStarted","Data":"838b620ee70f5ced5fecf537721d400be59c4cfe985b56a65503fe7e0a658ced"} Nov 03 20:33:56.181345 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.177579 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:33:56.181345 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.177604 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4"] Nov 03 20:33:56.181345 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.177625 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" event={"ID":"82894901-9901-4dc4-948a-321782ef8288","Type":"ContainerStarted","Data":"75098182ba8d8b64866d00d509377e1d942976753e6779545667b12a2d2314dd"} Nov 03 20:33:56.181345 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.177715 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn"] Nov 03 20:33:56.181345 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.177755 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" event={"ID":"d4fdd390-7fe2-406d-832a-7ecc545d97f4","Type":"ContainerStarted","Data":"4255477aa69a37bca85ef94bde9dc14fba2598777f972bf06e1248d62af4c20d"} Nov 03 20:33:56.183194 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.182391 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Nov 03 20:33:56.183194 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.182444 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Nov 03 20:33:56.183194 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.182915 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" start-of-body= Nov 03 20:33:56.183194 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.182951 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" Nov 03 20:33:56.183194 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.183015 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Nov 03 20:33:56.183194 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.183036 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Nov 03 20:33:56.183194 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.183112 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 20:33:56.183194 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.183132 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 20:33:56.196728 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.196166 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:33:56.196728 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.196218 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:33:56.206375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.197944 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:33:56.206375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.197984 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:33:56.239864 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.234446 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podStartSLOduration=114.234427007 podStartE2EDuration="1m54.234427007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:52.878257978 +0000 UTC m=+3427.136055236" watchObservedRunningTime="2026-11-03 20:33:56.234427007 +0000 UTC m=+3430.492224265" Nov 03 20:33:56.268726 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.241963 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" podStartSLOduration=111.796005586 podStartE2EDuration="1m53.241946736s" totalImagesPullingTime="1.44594115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:52.944694089 +0000 UTC m=+3427.202491348" watchObservedRunningTime="2026-11-03 20:33:56.241946736 +0000 UTC m=+3430.499743995" Nov 03 20:33:56.354835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.354319 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Nov 03 20:33:56.354835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.354379 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Nov 03 20:33:56.354835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.354684 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Nov 03 20:33:56.354835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.354721 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Nov 03 20:33:56.595463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.574433 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:33:56.595463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.574491 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:33:56.765716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.749103 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:33:56.765716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.749171 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" podUID="9c4b9c59-a980-4ec8-af7b-fc2c08133f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:56.765716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.764851 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" start-of-body= Nov 03 20:33:56.765716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.764910 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" podUID="9bff4885-a4dc-4b17-b596-e745e86fbccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" Nov 03 20:33:56.777447 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.767102 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:33:56.777447 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.767152 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:33:56.804523 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.804153 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Nov 03 20:33:56.804523 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:56.804213 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Nov 03 20:33:57.127809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.126789 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" start-of-body= Nov 03 20:33:57.127809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.126860 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" Nov 03 20:33:57.178818 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.169778 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 20:33:57.178818 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.169841 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 20:33:57.178818 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.170439 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" start-of-body= Nov 03 20:33:57.178818 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.170485 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" Nov 03 20:33:57.178818 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.170567 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Nov 03 20:33:57.178818 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.170593 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Nov 03 20:33:57.178818 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.170651 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Nov 03 20:33:57.178818 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.170694 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Nov 03 20:33:57.202417 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.202044 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:33:57.202417 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.202105 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:33:57.204885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.204329 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:33:57.204885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.204373 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:33:57.375715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.361831 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Nov 03 20:33:57.375715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.361890 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Nov 03 20:33:57.375715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.362359 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Nov 03 20:33:57.375715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.362398 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Nov 03 20:33:57.546947 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.546410 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:33:57.546947 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.546477 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:33:57.746238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.744888 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" start-of-body= Nov 03 20:33:57.746238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.744945 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" podUID="9bff4885-a4dc-4b17-b596-e745e86fbccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.131:15021/healthz/ready\": dial tcp 10.132.1.131:15021: connect: connection refused" Nov 03 20:33:57.761111 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.760610 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:33:57.761111 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.760709 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:33:57.832840 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.830793 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Nov 03 20:33:57.832840 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:57.830856 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Nov 03 20:33:58.058697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.055110 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" start-of-body= Nov 03 20:33:58.058697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.055179 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" Nov 03 20:33:58.144765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.117955 5059 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","podfc57130b-c950-49f2-8a98-22c47bd98288"] err="unable to destroy cgroup paths for cgroup [kubepods burstable podfc57130b-c950-49f2-8a98-22c47bd98288] : Timed out while waiting for systemd to remove kubepods-burstable-podfc57130b_c950_49f2_8a98_22c47bd98288.slice" Nov 03 20:33:58.144765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.117999 5059 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="fc57130b-c950-49f2-8a98-22c47bd98288" err="failed to delete cgroup paths for [kubepods burstable podfc57130b-c950-49f2-8a98-22c47bd98288] : unable to destroy cgroup paths for cgroup [kubepods burstable podfc57130b-c950-49f2-8a98-22c47bd98288] : Timed out while waiting for systemd to remove kubepods-burstable-podfc57130b_c950_49f2_8a98_22c47bd98288.slice" Nov 03 20:33:58.144765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.129756 5059 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod2c363281-0226-420f-a963-5922d5ca609e"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod2c363281-0226-420f-a963-5922d5ca609e] : Timed out while waiting for systemd to remove kubepods-burstable-pod2c363281_0226_420f_a963_5922d5ca609e.slice" Nov 03 20:33:58.144765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.129796 5059 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="2c363281-0226-420f-a963-5922d5ca609e" err="failed to delete cgroup paths for [kubepods burstable pod2c363281-0226-420f-a963-5922d5ca609e] : unable to destroy cgroup paths for cgroup [kubepods burstable pod2c363281-0226-420f-a963-5922d5ca609e] : Timed out while waiting for systemd to remove kubepods-burstable-pod2c363281_0226_420f_a963_5922d5ca609e.slice" Nov 03 20:33:58.160424 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.159751 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 20:33:58.160424 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.159814 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 20:33:58.160766 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.160741 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" start-of-body= Nov 03 20:33:58.160910 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.160887 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" Nov 03 20:33:58.169235 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.164952 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Nov 03 20:33:58.169235 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.165001 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Nov 03 20:33:58.169235 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.165072 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Nov 03 20:33:58.169235 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.165096 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Nov 03 20:33:58.182797 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.182132 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:33:58.182797 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.182192 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:33:58.250446 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.197015 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:33:58.250446 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.197068 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:33:58.352695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.349053 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Nov 03 20:33:58.352695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.349114 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Nov 03 20:33:58.352695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.349633 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Nov 03 20:33:58.352695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.349696 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Nov 03 20:33:58.540889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.539290 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:33:58.540889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.539349 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:33:58.554165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.554021 5059 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod26375297-81f4-427f-94c3-d56f7a6a11fb"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod26375297-81f4-427f-94c3-d56f7a6a11fb] : Timed out while waiting for systemd to remove kubepods-burstable-pod26375297_81f4_427f_94c3_d56f7a6a11fb.slice" Nov 03 20:33:58.554165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.554067 5059 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="26375297-81f4-427f-94c3-d56f7a6a11fb" err="failed to delete cgroup paths for [kubepods burstable pod26375297-81f4-427f-94c3-d56f7a6a11fb] : unable to destroy cgroup paths for cgroup [kubepods burstable pod26375297-81f4-427f-94c3-d56f7a6a11fb] : Timed out while waiting for systemd to remove kubepods-burstable-pod26375297_81f4_427f_94c3_d56f7a6a11fb.slice" Nov 03 20:33:58.761544 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:58.761351 5059 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.584s" Nov 03 20:33:58.761544 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.761397 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95" event={"ID":"0298aa52-aec0-481a-b7d1-12964e8137c8","Type":"ContainerStarted","Data":"1e7e9fb8cc6a318af498496044b7454b1a02e855c06f203301574493f60166da"} Nov 03 20:33:58.761544 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.761452 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-r5fd4"] Nov 03 20:33:58.812711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.808061 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:33:58.812711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.808127 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:33:58.812711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.809084 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Nov 03 20:33:58.827172 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.809136 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Nov 03 20:33:58.935699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:58.904616 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podStartSLOduration=115.904593306 podStartE2EDuration="1m55.904593306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:53.056122404 +0000 UTC m=+3427.313919663" watchObservedRunningTime="2026-11-03 20:33:58.904593306 +0000 UTC m=+3433.162390589" Nov 03 20:33:59.045833 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.045785 5059 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","podac6d56f2-a6c4-4d0e-86d5-64360d5e4a51"] err="unable to destroy cgroup paths for cgroup [kubepods burstable podac6d56f2-a6c4-4d0e-86d5-64360d5e4a51] : Timed out while waiting for systemd to remove kubepods-burstable-podac6d56f2_a6c4_4d0e_86d5_64360d5e4a51.slice" Nov 03 20:33:59.052764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.052714 5059 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="ac6d56f2-a6c4-4d0e-86d5-64360d5e4a51" err="failed to delete cgroup paths for [kubepods burstable podac6d56f2-a6c4-4d0e-86d5-64360d5e4a51] : unable to destroy cgroup paths for cgroup [kubepods burstable podac6d56f2-a6c4-4d0e-86d5-64360d5e4a51] : Timed out while waiting for systemd to remove kubepods-burstable-podac6d56f2_a6c4_4d0e_86d5_64360d5e4a51.slice" Nov 03 20:33:59.065847 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.065735 5059 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","podade5b9f4-d822-4418-a31b-f0c0a21ddb0c"] err="unable to destroy cgroup paths for cgroup [kubepods burstable podade5b9f4-d822-4418-a31b-f0c0a21ddb0c] : Timed out while waiting for systemd to remove kubepods-burstable-podade5b9f4_d822_4418_a31b_f0c0a21ddb0c.slice" Nov 03 20:33:59.066119 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.066091 5059 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="ade5b9f4-d822-4418-a31b-f0c0a21ddb0c" err="failed to delete cgroup paths for [kubepods burstable podade5b9f4-d822-4418-a31b-f0c0a21ddb0c] : unable to destroy cgroup paths for cgroup [kubepods burstable podade5b9f4-d822-4418-a31b-f0c0a21ddb0c] : Timed out while waiting for systemd to remove kubepods-burstable-podade5b9f4_d822_4418_a31b_f0c0a21ddb0c.slice" Nov 03 20:33:59.097828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.078032 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" start-of-body= Nov 03 20:33:59.097828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.078105 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" Nov 03 20:33:59.174911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.151787 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" start-of-body= Nov 03 20:33:59.174911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.151866 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" Nov 03 20:33:59.174911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.153058 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 20:33:59.174911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.153109 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 20:33:59.174911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.153410 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Nov 03 20:33:59.174911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.153450 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Nov 03 20:33:59.220686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.179707 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:33:59.220686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.179766 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:33:59.220686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.180101 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Nov 03 20:33:59.220686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.180147 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Nov 03 20:33:59.220686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.207117 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:33:59.220686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.207189 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:33:59.426937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.420903 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Nov 03 20:33:59.426937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.420972 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Nov 03 20:33:59.426937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.421071 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Nov 03 20:33:59.426937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.421096 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Nov 03 20:33:59.538904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.538775 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:33:59.538904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.538843 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:33:59.782740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.761774 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:33:59.782740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.761834 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:33:59.803170 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.803131 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Nov 03 20:33:59.803424 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.803400 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Nov 03 20:33:59.964103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.941685 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc10f92e-e902-48c4-9cd5-4f768c4611be" path="/var/lib/kubelet/pods/dc10f92e-e902-48c4-9cd5-4f768c4611be/volumes" Nov 03 20:33:59.964103 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:33:59.942240 5059 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.181s" Nov 03 20:33:59.964103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.942279 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:33:59.964103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.942397 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-4vl4x" event={"ID":"94e83c10-58b0-4bc7-9abd-3e144ba6e258","Type":"ContainerDied","Data":"735b32748a8b907b9eed5ef21efd8284798ec21dd97653a226fd406ff5d16e6e"} Nov 03 20:33:59.964103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.942436 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:33:59.964103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.942463 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" event={"ID":"9069949b-78ef-42cc-99b0-1feffc84b679","Type":"ContainerStarted","Data":"b3821841d889962619728e3cfd3742dffe6108759aefe0be61ddae8c78d55106"} Nov 03 20:33:59.964103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.942498 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" event={"ID":"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507","Type":"ContainerStarted","Data":"47973061462d7b92d288da1a98e63008b76ad7ea55f5bbc305750cfd1fd35ce0"} Nov 03 20:33:59.964103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.942528 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:33:59.964103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.942553 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" event={"ID":"569b2103-147f-4ea1-9b55-0e859b1c5e52","Type":"ContainerStarted","Data":"52d9dfafb55bc5b6dad5352af2151b15f85f2c035a91e44f089c01883269dcbc"} Nov 03 20:33:59.964103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.942574 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" event={"ID":"7f438514-df13-4871-9c2a-ef8e56a570c9","Type":"ContainerStarted","Data":"535c97f981aaf14f23e0940aa42d33bed0f295ee3f35cefd0fb15098b2c7793f"} Nov 03 20:33:59.964103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.942602 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:33:59.964103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.942624 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" event={"ID":"6b615028-a2db-4ad0-8a57-53efc040d6a6","Type":"ContainerStarted","Data":"608a9becc0865d2fa635a042e78e2d293a90887169f9f7a9a78ac4715c431fc8"} Nov 03 20:33:59.964103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.942650 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:33:59.965202 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.965170 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4"] Nov 03 20:33:59.965261 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.965229 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" event={"ID":"e2fe4277-3acc-4378-86d9-374b69918ec9","Type":"ContainerStarted","Data":"b7cacec852af6354300c159e158582fcff3f1d2e974a9d742f79b212daaff852"} Nov 03 20:34:00.027743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:33:59.969893 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podStartSLOduration=116.969877314 podStartE2EDuration="1m56.969877314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:53.254769581 +0000 UTC m=+3427.512566840" watchObservedRunningTime="2026-11-03 20:33:59.969877314 +0000 UTC m=+3434.227674574" Nov 03 20:34:00.085553 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.085516 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" start-of-body= Nov 03 20:34:00.085822 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.085789 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" Nov 03 20:34:00.213293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.207179 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 20:34:00.213293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.207236 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 20:34:00.213293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.208524 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:34:00.213293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.208569 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:34:00.213293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.209239 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Nov 03 20:34:00.213293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.209277 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Nov 03 20:34:00.213293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.209341 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Nov 03 20:34:00.213293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.209364 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Nov 03 20:34:00.213293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.209419 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" start-of-body= Nov 03 20:34:00.213293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.209440 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" Nov 03 20:34:00.373045 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.239286 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:34:00.373045 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.239340 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:34:00.814388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.374825 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Nov 03 20:34:00.814388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.374880 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Nov 03 20:34:00.814388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.375698 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Nov 03 20:34:00.814388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.375738 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Nov 03 20:34:00.814388 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:34:00.447979 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32deeada_f159_4266_8026_9b0a9ffa5eb1.slice/crio-conmon-6781a235f1334ac564063212a1f32f4190a943269e07f489ccd0b7a501738895.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32deeada_f159_4266_8026_9b0a9ffa5eb1.slice/crio-6781a235f1334ac564063212a1f32f4190a943269e07f489ccd0b7a501738895.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:34:00.814388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.461606 5059 scope.go:122] "RemoveContainer" containerID="5ac8799b34596375075829e9ebddeeb7da35d54820c64c585e69600c30f1d815" Nov 03 20:34:00.814388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.546739 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:34:00.814388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.546790 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:34:00.937690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.890992 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 20:34:00.937690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.916858 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Nov 03 20:34:00.937690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.916916 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.964991 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965059 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965086 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" event={"ID":"32deeada-f159-4266-8026-9b0a9ffa5eb1","Type":"ContainerDied","Data":"6781a235f1334ac564063212a1f32f4190a943269e07f489ccd0b7a501738895"} Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965127 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-9wxx4"] Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965160 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965284 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965314 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-v5xhc" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965338 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965360 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965384 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965411 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965434 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965458 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965487 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5"] Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965508 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" event={"ID":"75efd589-b943-46c2-add0-d44ccdaff537","Type":"ContainerStarted","Data":"2d9824c6fbca0342e936df0e333132fa3d4104bf9d7e7be14d38328d09ab2343"} Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965537 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-xqmv5"] Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965558 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" event={"ID":"89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5","Type":"ContainerStarted","Data":"bd742b87f85f7ab103acd541cd629b18903f06c2429861482a188c561b10530b"} Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965583 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965696 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965718 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" event={"ID":"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8","Type":"ContainerStarted","Data":"ba8b1d44e1eaed0f3e420da024ef6f5f654842540fe5435339c0ff9078b7ee8c"} Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965741 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" event={"ID":"ecab4d7a-4034-4b59-8500-8434baacafaf","Type":"ContainerStarted","Data":"26d622763433e981c524cf0ddaec2006cc9ffcb67eb091acf6792a109cd90315"} Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965764 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" event={"ID":"d75a00ce-f97c-431f-be9e-aac200bcd9d4","Type":"ContainerStarted","Data":"1119a404faca34d41ecebf28afb622f4f839d6f1e5fd27a2976df8db31439851"} Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965792 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62"] Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965865 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965889 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-f7b87d695-cwlfq" event={"ID":"603868b0-8628-4227-8471-1894e2e1de50","Type":"ContainerDied","Data":"56ca0571aedc921d703ca9a0d179a99f24a724e4d9f206adc2364d70071a3b71"} Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965954 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.965983 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g"] Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.966013 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w"] Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.966041 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc"] Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.966070 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj"] Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.966099 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr"] Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.966127 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95"] Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.966159 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf"] Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.966195 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7"] Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.966217 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x"] Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.966260 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-7w5r9"] Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967278 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1326708b-4642-4992-91ea-8a590863a703" containerName="revoke-keys" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967298 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1326708b-4642-4992-91ea-8a590863a703" containerName="revoke-keys" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967313 5059 cpu_manager.go:414] "containerMap: removing container" podUID="e2155512-2116-44f6-be5e-77cef72f9e14" containerName="connection-checker" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967321 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2155512-2116-44f6-be5e-77cef72f9e14" containerName="connection-checker" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967338 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1326708b-4642-4992-91ea-8a590863a703" containerName="revoke-keys" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967346 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1326708b-4642-4992-91ea-8a590863a703" containerName="revoke-keys" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967359 5059 cpu_manager.go:414] "containerMap: removing container" podUID="6b9428a2-ab50-4e44-a725-e0da980bdb93" containerName="revoke-keys" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967368 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b9428a2-ab50-4e44-a725-e0da980bdb93" containerName="revoke-keys" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967381 5059 cpu_manager.go:414] "containerMap: removing container" podUID="dc10f92e-e902-48c4-9cd5-4f768c4611be" containerName="revoke-keys" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967388 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc10f92e-e902-48c4-9cd5-4f768c4611be" containerName="revoke-keys" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967399 5059 cpu_manager.go:414] "containerMap: removing container" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967406 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967421 5059 cpu_manager.go:414] "containerMap: removing container" podUID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" containerName="main" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967428 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" containerName="main" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967439 5059 cpu_manager.go:414] "containerMap: removing container" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967446 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967466 5059 cpu_manager.go:414] "containerMap: removing container" podUID="6b9428a2-ab50-4e44-a725-e0da980bdb93" containerName="revoke-keys" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967474 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b9428a2-ab50-4e44-a725-e0da980bdb93" containerName="revoke-keys" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967501 5059 cpu_manager.go:414] "containerMap: removing container" podUID="0fc518fc-6302-4a1a-9ff9-f1e036a2199b" containerName="connection-checker" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967508 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc518fc-6302-4a1a-9ff9-f1e036a2199b" containerName="connection-checker" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967520 5059 cpu_manager.go:414] "containerMap: removing container" podUID="3265bd58-d654-467c-9b2a-14cedc27b754" containerName="registry" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967528 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3265bd58-d654-467c-9b2a-14cedc27b754" containerName="registry" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967549 5059 cpu_manager.go:414] "containerMap: removing container" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967556 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967566 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967572 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967588 5059 cpu_manager.go:414] "containerMap: removing container" podUID="f9d935b2-aa51-474c-836d-a74f91efab07" containerName="connection-checker" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967595 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d935b2-aa51-474c-836d-a74f91efab07" containerName="connection-checker" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967611 5059 cpu_manager.go:414] "containerMap: removing container" podUID="603868b0-8628-4227-8471-1894e2e1de50" containerName="router" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967618 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="603868b0-8628-4227-8471-1894e2e1de50" containerName="router" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967631 5059 cpu_manager.go:414] "containerMap: removing container" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967639 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" Nov 03 20:34:00.989334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:00.967651 5059 cpu_manager.go:414] "containerMap: removing container" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" Nov 03 20:34:01.429090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.386155 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" Nov 03 20:34:01.429090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.386204 5059 cpu_manager.go:414] "containerMap: removing container" podUID="dc10f92e-e902-48c4-9cd5-4f768c4611be" containerName="revoke-keys" Nov 03 20:34:01.429090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.386216 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc10f92e-e902-48c4-9cd5-4f768c4611be" containerName="revoke-keys" Nov 03 20:34:01.429090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.386439 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1326708b-4642-4992-91ea-8a590863a703" containerName="revoke-keys" Nov 03 20:34:01.429090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.386461 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2155512-2116-44f6-be5e-77cef72f9e14" containerName="connection-checker" Nov 03 20:34:01.429090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.386473 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1326708b-4642-4992-91ea-8a590863a703" containerName="revoke-keys" Nov 03 20:34:01.429090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.386486 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b9428a2-ab50-4e44-a725-e0da980bdb93" containerName="revoke-keys" Nov 03 20:34:01.429090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.386496 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc10f92e-e902-48c4-9cd5-4f768c4611be" containerName="revoke-keys" Nov 03 20:34:01.429090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.386505 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="cde430c2-2518-49a0-9851-42356d3f2fe9" containerName="main" Nov 03 20:34:01.429090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.386516 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7cf1267-7d06-459b-9df7-fc7c477d36e1" containerName="main" Nov 03 20:34:01.429090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.386525 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="541bc63e-66f1-42ca-a946-aedc34beb62f" containerName="main" Nov 03 20:34:01.429090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.386538 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b9428a2-ab50-4e44-a725-e0da980bdb93" containerName="revoke-keys" Nov 03 20:34:01.429090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.386553 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fc518fc-6302-4a1a-9ff9-f1e036a2199b" containerName="connection-checker" Nov 03 20:34:01.429090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.386562 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3265bd58-d654-467c-9b2a-14cedc27b754" containerName="registry" Nov 03 20:34:01.429090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.386574 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="16c463fa-8ddf-43c8-b00f-3f5f9ca33c02" containerName="main" Nov 03 20:34:01.429090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.386585 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a57d851-12b3-4535-b12c-b7d2c579d0fb" containerName="main" Nov 03 20:34:01.429090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.386597 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b9428a2-ab50-4e44-a725-e0da980bdb93" containerName="revoke-keys" Nov 03 20:34:01.429090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.386607 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d935b2-aa51-474c-836d-a74f91efab07" containerName="connection-checker" Nov 03 20:34:01.429090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.386618 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="603868b0-8628-4227-8471-1894e2e1de50" containerName="router" Nov 03 20:34:01.429090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.386628 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f14445d5-52ed-487d-972c-4499966065fa" containerName="main" Nov 03 20:34:01.429090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.386637 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="20690fd2-7022-4657-8c8c-bd738493ffc0" containerName="main" Nov 03 20:34:01.429090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.386646 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc10f92e-e902-48c4-9cd5-4f768c4611be" containerName="revoke-keys" Nov 03 20:34:01.506846 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:34:01.502194 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32deeada_f159_4266_8026_9b0a9ffa5eb1.slice/crio-conmon-37953d25de0f94b8fd21a2b83b892893b8a427a680e23e835c4c6768d0a1cb53.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:34:01.556956 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.036312 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-metrics-certs\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-metrics-certs\") pod \"603868b0-8628-4227-8471-1894e2e1de50\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " Nov 03 20:34:01.556956 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.507239 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle\") pod \"603868b0-8628-4227-8471-1894e2e1de50\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " Nov 03 20:34:01.556956 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.507535 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/603868b0-8628-4227-8471-1894e2e1de50-kube-api-access-89g8g\" (UniqueName: \"kubernetes.io/projected/603868b0-8628-4227-8471-1894e2e1de50-kube-api-access-89g8g\") pod \"603868b0-8628-4227-8471-1894e2e1de50\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " Nov 03 20:34:01.556956 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.507730 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-stats-auth\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-stats-auth\") pod \"603868b0-8628-4227-8471-1894e2e1de50\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " Nov 03 20:34:01.556956 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.508288 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-default-certificate\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-default-certificate\") pod \"603868b0-8628-4227-8471-1894e2e1de50\" (UID: \"603868b0-8628-4227-8471-1894e2e1de50\") " Nov 03 20:34:01.619894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.160011 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:34:01.619894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.613386 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:34:01.619894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.218007 5059 scope.go:122] "RemoveContainer" containerID="a75a1e78b8fa43ee7161dbf1263337919ca861621fb782933da9fe9d87a6c4b1" Nov 03 20:34:01.619894 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:34:01.253557 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75efd589_b943_46c2_add0_d44ccdaff537.slice/crio-2d9824c6fbca0342e936df0e333132fa3d4104bf9d7e7be14d38328d09ab2343.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b615028_a2db_4ad0_8a57_53efc040d6a6.slice/crio-conmon-608a9becc0865d2fa635a042e78e2d293a90887169f9f7a9a78ac4715c431fc8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75efd589_b943_46c2_add0_d44ccdaff537.slice/crio-conmon-2d9824c6fbca0342e936df0e333132fa3d4104bf9d7e7be14d38328d09ab2343.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82894901_9901_4dc4_948a_321782ef8288.slice/crio-conmon-dfba132bc5182982715f89fef8a8bee5073dca359534056d29abfab70c0a5c2b.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:34:01.707882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.303969 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Nov 03 20:34:01.707882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.689795 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Nov 03 20:34:01.707882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.304008 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Nov 03 20:34:01.707882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.689859 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Nov 03 20:34:01.707882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.304044 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" start-of-body= Nov 03 20:34:01.707882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.689906 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" Nov 03 20:34:01.707882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.388115 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:34:01.707882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.689947 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:34:01.707882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.397418 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Nov 03 20:34:01.707882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.689988 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Nov 03 20:34:01.707882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.435292 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" start-of-body= Nov 03 20:34:01.707882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.690026 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" Nov 03 20:34:01.707882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.435336 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Nov 03 20:34:01.707882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.690065 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Nov 03 20:34:01.707882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.464855 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 20:34:01.707882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.690102 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 20:34:01.707882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.577266 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:34:01.707882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.690138 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:34:01.707882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.611812 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-default-certificate" pod "603868b0-8628-4227-8471-1894e2e1de50" (UID: "603868b0-8628-4227-8471-1894e2e1de50"). InnerVolumeSpecName "default-certificate". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:01.879747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:01.863395 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-7w5r9"] Nov 03 20:34:02.030687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.018312 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle" pod "603868b0-8628-4227-8471-1894e2e1de50" (UID: "603868b0-8628-4227-8471-1894e2e1de50"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:02.030687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.024554 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:34:02.030687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.024625 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:34:02.114688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.075771 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-stats-auth" pod "603868b0-8628-4227-8471-1894e2e1de50" (UID: "603868b0-8628-4227-8471-1894e2e1de50"). InnerVolumeSpecName "stats-auth". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:02.269936 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.120504 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-7w5r9" Nov 03 20:34:02.519184 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.136840 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42fe8135-7b7e-4292-ab0d-5ba57bbd3699-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-7w5r9\" (UID: \"42fe8135-7b7e-4292-ab0d-5ba57bbd3699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-7w5r9" Nov 03 20:34:02.653689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.082077 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Nov 03 20:34:02.653689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.610162 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Nov 03 20:34:02.653689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.572207 5059 scope.go:122] "RemoveContainer" containerID="3904c49bb7ab63c515d01a6052936b8d990a8f9bb352ab2810819de54dc7b74c" Nov 03 20:34:02.653689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.109624 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" start-of-body= Nov 03 20:34:02.653689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.611016 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" Nov 03 20:34:02.653689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.137177 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603868b0-8628-4227-8471-1894e2e1de50-kube-api-access-89g8g" pod "603868b0-8628-4227-8471-1894e2e1de50" (UID: "603868b0-8628-4227-8471-1894e2e1de50"). InnerVolumeSpecName "kube-api-access-89g8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:02.653689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.227242 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-metrics-certs" pod "603868b0-8628-4227-8471-1894e2e1de50" (UID: "603868b0-8628-4227-8471-1894e2e1de50"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:02.653689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.249526 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:34:02.653689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.611251 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:34:02.653689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.482707 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Nov 03 20:34:02.653689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.611432 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Nov 03 20:34:02.687908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.687863 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtc4t\" (UniqueName: \"kubernetes.io/projected/42fe8135-7b7e-4292-ab0d-5ba57bbd3699-kube-api-access-xtc4t\") pod \"maas-api-revoke-sub-abad4031-7w5r9\" (UID: \"42fe8135-7b7e-4292-ab0d-5ba57bbd3699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-7w5r9" Nov 03 20:34:02.709801 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.709764 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42fe8135-7b7e-4292-ab0d-5ba57bbd3699-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-7w5r9\" (UID: \"42fe8135-7b7e-4292-ab0d-5ba57bbd3699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-7w5r9" Nov 03 20:34:02.710650 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.710626 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89g8g\" (UniqueName: \"kubernetes.io/projected/603868b0-8628-4227-8471-1894e2e1de50-kube-api-access-89g8g\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:02.710839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.710824 5059 reconciler_common.go:299] "Volume detached for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-stats-auth\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:02.710961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.710946 5059 reconciler_common.go:299] "Volume detached for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-default-certificate\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:02.711058 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.711046 5059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/603868b0-8628-4227-8471-1894e2e1de50-metrics-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:02.711160 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.711149 5059 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/603868b0-8628-4227-8471-1894e2e1de50-service-ca-bundle\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:02.714753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.708833 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 20:34:02.714951 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.714914 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 20:34:02.757037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.748451 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:34:02.757037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.748522 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:34:02.782204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.780745 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:34:02.782204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.780815 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" podUID="82894901-9901-4dc4-948a-321782ef8288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:34:02.782204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.781066 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1326708b-4642-4992-91ea-8a590863a703" path="/var/lib/kubelet/pods/1326708b-4642-4992-91ea-8a590863a703/volumes" Nov 03 20:34:02.782453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.782229 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b9428a2-ab50-4e44-a725-e0da980bdb93" path="/var/lib/kubelet/pods/6b9428a2-ab50-4e44-a725-e0da980bdb93/volumes" Nov 03 20:34:02.790686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.783356 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-nw2ph"] Nov 03 20:34:02.814901 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.813417 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Nov 03 20:34:02.814901 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.813474 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Nov 03 20:34:02.814901 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.813569 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Nov 03 20:34:02.814901 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.813591 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Nov 03 20:34:02.814901 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.813685 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Nov 03 20:34:02.814901 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.813712 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Nov 03 20:34:02.814901 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.813790 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" start-of-body= Nov 03 20:34:02.814901 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.813812 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" Nov 03 20:34:02.814901 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.813884 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:34:02.814901 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.813907 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:34:02.895041 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.879699 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body= Nov 03 20:34:02.895041 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.879765 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" Nov 03 20:34:02.909908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.904127 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtc4t\" (UniqueName: \"kubernetes.io/projected/42fe8135-7b7e-4292-ab0d-5ba57bbd3699-kube-api-access-xtc4t\") pod \"maas-api-revoke-sub-abad4031-7w5r9\" (UID: \"42fe8135-7b7e-4292-ab0d-5ba57bbd3699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-7w5r9" Nov 03 20:34:02.909908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.904253 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42fe8135-7b7e-4292-ab0d-5ba57bbd3699-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-7w5r9\" (UID: \"42fe8135-7b7e-4292-ab0d-5ba57bbd3699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-7w5r9" Nov 03 20:34:02.909908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.904519 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42fe8135-7b7e-4292-ab0d-5ba57bbd3699-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-7w5r9\" (UID: \"42fe8135-7b7e-4292-ab0d-5ba57bbd3699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-7w5r9" Nov 03 20:34:02.914335 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.910762 5059 cpu_manager.go:414] "containerMap: removing container" podUID="6b9428a2-ab50-4e44-a725-e0da980bdb93" containerName="revoke-keys" Nov 03 20:34:02.914335 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.910792 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b9428a2-ab50-4e44-a725-e0da980bdb93" containerName="revoke-keys" Nov 03 20:34:02.914335 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.910919 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1326708b-4642-4992-91ea-8a590863a703" containerName="revoke-keys" Nov 03 20:34:02.914335 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.910928 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1326708b-4642-4992-91ea-8a590863a703" containerName="revoke-keys" Nov 03 20:34:02.914335 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.910952 5059 cpu_manager.go:414] "containerMap: removing container" podUID="dc10f92e-e902-48c4-9cd5-4f768c4611be" containerName="revoke-keys" Nov 03 20:34:02.914335 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.910962 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc10f92e-e902-48c4-9cd5-4f768c4611be" containerName="revoke-keys" Nov 03 20:34:02.914335 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.911464 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1326708b-4642-4992-91ea-8a590863a703" containerName="revoke-keys" Nov 03 20:34:02.914335 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:02.911487 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc10f92e-e902-48c4-9cd5-4f768c4611be" containerName="revoke-keys" Nov 03 20:34:03.018536 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.018481 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:34:03.018791 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.018762 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:34:03.058077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.054978 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42fe8135-7b7e-4292-ab0d-5ba57bbd3699-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-7w5r9\" (UID: \"42fe8135-7b7e-4292-ab0d-5ba57bbd3699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-7w5r9" Nov 03 20:34:03.072957 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.072898 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:34:03.075871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.075838 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:34:03.115836 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.115799 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:34:03.177028 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.176978 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" podUID="82894901-9901-4dc4-948a-321782ef8288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:34:03.177283 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.116833 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" start-of-body= Nov 03 20:34:03.177410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.177380 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" Nov 03 20:34:03.177840 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.177810 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 20:34:03.198455 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.178060 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 20:34:03.376172 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.376080 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Nov 03 20:34:03.376426 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.376388 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Nov 03 20:34:03.376684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.376633 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Nov 03 20:34:03.376847 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.376826 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Nov 03 20:34:03.377033 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.377016 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" start-of-body= Nov 03 20:34:03.377145 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.377125 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" Nov 03 20:34:03.392687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.392638 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:34:03.392922 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.392880 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:34:03.393151 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.393134 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Nov 03 20:34:03.393299 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.393280 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Nov 03 20:34:03.412943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.412910 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42fe8135-7b7e-4292-ab0d-5ba57bbd3699-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-7w5r9\" (UID: \"42fe8135-7b7e-4292-ab0d-5ba57bbd3699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-7w5r9" Nov 03 20:34:03.476045 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.476005 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-nw2ph"] Nov 03 20:34:03.476455 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.476435 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:34:03.476599 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.476588 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:34:03.476800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.476786 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:34:03.476970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.476957 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" Nov 03 20:34:03.477146 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.477130 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-8sgnl"] Nov 03 20:34:03.532674 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.526504 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtc4t\" (UniqueName: \"kubernetes.io/projected/42fe8135-7b7e-4292-ab0d-5ba57bbd3699-kube-api-access-xtc4t\") pod \"maas-api-revoke-sub-abad4031-7w5r9\" (UID: \"42fe8135-7b7e-4292-ab0d-5ba57bbd3699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-7w5r9" Nov 03 20:34:03.556806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.533824 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:34:03.556806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.533873 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:34:03.556806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.533973 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7dmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body= Nov 03 20:34:03.556806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.534013 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" Nov 03 20:34:03.556806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.534160 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-nw2ph" Nov 03 20:34:03.556806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.535139 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Nov 03 20:34:03.556806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.535186 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Nov 03 20:34:03.561134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.561106 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-7w5r9" Nov 03 20:34:03.573961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.562723 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:34:03.573961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:03.562781 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:34:04.128771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.123021 5059 scope.go:122] "RemoveContainer" containerID="f35a41707de2c0253145d89c0f1f6796087f8f0d6d98dc9ca16c8dcbf8e24a1e" Nov 03 20:34:04.176687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.169586 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" start-of-body= Nov 03 20:34:04.176687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.169643 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" Nov 03 20:34:04.295688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.245060 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Nov 03 20:34:04.335359 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.335308 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Nov 03 20:34:04.336681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.257896 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Nov 03 20:34:04.336885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.336860 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Nov 03 20:34:04.349745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.261734 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" start-of-body= Nov 03 20:34:04.517102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.454090 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" Nov 03 20:34:04.517102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.366339 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31725d6d-9a91-44ba-a08b-1520f5853190-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bdd0b59-nw2ph\" (UID: \"31725d6d-9a91-44ba-a08b-1520f5853190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-nw2ph" Nov 03 20:34:04.517102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.454288 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxmhs\" (UniqueName: \"kubernetes.io/projected/31725d6d-9a91-44ba-a08b-1520f5853190-kube-api-access-gxmhs\") pod \"maas-api-revoke-sub-4bdd0b59-nw2ph\" (UID: \"31725d6d-9a91-44ba-a08b-1520f5853190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-nw2ph" Nov 03 20:34:04.517102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.288428 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7dmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body= Nov 03 20:34:04.517102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.457525 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" Nov 03 20:34:04.517102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.288520 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:34:04.517102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.458079 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:34:04.517102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.292377 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:34:04.517102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.458153 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:34:04.517102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.329456 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 20:34:04.517102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.458219 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 20:34:04.517102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.365899 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Nov 03 20:34:04.517102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.458275 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Nov 03 20:34:04.517102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.397164 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Nov 03 20:34:04.517102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.458316 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Nov 03 20:34:04.517102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.412515 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:34:04.517102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.458379 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" podUID="82894901-9901-4dc4-948a-321782ef8288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:34:04.517102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.500073 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31725d6d-9a91-44ba-a08b-1520f5853190-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bdd0b59-nw2ph\" (UID: \"31725d6d-9a91-44ba-a08b-1520f5853190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-nw2ph" Nov 03 20:34:04.543362 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.543324 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:34:04.545470 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.545433 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:34:04.950896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.658875 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31725d6d-9a91-44ba-a08b-1520f5853190-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bdd0b59-nw2ph\" (UID: \"31725d6d-9a91-44ba-a08b-1520f5853190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-nw2ph" Nov 03 20:34:04.950896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.659030 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31725d6d-9a91-44ba-a08b-1520f5853190-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bdd0b59-nw2ph\" (UID: \"31725d6d-9a91-44ba-a08b-1520f5853190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-nw2ph" Nov 03 20:34:04.950896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.659119 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxmhs\" (UniqueName: \"kubernetes.io/projected/31725d6d-9a91-44ba-a08b-1520f5853190-kube-api-access-gxmhs\") pod \"maas-api-revoke-sub-4bdd0b59-nw2ph\" (UID: \"31725d6d-9a91-44ba-a08b-1520f5853190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-nw2ph" Nov 03 20:34:04.950896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.678567 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31725d6d-9a91-44ba-a08b-1520f5853190-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bdd0b59-nw2ph\" (UID: \"31725d6d-9a91-44ba-a08b-1520f5853190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-nw2ph" Nov 03 20:34:04.950896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.689195 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-8sgnl" Nov 03 20:34:04.950896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.841686 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:34:04.950896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.841750 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused (Client.Timeout exceeded while awaiting headers)" Nov 03 20:34:04.950896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.850255 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:34:04.950896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.850311 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:34:04.950896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.858238 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmn6k\" (UniqueName: \"kubernetes.io/projected/b39ff65e-aa88-4534-a0f1-cd2af557c4ed-kube-api-access-gmn6k\") pod \"maas-api-revoke-sub-266d0609-8sgnl\" (UID: \"b39ff65e-aa88-4534-a0f1-cd2af557c4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-8sgnl" Nov 03 20:34:04.950896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.858317 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39ff65e-aa88-4534-a0f1-cd2af557c4ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-266d0609-8sgnl\" (UID: \"b39ff65e-aa88-4534-a0f1-cd2af557c4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-8sgnl" Nov 03 20:34:04.950896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.858424 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39ff65e-aa88-4534-a0f1-cd2af557c4ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-266d0609-8sgnl\" (UID: \"b39ff65e-aa88-4534-a0f1-cd2af557c4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-8sgnl" Nov 03 20:34:04.950896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.863090 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:34:04.950896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.863136 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" podUID="82894901-9901-4dc4-948a-321782ef8288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:34:05.028885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:04.977645 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxmhs\" (UniqueName: \"kubernetes.io/projected/31725d6d-9a91-44ba-a08b-1520f5853190-kube-api-access-gxmhs\") pod \"maas-api-revoke-sub-4bdd0b59-nw2ph\" (UID: \"31725d6d-9a91-44ba-a08b-1520f5853190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-nw2ph" Nov 03 20:34:05.028885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.010843 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-8sgnl"] Nov 03 20:34:05.028885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.010942 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:34:05.028885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.011030 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:34:05.028885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.011058 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:34:05.028885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.011085 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:34:05.028885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.011110 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:34:05.075689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.072579 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:34:05.111686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.107643 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39ff65e-aa88-4534-a0f1-cd2af557c4ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-266d0609-8sgnl\" (UID: \"b39ff65e-aa88-4534-a0f1-cd2af557c4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-8sgnl" Nov 03 20:34:05.111686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.107797 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39ff65e-aa88-4534-a0f1-cd2af557c4ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-266d0609-8sgnl\" (UID: \"b39ff65e-aa88-4534-a0f1-cd2af557c4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-8sgnl" Nov 03 20:34:05.111686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.108125 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmn6k\" (UniqueName: \"kubernetes.io/projected/b39ff65e-aa88-4534-a0f1-cd2af557c4ed-kube-api-access-gmn6k\") pod \"maas-api-revoke-sub-266d0609-8sgnl\" (UID: \"b39ff65e-aa88-4534-a0f1-cd2af557c4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-8sgnl" Nov 03 20:34:05.113217 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.113167 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" start-of-body= Nov 03 20:34:05.113522 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.113461 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" Nov 03 20:34:05.116485 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.116462 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39ff65e-aa88-4534-a0f1-cd2af557c4ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-266d0609-8sgnl\" (UID: \"b39ff65e-aa88-4534-a0f1-cd2af557c4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-8sgnl" Nov 03 20:34:05.154954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.139020 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31725d6d-9a91-44ba-a08b-1520f5853190-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bdd0b59-nw2ph\" (UID: \"31725d6d-9a91-44ba-a08b-1520f5853190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-nw2ph" Nov 03 20:34:05.154954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.139690 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmn6k\" (UniqueName: \"kubernetes.io/projected/b39ff65e-aa88-4534-a0f1-cd2af557c4ed-kube-api-access-gmn6k\") pod \"maas-api-revoke-sub-266d0609-8sgnl\" (UID: \"b39ff65e-aa88-4534-a0f1-cd2af557c4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-8sgnl" Nov 03 20:34:05.154954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.140144 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" start-of-body= Nov 03 20:34:05.154954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.140189 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" Nov 03 20:34:05.154954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.140832 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 20:34:05.154954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.140869 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 20:34:05.186752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.183255 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Nov 03 20:34:05.186752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.183316 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Nov 03 20:34:05.186752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.183580 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Nov 03 20:34:05.186752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.183616 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Nov 03 20:34:05.186752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.184212 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:34:05.186752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.184249 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:34:05.186752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.184319 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:34:05.257946 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.230787 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:34:05.317583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.317547 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:34:05.317824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.317458 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx" event={"ID":"847e7b38-8fb0-4c19-8943-f43e364f989b","Type":"ContainerStarted","Data":"8452ce584ec710dade06364894274fbcd2c189f440545b82d81e7ad1fc47dfb7"} Nov 03 20:34:05.328790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.234399 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7dmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body= Nov 03 20:34:05.396156 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.396110 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" Nov 03 20:34:05.396514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.396486 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Nov 03 20:34:05.396764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.396646 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Nov 03 20:34:05.439910 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.397746 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Nov 03 20:34:05.484818 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.484769 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Nov 03 20:34:05.485041 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.452161 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-nw2ph" Nov 03 20:34:05.502382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.502331 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39ff65e-aa88-4534-a0f1-cd2af557c4ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-266d0609-8sgnl\" (UID: \"b39ff65e-aa88-4534-a0f1-cd2af557c4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-8sgnl" Nov 03 20:34:05.520805 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.508600 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" podStartSLOduration=122.508579935 podStartE2EDuration="2m2.508579935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:02.338971977 +0000 UTC m=+3436.596769236" watchObservedRunningTime="2026-11-03 20:34:05.508579935 +0000 UTC m=+3439.766377195" Nov 03 20:34:05.626727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.575083 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:34:05.681452 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.638839 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" podStartSLOduration=121.638816763 podStartE2EDuration="2m1.638816763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:03.560949546 +0000 UTC m=+3437.818746828" watchObservedRunningTime="2026-11-03 20:34:05.638816763 +0000 UTC m=+3439.896614021" Nov 03 20:34:05.691850 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.681220 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:34:05.692125 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.640193 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-8sgnl" Nov 03 20:34:05.699704 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.694216 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" event={"ID":"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06","Type":"ContainerStarted","Data":"5932a6c7fb7e966629bc5438e3483e6beb7b50f48ede3eee43850147c22bdddd"} Nov 03 20:34:05.723706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.723560 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" event={"ID":"82894901-9901-4dc4-948a-321782ef8288","Type":"ContainerStarted","Data":"dfba132bc5182982715f89fef8a8bee5073dca359534056d29abfab70c0a5c2b"} Nov 03 20:34:05.773687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.770407 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:34:05.778835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.778795 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:34:05.779019 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.776961 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" event={"ID":"7101a242-11ce-4b8d-9a74-dae3baf6444b","Type":"ContainerStarted","Data":"fb62268796fa9f889a9bbeab400464d27f27b14dca4f08044b07d6c2d0ba642b"} Nov 03 20:34:05.820699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.810873 5059 scope.go:122] "RemoveContainer" containerID="6739ff178c81c67d823d4e8761f1f61d4d648a31a61e02195fb24f7ce8ca0bdf" Nov 03 20:34:05.888725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.853979 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:34:05.888725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.879098 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:34:05.888725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:05.879154 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" podUID="82894901-9901-4dc4-948a-321782ef8288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:34:06.284698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.284093 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:34:06.284698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.284160 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:34:06.284698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.284187 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:34:06.284698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.284209 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" event={"ID":"d4fdd390-7fe2-406d-832a-7ecc545d97f4","Type":"ContainerStarted","Data":"c9577eca08e6a70c108edb2ac96c1b9d5f54df9956d0d1467ddc72fffdf59343"} Nov 03 20:34:06.284698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.284238 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:34:06.284698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.284260 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95" event={"ID":"0298aa52-aec0-481a-b7d1-12964e8137c8","Type":"ContainerStarted","Data":"97b5e89796f952bd7b97823c3ecf143080a6c7e2492e836cfba674e2901c0056"} Nov 03 20:34:06.284698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.284286 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:34:06.284698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.284309 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:34:06.284698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.284329 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95" Nov 03 20:34:06.365362 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.364086 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:34:06.365362 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.365293 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:34:06.422803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.421025 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7dmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body= Nov 03 20:34:06.422803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.421087 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" Nov 03 20:34:06.422803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.421166 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:34:06.422803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.421187 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:34:06.422803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.421246 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Nov 03 20:34:06.422803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.421266 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Nov 03 20:34:06.422803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.421329 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Nov 03 20:34:06.422803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.421347 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Nov 03 20:34:06.422803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.421416 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Nov 03 20:34:06.422803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.421464 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Nov 03 20:34:06.425165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.423500 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 20:34:06.425165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.423539 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 20:34:06.425165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.423607 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Nov 03 20:34:06.425165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.423627 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Nov 03 20:34:06.425165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.423705 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" start-of-body= Nov 03 20:34:06.425165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.423725 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" Nov 03 20:34:06.425165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.423779 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:34:06.425165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.423797 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:34:06.425165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.423850 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" start-of-body= Nov 03 20:34:06.425165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.423869 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" Nov 03 20:34:06.492686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.492606 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Nov 03 20:34:06.554748 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.522730 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" podUID="35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Nov 03 20:34:06.554748 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.538993 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:34:06.554748 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.539046 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:34:06.583627 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.583502 5059 scope.go:122] "RemoveContainer" containerID="90b3ff7ea089036c1955493d6ad07b652eb65d8c482b1abd7f0dc8c58bea2751" Nov 03 20:34:06.609149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.609120 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:34:06.639688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.631047 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body= Nov 03 20:34:06.639688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.631104 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" podUID="d4fdd390-7fe2-406d-832a-7ecc545d97f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" Nov 03 20:34:06.727573 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.702414 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-4vl4x" event={"ID":"94e83c10-58b0-4bc7-9abd-3e144ba6e258","Type":"ContainerStarted","Data":"7dbfa1ff25e2a926d9d9feaa2ace0d152c5e6229ded2d8b9803f7a1302037649"} Nov 03 20:34:06.817790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.812594 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:34:06.817790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.812648 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:34:06.878706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.862384 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:34:06.878706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.862444 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" podUID="82894901-9901-4dc4-948a-321782ef8288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:34:06.878706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.864874 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" event={"ID":"68bc094a-84e2-41c2-9fb5-081682413f27","Type":"ContainerStarted","Data":"a96237ecf3f4902402f80b450ff7a22b4fcd8a6500b222e4be665f0e434b7f54"} Nov 03 20:34:06.917497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.917457 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" event={"ID":"32deeada-f159-4266-8026-9b0a9ffa5eb1","Type":"ContainerStarted","Data":"37953d25de0f94b8fd21a2b83b892893b8a427a680e23e835c4c6768d0a1cb53"} Nov 03 20:34:06.918809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:06.918780 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" Nov 03 20:34:07.038969 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.011911 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:34:07.107889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.101807 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" start-of-body= Nov 03 20:34:07.107889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.101871 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" podUID="68bc094a-84e2-41c2-9fb5-081682413f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" Nov 03 20:34:07.118721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.117827 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" start-of-body= Nov 03 20:34:07.118721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.117889 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" Nov 03 20:34:07.141686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.141367 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 20:34:07.141686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.141437 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 20:34:07.141686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.141517 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" start-of-body= Nov 03 20:34:07.141686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.141539 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" Nov 03 20:34:07.158568 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.142564 5059 scope.go:122] "RemoveContainer" containerID="6abcd4b68690b0791ce17fbaaa16ead4fd929c811755fa8d2211289ee770f9bd" Nov 03 20:34:07.162815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.158961 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Nov 03 20:34:07.162815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.159018 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Nov 03 20:34:07.162815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.159102 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Nov 03 20:34:07.162815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.159123 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Nov 03 20:34:07.174818 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.174786 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f7b87d695-cwlfq" Nov 03 20:34:07.212681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.212020 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:34:07.212681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.212040 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:34:07.212681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.212079 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:34:07.212681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.212105 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:34:07.212681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.212519 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7dmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body= Nov 03 20:34:07.212681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.212553 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" Nov 03 20:34:07.257859 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.225431 5059 scope.go:122] "RemoveContainer" containerID="dccc4c886ebdb414a11a97a53696b63ef424a4fe2c14f42ce427a8a3da8f9b10" Nov 03 20:34:07.257859 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:34:07.225896 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-fb827799-ndv2h_odh-ai-gateway-infra(28034528-e342-417a-a9df-050d7ea91453)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h" podUID="28034528-e342-417a-a9df-050d7ea91453" Nov 03 20:34:07.257859 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.225955 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h" event={"ID":"28034528-e342-417a-a9df-050d7ea91453","Type":"ContainerStarted","Data":"dccc4c886ebdb414a11a97a53696b63ef424a4fe2c14f42ce427a8a3da8f9b10"} Nov 03 20:34:07.257859 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.256558 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Nov 03 20:34:07.257859 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.256619 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" podUID="35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Nov 03 20:34:07.359608 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.359508 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Nov 03 20:34:07.360293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.360259 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Nov 03 20:34:07.360798 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.360774 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Nov 03 20:34:07.360980 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.360958 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Nov 03 20:34:07.610768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.595765 5059 scope.go:122] "RemoveContainer" containerID="bc571c0f9d14acdd1fada8fd767a2f23f5170cfca0dc658da6d8f8ab02daf45d" Nov 03 20:34:07.610768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.606356 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body= Nov 03 20:34:07.610768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.606419 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" podUID="d4fdd390-7fe2-406d-832a-7ecc545d97f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" Nov 03 20:34:07.637786 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.635554 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:34:07.637786 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.635623 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:34:07.809902 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.809856 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:34:07.810233 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.810199 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:34:08.008603 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:07.963020 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:34:08.528267 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.528219 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" podUID="82894901-9901-4dc4-948a-321782ef8288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:34:08.529789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.145655 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" podStartSLOduration=125.14563329 podStartE2EDuration="2m5.14563329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:05.775669919 +0000 UTC m=+3440.033467169" watchObservedRunningTime="2026-11-03 20:34:08.14563329 +0000 UTC m=+3442.403430561" Nov 03 20:34:08.549775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.076512 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" start-of-body= Nov 03 20:34:08.550066 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.550027 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" Nov 03 20:34:08.550225 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.078769 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" start-of-body= Nov 03 20:34:08.550335 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.550318 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" podUID="68bc094a-84e2-41c2-9fb5-081682413f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" Nov 03 20:34:08.550453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.145254 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 20:34:08.550574 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.550552 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 20:34:08.550733 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.145307 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" start-of-body= Nov 03 20:34:08.550867 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.550848 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.134:15021/healthz/ready\": dial tcp 10.132.1.134:15021: connect: connection refused" Nov 03 20:34:08.550971 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.245903 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Nov 03 20:34:08.557725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.551068 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Nov 03 20:34:08.557938 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.274870 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:34:08.558085 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.558063 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:34:08.558208 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.274953 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7dmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body= Nov 03 20:34:08.558311 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.558294 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" Nov 03 20:34:08.558570 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.285629 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Nov 03 20:34:08.558744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.558721 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Nov 03 20:34:08.558862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.412204 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Nov 03 20:34:08.558978 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.558957 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" podUID="35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Nov 03 20:34:08.640137 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.623162 5059 scope.go:122] "RemoveContainer" containerID="502104f017af359d04ca3c8ec6ed2634250c518da5bf724e38ed745f86c73474" Nov 03 20:34:08.640137 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.623654 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:34:08.640137 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.623732 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:34:08.726728 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.696527 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" podStartSLOduration=125.69650089 podStartE2EDuration="2m5.69650089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:06.735366557 +0000 UTC m=+3440.993163815" watchObservedRunningTime="2026-11-03 20:34:08.69650089 +0000 UTC m=+3442.954298149" Nov 03 20:34:08.727005 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.726912 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Nov 03 20:34:08.727082 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:08.726996 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Nov 03 20:34:09.272688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:09.243923 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-wz2w4" podUID="d098ef77-addb-4077-b6d4-f0ae477409b8" containerName="main" probeResult="failure" output="Get \"https://10.132.1.82:8000/ready\": context deadline exceeded" Nov 03 20:34:09.272688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:09.272062 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body= Nov 03 20:34:09.272688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:09.272129 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" podUID="d4fdd390-7fe2-406d-832a-7ecc545d97f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" Nov 03 20:34:09.442688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:09.433820 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:34:09.546602 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:09.546547 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:34:09.654689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:09.494829 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 20:34:09.658685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:09.654963 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 20:34:09.658903 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:09.494894 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Nov 03 20:34:09.659032 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:09.659008 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Nov 03 20:34:09.659155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:09.494930 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Nov 03 20:34:09.659268 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:09.659250 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Nov 03 20:34:09.662739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:09.546185 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Nov 03 20:34:09.662906 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:09.662885 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" podUID="35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Nov 03 20:34:09.663017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:09.650655 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:34:09.663109 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:09.663093 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:34:09.778711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:09.706774 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" start-of-body= Nov 03 20:34:09.778711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:09.758264 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" Nov 03 20:34:09.778711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:09.754079 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:34:09.778711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:09.758342 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:34:09.778711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:09.758121 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body= Nov 03 20:34:09.778711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:09.758404 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" podUID="d4fdd390-7fe2-406d-832a-7ecc545d97f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" Nov 03 20:34:09.780555 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:09.780509 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Nov 03 20:34:09.780813 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:09.780771 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Nov 03 20:34:09.806138 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:09.806064 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95" podStartSLOduration=121.130054059 podStartE2EDuration="2m7.806028533s" totalImagesPullingTime="6.675974474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:06.846917298 +0000 UTC m=+3441.104714557" watchObservedRunningTime="2026-11-03 20:34:09.806028533 +0000 UTC m=+3444.063825791" Nov 03 20:34:10.073691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.046958 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Nov 03 20:34:10.073691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.047042 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Nov 03 20:34:10.099689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.075292 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:34:10.099689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.075361 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:34:10.099689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.075471 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" start-of-body= Nov 03 20:34:10.099689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.075494 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" podUID="68bc094a-84e2-41c2-9fb5-081682413f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" Nov 03 20:34:10.099689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.076061 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7dmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body= Nov 03 20:34:10.099689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.076102 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" Nov 03 20:34:10.103256 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:34:10.103222 5059 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.019s" Nov 03 20:34:10.196763 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.189103 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Nov 03 20:34:10.196763 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.189189 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Nov 03 20:34:10.196763 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.189295 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Nov 03 20:34:10.196763 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.189330 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Nov 03 20:34:10.196763 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.189403 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:34:10.196763 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.189440 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" podUID="82894901-9901-4dc4-948a-321782ef8288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:34:10.213506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.213463 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:34:10.246442 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.246393 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:34:10.371011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.226886 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:34:10.371011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.338856 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" podUID="82894901-9901-4dc4-948a-321782ef8288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:34:10.371011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.242250 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" start-of-body= Nov 03 20:34:10.371011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.338950 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" Nov 03 20:34:10.371011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.313051 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Nov 03 20:34:10.371011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.339008 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" podUID="35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Nov 03 20:34:10.418385 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.413023 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:34:10.418385 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.413096 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:34:10.427059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.419879 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Nov 03 20:34:10.427059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.419940 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Nov 03 20:34:10.440693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.439808 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7dmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body= Nov 03 20:34:10.440693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.439894 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" Nov 03 20:34:10.440693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.439976 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:34:10.440693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.439998 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:34:10.466691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.449383 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Nov 03 20:34:10.466691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.449438 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Nov 03 20:34:10.543691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.538368 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:34:10.543691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.538430 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:34:10.647092 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.595350 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body= Nov 03 20:34:10.647092 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.595432 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" podUID="d4fdd390-7fe2-406d-832a-7ecc545d97f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" Nov 03 20:34:10.647092 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.626043 5059 generic.go:356] "Generic (PLEG): container finished" podID="6c29d53f-b738-49ec-9b15-1c88e5950008" containerID="af2790ddecc29d9fe9f64c8e69ec5c674fb2d6254584e35b58eeafbca6d951e5" exitCode=0 Nov 03 20:34:10.789689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.763540 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:34:10.789689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.763602 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:34:10.815690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.790437 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47" event={"ID":"6c29d53f-b738-49ec-9b15-1c88e5950008","Type":"ContainerDied","Data":"af2790ddecc29d9fe9f64c8e69ec5c674fb2d6254584e35b58eeafbca6d951e5"} Nov 03 20:34:10.815690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.790583 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:34:10.815690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.790651 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:34:10.815690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.797287 5059 scope.go:122] "RemoveContainer" containerID="ca6370d9da34484b1b31be792a765188027ebdb4619fdaee7682eef377ae36b3" Nov 03 20:34:10.820959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.820881 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:34:10.843697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.836979 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" podStartSLOduration=128.836958033 podStartE2EDuration="2m8.836958033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:06.94118099 +0000 UTC m=+3441.198978245" watchObservedRunningTime="2026-11-03 20:34:10.836958033 +0000 UTC m=+3445.094755290" Nov 03 20:34:10.889713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.879153 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:34:10.889713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.879217 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" podUID="82894901-9901-4dc4-948a-321782ef8288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:34:10.896512 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.896477 5059 generic.go:356] "Generic (PLEG): container finished" podID="28034528-e342-417a-a9df-050d7ea91453" containerID="dccc4c886ebdb414a11a97a53696b63ef424a4fe2c14f42ce427a8a3da8f9b10" exitCode=7 Nov 03 20:34:10.926744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.901270 5059 scope.go:122] "RemoveContainer" containerID="dccc4c886ebdb414a11a97a53696b63ef424a4fe2c14f42ce427a8a3da8f9b10" Nov 03 20:34:10.926744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:10.901208 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h" event={"ID":"28034528-e342-417a-a9df-050d7ea91453","Type":"ContainerDied","Data":"dccc4c886ebdb414a11a97a53696b63ef424a4fe2c14f42ce427a8a3da8f9b10"} Nov 03 20:34:10.926744 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:34:10.901708 5059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-fb827799-ndv2h_odh-ai-gateway-infra(28034528-e342-417a-a9df-050d7ea91453)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h" podUID="28034528-e342-417a-a9df-050d7ea91453" Nov 03 20:34:11.019987 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.002070 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" start-of-body= Nov 03 20:34:11.019987 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.002150 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" podUID="68bc094a-84e2-41c2-9fb5-081682413f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" Nov 03 20:34:11.076885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.074775 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" start-of-body= Nov 03 20:34:11.076885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.074842 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" Nov 03 20:34:11.155141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.150268 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Nov 03 20:34:11.155141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.150510 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Nov 03 20:34:11.478222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.461337 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7dmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body= Nov 03 20:34:11.478222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.461399 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" Nov 03 20:34:11.478222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.461853 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:34:11.478222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.461897 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:34:11.478222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.461966 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Nov 03 20:34:11.478222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.461986 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Nov 03 20:34:11.478222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.467755 5059 scope.go:122] "RemoveContainer" containerID="b13a3b8cfa23ddc9815549092296b81b15bde977434b7a3b30d1385e768e8fb0" Nov 03 20:34:11.478222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.476682 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Nov 03 20:34:11.478222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.476726 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" podUID="35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Nov 03 20:34:11.478222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.477592 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Nov 03 20:34:11.478222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.477632 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Nov 03 20:34:11.478222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.477901 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:34:11.478222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.477931 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:34:11.612929 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.590126 5059 generic.go:356] "Generic (PLEG): container finished" podID="847e7b38-8fb0-4c19-8943-f43e364f989b" containerID="8452ce584ec710dade06364894274fbcd2c189f440545b82d81e7ad1fc47dfb7" exitCode=0 Nov 03 20:34:11.612929 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.590424 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx" event={"ID":"847e7b38-8fb0-4c19-8943-f43e364f989b","Type":"ContainerDied","Data":"8452ce584ec710dade06364894274fbcd2c189f440545b82d81e7ad1fc47dfb7"} Nov 03 20:34:11.612929 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.590992 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Nov 03 20:34:11.612929 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.591042 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Nov 03 20:34:11.612929 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.607653 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:34:11.612929 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.607723 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:34:11.709755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.632566 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" podStartSLOduration=124.846463925 podStartE2EDuration="2m8.632544223s" totalImagesPullingTime="2.417668336s" totalInitContainerRuntime="1.368411962s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:07.103351923 +0000 UTC m=+3441.361149210" watchObservedRunningTime="2026-11-03 20:34:11.632544223 +0000 UTC m=+3445.890341482" Nov 03 20:34:11.839805 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.830772 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:34:11.839805 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.831093 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:34:11.906688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.864798 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body= Nov 03 20:34:11.906688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.864861 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" podUID="d4fdd390-7fe2-406d-832a-7ecc545d97f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" Nov 03 20:34:11.906688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:11.865753 5059 scope.go:122] "RemoveContainer" containerID="c63223667135eb8a17f59b5174cf095c231af49811bd42e93ad4f78c3ee6ba55" Nov 03 20:34:12.018064 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.009966 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" start-of-body= Nov 03 20:34:12.018064 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.010025 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" podUID="68bc094a-84e2-41c2-9fb5-081682413f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" Nov 03 20:34:12.018064 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:34:12.014446 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:34:12.018064 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.015850 5059 generic.go:356] "Generic (PLEG): container finished" podID="94e83c10-58b0-4bc7-9abd-3e144ba6e258" containerID="7dbfa1ff25e2a926d9d9feaa2ace0d152c5e6229ded2d8b9803f7a1302037649" exitCode=0 Nov 03 20:34:12.044138 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.043222 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-4vl4x" event={"ID":"94e83c10-58b0-4bc7-9abd-3e144ba6e258","Type":"ContainerDied","Data":"7dbfa1ff25e2a926d9d9feaa2ace0d152c5e6229ded2d8b9803f7a1302037649"} Nov 03 20:34:12.062706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.059031 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" start-of-body= Nov 03 20:34:12.099702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.059086 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" Nov 03 20:34:12.136984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.136949 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95" Nov 03 20:34:12.212292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.200540 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:34:12.212292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.200706 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" podUID="82894901-9901-4dc4-948a-321782ef8288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:34:12.212292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.200584 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Nov 03 20:34:12.212292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.201131 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Nov 03 20:34:12.212292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.201509 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Nov 03 20:34:12.212292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.201543 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Nov 03 20:34:12.212292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.201602 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7dmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body= Nov 03 20:34:12.212292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.201622 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" Nov 03 20:34:12.299911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.215207 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:34:12.299911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.215259 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:34:12.403854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.399217 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Nov 03 20:34:12.403854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.399274 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Nov 03 20:34:12.403854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.399217 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Nov 03 20:34:12.403854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.399334 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Nov 03 20:34:12.478732 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.435821 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:34:12.679698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.635528 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:34:12.679698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.635592 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:34:12.729732 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.722848 5059 scope.go:122] "RemoveContainer" containerID="1638b310e0522c5f2a20ee3d226436456869be0ba4a5f3e1f6c3bdebb4a48c87" Nov 03 20:34:12.865066 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.833555 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body= Nov 03 20:34:12.865066 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.833620 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" podUID="d4fdd390-7fe2-406d-832a-7ecc545d97f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" Nov 03 20:34:12.897758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.885063 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:34:12.897758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.885123 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:34:12.931645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.913195 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:34:12.931645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.913248 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" podUID="82894901-9901-4dc4-948a-321782ef8288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:34:12.977574 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.977536 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Nov 03 20:34:12.978471 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.978437 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" podUID="35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Nov 03 20:34:12.979144 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.979123 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:34:12.979321 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:12.979300 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:34:13.019767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:13.015115 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-dgdfv" Nov 03 20:34:13.037752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:13.034981 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" start-of-body= Nov 03 20:34:13.037752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:13.035040 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" podUID="68bc094a-84e2-41c2-9fb5-081682413f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" Nov 03 20:34:13.086687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:13.084623 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" start-of-body= Nov 03 20:34:13.086687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:13.084709 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" Nov 03 20:34:13.236837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:13.236785 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Nov 03 20:34:13.240685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:13.237058 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Nov 03 20:34:13.331737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:13.280880 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Nov 03 20:34:13.447838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:13.447797 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" podUID="35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Nov 03 20:34:13.648922 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:13.293791 5059 scope.go:122] "RemoveContainer" containerID="6638b2c26dcbd2beddb822cddbf45294c8f85f17724b2ee963373d4a3c6be587" Nov 03 20:34:13.648922 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:13.290989 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:34:13.648922 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:13.634933 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:34:13.648922 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:13.453347 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:34:13.648922 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:13.380821 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:34:13.648922 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:13.636442 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:34:13.648922 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:13.393794 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7dmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body= Nov 03 20:34:13.648922 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:13.636496 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" Nov 03 20:34:13.648922 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:13.428807 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Nov 03 20:34:13.648922 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:13.636536 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Nov 03 20:34:13.906696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:13.871005 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body= Nov 03 20:34:13.906696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:13.871069 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" podUID="d4fdd390-7fe2-406d-832a-7ecc545d97f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" Nov 03 20:34:14.379690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:14.369154 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:34:14.379690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:14.369225 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:34:14.379690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:14.370389 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Nov 03 20:34:14.379690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:14.370432 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Nov 03 20:34:14.529765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:14.500529 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" start-of-body= Nov 03 20:34:14.741726 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:14.741356 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.122:15021/healthz/ready\": dial tcp 10.132.1.122:15021: connect: connection refused" Nov 03 20:34:14.741726 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:14.739032 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" podUID="60b2ec36-7e99-46e0-9671-1c500c402e45" containerName="main" probeResult="failure" output="Get \"https://10.132.1.89:8000/ready\": context deadline exceeded" Nov 03 20:34:14.741975 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:14.577799 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Nov 03 20:34:14.741975 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:14.741815 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" podUID="35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Nov 03 20:34:14.741975 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:14.577858 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Nov 03 20:34:14.741975 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:14.741878 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Nov 03 20:34:14.741975 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:14.577910 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:34:14.741975 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:14.741925 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:34:14.741975 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:14.609921 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:34:14.742353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:14.741980 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" podUID="82894901-9901-4dc4-948a-321782ef8288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:34:14.742353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:14.611583 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" start-of-body= Nov 03 20:34:14.742353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:14.742024 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" Nov 03 20:34:14.742353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:14.704831 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:34:14.742353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:14.742060 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:34:14.742353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:14.705012 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" start-of-body= Nov 03 20:34:14.742353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:14.742105 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" podUID="68bc094a-84e2-41c2-9fb5-081682413f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" Nov 03 20:34:14.772338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:14.772304 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body= Nov 03 20:34:14.772612 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:14.772567 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" podUID="d4fdd390-7fe2-406d-832a-7ecc545d97f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" Nov 03 20:34:14.875699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:14.873835 5059 scope.go:122] "RemoveContainer" containerID="f0ca6f95c31b4ccad9131b19d71014e3f54afa9b79e7f53f9d77d4b861fb6af9" Nov 03 20:34:15.197210 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.054159 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" start-of-body= Nov 03 20:34:15.217263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.217014 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.132:15021/healthz/ready\": dial tcp 10.132.1.132:15021: connect: connection refused" Nov 03 20:34:15.245864 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.135501 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-4vl4x" Nov 03 20:34:15.247490 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:34:15.247457 5059 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.194s" Nov 03 20:34:15.380718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.373337 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:34:15.380718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.373413 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/router-default-f7b87d695-cwlfq"] Nov 03 20:34:15.380718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.373441 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/router-default-f7b87d695-cwlfq"] Nov 03 20:34:15.380718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.373526 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:34:15.380718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.113352 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7dmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body= Nov 03 20:34:15.380718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.373728 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" Nov 03 20:34:15.380718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.114103 5059 scope.go:122] "RemoveContainer" containerID="735b32748a8b907b9eed5ef21efd8284798ec21dd97653a226fd406ff5d16e6e" Nov 03 20:34:15.380718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.113431 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:34:15.380718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.378796 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:34:15.380718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.113702 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Nov 03 20:34:15.380718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.379138 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Nov 03 20:34:15.380718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.141428 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" start-of-body= Nov 03 20:34:15.380718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.379417 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" podUID="68bc094a-84e2-41c2-9fb5-081682413f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" Nov 03 20:34:15.380718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.152524 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:34:15.380718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.379478 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" podUID="82894901-9901-4dc4-948a-321782ef8288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:34:15.380718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.199341 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" start-of-body= Nov 03 20:34:15.380718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.379534 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.123:15021/healthz/ready\": dial tcp 10.132.1.123:15021: connect: connection refused" Nov 03 20:34:15.380718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.199601 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:34:15.380718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.379592 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:34:15.380718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.353549 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Nov 03 20:34:15.380718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.379640 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" podUID="35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Nov 03 20:34:15.382878 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.382854 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Nov 03 20:34:15.421884 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.421784 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Nov 03 20:34:15.483818 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.383035 5059 scope.go:122] "RemoveContainer" containerID="dccc4c886ebdb414a11a97a53696b63ef424a4fe2c14f42ce427a8a3da8f9b10" Nov 03 20:34:15.484110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.383165 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7dmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body= Nov 03 20:34:15.484276 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.484251 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" Nov 03 20:34:15.484408 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.483262 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:34:15.497690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.444859 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:34:15.498355 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.498321 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:34:15.558879 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.542453 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:34:15.717386 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.559644 5059 scope.go:122] "RemoveContainer" containerID="5c19ca06fbb1d449a708204a75d839cc487a9ea18245ce5649dd7dfdd316e86f" Nov 03 20:34:15.717852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.575717 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:34:15.718094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.718050 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:34:15.718275 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.652699 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body= Nov 03 20:34:15.718441 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.718402 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" podUID="d4fdd390-7fe2-406d-832a-7ecc545d97f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" Nov 03 20:34:15.763255 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.762983 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:34:15.877908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.821684 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-7w5r9"] Nov 03 20:34:15.877908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.862026 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:34:15.877908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.862101 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" podUID="82894901-9901-4dc4-948a-321782ef8288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:34:15.922697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:15.902709 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e83c10-58b0-4bc7-9abd-3e144ba6e258-kube-api-access-nksbf\" (UniqueName: \"kubernetes.io/projected/94e83c10-58b0-4bc7-9abd-3e144ba6e258-kube-api-access-nksbf\") pod \"94e83c10-58b0-4bc7-9abd-3e144ba6e258\" (UID: \"94e83c10-58b0-4bc7-9abd-3e144ba6e258\") " Nov 03 20:34:16.297560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.092866 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" start-of-body= Nov 03 20:34:16.303241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.303198 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" podUID="68bc094a-84e2-41c2-9fb5-081682413f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" Nov 03 20:34:16.303442 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.171152 5059 generic.go:356] "Generic (PLEG): container finished" podID="42fe8135-7b7e-4292-ab0d-5ba57bbd3699" containerID="bf14b10e6eed091103f0a91d03c839e8e63da4240c83391a08fcc3b818d4bdc8" exitCode=0 Nov 03 20:34:16.310742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.194809 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:34:16.310942 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.310915 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:34:16.311106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.194892 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7dmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body= Nov 03 20:34:16.328876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.328851 5059 scope.go:122] "RemoveContainer" containerID="a75a1e78b8fa43ee7161dbf1263337919ca861621fb782933da9fe9d87a6c4b1" Nov 03 20:34:16.340003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.339969 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" Nov 03 20:34:16.361032 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.360990 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:34:16.361273 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.361242 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:34:16.361913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.361875 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Nov 03 20:34:16.362179 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.362153 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" podUID="35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Nov 03 20:34:16.387121 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:34:16.387075 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a75a1e78b8fa43ee7161dbf1263337919ca861621fb782933da9fe9d87a6c4b1\": container with ID starting with a75a1e78b8fa43ee7161dbf1263337919ca861621fb782933da9fe9d87a6c4b1 not found: ID does not exist" containerID="a75a1e78b8fa43ee7161dbf1263337919ca861621fb782933da9fe9d87a6c4b1" Nov 03 20:34:16.387378 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:34:16.387349 5059 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"a75a1e78b8fa43ee7161dbf1263337919ca861621fb782933da9fe9d87a6c4b1\": rpc error: code = NotFound desc = could not find container \"a75a1e78b8fa43ee7161dbf1263337919ca861621fb782933da9fe9d87a6c4b1\": container with ID starting with a75a1e78b8fa43ee7161dbf1263337919ca861621fb782933da9fe9d87a6c4b1 not found: ID does not exist" containerID="a75a1e78b8fa43ee7161dbf1263337919ca861621fb782933da9fe9d87a6c4b1" Nov 03 20:34:16.388103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.388073 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" start-of-body= Nov 03 20:34:16.388395 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.388363 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.128:15021/healthz/ready\": dial tcp 10.132.1.128:15021: connect: connection refused" Nov 03 20:34:16.459683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.456842 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94e83c10-58b0-4bc7-9abd-3e144ba6e258-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e83c10-58b0-4bc7-9abd-3e144ba6e258-maas-api-xks-ca\") pod \"94e83c10-58b0-4bc7-9abd-3e144ba6e258\" (UID: \"94e83c10-58b0-4bc7-9abd-3e144ba6e258\") " Nov 03 20:34:16.459683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.457056 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94e83c10-58b0-4bc7-9abd-3e144ba6e258-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e83c10-58b0-4bc7-9abd-3e144ba6e258-maas-api-openshift-ca\") pod \"94e83c10-58b0-4bc7-9abd-3e144ba6e258\" (UID: \"94e83c10-58b0-4bc7-9abd-3e144ba6e258\") " Nov 03 20:34:16.572949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.571984 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-mnmjc" podUID="89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5" containerName="main" probeResult="failure" output="Get \"https://10.132.1.115:8000/ready\": context deadline exceeded" Nov 03 20:34:16.577870 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.577830 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e83c10-58b0-4bc7-9abd-3e144ba6e258-kube-api-access-nksbf" pod "94e83c10-58b0-4bc7-9abd-3e144ba6e258" (UID: "94e83c10-58b0-4bc7-9abd-3e144ba6e258"). InnerVolumeSpecName "kube-api-access-nksbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:16.639996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.632239 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r" podUID="60b2ec36-7e99-46e0-9671-1c500c402e45" containerName="main" probeResult="failure" output="Get \"https://10.132.1.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:34:16.640168 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.640043 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:34:16.640168 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.640092 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:34:16.642243 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.642211 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="603868b0-8628-4227-8471-1894e2e1de50" path="/var/lib/kubelet/pods/603868b0-8628-4227-8471-1894e2e1de50/volumes" Nov 03 20:34:16.643381 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.643361 5059 scope.go:122] "RemoveContainer" containerID="f8323dfa4e2c1a85ed81f607b727475c16361a359dbb2bb285fe28a219d7a6eb" Nov 03 20:34:16.650637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.650611 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-nw2ph" event={"ID":"31725d6d-9a91-44ba-a08b-1520f5853190","Type":"ContainerStarted","Data":"26f0370e8a062fc5d6b350cb6562de91ff7496aa5deead5b631ec0ec052ef2dc"} Nov 03 20:34:16.656691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.651075 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:34:16.658902 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.658883 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-8sgnl"] Nov 03 20:34:16.659101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.659067 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-nw2ph" event={"ID":"31725d6d-9a91-44ba-a08b-1520f5853190","Type":"ContainerStarted","Data":"3536854a95b53f56e0e8a2a2fcc916c38fe9d1155595b712a1a94897173fd39b"} Nov 03 20:34:16.691718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.691276 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nksbf\" (UniqueName: \"kubernetes.io/projected/94e83c10-58b0-4bc7-9abd-3e144ba6e258-kube-api-access-nksbf\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:16.695802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.695773 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-7w5r9" event={"ID":"42fe8135-7b7e-4292-ab0d-5ba57bbd3699","Type":"ContainerDied","Data":"bf14b10e6eed091103f0a91d03c839e8e63da4240c83391a08fcc3b818d4bdc8"} Nov 03 20:34:16.696022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.695996 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-7w5r9" event={"ID":"42fe8135-7b7e-4292-ab0d-5ba57bbd3699","Type":"ContainerStarted","Data":"e9b161d400ab53eb83dfceccb6bc0b09de85aef83c06c7b21ea4a11ff6c7e571"} Nov 03 20:34:17.458800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.443843 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-nw2ph"] Nov 03 20:34:17.458800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.764246 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body= Nov 03 20:34:17.458800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.444340 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" podUID="d4fdd390-7fe2-406d-832a-7ecc545d97f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" Nov 03 20:34:17.458800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:16.980195 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e83c10-58b0-4bc7-9abd-3e144ba6e258-maas-api-openshift-ca" pod "94e83c10-58b0-4bc7-9abd-3e144ba6e258" (UID: "94e83c10-58b0-4bc7-9abd-3e144ba6e258"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:17.458800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.012858 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" start-of-body= Nov 03 20:34:17.458800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.444410 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" podUID="68bc094a-84e2-41c2-9fb5-081682413f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" Nov 03 20:34:17.458800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.066778 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:34:17.458800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.444774 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" podUID="82894901-9901-4dc4-948a-321782ef8288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:34:17.458800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.214944 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7dmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body= Nov 03 20:34:17.458800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.444833 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" Nov 03 20:34:17.458800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.244174 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Nov 03 20:34:17.458800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.444885 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Nov 03 20:34:17.458800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.330155 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Nov 03 20:34:17.458800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.444924 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" podUID="35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Nov 03 20:34:17.488131 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.464775 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e83c10-58b0-4bc7-9abd-3e144ba6e258-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:17.488131 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.471762 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:34:17.488131 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.471878 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:34:17.488131 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.471946 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:34:17.594325 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.594291 5059 scope.go:122] "RemoveContainer" containerID="8eba0c0d93805e32f585be9b5dffe0df1cf0f78d3928ac2e35a5a76d60ddd0e3" Nov 03 20:34:17.596397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.596361 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:34:17.618976 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.618954 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:34:17.619417 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.619380 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:34:17.619596 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.619569 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:34:17.621775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.621735 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-8sgnl" event={"ID":"b39ff65e-aa88-4534-a0f1-cd2af557c4ed","Type":"ContainerStarted","Data":"34b644e595dd4c7fa011e16e93e0d748f3b8f70ba73413e7d75dd921cc60c8b5"} Nov 03 20:34:17.621954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.621934 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-8sgnl" event={"ID":"b39ff65e-aa88-4534-a0f1-cd2af557c4ed","Type":"ContainerStarted","Data":"997914bb2fef476abb0320da6c2569885574565fc8cce86ac419f8b8f03c91ba"} Nov 03 20:34:17.794706 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:34:17.775276 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42fe8135_7b7e_4292_ab0d_5ba57bbd3699.slice/crio-bf14b10e6eed091103f0a91d03c839e8e63da4240c83391a08fcc3b818d4bdc8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31725d6d_9a91_44ba_a08b_1520f5853190.slice/crio-26f0370e8a062fc5d6b350cb6562de91ff7496aa5deead5b631ec0ec052ef2dc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42fe8135_7b7e_4292_ab0d_5ba57bbd3699.slice/crio-conmon-bf14b10e6eed091103f0a91d03c839e8e63da4240c83391a08fcc3b818d4bdc8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39ff65e_aa88_4534_a0f1_cd2af557c4ed.slice/crio-conmon-34b644e595dd4c7fa011e16e93e0d748f3b8f70ba73413e7d75dd921cc60c8b5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31725d6d_9a91_44ba_a08b_1520f5853190.slice/crio-conmon-26f0370e8a062fc5d6b350cb6562de91ff7496aa5deead5b631ec0ec052ef2dc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39ff65e_aa88_4534_a0f1_cd2af557c4ed.slice/crio-34b644e595dd4c7fa011e16e93e0d748f3b8f70ba73413e7d75dd921cc60c8b5.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:34:17.874694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.873562 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:34:17.874694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.873784 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:34:17.959098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.958999 5059 scope.go:122] "RemoveContainer" containerID="c1c608afee5d8326b20645c4ef715824387423915b54182a5c217c34ffe83cf2" Nov 03 20:34:17.977698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.960905 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:34:17.986690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:17.977959 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" podUID="82894901-9901-4dc4-948a-321782ef8288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:34:18.033186 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:18.009506 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" start-of-body= Nov 03 20:34:18.033186 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:18.009576 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" podUID="68bc094a-84e2-41c2-9fb5-081682413f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" Nov 03 20:34:18.058962 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:18.058924 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body= Nov 03 20:34:18.059288 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:18.059234 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" podUID="d4fdd390-7fe2-406d-832a-7ecc545d97f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" Nov 03 20:34:18.106767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:18.106726 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94e83c10-58b0-4bc7-9abd-3e144ba6e258-maas-api-xks-ca" pod "94e83c10-58b0-4bc7-9abd-3e144ba6e258" (UID: "94e83c10-58b0-4bc7-9abd-3e144ba6e258"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:18.207499 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:18.204164 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7dmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body= Nov 03 20:34:18.240580 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:18.240484 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" Nov 03 20:34:18.256877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:18.253723 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Nov 03 20:34:18.256877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:18.253782 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" podUID="35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Nov 03 20:34:18.291431 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:18.272872 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e83c10-58b0-4bc7-9abd-3e144ba6e258-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:18.433898 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:18.423909 5059 scope.go:122] "RemoveContainer" containerID="ddfb36a61a4890d032f40a96fa7dc33ada9d7745461eee81dc563f215eda9eae" Nov 03 20:34:18.540655 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:18.540617 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body= Nov 03 20:34:18.556980 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:18.556935 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" Nov 03 20:34:18.604691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:18.599169 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body= Nov 03 20:34:18.604691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:18.599227 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" podUID="d4fdd390-7fe2-406d-832a-7ecc545d97f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" Nov 03 20:34:18.969856 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:18.969819 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:34:18.970321 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:18.970258 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" podUID="82894901-9901-4dc4-948a-321782ef8288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:34:19.022982 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:19.022944 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" start-of-body= Nov 03 20:34:19.023368 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:19.023293 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" podUID="68bc094a-84e2-41c2-9fb5-081682413f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" Nov 03 20:34:19.346325 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:19.254313 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7dmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body= Nov 03 20:34:19.346325 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:19.254375 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" Nov 03 20:34:19.528517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:19.528468 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Nov 03 20:34:19.528865 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:19.528830 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" podUID="35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Nov 03 20:34:19.612753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:19.587258 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:34:19.676775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:19.605793 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body= Nov 03 20:34:19.677077 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:19.677041 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" podUID="d4fdd390-7fe2-406d-832a-7ecc545d97f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" Nov 03 20:34:19.828730 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:19.815710 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47"] Nov 03 20:34:19.976739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:19.965431 5059 generic.go:356] "Generic (PLEG): container finished" podID="31725d6d-9a91-44ba-a08b-1520f5853190" containerID="26f0370e8a062fc5d6b350cb6562de91ff7496aa5deead5b631ec0ec052ef2dc" exitCode=0 Nov 03 20:34:20.001731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:19.998484 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" start-of-body= Nov 03 20:34:20.001731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:19.998542 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" podUID="68bc094a-84e2-41c2-9fb5-081682413f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" Nov 03 20:34:20.001731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:19.999393 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:34:20.001731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:19.999438 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" podUID="82894901-9901-4dc4-948a-321782ef8288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:34:20.022617 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:20.022580 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-nt297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:34:20.022890 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:20.022859 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:34:20.487690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:20.484558 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Nov 03 20:34:20.487690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:20.484638 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" podUID="35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Nov 03 20:34:20.487690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:20.485105 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7dmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body= Nov 03 20:34:20.487690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:20.485144 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" Nov 03 20:34:20.840905 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:20.840871 5059 scope.go:122] "RemoveContainer" containerID="386f7e5573035b544a49ee0875358940b7fed0c314140f35957f5ac8a63fe021" Nov 03 20:34:21.093314 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:34:21.005532 5059 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.04s" Nov 03 20:34:21.106687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:21.103802 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-nw2ph" event={"ID":"31725d6d-9a91-44ba-a08b-1520f5853190","Type":"ContainerDied","Data":"26f0370e8a062fc5d6b350cb6562de91ff7496aa5deead5b631ec0ec052ef2dc"} Nov 03 20:34:21.106687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:21.104083 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:34:21.106687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:21.104158 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:34:21.106687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:21.104315 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:34:21.148753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:21.061938 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" start-of-body= Nov 03 20:34:21.149076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:21.149029 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" podUID="68bc094a-84e2-41c2-9fb5-081682413f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" Nov 03 20:34:21.407462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:21.407350 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body= Nov 03 20:34:21.407743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:21.407697 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" podUID="d4fdd390-7fe2-406d-832a-7ecc545d97f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" Nov 03 20:34:21.572854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:21.569126 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7dmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body= Nov 03 20:34:21.572854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:21.569190 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" Nov 03 20:34:21.709382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:21.707800 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-4vl4x" Nov 03 20:34:21.709382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:21.708221 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-4vl4x" event={"ID":"94e83c10-58b0-4bc7-9abd-3e144ba6e258","Type":"ContainerDied","Data":"16c66826c64dd74f6d30973c00c34771034c241ce9bba4601580e67e603b88b9"} Nov 03 20:34:21.709382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:21.708286 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16c66826c64dd74f6d30973c00c34771034c241ce9bba4601580e67e603b88b9" Nov 03 20:34:21.709382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:21.708370 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:34:21.709382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:21.708445 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:34:21.884688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:21.866686 5059 scope.go:122] "RemoveContainer" containerID="e7fcbceef75077181869de0f40c073966ec17006f2685b45f17586d4fcf3eeb3" Nov 03 20:34:21.957894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:21.950167 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Nov 03 20:34:21.957894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:21.950598 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" podUID="35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Nov 03 20:34:22.435843 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.172092 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" start-of-body= Nov 03 20:34:22.436222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.436183 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" podUID="68bc094a-84e2-41c2-9fb5-081682413f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" Nov 03 20:34:22.436412 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.333031 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47" Nov 03 20:34:22.442364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.215055 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx" Nov 03 20:34:22.442364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.333213 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7dmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body= Nov 03 20:34:22.442364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.439995 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" Nov 03 20:34:22.456172 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.456137 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c29d53f-b738-49ec-9b15-1c88e5950008-kube-api-access-rxxr7\" (UniqueName: \"kubernetes.io/projected/6c29d53f-b738-49ec-9b15-1c88e5950008-kube-api-access-rxxr7\") pod \"6c29d53f-b738-49ec-9b15-1c88e5950008\" (UID: \"6c29d53f-b738-49ec-9b15-1c88e5950008\") " Nov 03 20:34:22.456340 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.456214 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/847e7b38-8fb0-4c19-8943-f43e364f989b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/847e7b38-8fb0-4c19-8943-f43e364f989b-maas-api-xks-ca\") pod \"847e7b38-8fb0-4c19-8943-f43e364f989b\" (UID: \"847e7b38-8fb0-4c19-8943-f43e364f989b\") " Nov 03 20:34:22.456340 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.456297 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c29d53f-b738-49ec-9b15-1c88e5950008-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c29d53f-b738-49ec-9b15-1c88e5950008-maas-api-xks-ca\") pod \"6c29d53f-b738-49ec-9b15-1c88e5950008\" (UID: \"6c29d53f-b738-49ec-9b15-1c88e5950008\") " Nov 03 20:34:22.456459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.456414 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/847e7b38-8fb0-4c19-8943-f43e364f989b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/847e7b38-8fb0-4c19-8943-f43e364f989b-maas-api-openshift-ca\") pod \"847e7b38-8fb0-4c19-8943-f43e364f989b\" (UID: \"847e7b38-8fb0-4c19-8943-f43e364f989b\") " Nov 03 20:34:22.456459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.456450 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/847e7b38-8fb0-4c19-8943-f43e364f989b-kube-api-access-nlhpl\" (UniqueName: \"kubernetes.io/projected/847e7b38-8fb0-4c19-8943-f43e364f989b-kube-api-access-nlhpl\") pod \"847e7b38-8fb0-4c19-8943-f43e364f989b\" (UID: \"847e7b38-8fb0-4c19-8943-f43e364f989b\") " Nov 03 20:34:22.456562 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.456538 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c29d53f-b738-49ec-9b15-1c88e5950008-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c29d53f-b738-49ec-9b15-1c88e5950008-maas-api-openshift-ca\") pod \"6c29d53f-b738-49ec-9b15-1c88e5950008\" (UID: \"6c29d53f-b738-49ec-9b15-1c88e5950008\") " Nov 03 20:34:22.568582 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.568530 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body= Nov 03 20:34:22.568931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.568896 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" podUID="d4fdd390-7fe2-406d-832a-7ecc545d97f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" Nov 03 20:34:22.569222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.569190 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" start-of-body= Nov 03 20:34:22.569325 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.569279 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" podUID="35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.108:15021/healthz/ready\": dial tcp 10.132.1.108:15021: connect: connection refused" Nov 03 20:34:22.650118 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.649176 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-nw2ph" Nov 03 20:34:22.654950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.654895 5059 generic.go:356] "Generic (PLEG): container finished" podID="b39ff65e-aa88-4534-a0f1-cd2af557c4ed" containerID="34b644e595dd4c7fa011e16e93e0d748f3b8f70ba73413e7d75dd921cc60c8b5" exitCode=0 Nov 03 20:34:22.743181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.738294 5059 scope.go:122] "RemoveContainer" containerID="50a334bf18a40ce26b6f3aa885ab25406b24910372b8a40e1242ebd2b8c1f4b1" Nov 03 20:34:22.743181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.740857 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c29d53f-b738-49ec-9b15-1c88e5950008-maas-api-openshift-ca" pod "6c29d53f-b738-49ec-9b15-1c88e5950008" (UID: "6c29d53f-b738-49ec-9b15-1c88e5950008"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:22.746412 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.746379 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:34:22.746556 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.746441 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-8sgnl" event={"ID":"b39ff65e-aa88-4534-a0f1-cd2af557c4ed","Type":"ContainerDied","Data":"34b644e595dd4c7fa011e16e93e0d748f3b8f70ba73413e7d75dd921cc60c8b5"} Nov 03 20:34:22.749284 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.749241 5059 generic.go:356] "Generic (PLEG): container finished" podID="28034528-e342-417a-a9df-050d7ea91453" containerID="9dfc60c513c642a762a15433fbf7ab06f633fb6d150365a70a84fd6ecc7aeee0" exitCode=0 Nov 03 20:34:22.749597 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.749575 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h" event={"ID":"28034528-e342-417a-a9df-050d7ea91453","Type":"ContainerDied","Data":"9dfc60c513c642a762a15433fbf7ab06f633fb6d150365a70a84fd6ecc7aeee0"} Nov 03 20:34:22.749790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.749764 5059 scope.go:122] "RemoveContainer" containerID="dccc4c886ebdb414a11a97a53696b63ef424a4fe2c14f42ce427a8a3da8f9b10" Nov 03 20:34:22.754507 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.754476 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c29d53f-b738-49ec-9b15-1c88e5950008-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:22.774777 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.757763 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx" Nov 03 20:34:22.798968 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.797557 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/847e7b38-8fb0-4c19-8943-f43e364f989b-maas-api-openshift-ca" pod "847e7b38-8fb0-4c19-8943-f43e364f989b" (UID: "847e7b38-8fb0-4c19-8943-f43e364f989b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:22.857525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.809459 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c29d53f-b738-49ec-9b15-1c88e5950008-maas-api-xks-ca" pod "6c29d53f-b738-49ec-9b15-1c88e5950008" (UID: "6c29d53f-b738-49ec-9b15-1c88e5950008"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:22.857525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.821297 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847e7b38-8fb0-4c19-8943-f43e364f989b-kube-api-access-nlhpl" pod "847e7b38-8fb0-4c19-8943-f43e364f989b" (UID: "847e7b38-8fb0-4c19-8943-f43e364f989b"). InnerVolumeSpecName "kube-api-access-nlhpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:22.857525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.826718 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/847e7b38-8fb0-4c19-8943-f43e364f989b-maas-api-xks-ca" pod "847e7b38-8fb0-4c19-8943-f43e364f989b" (UID: "847e7b38-8fb0-4c19-8943-f43e364f989b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:22.868726 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.867462 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31725d6d-9a91-44ba-a08b-1520f5853190-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31725d6d-9a91-44ba-a08b-1520f5853190-maas-api-openshift-ca\") pod \"31725d6d-9a91-44ba-a08b-1520f5853190\" (UID: \"31725d6d-9a91-44ba-a08b-1520f5853190\") " Nov 03 20:34:22.868726 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.867619 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31725d6d-9a91-44ba-a08b-1520f5853190-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31725d6d-9a91-44ba-a08b-1520f5853190-maas-api-xks-ca\") pod \"31725d6d-9a91-44ba-a08b-1520f5853190\" (UID: \"31725d6d-9a91-44ba-a08b-1520f5853190\") " Nov 03 20:34:22.868726 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.867820 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31725d6d-9a91-44ba-a08b-1520f5853190-kube-api-access-gxmhs\" (UniqueName: \"kubernetes.io/projected/31725d6d-9a91-44ba-a08b-1520f5853190-kube-api-access-gxmhs\") pod \"31725d6d-9a91-44ba-a08b-1520f5853190\" (UID: \"31725d6d-9a91-44ba-a08b-1520f5853190\") " Nov 03 20:34:22.881205 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.876421 5059 scope.go:122] "RemoveContainer" containerID="3a0b8447bfda8a58216cc853554baf15762b4ccc77d0d98ce90bfe2224c3b122" Nov 03 20:34:22.897900 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.895820 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31725d6d-9a91-44ba-a08b-1520f5853190-maas-api-xks-ca" pod "31725d6d-9a91-44ba-a08b-1520f5853190" (UID: "31725d6d-9a91-44ba-a08b-1520f5853190"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:22.898518 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.898467 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31725d6d-9a91-44ba-a08b-1520f5853190-maas-api-openshift-ca" pod "31725d6d-9a91-44ba-a08b-1520f5853190" (UID: "31725d6d-9a91-44ba-a08b-1520f5853190"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:22.914893 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.914513 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/847e7b38-8fb0-4c19-8943-f43e364f989b-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:22.914893 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.914549 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31725d6d-9a91-44ba-a08b-1520f5853190-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:22.914893 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.914566 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31725d6d-9a91-44ba-a08b-1520f5853190-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:22.914893 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.914582 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c29d53f-b738-49ec-9b15-1c88e5950008-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:22.914893 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.914597 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/847e7b38-8fb0-4c19-8943-f43e364f989b-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:22.914893 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.914614 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlhpl\" (UniqueName: \"kubernetes.io/projected/847e7b38-8fb0-4c19-8943-f43e364f989b-kube-api-access-nlhpl\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:22.936949 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.936908 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c29d53f-b738-49ec-9b15-1c88e5950008-kube-api-access-rxxr7" pod "6c29d53f-b738-49ec-9b15-1c88e5950008" (UID: "6c29d53f-b738-49ec-9b15-1c88e5950008"). InnerVolumeSpecName "kube-api-access-rxxr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:22.938493 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.938463 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:34:22.938904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.938880 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31725d6d-9a91-44ba-a08b-1520f5853190-kube-api-access-gxmhs" pod "31725d6d-9a91-44ba-a08b-1520f5853190" (UID: "31725d6d-9a91-44ba-a08b-1520f5853190"). InnerVolumeSpecName "kube-api-access-gxmhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:22.993457 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.993390 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:34:22.993797 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.993776 5059 scope.go:122] "RemoveContainer" containerID="a1545bd75166fa5a492a8e6723253de1761cd417b14f627fbdab40e21ab83652" Nov 03 20:34:22.994293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.994266 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" start-of-body= Nov 03 20:34:22.994451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:22.994426 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" podUID="68bc094a-84e2-41c2-9fb5-081682413f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.133:15021/healthz/ready\": dial tcp 10.132.1.133:15021: connect: connection refused" Nov 03 20:34:23.020181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.020149 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxxr7\" (UniqueName: \"kubernetes.io/projected/6c29d53f-b738-49ec-9b15-1c88e5950008-kube-api-access-rxxr7\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.020382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.020364 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxmhs\" (UniqueName: \"kubernetes.io/projected/31725d6d-9a91-44ba-a08b-1520f5853190-kube-api-access-gxmhs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.198101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.198062 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7dmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body= Nov 03 20:34:23.198596 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.198538 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" Nov 03 20:34:23.209079 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.209050 5059 scope.go:122] "RemoveContainer" containerID="083bf41b34c7c17d7a323c71536ca53488c9b5d6024a331fe82c42a4a9894a43" Nov 03 20:34:23.246852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.246788 5059 scope.go:122] "RemoveContainer" containerID="4436c68602fe345d6bb4fb9dfe6e2ac4dec763399336a2db74a54312609ee767" Nov 03 20:34:23.250141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.250110 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-8sgnl" Nov 03 20:34:23.255744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.255243 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr"] Nov 03 20:34:23.255744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.255462 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" podUID="d931dc13-4399-48dc-80e0-b0b304c5dda0" containerName="istio-proxy" containerID="cri-o://51e5dd7d49f3191cbe26fa7fea29e5e5bd18af15599c15f1e53810285e8f9294" gracePeriod=30 Nov 03 20:34:23.269039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.268905 5059 scope.go:122] "RemoveContainer" containerID="6e4d31c85e8cd1b2b8591eeb7383bfd43ae2ffe368b7b0f552663b7b668eb233" Nov 03 20:34:23.325644 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.325612 5059 scope.go:122] "RemoveContainer" containerID="04da4e840db9b36ab74f969a24bc9303a68de3283b026a9e0deacd6cef05ae43" Nov 03 20:34:23.328937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.328908 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b39ff65e-aa88-4534-a0f1-cd2af557c4ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39ff65e-aa88-4534-a0f1-cd2af557c4ed-maas-api-openshift-ca\") pod \"b39ff65e-aa88-4534-a0f1-cd2af557c4ed\" (UID: \"b39ff65e-aa88-4534-a0f1-cd2af557c4ed\") " Nov 03 20:34:23.329324 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.329299 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39ff65e-aa88-4534-a0f1-cd2af557c4ed-kube-api-access-gmn6k\" (UniqueName: \"kubernetes.io/projected/b39ff65e-aa88-4534-a0f1-cd2af557c4ed-kube-api-access-gmn6k\") pod \"b39ff65e-aa88-4534-a0f1-cd2af557c4ed\" (UID: \"b39ff65e-aa88-4534-a0f1-cd2af557c4ed\") " Nov 03 20:34:23.329486 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.329467 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b39ff65e-aa88-4534-a0f1-cd2af557c4ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39ff65e-aa88-4534-a0f1-cd2af557c4ed-maas-api-xks-ca\") pod \"b39ff65e-aa88-4534-a0f1-cd2af557c4ed\" (UID: \"b39ff65e-aa88-4534-a0f1-cd2af557c4ed\") " Nov 03 20:34:23.335557 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.334109 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b39ff65e-aa88-4534-a0f1-cd2af557c4ed-maas-api-xks-ca" pod "b39ff65e-aa88-4534-a0f1-cd2af557c4ed" (UID: "b39ff65e-aa88-4534-a0f1-cd2af557c4ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:23.336121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.336098 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b39ff65e-aa88-4534-a0f1-cd2af557c4ed-maas-api-openshift-ca" pod "b39ff65e-aa88-4534-a0f1-cd2af557c4ed" (UID: "b39ff65e-aa88-4534-a0f1-cd2af557c4ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:23.339701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.339681 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:34:23.351741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.351712 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39ff65e-aa88-4534-a0f1-cd2af557c4ed-kube-api-access-gmn6k" pod "b39ff65e-aa88-4534-a0f1-cd2af557c4ed" (UID: "b39ff65e-aa88-4534-a0f1-cd2af557c4ed"). InnerVolumeSpecName "kube-api-access-gmn6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:23.352321 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.352260 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:34:23.400073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.399922 5059 scope.go:122] "RemoveContainer" containerID="1317df3f930488af045a4ea5f2ffd5b2d0b6bac1972a73208461be0692b6e2b2" Nov 03 20:34:23.433437 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.432335 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39ff65e-aa88-4534-a0f1-cd2af557c4ed-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.433437 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.432374 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmn6k\" (UniqueName: \"kubernetes.io/projected/b39ff65e-aa88-4534-a0f1-cd2af557c4ed-kube-api-access-gmn6k\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.433437 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.432396 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39ff65e-aa88-4534-a0f1-cd2af557c4ed-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.438320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.437889 5059 scope.go:122] "RemoveContainer" containerID="27b2092e2b1ae773885e5c47b4223f33ad58ad3523d44d0fdfca69cbbee17e22" Nov 03 20:34:23.467835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.467614 5059 scope.go:122] "RemoveContainer" containerID="981dfae86baecb93fdc63c743ed0c803df2f98640cac0233f4bc9661252407bf" Nov 03 20:34:23.504391 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.504315 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-7w5r9" Nov 03 20:34:23.510496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.509075 5059 scope.go:122] "RemoveContainer" containerID="6c9d6b78a3dbde46dbddcc1222495b4fce407ba8a3afc87ef4704aadd4524702" Nov 03 20:34:23.566793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.566758 5059 scope.go:122] "RemoveContainer" containerID="cdc8ec4afacb060d845fdcbcf9f9a36756ac0c3cf27302051a5a1dae638f96f6" Nov 03 20:34:23.599261 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.599234 5059 scope.go:122] "RemoveContainer" containerID="3ed08a9a04c4bb41c1603c5fbc1967f1823b1e4aefed23d4755b36d6e13d899e" Nov 03 20:34:23.618530 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.618497 5059 scope.go:122] "RemoveContainer" containerID="d610230fd1e961d1b794119c73b53766e612967c8196f325df7359012d12af34" Nov 03 20:34:23.638294 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.638249 5059 scope.go:122] "RemoveContainer" containerID="89e013c2e4eeadb512ea4a0824409f67321420e7c404d52597c8ba458a34976b" Nov 03 20:34:23.675697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.662438 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42fe8135-7b7e-4292-ab0d-5ba57bbd3699-kube-api-access-xtc4t\" (UniqueName: \"kubernetes.io/projected/42fe8135-7b7e-4292-ab0d-5ba57bbd3699-kube-api-access-xtc4t\") pod \"42fe8135-7b7e-4292-ab0d-5ba57bbd3699\" (UID: \"42fe8135-7b7e-4292-ab0d-5ba57bbd3699\") " Nov 03 20:34:23.675697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.662765 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42fe8135-7b7e-4292-ab0d-5ba57bbd3699-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42fe8135-7b7e-4292-ab0d-5ba57bbd3699-maas-api-xks-ca\") pod \"42fe8135-7b7e-4292-ab0d-5ba57bbd3699\" (UID: \"42fe8135-7b7e-4292-ab0d-5ba57bbd3699\") " Nov 03 20:34:23.675697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.662868 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42fe8135-7b7e-4292-ab0d-5ba57bbd3699-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42fe8135-7b7e-4292-ab0d-5ba57bbd3699-maas-api-openshift-ca\") pod \"42fe8135-7b7e-4292-ab0d-5ba57bbd3699\" (UID: \"42fe8135-7b7e-4292-ab0d-5ba57bbd3699\") " Nov 03 20:34:23.675697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.674234 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42fe8135-7b7e-4292-ab0d-5ba57bbd3699-maas-api-openshift-ca" pod "42fe8135-7b7e-4292-ab0d-5ba57bbd3699" (UID: "42fe8135-7b7e-4292-ab0d-5ba57bbd3699"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:23.694420 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.681007 5059 scope.go:122] "RemoveContainer" containerID="39fd5a87217d3b8146ccfe78173093985a26712d776696de4ab3b763f50a762f" Nov 03 20:34:23.694420 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.686835 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42fe8135-7b7e-4292-ab0d-5ba57bbd3699-maas-api-xks-ca" pod "42fe8135-7b7e-4292-ab0d-5ba57bbd3699" (UID: "42fe8135-7b7e-4292-ab0d-5ba57bbd3699"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:23.695493 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.695449 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42fe8135-7b7e-4292-ab0d-5ba57bbd3699-kube-api-access-xtc4t" pod "42fe8135-7b7e-4292-ab0d-5ba57bbd3699" (UID: "42fe8135-7b7e-4292-ab0d-5ba57bbd3699"). InnerVolumeSpecName "kube-api-access-xtc4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:23.719040 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.718990 5059 scope.go:122] "RemoveContainer" containerID="926702d6bfa8f7006addfc1ed714e4470d66af979102156b85f6cad5b34d1221" Nov 03 20:34:23.747912 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.747880 5059 scope.go:122] "RemoveContainer" containerID="93172d79a58c2d8458ec95cdf845557fbb10e251f6cf98dda510ae64f827325b" Nov 03 20:34:23.765467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.765394 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42fe8135-7b7e-4292-ab0d-5ba57bbd3699-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.765467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.765433 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42fe8135-7b7e-4292-ab0d-5ba57bbd3699-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.765467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.765450 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtc4t\" (UniqueName: \"kubernetes.io/projected/42fe8135-7b7e-4292-ab0d-5ba57bbd3699-kube-api-access-xtc4t\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.770024 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.769971 5059 scope.go:122] "RemoveContainer" containerID="7598d2be586d16550aff9c80d082d04770669cb98c756f3811527543a1ac8d0f" Nov 03 20:34:23.797092 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.797065 5059 scope.go:122] "RemoveContainer" containerID="06775cb2b6311ab6024c5cf0da37eef3631ba5e53f2a44ad31bd33c7d778d26f" Nov 03 20:34:23.813261 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.813238 5059 scope.go:122] "RemoveContainer" containerID="b55d8d94b1bfd091549e5f64c5b4ba80d22392a32e05076a92b999b27dc44f7a" Nov 03 20:34:23.828432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.828411 5059 scope.go:122] "RemoveContainer" containerID="2ce7833679c135af351a934eeaaf542087c31098688f1a7b779e25f9a2bf4bc1" Nov 03 20:34:23.844515 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.844408 5059 scope.go:122] "RemoveContainer" containerID="07485e5a1a9bd8f3e2278bef20b0f805688a601c1ec005e9ee1b9f8abb449cd3" Nov 03 20:34:23.848453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.848415 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-nw2ph" event={"ID":"31725d6d-9a91-44ba-a08b-1520f5853190","Type":"ContainerDied","Data":"3536854a95b53f56e0e8a2a2fcc916c38fe9d1155595b712a1a94897173fd39b"} Nov 03 20:34:23.848756 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.848460 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-nw2ph" Nov 03 20:34:23.848756 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.848537 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3536854a95b53f56e0e8a2a2fcc916c38fe9d1155595b712a1a94897173fd39b" Nov 03 20:34:23.868096 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.868071 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-7w5r9" Nov 03 20:34:23.874857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.874828 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-7w5r9" event={"ID":"42fe8135-7b7e-4292-ab0d-5ba57bbd3699","Type":"ContainerDied","Data":"e9b161d400ab53eb83dfceccb6bc0b09de85aef83c06c7b21ea4a11ff6c7e571"} Nov 03 20:34:23.875020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.874999 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9b161d400ab53eb83dfceccb6bc0b09de85aef83c06c7b21ea4a11ff6c7e571" Nov 03 20:34:23.882320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.882294 5059 scope.go:122] "RemoveContainer" containerID="7bef2c62c2b96701d07f59fbf898023344294a1924ab4eb7ca9da4476e94db98" Nov 03 20:34:23.892022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.891915 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47" event={"ID":"6c29d53f-b738-49ec-9b15-1c88e5950008","Type":"ContainerDied","Data":"c0d6aef855ff12a4c072c15a072a0bfa3716b31ab4083ee5af5263844754bb97"} Nov 03 20:34:23.892022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.891956 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0d6aef855ff12a4c072c15a072a0bfa3716b31ab4083ee5af5263844754bb97" Nov 03 20:34:23.892022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.891925 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47" Nov 03 20:34:23.893698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.893679 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-8sgnl" Nov 03 20:34:23.893785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.893716 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-8sgnl" event={"ID":"b39ff65e-aa88-4534-a0f1-cd2af557c4ed","Type":"ContainerDied","Data":"997914bb2fef476abb0320da6c2569885574565fc8cce86ac419f8b8f03c91ba"} Nov 03 20:34:23.893785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.893774 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="997914bb2fef476abb0320da6c2569885574565fc8cce86ac419f8b8f03c91ba" Nov 03 20:34:23.903953 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.903926 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx" event={"ID":"847e7b38-8fb0-4c19-8943-f43e364f989b","Type":"ContainerDied","Data":"e32f190c4a0f1b9ade21ae5829e9ac0f0a81a815dc4a906aabec877a5680692a"} Nov 03 20:34:23.904323 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.903956 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e32f190c4a0f1b9ade21ae5829e9ac0f0a81a815dc4a906aabec877a5680692a" Nov 03 20:34:23.913104 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.913082 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:34:23.914200 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.914171 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h"] Nov 03 20:34:23.935837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.935794 5059 scope.go:122] "RemoveContainer" containerID="fa697f01be324e223c752a12316d9a9c14ae9cae343d4fc9f472c399735cd9ad" Nov 03 20:34:23.968719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:23.968648 5059 scope.go:122] "RemoveContainer" containerID="e7118a165e1c692020e00ab701f84d22189feb4501a06e12ef7daecc27e93292" Nov 03 20:34:24.010577 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.002800 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:34:24.010577 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.009221 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:34:24.022628 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.022233 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47"] Nov 03 20:34:24.026310 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.026283 5059 scope.go:122] "RemoveContainer" containerID="034deb3d094b43323ad58f8415dd48a1c89f624ddffe2676cc27ac841dc9e817" Nov 03 20:34:24.029102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.028974 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-gqz47"] Nov 03 20:34:24.103515 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.103337 5059 scope.go:122] "RemoveContainer" containerID="3d68de58f4e860642d4d4255a14fe00815183f88da48e5f851f0a0c70e971419" Nov 03 20:34:24.147957 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.147935 5059 scope.go:122] "RemoveContainer" containerID="1c69adcd1fc6db5a42ddeb00863f24a144f0cf528222cad083ac5a885e389943" Nov 03 20:34:24.149483 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.149457 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h" Nov 03 20:34:24.167187 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.163604 5059 scope.go:122] "RemoveContainer" containerID="ab75e4ee856c98aa1f2cad9eecfea81d6df844e47094122246ef45af739d37d5" Nov 03 20:34:24.192150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.192123 5059 scope.go:122] "RemoveContainer" containerID="5b05929b67ea95a6b627445e0a3fa63ef0a0d7ef033293a27edabc525d4bd1cd" Nov 03 20:34:24.194011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.193985 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-7dmwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" start-of-body= Nov 03 20:34:24.194183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.194156 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.124:15021/healthz/ready\": dial tcp 10.132.1.124:15021: connect: connection refused" Nov 03 20:34:24.205605 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.205570 5059 scope.go:122] "RemoveContainer" containerID="7bcd4a82a69aea94af0759459ddbccaebf9527ac1e3814ef792bac83392b4aa6" Nov 03 20:34:24.231100 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.231071 5059 scope.go:122] "RemoveContainer" containerID="b4dc2ca466f30e8024a88160d858421a5e6e6e115a118109fb3f8569f83e3ae2" Nov 03 20:34:24.245746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.245721 5059 scope.go:122] "RemoveContainer" containerID="4cc323b3fee429ec7bc1ab6e19bfbfe5f3926b292a99afff5f656543c7dd8e48" Nov 03 20:34:24.259378 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.259353 5059 scope.go:122] "RemoveContainer" containerID="1800f61ab969d2f54568614237d89ac56416a680367326b1f485f3183aebbfe9" Nov 03 20:34:24.276974 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.276951 5059 scope.go:122] "RemoveContainer" containerID="44d16c3b122733147748e39e219fb00f390ff6e37a49c47ed3e51ba2960a62c1" Nov 03 20:34:24.287873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.287844 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28034528-e342-417a-a9df-050d7ea91453-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28034528-e342-417a-a9df-050d7ea91453-maas-api-xks-ca\") pod \"28034528-e342-417a-a9df-050d7ea91453\" (UID: \"28034528-e342-417a-a9df-050d7ea91453\") " Nov 03 20:34:24.288152 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.288130 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28034528-e342-417a-a9df-050d7ea91453-kube-api-access-kmvvg\" (UniqueName: \"kubernetes.io/projected/28034528-e342-417a-a9df-050d7ea91453-kube-api-access-kmvvg\") pod \"28034528-e342-417a-a9df-050d7ea91453\" (UID: \"28034528-e342-417a-a9df-050d7ea91453\") " Nov 03 20:34:24.288341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.288314 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28034528-e342-417a-a9df-050d7ea91453-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28034528-e342-417a-a9df-050d7ea91453-maas-api-openshift-ca\") pod \"28034528-e342-417a-a9df-050d7ea91453\" (UID: \"28034528-e342-417a-a9df-050d7ea91453\") " Nov 03 20:34:24.288806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.288570 5059 scope.go:122] "RemoveContainer" containerID="6a213e69e74bbe139a2688d93452e7d59e19f98c3c9a8e404648454c84993ae7" Nov 03 20:34:24.288900 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.288837 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28034528-e342-417a-a9df-050d7ea91453-maas-api-openshift-ca" pod "28034528-e342-417a-a9df-050d7ea91453" (UID: "28034528-e342-417a-a9df-050d7ea91453"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:24.289119 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.289097 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28034528-e342-417a-a9df-050d7ea91453-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:24.290433 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.290400 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28034528-e342-417a-a9df-050d7ea91453-maas-api-xks-ca" pod "28034528-e342-417a-a9df-050d7ea91453" (UID: "28034528-e342-417a-a9df-050d7ea91453"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:24.290811 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.290784 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28034528-e342-417a-a9df-050d7ea91453-kube-api-access-kmvvg" pod "28034528-e342-417a-a9df-050d7ea91453" (UID: "28034528-e342-417a-a9df-050d7ea91453"). InnerVolumeSpecName "kube-api-access-kmvvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:24.315293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.315269 5059 scope.go:122] "RemoveContainer" containerID="c00733af03a11c26991be300ebced8bc887b057aa45430c49b4b704481238514" Nov 03 20:34:24.326918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.326890 5059 scope.go:122] "RemoveContainer" containerID="13a83c66d1721d1a4304250d96ba6c4a613cfb0ba59048a8e63f219d8a6f74c8" Nov 03 20:34:24.337542 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.337520 5059 scope.go:122] "RemoveContainer" containerID="63bc22494254a32bee2bb7f180343e815aa6a4b2d9c80eef741cbee37519d4fe" Nov 03 20:34:24.348221 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.348189 5059 scope.go:122] "RemoveContainer" containerID="25b28b428ebc85ad2241009e2e20a1a14a4f98588d071bd9785777b228e75310" Nov 03 20:34:24.358390 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.358371 5059 scope.go:122] "RemoveContainer" containerID="b8d82ca6d1d4fd46ae40cb0b5685d96dc2c2f3703f2b90b33d10896376edb8cc" Nov 03 20:34:24.368437 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.368416 5059 scope.go:122] "RemoveContainer" containerID="238b88770b80cec7d339089cb9d69cc1f7ef5a1fdcefa3eb790154cc7f3289f1" Nov 03 20:34:24.378555 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.378528 5059 scope.go:122] "RemoveContainer" containerID="e9d8391fb50e417bbc209099a26beaeb42cc1de9591126f570e39ea8262098a0" Nov 03 20:34:24.388788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.388762 5059 scope.go:122] "RemoveContainer" containerID="2bb96bf56e8cc17a793ce1ddb74c44ae1dca020bdfeee2d770dda49eb3c3d502" Nov 03 20:34:24.390257 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.390235 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28034528-e342-417a-a9df-050d7ea91453-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:24.390257 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.390257 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmvvg\" (UniqueName: \"kubernetes.io/projected/28034528-e342-417a-a9df-050d7ea91453-kube-api-access-kmvvg\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:24.401107 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.401091 5059 scope.go:122] "RemoveContainer" containerID="8880ed12964f51d73ab467040324e0b739812c728f7782757374b9b0f025d69e" Nov 03 20:34:24.924384 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.923697 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-fzr95" event={"ID":"191c7922-07dd-4c9c-89d2-d804f638854d","Type":"ContainerStarted","Data":"4954f58af4b62e9c8562a7ea22b05034f5f25a62474b277d649acb4cef5910bb"} Nov 03 20:34:24.924384 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.923805 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-fzr95" Nov 03 20:34:24.937227 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.937197 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h" Nov 03 20:34:24.937331 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.937238 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h" event={"ID":"28034528-e342-417a-a9df-050d7ea91453","Type":"ContainerDied","Data":"7553fcdab810b85acc004e15950eef060706caf88e5df5ba2f9e43e9ce29c977"} Nov 03 20:34:24.937331 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.937277 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7553fcdab810b85acc004e15950eef060706caf88e5df5ba2f9e43e9ce29c977" Nov 03 20:34:24.946404 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.946347 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-fzr95" podStartSLOduration=62.223922148 podStartE2EDuration="2m20.946329108s" totalImagesPullingTime="1m18.72240696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:24.943924473 +0000 UTC m=+3459.201721730" watchObservedRunningTime="2026-11-03 20:34:24.946329108 +0000 UTC m=+3459.204126369" Nov 03 20:34:24.982837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.982808 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h"] Nov 03 20:34:24.988182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:24.988159 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-ndv2h"] Nov 03 20:34:25.335740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:25.335717 5059 kubelet.go:1658] "Container garbage collection succeeded" Nov 03 20:34:25.657994 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:25.657959 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:34:25.659556 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:25.659527 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:34:25.734569 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:25.734521 5059 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:34:25.734769 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:25.734587 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" podUID="d931dc13-4399-48dc-80e0-b0b304c5dda0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:34:25.861263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:25.861222 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28034528-e342-417a-a9df-050d7ea91453" path="/var/lib/kubelet/pods/28034528-e342-417a-a9df-050d7ea91453/volumes" Nov 03 20:34:25.861994 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:25.861973 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c29d53f-b738-49ec-9b15-1c88e5950008" path="/var/lib/kubelet/pods/6c29d53f-b738-49ec-9b15-1c88e5950008/volumes" Nov 03 20:34:28.356994 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:34:28.356952 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd931dc13_4399_48dc_80e0_b0b304c5dda0.slice/crio-conmon-51e5dd7d49f3191cbe26fa7fea29e5e5bd18af15599c15f1e53810285e8f9294.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:34:28.547502 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.547477 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:34:28.593377 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.593340 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-credential-socket\") pod \"d931dc13-4399-48dc-80e0-b0b304c5dda0\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " Nov 03 20:34:28.594069 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.594038 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-workload-socket\") pod \"d931dc13-4399-48dc-80e0-b0b304c5dda0\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " Nov 03 20:34:28.594426 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.594400 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d931dc13-4399-48dc-80e0-b0b304c5dda0-kube-api-access-s2n95\" (UniqueName: \"kubernetes.io/projected/d931dc13-4399-48dc-80e0-b0b304c5dda0-kube-api-access-s2n95\") pod \"d931dc13-4399-48dc-80e0-b0b304c5dda0\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " Nov 03 20:34:28.594580 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.594563 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-envoy\") pod \"d931dc13-4399-48dc-80e0-b0b304c5dda0\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " Nov 03 20:34:28.594724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.594706 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-workload-certs\") pod \"d931dc13-4399-48dc-80e0-b0b304c5dda0\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " Nov 03 20:34:28.594870 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.594853 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-token\" (UniqueName: \"kubernetes.io/projected/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-token\") pod \"d931dc13-4399-48dc-80e0-b0b304c5dda0\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " Nov 03 20:34:28.595045 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.595026 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d931dc13-4399-48dc-80e0-b0b304c5dda0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d931dc13-4399-48dc-80e0-b0b304c5dda0-istiod-ca-cert\") pod \"d931dc13-4399-48dc-80e0-b0b304c5dda0\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " Nov 03 20:34:28.595158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.595143 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-data\") pod \"d931dc13-4399-48dc-80e0-b0b304c5dda0\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " Nov 03 20:34:28.596106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.594141 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-credential-socket" pod "d931dc13-4399-48dc-80e0-b0b304c5dda0" (UID: "d931dc13-4399-48dc-80e0-b0b304c5dda0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:28.596276 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.596253 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-workload-certs" pod "d931dc13-4399-48dc-80e0-b0b304c5dda0" (UID: "d931dc13-4399-48dc-80e0-b0b304c5dda0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:28.596385 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.596270 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-workload-socket" pod "d931dc13-4399-48dc-80e0-b0b304c5dda0" (UID: "d931dc13-4399-48dc-80e0-b0b304c5dda0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:28.596504 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.596480 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-data" pod "d931dc13-4399-48dc-80e0-b0b304c5dda0" (UID: "d931dc13-4399-48dc-80e0-b0b304c5dda0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:28.596647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.596298 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d931dc13-4399-48dc-80e0-b0b304c5dda0-istiod-ca-cert" pod "d931dc13-4399-48dc-80e0-b0b304c5dda0" (UID: "d931dc13-4399-48dc-80e0-b0b304c5dda0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:28.600170 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.600148 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-token" pod "d931dc13-4399-48dc-80e0-b0b304c5dda0" (UID: "d931dc13-4399-48dc-80e0-b0b304c5dda0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:28.600253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.600185 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d931dc13-4399-48dc-80e0-b0b304c5dda0-kube-api-access-s2n95" pod "d931dc13-4399-48dc-80e0-b0b304c5dda0" (UID: "d931dc13-4399-48dc-80e0-b0b304c5dda0"). InnerVolumeSpecName "kube-api-access-s2n95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:28.601117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.601082 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-envoy" pod "d931dc13-4399-48dc-80e0-b0b304c5dda0" (UID: "d931dc13-4399-48dc-80e0-b0b304c5dda0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:28.696975 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.696941 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-podinfo\") pod \"d931dc13-4399-48dc-80e0-b0b304c5dda0\" (UID: \"d931dc13-4399-48dc-80e0-b0b304c5dda0\") " Nov 03 20:34:28.697593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.697568 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:28.697593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.697594 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2n95\" (UniqueName: \"kubernetes.io/projected/d931dc13-4399-48dc-80e0-b0b304c5dda0-kube-api-access-s2n95\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:28.697762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.697605 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:28.697762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.697615 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:28.697762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.697624 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:28.697762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.697633 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d931dc13-4399-48dc-80e0-b0b304c5dda0-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:28.697762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.697643 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:28.697762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.697653 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d931dc13-4399-48dc-80e0-b0b304c5dda0-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:28.699064 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.699034 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-podinfo" pod "d931dc13-4399-48dc-80e0-b0b304c5dda0" (UID: "d931dc13-4399-48dc-80e0-b0b304c5dda0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:34:28.799627 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.799559 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d931dc13-4399-48dc-80e0-b0b304c5dda0-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:28.979530 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.979486 5059 generic.go:356] "Generic (PLEG): container finished" podID="d931dc13-4399-48dc-80e0-b0b304c5dda0" containerID="51e5dd7d49f3191cbe26fa7fea29e5e5bd18af15599c15f1e53810285e8f9294" exitCode=0 Nov 03 20:34:28.979803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.979570 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" Nov 03 20:34:28.979803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.979575 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" event={"ID":"d931dc13-4399-48dc-80e0-b0b304c5dda0","Type":"ContainerDied","Data":"51e5dd7d49f3191cbe26fa7fea29e5e5bd18af15599c15f1e53810285e8f9294"} Nov 03 20:34:28.979803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.979655 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr" event={"ID":"d931dc13-4399-48dc-80e0-b0b304c5dda0","Type":"ContainerDied","Data":"31b6388801c2c480e702c4985e3f6638dbac4103cebd0a5bf86216cac78f3f2f"} Nov 03 20:34:28.979803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.979702 5059 scope.go:122] "RemoveContainer" containerID="51e5dd7d49f3191cbe26fa7fea29e5e5bd18af15599c15f1e53810285e8f9294" Nov 03 20:34:28.992059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.992041 5059 scope.go:122] "RemoveContainer" containerID="51e5dd7d49f3191cbe26fa7fea29e5e5bd18af15599c15f1e53810285e8f9294" Nov 03 20:34:28.992338 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:34:28.992316 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51e5dd7d49f3191cbe26fa7fea29e5e5bd18af15599c15f1e53810285e8f9294\": container with ID starting with 51e5dd7d49f3191cbe26fa7fea29e5e5bd18af15599c15f1e53810285e8f9294 not found: ID does not exist" containerID="51e5dd7d49f3191cbe26fa7fea29e5e5bd18af15599c15f1e53810285e8f9294" Nov 03 20:34:28.992387 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:28.992346 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51e5dd7d49f3191cbe26fa7fea29e5e5bd18af15599c15f1e53810285e8f9294"} err="failed to get container status \"51e5dd7d49f3191cbe26fa7fea29e5e5bd18af15599c15f1e53810285e8f9294\": rpc error: code = NotFound desc = could not find container \"51e5dd7d49f3191cbe26fa7fea29e5e5bd18af15599c15f1e53810285e8f9294\": container with ID starting with 51e5dd7d49f3191cbe26fa7fea29e5e5bd18af15599c15f1e53810285e8f9294 not found: ID does not exist" Nov 03 20:34:29.008381 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:29.008348 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr"] Nov 03 20:34:29.012830 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:29.012804 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-vmgsr"] Nov 03 20:34:29.860000 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:29.859968 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d931dc13-4399-48dc-80e0-b0b304c5dda0" path="/var/lib/kubelet/pods/d931dc13-4399-48dc-80e0-b0b304c5dda0/volumes" Nov 03 20:34:32.976721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.976680 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j8h6z"] Nov 03 20:34:32.977557 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977534 5059 cpu_manager.go:414] "containerMap: removing container" podUID="6c29d53f-b738-49ec-9b15-1c88e5950008" containerName="revoke-keys" Nov 03 20:34:32.977557 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977557 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c29d53f-b738-49ec-9b15-1c88e5950008" containerName="revoke-keys" Nov 03 20:34:32.977751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977571 5059 cpu_manager.go:414] "containerMap: removing container" podUID="6c29d53f-b738-49ec-9b15-1c88e5950008" containerName="revoke-keys" Nov 03 20:34:32.977751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977580 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c29d53f-b738-49ec-9b15-1c88e5950008" containerName="revoke-keys" Nov 03 20:34:32.977751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977598 5059 cpu_manager.go:414] "containerMap: removing container" podUID="94e83c10-58b0-4bc7-9abd-3e144ba6e258" containerName="revoke-keys" Nov 03 20:34:32.977751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977606 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e83c10-58b0-4bc7-9abd-3e144ba6e258" containerName="revoke-keys" Nov 03 20:34:32.977751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977622 5059 cpu_manager.go:414] "containerMap: removing container" podUID="31725d6d-9a91-44ba-a08b-1520f5853190" containerName="revoke-keys" Nov 03 20:34:32.977751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977629 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31725d6d-9a91-44ba-a08b-1520f5853190" containerName="revoke-keys" Nov 03 20:34:32.977751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977637 5059 cpu_manager.go:414] "containerMap: removing container" podUID="847e7b38-8fb0-4c19-8943-f43e364f989b" containerName="revoke-keys" Nov 03 20:34:32.977751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977645 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="847e7b38-8fb0-4c19-8943-f43e364f989b" containerName="revoke-keys" Nov 03 20:34:32.977751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977654 5059 cpu_manager.go:414] "containerMap: removing container" podUID="28034528-e342-417a-a9df-050d7ea91453" containerName="revoke-keys" Nov 03 20:34:32.977751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977676 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28034528-e342-417a-a9df-050d7ea91453" containerName="revoke-keys" Nov 03 20:34:32.977751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977692 5059 cpu_manager.go:414] "containerMap: removing container" podUID="94e83c10-58b0-4bc7-9abd-3e144ba6e258" containerName="revoke-keys" Nov 03 20:34:32.977751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977700 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e83c10-58b0-4bc7-9abd-3e144ba6e258" containerName="revoke-keys" Nov 03 20:34:32.977751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977716 5059 cpu_manager.go:414] "containerMap: removing container" podUID="28034528-e342-417a-a9df-050d7ea91453" containerName="revoke-keys" Nov 03 20:34:32.977751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977725 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28034528-e342-417a-a9df-050d7ea91453" containerName="revoke-keys" Nov 03 20:34:32.977751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977740 5059 cpu_manager.go:414] "containerMap: removing container" podUID="b39ff65e-aa88-4534-a0f1-cd2af557c4ed" containerName="revoke-keys" Nov 03 20:34:32.977751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977747 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39ff65e-aa88-4534-a0f1-cd2af557c4ed" containerName="revoke-keys" Nov 03 20:34:32.978467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977766 5059 cpu_manager.go:414] "containerMap: removing container" podUID="42fe8135-7b7e-4292-ab0d-5ba57bbd3699" containerName="revoke-keys" Nov 03 20:34:32.978467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977773 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42fe8135-7b7e-4292-ab0d-5ba57bbd3699" containerName="revoke-keys" Nov 03 20:34:32.978467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977804 5059 cpu_manager.go:414] "containerMap: removing container" podUID="847e7b38-8fb0-4c19-8943-f43e364f989b" containerName="revoke-keys" Nov 03 20:34:32.978467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977811 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="847e7b38-8fb0-4c19-8943-f43e364f989b" containerName="revoke-keys" Nov 03 20:34:32.978467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977824 5059 cpu_manager.go:414] "containerMap: removing container" podUID="d931dc13-4399-48dc-80e0-b0b304c5dda0" containerName="istio-proxy" Nov 03 20:34:32.978467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977830 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d931dc13-4399-48dc-80e0-b0b304c5dda0" containerName="istio-proxy" Nov 03 20:34:32.978467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.977990 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c29d53f-b738-49ec-9b15-1c88e5950008" containerName="revoke-keys" Nov 03 20:34:32.978467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.978004 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c29d53f-b738-49ec-9b15-1c88e5950008" containerName="revoke-keys" Nov 03 20:34:32.978467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.978017 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e83c10-58b0-4bc7-9abd-3e144ba6e258" containerName="revoke-keys" Nov 03 20:34:32.978467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.978029 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="31725d6d-9a91-44ba-a08b-1520f5853190" containerName="revoke-keys" Nov 03 20:34:32.978467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.978037 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="847e7b38-8fb0-4c19-8943-f43e364f989b" containerName="revoke-keys" Nov 03 20:34:32.978467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.978046 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="28034528-e342-417a-a9df-050d7ea91453" containerName="revoke-keys" Nov 03 20:34:32.978467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.978055 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e83c10-58b0-4bc7-9abd-3e144ba6e258" containerName="revoke-keys" Nov 03 20:34:32.978467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.978066 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39ff65e-aa88-4534-a0f1-cd2af557c4ed" containerName="revoke-keys" Nov 03 20:34:32.978467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.978075 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="42fe8135-7b7e-4292-ab0d-5ba57bbd3699" containerName="revoke-keys" Nov 03 20:34:32.978467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.978086 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="28034528-e342-417a-a9df-050d7ea91453" containerName="revoke-keys" Nov 03 20:34:32.978467 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.978095 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d931dc13-4399-48dc-80e0-b0b304c5dda0" containerName="istio-proxy" Nov 03 20:34:32.983590 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.983570 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j8h6z" Nov 03 20:34:32.987485 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.987458 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:34:32.988578 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.988419 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:34:32.990330 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:32.990304 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j8h6z"] Nov 03 20:34:33.086808 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:33.086776 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddf9e671-9f34-424b-976f-2b9a25deddbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-j8h6z\" (UID: \"ddf9e671-9f34-424b-976f-2b9a25deddbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j8h6z" Nov 03 20:34:33.086936 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:33.086813 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddf9e671-9f34-424b-976f-2b9a25deddbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-j8h6z\" (UID: \"ddf9e671-9f34-424b-976f-2b9a25deddbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j8h6z" Nov 03 20:34:33.086987 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:33.086967 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88qvp\" (UniqueName: \"kubernetes.io/projected/ddf9e671-9f34-424b-976f-2b9a25deddbf-kube-api-access-88qvp\") pod \"maas-api-revoke-sub-5bac6f1e-j8h6z\" (UID: \"ddf9e671-9f34-424b-976f-2b9a25deddbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j8h6z" Nov 03 20:34:33.187947 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:33.187915 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddf9e671-9f34-424b-976f-2b9a25deddbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-j8h6z\" (UID: \"ddf9e671-9f34-424b-976f-2b9a25deddbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j8h6z" Nov 03 20:34:33.188102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:33.187960 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddf9e671-9f34-424b-976f-2b9a25deddbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-j8h6z\" (UID: \"ddf9e671-9f34-424b-976f-2b9a25deddbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j8h6z" Nov 03 20:34:33.188102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:33.188060 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88qvp\" (UniqueName: \"kubernetes.io/projected/ddf9e671-9f34-424b-976f-2b9a25deddbf-kube-api-access-88qvp\") pod \"maas-api-revoke-sub-5bac6f1e-j8h6z\" (UID: \"ddf9e671-9f34-424b-976f-2b9a25deddbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j8h6z" Nov 03 20:34:33.188771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:33.188750 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddf9e671-9f34-424b-976f-2b9a25deddbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-j8h6z\" (UID: \"ddf9e671-9f34-424b-976f-2b9a25deddbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j8h6z" Nov 03 20:34:33.190381 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:33.190358 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddf9e671-9f34-424b-976f-2b9a25deddbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-j8h6z\" (UID: \"ddf9e671-9f34-424b-976f-2b9a25deddbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j8h6z" Nov 03 20:34:33.200308 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:33.200284 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88qvp\" (UniqueName: \"kubernetes.io/projected/ddf9e671-9f34-424b-976f-2b9a25deddbf-kube-api-access-88qvp\") pod \"maas-api-revoke-sub-5bac6f1e-j8h6z\" (UID: \"ddf9e671-9f34-424b-976f-2b9a25deddbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j8h6z" Nov 03 20:34:33.307222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:33.307131 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j8h6z" Nov 03 20:34:33.594825 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:33.594789 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j8h6z"] Nov 03 20:34:33.595042 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:34:33.594851 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf9e671_9f34_424b_976f_2b9a25deddbf.slice/crio-89989ad1072209448a417d417e20ae77b3df256262d7163e44d5c47301c295dd WatchSource:0}: Error finding container 89989ad1072209448a417d417e20ae77b3df256262d7163e44d5c47301c295dd: Status 404 returned error can't find the container with id 89989ad1072209448a417d417e20ae77b3df256262d7163e44d5c47301c295dd Nov 03 20:34:33.932791 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:33.932613 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-wvkx8"] Nov 03 20:34:33.933791 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:33.933762 5059 cpu_manager.go:414] "containerMap: removing container" podUID="28034528-e342-417a-a9df-050d7ea91453" containerName="revoke-keys" Nov 03 20:34:33.933791 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:33.933791 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28034528-e342-417a-a9df-050d7ea91453" containerName="revoke-keys" Nov 03 20:34:33.933995 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:33.933862 5059 cpu_manager.go:414] "containerMap: removing container" podUID="6c29d53f-b738-49ec-9b15-1c88e5950008" containerName="revoke-keys" Nov 03 20:34:33.933995 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:33.933871 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c29d53f-b738-49ec-9b15-1c88e5950008" containerName="revoke-keys" Nov 03 20:34:33.934100 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:33.934020 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c29d53f-b738-49ec-9b15-1c88e5950008" containerName="revoke-keys" Nov 03 20:34:33.934100 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:33.934036 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="847e7b38-8fb0-4c19-8943-f43e364f989b" containerName="revoke-keys" Nov 03 20:34:33.934100 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:33.934056 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="28034528-e342-417a-a9df-050d7ea91453" containerName="revoke-keys" Nov 03 20:34:33.938432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:33.938408 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-wvkx8" Nov 03 20:34:33.940291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:33.940266 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-wvkx8"] Nov 03 20:34:34.029342 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:34.029303 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j8h6z" event={"ID":"ddf9e671-9f34-424b-976f-2b9a25deddbf","Type":"ContainerStarted","Data":"a61b0ec38265a7b62ed139983e4c7f404bbb92940a0beb66d08a48f6a22de06b"} Nov 03 20:34:34.029731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:34.029398 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j8h6z" event={"ID":"ddf9e671-9f34-424b-976f-2b9a25deddbf","Type":"ContainerStarted","Data":"89989ad1072209448a417d417e20ae77b3df256262d7163e44d5c47301c295dd"} Nov 03 20:34:34.049819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:34.049771 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j8h6z" podStartSLOduration=2.049757405 podStartE2EDuration="2.049757405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:34.047497702 +0000 UTC m=+3468.305294959" watchObservedRunningTime="2026-11-03 20:34:34.049757405 +0000 UTC m=+3468.307554662" Nov 03 20:34:34.109190 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:34.109162 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8767\" (UniqueName: \"kubernetes.io/projected/f544668b-01c9-4c84-bffe-d6f09f4566e5-kube-api-access-k8767\") pod \"maas-api-revoke-sub-fb827799-wvkx8\" (UID: \"f544668b-01c9-4c84-bffe-d6f09f4566e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-wvkx8" Nov 03 20:34:34.109326 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:34.109199 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f544668b-01c9-4c84-bffe-d6f09f4566e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb827799-wvkx8\" (UID: \"f544668b-01c9-4c84-bffe-d6f09f4566e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-wvkx8" Nov 03 20:34:34.109442 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:34.109425 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f544668b-01c9-4c84-bffe-d6f09f4566e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb827799-wvkx8\" (UID: \"f544668b-01c9-4c84-bffe-d6f09f4566e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-wvkx8" Nov 03 20:34:34.210367 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:34.210301 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f544668b-01c9-4c84-bffe-d6f09f4566e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb827799-wvkx8\" (UID: \"f544668b-01c9-4c84-bffe-d6f09f4566e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-wvkx8" Nov 03 20:34:34.210541 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:34.210521 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8767\" (UniqueName: \"kubernetes.io/projected/f544668b-01c9-4c84-bffe-d6f09f4566e5-kube-api-access-k8767\") pod \"maas-api-revoke-sub-fb827799-wvkx8\" (UID: \"f544668b-01c9-4c84-bffe-d6f09f4566e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-wvkx8" Nov 03 20:34:34.210587 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:34.210560 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f544668b-01c9-4c84-bffe-d6f09f4566e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb827799-wvkx8\" (UID: \"f544668b-01c9-4c84-bffe-d6f09f4566e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-wvkx8" Nov 03 20:34:34.211212 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:34.211186 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f544668b-01c9-4c84-bffe-d6f09f4566e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb827799-wvkx8\" (UID: \"f544668b-01c9-4c84-bffe-d6f09f4566e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-wvkx8" Nov 03 20:34:34.212789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:34.212769 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f544668b-01c9-4c84-bffe-d6f09f4566e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb827799-wvkx8\" (UID: \"f544668b-01c9-4c84-bffe-d6f09f4566e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-wvkx8" Nov 03 20:34:34.223899 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:34.223878 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8767\" (UniqueName: \"kubernetes.io/projected/f544668b-01c9-4c84-bffe-d6f09f4566e5-kube-api-access-k8767\") pod \"maas-api-revoke-sub-fb827799-wvkx8\" (UID: \"f544668b-01c9-4c84-bffe-d6f09f4566e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-wvkx8" Nov 03 20:34:34.245843 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:34.245820 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk"] Nov 03 20:34:34.246062 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:34.246019 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" containerID="cri-o://ba8b1d44e1eaed0f3e420da024ef6f5f654842540fe5435339c0ff9078b7ee8c" gracePeriod=30 Nov 03 20:34:34.267677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:34.267645 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-wvkx8" Nov 03 20:34:34.563547 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:34.563520 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-wvkx8"] Nov 03 20:34:34.619573 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:34:34.619544 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf544668b_01c9_4c84_bffe_d6f09f4566e5.slice/crio-7ad92a29cef53e7120ca7e66e87043a82a508d371edb0c758a22375ae75f0394 WatchSource:0}: Error finding container 7ad92a29cef53e7120ca7e66e87043a82a508d371edb0c758a22375ae75f0394: Status 404 returned error can't find the container with id 7ad92a29cef53e7120ca7e66e87043a82a508d371edb0c758a22375ae75f0394 Nov 03 20:34:35.037639 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:35.037606 5059 generic.go:356] "Generic (PLEG): container finished" podID="ddf9e671-9f34-424b-976f-2b9a25deddbf" containerID="a61b0ec38265a7b62ed139983e4c7f404bbb92940a0beb66d08a48f6a22de06b" exitCode=0 Nov 03 20:34:35.038041 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:35.037688 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j8h6z" event={"ID":"ddf9e671-9f34-424b-976f-2b9a25deddbf","Type":"ContainerDied","Data":"a61b0ec38265a7b62ed139983e4c7f404bbb92940a0beb66d08a48f6a22de06b"} Nov 03 20:34:35.039224 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:35.039202 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-wvkx8" event={"ID":"f544668b-01c9-4c84-bffe-d6f09f4566e5","Type":"ContainerStarted","Data":"4b50d662eaf88e93735d86eb301d9af01f88938380cdae493dd558f07e9ddfaf"} Nov 03 20:34:35.039305 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:35.039245 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-wvkx8" event={"ID":"f544668b-01c9-4c84-bffe-d6f09f4566e5","Type":"ContainerStarted","Data":"7ad92a29cef53e7120ca7e66e87043a82a508d371edb0c758a22375ae75f0394"} Nov 03 20:34:35.075047 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:35.075005 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-wvkx8" podStartSLOduration=2.074992708 podStartE2EDuration="2.074992708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:35.073594816 +0000 UTC m=+3469.331392073" watchObservedRunningTime="2026-11-03 20:34:35.074992708 +0000 UTC m=+3469.332789961" Nov 03 20:34:35.946906 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:35.946828 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-fzr95" Nov 03 20:34:36.047712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:36.047649 5059 generic.go:356] "Generic (PLEG): container finished" podID="f544668b-01c9-4c84-bffe-d6f09f4566e5" containerID="4b50d662eaf88e93735d86eb301d9af01f88938380cdae493dd558f07e9ddfaf" exitCode=0 Nov 03 20:34:36.048144 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:36.047735 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-wvkx8" event={"ID":"f544668b-01c9-4c84-bffe-d6f09f4566e5","Type":"ContainerDied","Data":"4b50d662eaf88e93735d86eb301d9af01f88938380cdae493dd558f07e9ddfaf"} Nov 03 20:34:36.268926 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:36.268897 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j8h6z" Nov 03 20:34:36.342153 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:36.342118 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddf9e671-9f34-424b-976f-2b9a25deddbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddf9e671-9f34-424b-976f-2b9a25deddbf-maas-api-xks-ca\") pod \"ddf9e671-9f34-424b-976f-2b9a25deddbf\" (UID: \"ddf9e671-9f34-424b-976f-2b9a25deddbf\") " Nov 03 20:34:36.342292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:36.342176 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddf9e671-9f34-424b-976f-2b9a25deddbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddf9e671-9f34-424b-976f-2b9a25deddbf-maas-api-openshift-ca\") pod \"ddf9e671-9f34-424b-976f-2b9a25deddbf\" (UID: \"ddf9e671-9f34-424b-976f-2b9a25deddbf\") " Nov 03 20:34:36.342292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:36.342276 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddf9e671-9f34-424b-976f-2b9a25deddbf-kube-api-access-88qvp\" (UniqueName: \"kubernetes.io/projected/ddf9e671-9f34-424b-976f-2b9a25deddbf-kube-api-access-88qvp\") pod \"ddf9e671-9f34-424b-976f-2b9a25deddbf\" (UID: \"ddf9e671-9f34-424b-976f-2b9a25deddbf\") " Nov 03 20:34:36.342565 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:36.342540 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddf9e671-9f34-424b-976f-2b9a25deddbf-maas-api-openshift-ca" pod "ddf9e671-9f34-424b-976f-2b9a25deddbf" (UID: "ddf9e671-9f34-424b-976f-2b9a25deddbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:36.342871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:36.342849 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddf9e671-9f34-424b-976f-2b9a25deddbf-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:36.344438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:36.344414 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddf9e671-9f34-424b-976f-2b9a25deddbf-maas-api-xks-ca" pod "ddf9e671-9f34-424b-976f-2b9a25deddbf" (UID: "ddf9e671-9f34-424b-976f-2b9a25deddbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:36.344552 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:36.344527 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf9e671-9f34-424b-976f-2b9a25deddbf-kube-api-access-88qvp" pod "ddf9e671-9f34-424b-976f-2b9a25deddbf" (UID: "ddf9e671-9f34-424b-976f-2b9a25deddbf"). InnerVolumeSpecName "kube-api-access-88qvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:36.444494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:36.444464 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddf9e671-9f34-424b-976f-2b9a25deddbf-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:36.444494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:36.444492 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88qvp\" (UniqueName: \"kubernetes.io/projected/ddf9e671-9f34-424b-976f-2b9a25deddbf-kube-api-access-88qvp\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:37.056732 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:37.056692 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j8h6z" Nov 03 20:34:37.056732 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:37.056705 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j8h6z" event={"ID":"ddf9e671-9f34-424b-976f-2b9a25deddbf","Type":"ContainerDied","Data":"89989ad1072209448a417d417e20ae77b3df256262d7163e44d5c47301c295dd"} Nov 03 20:34:37.057178 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:37.056765 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89989ad1072209448a417d417e20ae77b3df256262d7163e44d5c47301c295dd" Nov 03 20:34:37.192577 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:37.192549 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-wvkx8" Nov 03 20:34:37.258770 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:37.258720 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f544668b-01c9-4c84-bffe-d6f09f4566e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f544668b-01c9-4c84-bffe-d6f09f4566e5-maas-api-openshift-ca\") pod \"f544668b-01c9-4c84-bffe-d6f09f4566e5\" (UID: \"f544668b-01c9-4c84-bffe-d6f09f4566e5\") " Nov 03 20:34:37.258913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:37.258820 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f544668b-01c9-4c84-bffe-d6f09f4566e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f544668b-01c9-4c84-bffe-d6f09f4566e5-maas-api-xks-ca\") pod \"f544668b-01c9-4c84-bffe-d6f09f4566e5\" (UID: \"f544668b-01c9-4c84-bffe-d6f09f4566e5\") " Nov 03 20:34:37.259119 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:37.259097 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f544668b-01c9-4c84-bffe-d6f09f4566e5-kube-api-access-k8767\" (UniqueName: \"kubernetes.io/projected/f544668b-01c9-4c84-bffe-d6f09f4566e5-kube-api-access-k8767\") pod \"f544668b-01c9-4c84-bffe-d6f09f4566e5\" (UID: \"f544668b-01c9-4c84-bffe-d6f09f4566e5\") " Nov 03 20:34:37.259119 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:37.259107 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f544668b-01c9-4c84-bffe-d6f09f4566e5-maas-api-openshift-ca" pod "f544668b-01c9-4c84-bffe-d6f09f4566e5" (UID: "f544668b-01c9-4c84-bffe-d6f09f4566e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:37.260081 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:37.260052 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f544668b-01c9-4c84-bffe-d6f09f4566e5-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:37.261217 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:37.261189 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f544668b-01c9-4c84-bffe-d6f09f4566e5-kube-api-access-k8767" pod "f544668b-01c9-4c84-bffe-d6f09f4566e5" (UID: "f544668b-01c9-4c84-bffe-d6f09f4566e5"). InnerVolumeSpecName "kube-api-access-k8767". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:37.261478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:37.261458 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f544668b-01c9-4c84-bffe-d6f09f4566e5-maas-api-xks-ca" pod "f544668b-01c9-4c84-bffe-d6f09f4566e5" (UID: "f544668b-01c9-4c84-bffe-d6f09f4566e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:37.361959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:37.361933 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8767\" (UniqueName: \"kubernetes.io/projected/f544668b-01c9-4c84-bffe-d6f09f4566e5-kube-api-access-k8767\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:37.361959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:37.361958 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f544668b-01c9-4c84-bffe-d6f09f4566e5-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:38.064260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:38.064217 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-wvkx8" Nov 03 20:34:38.064260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:38.064228 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-wvkx8" event={"ID":"f544668b-01c9-4c84-bffe-d6f09f4566e5","Type":"ContainerDied","Data":"7ad92a29cef53e7120ca7e66e87043a82a508d371edb0c758a22375ae75f0394"} Nov 03 20:34:38.064767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:38.064305 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ad92a29cef53e7120ca7e66e87043a82a508d371edb0c758a22375ae75f0394" Nov 03 20:34:38.174766 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:38.174692 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst"] Nov 03 20:34:39.530457 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.530430 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:34:39.606176 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.606145 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-envoy\") pod \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " Nov 03 20:34:39.606359 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.606200 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istiod-ca-cert\") pod \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " Nov 03 20:34:39.606359 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.606235 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-token\" (UniqueName: \"kubernetes.io/projected/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-token\") pod \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " Nov 03 20:34:39.606490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.606362 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-kube-api-access-cl7tt\" (UniqueName: \"kubernetes.io/projected/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-kube-api-access-cl7tt\") pod \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " Nov 03 20:34:39.606490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.606449 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-workload-certs\") pod \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " Nov 03 20:34:39.606490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.606481 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-workload-socket\") pod \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " Nov 03 20:34:39.606638 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.606520 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-credential-socket\") pod \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " Nov 03 20:34:39.606638 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.606554 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-podinfo\") pod \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " Nov 03 20:34:39.606638 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.606594 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-data\") pod \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\" (UID: \"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8\") " Nov 03 20:34:39.607342 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.607180 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-credential-socket" pod "1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" (UID: "1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:39.607489 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.607370 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-data" pod "1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" (UID: "1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:39.607826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.607605 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-workload-certs" pod "1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" (UID: "1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:39.608393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.608052 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istiod-ca-cert" pod "1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" (UID: "1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:39.608393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.608315 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:39.608393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.608339 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:39.608393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.608354 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:39.608393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.608369 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:39.608805 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.608773 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-workload-socket" pod "1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" (UID: "1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:39.610278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.610248 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-podinfo" pod "1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" (UID: "1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:34:39.610414 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.610391 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-token" pod "1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" (UID: "1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:39.611320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.611297 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-kube-api-access-cl7tt" pod "1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" (UID: "1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8"). InnerVolumeSpecName "kube-api-access-cl7tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:39.611407 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.611366 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-envoy" pod "1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" (UID: "1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:39.709583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.709545 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:39.709583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.709582 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:39.709815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.709597 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:39.709815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.709610 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl7tt\" (UniqueName: \"kubernetes.io/projected/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-kube-api-access-cl7tt\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:39.709815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:39.709623 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:40.080720 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.080679 5059 generic.go:356] "Generic (PLEG): container finished" podID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerID="ba8b1d44e1eaed0f3e420da024ef6f5f654842540fe5435339c0ff9078b7ee8c" exitCode=0 Nov 03 20:34:40.080920 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.080726 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" Nov 03 20:34:40.080920 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.080762 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" event={"ID":"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8","Type":"ContainerDied","Data":"ba8b1d44e1eaed0f3e420da024ef6f5f654842540fe5435339c0ff9078b7ee8c"} Nov 03 20:34:40.080920 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.080831 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk" event={"ID":"1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8","Type":"ContainerDied","Data":"7d87b96d63a6638c8038b372010e4b7d308315e17a3c3fb4ebb4d870363b129a"} Nov 03 20:34:40.080920 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.080863 5059 scope.go:122] "RemoveContainer" containerID="ba8b1d44e1eaed0f3e420da024ef6f5f654842540fe5435339c0ff9078b7ee8c" Nov 03 20:34:40.091628 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.091608 5059 scope.go:122] "RemoveContainer" containerID="ba8b1d44e1eaed0f3e420da024ef6f5f654842540fe5435339c0ff9078b7ee8c" Nov 03 20:34:40.091890 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:34:40.091865 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba8b1d44e1eaed0f3e420da024ef6f5f654842540fe5435339c0ff9078b7ee8c\": container with ID starting with ba8b1d44e1eaed0f3e420da024ef6f5f654842540fe5435339c0ff9078b7ee8c not found: ID does not exist" containerID="ba8b1d44e1eaed0f3e420da024ef6f5f654842540fe5435339c0ff9078b7ee8c" Nov 03 20:34:40.091952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.091901 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba8b1d44e1eaed0f3e420da024ef6f5f654842540fe5435339c0ff9078b7ee8c"} err="failed to get container status \"ba8b1d44e1eaed0f3e420da024ef6f5f654842540fe5435339c0ff9078b7ee8c\": rpc error: code = NotFound desc = could not find container \"ba8b1d44e1eaed0f3e420da024ef6f5f654842540fe5435339c0ff9078b7ee8c\": container with ID starting with ba8b1d44e1eaed0f3e420da024ef6f5f654842540fe5435339c0ff9078b7ee8c not found: ID does not exist" Nov 03 20:34:40.104230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.104207 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk"] Nov 03 20:34:40.109718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.109698 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-bm2lk"] Nov 03 20:34:40.362211 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.362172 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk"] Nov 03 20:34:40.363246 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.363222 5059 cpu_manager.go:414] "containerMap: removing container" podUID="f544668b-01c9-4c84-bffe-d6f09f4566e5" containerName="revoke-keys" Nov 03 20:34:40.363329 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.363249 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f544668b-01c9-4c84-bffe-d6f09f4566e5" containerName="revoke-keys" Nov 03 20:34:40.363329 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.363278 5059 cpu_manager.go:414] "containerMap: removing container" podUID="ddf9e671-9f34-424b-976f-2b9a25deddbf" containerName="revoke-keys" Nov 03 20:34:40.363329 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.363286 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddf9e671-9f34-424b-976f-2b9a25deddbf" containerName="revoke-keys" Nov 03 20:34:40.363329 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.363315 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" Nov 03 20:34:40.363329 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.363324 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" Nov 03 20:34:40.363520 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.363482 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddf9e671-9f34-424b-976f-2b9a25deddbf" containerName="revoke-keys" Nov 03 20:34:40.363520 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.363498 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" containerName="istio-proxy" Nov 03 20:34:40.363520 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.363509 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f544668b-01c9-4c84-bffe-d6f09f4566e5" containerName="revoke-keys" Nov 03 20:34:40.368760 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.368734 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:34:40.377198 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.377172 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk"] Nov 03 20:34:40.378649 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.378627 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bfc240\"/\"openshift-service-ca.crt\"" Nov 03 20:34:40.378776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.378733 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bfc240\"/\"kube-root-ca.crt\"" Nov 03 20:34:40.393281 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.393259 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bfc240\"/\"sub-test-model-bfc240-kserve-self-signed-certs\"" Nov 03 20:34:40.393370 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.393345 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bfc240\"/\"default-dockercfg-h25rf\"" Nov 03 20:34:40.420914 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.420891 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-tls-certs\") pod \"sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk\" (UID: \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:34:40.421095 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.421079 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-model-cache\") pod \"sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk\" (UID: \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:34:40.421152 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.421134 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7cbh\" (UniqueName: \"kubernetes.io/projected/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-kube-api-access-l7cbh\") pod \"sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk\" (UID: \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:34:40.421260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.421248 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-home\") pod \"sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk\" (UID: \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:34:40.421295 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.421279 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-dshm\") pod \"sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk\" (UID: \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:34:40.421410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.421381 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-tmp-dir\") pod \"sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk\" (UID: \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:34:40.522829 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.522802 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-tmp-dir\") pod \"sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk\" (UID: \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:34:40.522991 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.522878 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-tls-certs\") pod \"sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk\" (UID: \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:34:40.523075 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.523044 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-model-cache\") pod \"sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk\" (UID: \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:34:40.523148 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.523108 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7cbh\" (UniqueName: \"kubernetes.io/projected/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-kube-api-access-l7cbh\") pod \"sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk\" (UID: \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:34:40.523253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.523221 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-home\") pod \"sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk\" (UID: \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:34:40.523309 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.523269 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-dshm\") pod \"sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk\" (UID: \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:34:40.525968 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.524690 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-tmp-dir\") pod \"sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk\" (UID: \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:34:40.526346 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.526299 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-home\") pod \"sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk\" (UID: \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:34:40.574684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.556470 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-model-cache\") pod \"sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk\" (UID: \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:34:40.574684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.569322 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-tls-certs\") pod \"sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk\" (UID: \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:34:40.588688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.579782 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-dshm\") pod \"sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk\" (UID: \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:34:40.588688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.583136 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7cbh\" (UniqueName: \"kubernetes.io/projected/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-kube-api-access-l7cbh\") pod \"sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk\" (UID: \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:34:40.697329 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:40.696972 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:34:40.780518 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:34:40.780424 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:34:41.049842 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:41.049815 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk"] Nov 03 20:34:41.050364 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:34:41.050342 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb66cb90_2ccf_4325_80aa_d8e43ac54a22.slice/crio-5d31881ffd1668800f9e00f1343ca4f9463f990eb1ee3878260c553cc9f7433b WatchSource:0}: Error finding container 5d31881ffd1668800f9e00f1343ca4f9463f990eb1ee3878260c553cc9f7433b: Status 404 returned error can't find the container with id 5d31881ffd1668800f9e00f1343ca4f9463f990eb1ee3878260c553cc9f7433b Nov 03 20:34:41.090316 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:41.090285 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" event={"ID":"bb66cb90-2ccf-4325-80aa-d8e43ac54a22","Type":"ContainerStarted","Data":"5d31881ffd1668800f9e00f1343ca4f9463f990eb1ee3878260c553cc9f7433b"} Nov 03 20:34:41.860488 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:41.860451 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8" path="/var/lib/kubelet/pods/1ef66ce6-8b55-4ef9-8dce-fdfad6b1edb8/volumes" Nov 03 20:34:42.099076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:42.099039 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" event={"ID":"bb66cb90-2ccf-4325-80aa-d8e43ac54a22","Type":"ContainerStarted","Data":"7a8879573a59af72b0c503748f3635c55ffb0f90934ffc77fc4ef4f8f957e740"} Nov 03 20:34:42.121117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:42.121030 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" podStartSLOduration=2.121017892 podStartE2EDuration="2.121017892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:42.11811681 +0000 UTC m=+3476.375914071" watchObservedRunningTime="2026-11-03 20:34:42.121017892 +0000 UTC m=+3476.378815214" Nov 03 20:34:42.698085 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:42.698053 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:34:43.175609 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:43.175543 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst" podUID="977cec40-92df-410a-a3b6-68520b98aaf2" containerName="payload-processing" containerID="cri-o://22d805127f7df6029c93afae2159725cef5c3fda386692204c7376c78cf1f788" gracePeriod=25 Nov 03 20:34:43.477348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:43.477318 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst" Nov 03 20:34:43.595752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:43.595719 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/977cec40-92df-410a-a3b6-68520b98aaf2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/977cec40-92df-410a-a3b6-68520b98aaf2-plugins-config-volume\") pod \"977cec40-92df-410a-a3b6-68520b98aaf2\" (UID: \"977cec40-92df-410a-a3b6-68520b98aaf2\") " Nov 03 20:34:43.595910 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:43.595819 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/977cec40-92df-410a-a3b6-68520b98aaf2-kube-api-access-75tbk\" (UniqueName: \"kubernetes.io/projected/977cec40-92df-410a-a3b6-68520b98aaf2-kube-api-access-75tbk\") pod \"977cec40-92df-410a-a3b6-68520b98aaf2\" (UID: \"977cec40-92df-410a-a3b6-68520b98aaf2\") " Nov 03 20:34:43.596170 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:43.596139 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/977cec40-92df-410a-a3b6-68520b98aaf2-plugins-config-volume" pod "977cec40-92df-410a-a3b6-68520b98aaf2" (UID: "977cec40-92df-410a-a3b6-68520b98aaf2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:43.596539 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:43.596523 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/977cec40-92df-410a-a3b6-68520b98aaf2-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:43.598150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:43.598122 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/977cec40-92df-410a-a3b6-68520b98aaf2-kube-api-access-75tbk" pod "977cec40-92df-410a-a3b6-68520b98aaf2" (UID: "977cec40-92df-410a-a3b6-68520b98aaf2"). InnerVolumeSpecName "kube-api-access-75tbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:43.698438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:43.698365 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75tbk\" (UniqueName: \"kubernetes.io/projected/977cec40-92df-410a-a3b6-68520b98aaf2-kube-api-access-75tbk\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:44.126425 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:44.126384 5059 generic.go:356] "Generic (PLEG): container finished" podID="977cec40-92df-410a-a3b6-68520b98aaf2" containerID="22d805127f7df6029c93afae2159725cef5c3fda386692204c7376c78cf1f788" exitCode=0 Nov 03 20:34:44.126584 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:44.126426 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst" Nov 03 20:34:44.126584 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:44.126470 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst" event={"ID":"977cec40-92df-410a-a3b6-68520b98aaf2","Type":"ContainerDied","Data":"22d805127f7df6029c93afae2159725cef5c3fda386692204c7376c78cf1f788"} Nov 03 20:34:44.126584 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:44.126518 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst" event={"ID":"977cec40-92df-410a-a3b6-68520b98aaf2","Type":"ContainerDied","Data":"0e8dd4143aa1ca3e3e68053f35530077b364020b2c76a67ec8b8b8920aa8be5a"} Nov 03 20:34:44.126584 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:44.126545 5059 scope.go:122] "RemoveContainer" containerID="22d805127f7df6029c93afae2159725cef5c3fda386692204c7376c78cf1f788" Nov 03 20:34:44.137089 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:44.137071 5059 scope.go:122] "RemoveContainer" containerID="22d805127f7df6029c93afae2159725cef5c3fda386692204c7376c78cf1f788" Nov 03 20:34:44.137341 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:34:44.137316 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d805127f7df6029c93afae2159725cef5c3fda386692204c7376c78cf1f788\": container with ID starting with 22d805127f7df6029c93afae2159725cef5c3fda386692204c7376c78cf1f788 not found: ID does not exist" containerID="22d805127f7df6029c93afae2159725cef5c3fda386692204c7376c78cf1f788" Nov 03 20:34:44.137442 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:44.137349 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d805127f7df6029c93afae2159725cef5c3fda386692204c7376c78cf1f788"} err="failed to get container status \"22d805127f7df6029c93afae2159725cef5c3fda386692204c7376c78cf1f788\": rpc error: code = NotFound desc = could not find container \"22d805127f7df6029c93afae2159725cef5c3fda386692204c7376c78cf1f788\": container with ID starting with 22d805127f7df6029c93afae2159725cef5c3fda386692204c7376c78cf1f788 not found: ID does not exist" Nov 03 20:34:44.146850 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:44.146828 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst"] Nov 03 20:34:44.150946 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:44.150924 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-98529d-5857fc5786-qkmst"] Nov 03 20:34:45.861066 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:45.861030 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="977cec40-92df-410a-a3b6-68520b98aaf2" path="/var/lib/kubelet/pods/977cec40-92df-410a-a3b6-68520b98aaf2/volumes" Nov 03 20:34:50.697849 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:50.697810 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:34:50.710548 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:50.710518 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:34:51.196215 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:51.196185 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:34:54.782565 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:54.782520 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g"] Nov 03 20:34:54.783408 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:54.783366 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" podUID="35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" containerName="istio-proxy" containerID="cri-o://5932a6c7fb7e966629bc5438e3483e6beb7b50f48ede3eee43850147c22bdddd" gracePeriod=30 Nov 03 20:34:57.784904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:57.784858 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95"] Nov 03 20:34:57.785373 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:57.785114 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95" podUID="0298aa52-aec0-481a-b7d1-12964e8137c8" containerName="maas-api" containerID="cri-o://97b5e89796f952bd7b97823c3ecf143080a6c7e2492e836cfba674e2901c0056" gracePeriod=30 Nov 03 20:34:58.586759 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:58.573275 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk"] Nov 03 20:34:58.586759 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:58.573530 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" containerID="cri-o://52d9dfafb55bc5b6dad5352af2151b15f85f2c035a91e44f089c01883269dcbc" gracePeriod=30 Nov 03 20:34:58.750256 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:58.711109 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95" Nov 03 20:34:58.898696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:58.854384 5059 generic.go:356] "Generic (PLEG): container finished" podID="0298aa52-aec0-481a-b7d1-12964e8137c8" containerID="97b5e89796f952bd7b97823c3ecf143080a6c7e2492e836cfba674e2901c0056" exitCode=0 Nov 03 20:34:58.898696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:58.854626 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95" event={"ID":"0298aa52-aec0-481a-b7d1-12964e8137c8","Type":"ContainerDied","Data":"97b5e89796f952bd7b97823c3ecf143080a6c7e2492e836cfba674e2901c0056"} Nov 03 20:34:58.898696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:58.854822 5059 scope.go:122] "RemoveContainer" containerID="97b5e89796f952bd7b97823c3ecf143080a6c7e2492e836cfba674e2901c0056" Nov 03 20:34:59.141783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.141747 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0298aa52-aec0-481a-b7d1-12964e8137c8-kube-api-access-w6glj\" (UniqueName: \"kubernetes.io/projected/0298aa52-aec0-481a-b7d1-12964e8137c8-kube-api-access-w6glj\") pod \"0298aa52-aec0-481a-b7d1-12964e8137c8\" (UID: \"0298aa52-aec0-481a-b7d1-12964e8137c8\") " Nov 03 20:34:59.142321 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.142276 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0298aa52-aec0-481a-b7d1-12964e8137c8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0298aa52-aec0-481a-b7d1-12964e8137c8-maas-api-tls\") pod \"0298aa52-aec0-481a-b7d1-12964e8137c8\" (UID: \"0298aa52-aec0-481a-b7d1-12964e8137c8\") " Nov 03 20:34:59.142478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.142453 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0298aa52-aec0-481a-b7d1-12964e8137c8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0298aa52-aec0-481a-b7d1-12964e8137c8-openshift-service-ca\") pod \"0298aa52-aec0-481a-b7d1-12964e8137c8\" (UID: \"0298aa52-aec0-481a-b7d1-12964e8137c8\") " Nov 03 20:34:59.236930 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.215927 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0298aa52-aec0-481a-b7d1-12964e8137c8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0298aa52-aec0-481a-b7d1-12964e8137c8-metrics-certs\") pod \"0298aa52-aec0-481a-b7d1-12964e8137c8\" (UID: \"0298aa52-aec0-481a-b7d1-12964e8137c8\") " Nov 03 20:34:59.236930 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.224963 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0298aa52-aec0-481a-b7d1-12964e8137c8-openshift-service-ca" pod "0298aa52-aec0-481a-b7d1-12964e8137c8" (UID: "0298aa52-aec0-481a-b7d1-12964e8137c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:59.236930 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.225654 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf"] Nov 03 20:34:59.236930 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.225935 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" podUID="82894901-9901-4dc4-948a-321782ef8288" containerName="istio-proxy" containerID="cri-o://dfba132bc5182982715f89fef8a8bee5073dca359534056d29abfab70c0a5c2b" gracePeriod=30 Nov 03 20:34:59.236930 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.227185 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0298aa52-aec0-481a-b7d1-12964e8137c8-kube-api-access-w6glj" pod "0298aa52-aec0-481a-b7d1-12964e8137c8" (UID: "0298aa52-aec0-481a-b7d1-12964e8137c8"). InnerVolumeSpecName "kube-api-access-w6glj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:59.236930 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.232761 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0298aa52-aec0-481a-b7d1-12964e8137c8-metrics-certs" pod "0298aa52-aec0-481a-b7d1-12964e8137c8" (UID: "0298aa52-aec0-481a-b7d1-12964e8137c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:59.236930 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.234194 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0298aa52-aec0-481a-b7d1-12964e8137c8-maas-api-tls" pod "0298aa52-aec0-481a-b7d1-12964e8137c8" (UID: "0298aa52-aec0-481a-b7d1-12964e8137c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:59.241764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.241729 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w"] Nov 03 20:34:59.241999 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.241966 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" containerID="cri-o://608a9becc0865d2fa635a042e78e2d293a90887169f9f7a9a78ac4715c431fc8" gracePeriod=30 Nov 03 20:34:59.244496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.244466 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn"] Nov 03 20:34:59.244719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.244690 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" containerID="cri-o://1119a404faca34d41ecebf28afb622f4f839d6f1e5fd27a2976df8db31439851" gracePeriod=30 Nov 03 20:34:59.252903 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.252646 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x"] Nov 03 20:34:59.252903 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.252864 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" podUID="68bc094a-84e2-41c2-9fb5-081682413f27" containerName="istio-proxy" containerID="cri-o://a96237ecf3f4902402f80b450ff7a22b4fcd8a6500b222e4be665f0e434b7f54" gracePeriod=30 Nov 03 20:34:59.258972 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.258941 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws"] Nov 03 20:34:59.259167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.259136 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerName="istio-proxy" containerID="cri-o://a7892a98b6abfbe7e24a6a645e11e9dc24194d6039d51dac1b67f8117fb642b5" gracePeriod=30 Nov 03 20:34:59.262781 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.262759 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7"] Nov 03 20:34:59.262917 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.262897 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" containerID="cri-o://26d622763433e981c524cf0ddaec2006cc9ffcb67eb091acf6792a109cd90315" gracePeriod=30 Nov 03 20:34:59.266503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.266480 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr"] Nov 03 20:34:59.266645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.266620 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" podUID="d4fdd390-7fe2-406d-832a-7ecc545d97f4" containerName="istio-proxy" containerID="cri-o://c9577eca08e6a70c108edb2ac96c1b9d5f54df9956d0d1467ddc72fffdf59343" gracePeriod=30 Nov 03 20:34:59.319921 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.319876 5059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0298aa52-aec0-481a-b7d1-12964e8137c8-metrics-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:59.319921 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.319923 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6glj\" (UniqueName: \"kubernetes.io/projected/0298aa52-aec0-481a-b7d1-12964e8137c8-kube-api-access-w6glj\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:59.320155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.319941 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0298aa52-aec0-481a-b7d1-12964e8137c8-maas-api-tls\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:59.320155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.319956 5059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0298aa52-aec0-481a-b7d1-12964e8137c8-openshift-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:59.862804 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.862771 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95" event={"ID":"0298aa52-aec0-481a-b7d1-12964e8137c8","Type":"ContainerDied","Data":"1e7e9fb8cc6a318af498496044b7454b1a02e855c06f203301574493f60166da"} Nov 03 20:34:59.862909 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.862791 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95" Nov 03 20:34:59.907738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.907694 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95"] Nov 03 20:34:59.914421 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:34:59.914392 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7b70d-5b89775464-rpj95"] Nov 03 20:35:00.052745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.052690 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:35:00.131013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.130983 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-workload-socket\") pod \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " Nov 03 20:35:00.131149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.131095 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-podinfo\") pod \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " Nov 03 20:35:00.131149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.131117 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-data\") pod \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " Nov 03 20:35:00.131220 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.131190 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-token\" (UniqueName: \"kubernetes.io/projected/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-token\") pod \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " Nov 03 20:35:00.131269 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.131218 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-workload-socket" pod "35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" (UID: "35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:00.131269 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.131226 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-workload-certs\") pod \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " Nov 03 20:35:00.131371 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.131306 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-envoy\") pod \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " Nov 03 20:35:00.131371 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.131337 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istiod-ca-cert\") pod \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " Nov 03 20:35:00.131469 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.131422 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-credential-socket\") pod \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " Nov 03 20:35:00.131531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.131461 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-workload-certs" pod "35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" (UID: "35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:00.131531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.131485 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-data" pod "35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" (UID: "35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:00.132207 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.131628 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-kube-api-access-wd5rh\" (UniqueName: \"kubernetes.io/projected/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-kube-api-access-wd5rh\") pod \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\" (UID: \"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06\") " Nov 03 20:35:00.132207 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.131797 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-credential-socket" pod "35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" (UID: "35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:00.132207 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.131842 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istiod-ca-cert" pod "35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" (UID: "35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:00.132406 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.132382 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.132468 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.132411 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.132468 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.132427 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.132468 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.132441 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.132468 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.132457 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.133469 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.133444 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-podinfo" pod "35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" (UID: "35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:00.133715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.133656 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-envoy" pod "35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" (UID: "35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:00.133787 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.133713 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-kube-api-access-wd5rh" pod "35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" (UID: "35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06"). InnerVolumeSpecName "kube-api-access-wd5rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:00.133837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.133800 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-token" pod "35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" (UID: "35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:00.233511 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.233479 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.233511 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.233511 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.233692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.233521 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.233692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.233530 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd5rh\" (UniqueName: \"kubernetes.io/projected/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06-kube-api-access-wd5rh\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.409585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.409547 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:35:00.409772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.409613 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:35:00.487522 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.487483 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:35:00.487684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.487544 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:35:00.871332 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.871301 5059 generic.go:356] "Generic (PLEG): container finished" podID="35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" containerID="5932a6c7fb7e966629bc5438e3483e6beb7b50f48ede3eee43850147c22bdddd" exitCode=0 Nov 03 20:35:00.871486 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.871342 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" Nov 03 20:35:00.871486 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.871380 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" event={"ID":"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06","Type":"ContainerDied","Data":"5932a6c7fb7e966629bc5438e3483e6beb7b50f48ede3eee43850147c22bdddd"} Nov 03 20:35:00.871486 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.871427 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g" event={"ID":"35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06","Type":"ContainerDied","Data":"6f4798010a65daf71ca385fbaced28e5f1e5a941bca18ebf0548a3408aca06ed"} Nov 03 20:35:00.871486 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.871449 5059 scope.go:122] "RemoveContainer" containerID="5932a6c7fb7e966629bc5438e3483e6beb7b50f48ede3eee43850147c22bdddd" Nov 03 20:35:00.882715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.882697 5059 scope.go:122] "RemoveContainer" containerID="5932a6c7fb7e966629bc5438e3483e6beb7b50f48ede3eee43850147c22bdddd" Nov 03 20:35:00.882977 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:35:00.882957 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5932a6c7fb7e966629bc5438e3483e6beb7b50f48ede3eee43850147c22bdddd\": container with ID starting with 5932a6c7fb7e966629bc5438e3483e6beb7b50f48ede3eee43850147c22bdddd not found: ID does not exist" containerID="5932a6c7fb7e966629bc5438e3483e6beb7b50f48ede3eee43850147c22bdddd" Nov 03 20:35:00.883049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.882988 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5932a6c7fb7e966629bc5438e3483e6beb7b50f48ede3eee43850147c22bdddd"} err="failed to get container status \"5932a6c7fb7e966629bc5438e3483e6beb7b50f48ede3eee43850147c22bdddd\": rpc error: code = NotFound desc = could not find container \"5932a6c7fb7e966629bc5438e3483e6beb7b50f48ede3eee43850147c22bdddd\": container with ID starting with 5932a6c7fb7e966629bc5438e3483e6beb7b50f48ede3eee43850147c22bdddd not found: ID does not exist" Nov 03 20:35:00.898434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.898411 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g"] Nov 03 20:35:00.904605 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:00.904583 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-98529d-openshift-default-5bd85f96bf-46d2g"] Nov 03 20:35:01.927321 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:01.927280 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0298aa52-aec0-481a-b7d1-12964e8137c8" path="/var/lib/kubelet/pods/0298aa52-aec0-481a-b7d1-12964e8137c8/volumes" Nov 03 20:35:01.928580 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:01.928266 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" path="/var/lib/kubelet/pods/35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06/volumes" Nov 03 20:35:02.476081 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:02.475929 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 03 20:35:02.476081 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:02.475986 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:35:02.599715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:02.599641 5059 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:35:02.599873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:02.599727 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:35:03.490492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.490457 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx"] Nov 03 20:35:03.491233 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.491214 5059 cpu_manager.go:414] "containerMap: removing container" podUID="977cec40-92df-410a-a3b6-68520b98aaf2" containerName="payload-processing" Nov 03 20:35:03.491295 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.491235 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="977cec40-92df-410a-a3b6-68520b98aaf2" containerName="payload-processing" Nov 03 20:35:03.491295 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.491248 5059 cpu_manager.go:414] "containerMap: removing container" podUID="35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" containerName="istio-proxy" Nov 03 20:35:03.491295 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.491254 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" containerName="istio-proxy" Nov 03 20:35:03.491295 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.491282 5059 cpu_manager.go:414] "containerMap: removing container" podUID="0298aa52-aec0-481a-b7d1-12964e8137c8" containerName="maas-api" Nov 03 20:35:03.491295 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.491287 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0298aa52-aec0-481a-b7d1-12964e8137c8" containerName="maas-api" Nov 03 20:35:03.491447 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.491398 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0298aa52-aec0-481a-b7d1-12964e8137c8" containerName="maas-api" Nov 03 20:35:03.491447 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.491413 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="977cec40-92df-410a-a3b6-68520b98aaf2" containerName="payload-processing" Nov 03 20:35:03.491447 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.491419 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="35cec4f7-b7d9-49b4-b9a7-ff7eef2a1d06" containerName="istio-proxy" Nov 03 20:35:03.496543 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.496524 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:35:03.500513 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.500490 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-f154ac\"/\"kube-root-ca.crt\"" Nov 03 20:35:03.502360 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.502335 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-f154ac\"/\"sub-test-model-f154ac-kserve-self-signed-certs\"" Nov 03 20:35:03.502495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.502400 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-f154ac\"/\"default-dockercfg-d457l\"" Nov 03 20:35:03.502495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.502444 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-f154ac\"/\"openshift-service-ca.crt\"" Nov 03 20:35:03.509106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.509084 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx"] Nov 03 20:35:03.632950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.630731 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rxmm\" (UniqueName: \"kubernetes.io/projected/765cb609-3515-4d3d-8158-da836d306b6c-kube-api-access-2rxmm\") pod \"sub-test-model-f154ac-kserve-5846d75657-jrrlx\" (UID: \"765cb609-3515-4d3d-8158-da836d306b6c\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:35:03.632950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.631053 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-home\") pod \"sub-test-model-f154ac-kserve-5846d75657-jrrlx\" (UID: \"765cb609-3515-4d3d-8158-da836d306b6c\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:35:03.632950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.631191 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-dshm\") pod \"sub-test-model-f154ac-kserve-5846d75657-jrrlx\" (UID: \"765cb609-3515-4d3d-8158-da836d306b6c\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:35:03.632950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.631285 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/765cb609-3515-4d3d-8158-da836d306b6c-tls-certs\") pod \"sub-test-model-f154ac-kserve-5846d75657-jrrlx\" (UID: \"765cb609-3515-4d3d-8158-da836d306b6c\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:35:03.632950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.631580 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-model-cache\") pod \"sub-test-model-f154ac-kserve-5846d75657-jrrlx\" (UID: \"765cb609-3515-4d3d-8158-da836d306b6c\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:35:03.632950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.631702 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-tmp-dir\") pod \"sub-test-model-f154ac-kserve-5846d75657-jrrlx\" (UID: \"765cb609-3515-4d3d-8158-da836d306b6c\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:35:03.733634 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.733595 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-home\") pod \"sub-test-model-f154ac-kserve-5846d75657-jrrlx\" (UID: \"765cb609-3515-4d3d-8158-da836d306b6c\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:35:03.733837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.733777 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-dshm\") pod \"sub-test-model-f154ac-kserve-5846d75657-jrrlx\" (UID: \"765cb609-3515-4d3d-8158-da836d306b6c\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:35:03.733910 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.733871 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/765cb609-3515-4d3d-8158-da836d306b6c-tls-certs\") pod \"sub-test-model-f154ac-kserve-5846d75657-jrrlx\" (UID: \"765cb609-3515-4d3d-8158-da836d306b6c\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:35:03.734058 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.734036 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-home\") pod \"sub-test-model-f154ac-kserve-5846d75657-jrrlx\" (UID: \"765cb609-3515-4d3d-8158-da836d306b6c\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:35:03.734141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.734041 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-model-cache\") pod \"sub-test-model-f154ac-kserve-5846d75657-jrrlx\" (UID: \"765cb609-3515-4d3d-8158-da836d306b6c\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:35:03.734141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.734132 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-tmp-dir\") pod \"sub-test-model-f154ac-kserve-5846d75657-jrrlx\" (UID: \"765cb609-3515-4d3d-8158-da836d306b6c\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:35:03.734312 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.734294 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rxmm\" (UniqueName: \"kubernetes.io/projected/765cb609-3515-4d3d-8158-da836d306b6c-kube-api-access-2rxmm\") pod \"sub-test-model-f154ac-kserve-5846d75657-jrrlx\" (UID: \"765cb609-3515-4d3d-8158-da836d306b6c\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:35:03.734385 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.734332 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-model-cache\") pod \"sub-test-model-f154ac-kserve-5846d75657-jrrlx\" (UID: \"765cb609-3515-4d3d-8158-da836d306b6c\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:35:03.734437 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.734391 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-tmp-dir\") pod \"sub-test-model-f154ac-kserve-5846d75657-jrrlx\" (UID: \"765cb609-3515-4d3d-8158-da836d306b6c\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:35:03.736515 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.736494 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-dshm\") pod \"sub-test-model-f154ac-kserve-5846d75657-jrrlx\" (UID: \"765cb609-3515-4d3d-8158-da836d306b6c\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:35:03.736818 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.736800 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/765cb609-3515-4d3d-8158-da836d306b6c-tls-certs\") pod \"sub-test-model-f154ac-kserve-5846d75657-jrrlx\" (UID: \"765cb609-3515-4d3d-8158-da836d306b6c\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:35:03.746347 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.746292 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rxmm\" (UniqueName: \"kubernetes.io/projected/765cb609-3515-4d3d-8158-da836d306b6c-kube-api-access-2rxmm\") pod \"sub-test-model-f154ac-kserve-5846d75657-jrrlx\" (UID: \"765cb609-3515-4d3d-8158-da836d306b6c\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:35:03.813341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:03.813309 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:35:04.106941 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:35:04.106911 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod765cb609_3515_4d3d_8158_da836d306b6c.slice/crio-8b49fd665c473e908ea644249c843e4dc6beb1d996760a5818b09159babb5d73 WatchSource:0}: Error finding container 8b49fd665c473e908ea644249c843e4dc6beb1d996760a5818b09159babb5d73: Status 404 returned error can't find the container with id 8b49fd665c473e908ea644249c843e4dc6beb1d996760a5818b09159babb5d73 Nov 03 20:35:04.108562 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.108477 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx"] Nov 03 20:35:04.596738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.596715 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:35:04.634732 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.634703 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.834291 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82894901-9901-4dc4-948a-321782ef8288-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82894901-9901-4dc4-948a-321782ef8288-istio-podinfo\") pod \"82894901-9901-4dc4-948a-321782ef8288\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.834432 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-data\") pod \"569b2103-147f-4ea1-9b55-0e859b1c5e52\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.834496 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-workload-socket\") pod \"82894901-9901-4dc4-948a-321782ef8288\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.834567 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-workload-certs\") pod \"82894901-9901-4dc4-948a-321782ef8288\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.834685 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-credential-socket\") pod \"82894901-9901-4dc4-948a-321782ef8288\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.834745 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-token\" (UniqueName: \"kubernetes.io/projected/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-token\") pod \"569b2103-147f-4ea1-9b55-0e859b1c5e52\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.834786 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-podinfo\") pod \"569b2103-147f-4ea1-9b55-0e859b1c5e52\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.834819 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-workload-certs\") pod \"569b2103-147f-4ea1-9b55-0e859b1c5e52\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.834909 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82894901-9901-4dc4-948a-321782ef8288-istio-token\" (UniqueName: \"kubernetes.io/projected/82894901-9901-4dc4-948a-321782ef8288-istio-token\") pod \"82894901-9901-4dc4-948a-321782ef8288\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.834953 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569b2103-147f-4ea1-9b55-0e859b1c5e52-kube-api-access-5hzr5\" (UniqueName: \"kubernetes.io/projected/569b2103-147f-4ea1-9b55-0e859b1c5e52-kube-api-access-5hzr5\") pod \"569b2103-147f-4ea1-9b55-0e859b1c5e52\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.835005 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82894901-9901-4dc4-948a-321782ef8288-kube-api-access-jcptm\" (UniqueName: \"kubernetes.io/projected/82894901-9901-4dc4-948a-321782ef8288-kube-api-access-jcptm\") pod \"82894901-9901-4dc4-948a-321782ef8288\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.835156 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-workload-socket\") pod \"569b2103-147f-4ea1-9b55-0e859b1c5e52\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.835257 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-istio-envoy\") pod \"82894901-9901-4dc4-948a-321782ef8288\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.835315 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-istio-data\") pod \"82894901-9901-4dc4-948a-321782ef8288\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.835356 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-credential-socket\") pod \"569b2103-147f-4ea1-9b55-0e859b1c5e52\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.835386 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/569b2103-147f-4ea1-9b55-0e859b1c5e52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569b2103-147f-4ea1-9b55-0e859b1c5e52-istiod-ca-cert\") pod \"569b2103-147f-4ea1-9b55-0e859b1c5e52\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.835419 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82894901-9901-4dc4-948a-321782ef8288-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82894901-9901-4dc4-948a-321782ef8288-istiod-ca-cert\") pod \"82894901-9901-4dc4-948a-321782ef8288\" (UID: \"82894901-9901-4dc4-948a-321782ef8288\") " Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.835461 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-envoy\") pod \"569b2103-147f-4ea1-9b55-0e859b1c5e52\" (UID: \"569b2103-147f-4ea1-9b55-0e859b1c5e52\") " Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.938792 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-workload-certs" pod "569b2103-147f-4ea1-9b55-0e859b1c5e52" (UID: "569b2103-147f-4ea1-9b55-0e859b1c5e52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.939174 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-data" pod "569b2103-147f-4ea1-9b55-0e859b1c5e52" (UID: "569b2103-147f-4ea1-9b55-0e859b1c5e52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.939439 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-istio-data" pod "82894901-9901-4dc4-948a-321782ef8288" (UID: "82894901-9901-4dc4-948a-321782ef8288"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.939919 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-workload-certs" pod "82894901-9901-4dc4-948a-321782ef8288" (UID: "82894901-9901-4dc4-948a-321782ef8288"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.940102 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82894901-9901-4dc4-948a-321782ef8288-istiod-ca-cert" pod "82894901-9901-4dc4-948a-321782ef8288" (UID: "82894901-9901-4dc4-948a-321782ef8288"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.940194 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-workload-socket" pod "569b2103-147f-4ea1-9b55-0e859b1c5e52" (UID: "569b2103-147f-4ea1-9b55-0e859b1c5e52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.941786 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-workload-socket" pod "82894901-9901-4dc4-948a-321782ef8288" (UID: "82894901-9901-4dc4-948a-321782ef8288"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:04.946824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:04.941853 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-credential-socket" pod "82894901-9901-4dc4-948a-321782ef8288" (UID: "82894901-9901-4dc4-948a-321782ef8288"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.018138 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.016568 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-podinfo" pod "569b2103-147f-4ea1-9b55-0e859b1c5e52" (UID: "569b2103-147f-4ea1-9b55-0e859b1c5e52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:05.018138 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.017773 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569b2103-147f-4ea1-9b55-0e859b1c5e52-istiod-ca-cert" pod "569b2103-147f-4ea1-9b55-0e859b1c5e52" (UID: "569b2103-147f-4ea1-9b55-0e859b1c5e52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:05.018138 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.017838 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-credential-socket" pod "569b2103-147f-4ea1-9b55-0e859b1c5e52" (UID: "569b2103-147f-4ea1-9b55-0e859b1c5e52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.018991 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.018870 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-token" pod "569b2103-147f-4ea1-9b55-0e859b1c5e52" (UID: "569b2103-147f-4ea1-9b55-0e859b1c5e52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:05.020765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.020171 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.020765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.020207 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.020765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.020234 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.020765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.020250 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.020765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.020266 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.020765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.020286 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.020765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.020302 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.020765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.020318 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.020765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.020332 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.020765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.020352 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.020765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.020368 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569b2103-147f-4ea1-9b55-0e859b1c5e52-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.020765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.020384 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82894901-9901-4dc4-948a-321782ef8288-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.029904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.029848 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp"] Nov 03 20:35:05.043209 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.042003 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82894901-9901-4dc4-948a-321782ef8288-istio-token" pod "82894901-9901-4dc4-948a-321782ef8288" (UID: "82894901-9901-4dc4-948a-321782ef8288"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:05.048017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.047831 5059 generic.go:356] "Generic (PLEG): container finished" podID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerID="608a9becc0865d2fa635a042e78e2d293a90887169f9f7a9a78ac4715c431fc8" exitCode=0 Nov 03 20:35:05.048017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.047975 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" event={"ID":"6b615028-a2db-4ad0-8a57-53efc040d6a6","Type":"ContainerDied","Data":"608a9becc0865d2fa635a042e78e2d293a90887169f9f7a9a78ac4715c431fc8"} Nov 03 20:35:05.048216 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.048052 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" event={"ID":"6b615028-a2db-4ad0-8a57-53efc040d6a6","Type":"ContainerDied","Data":"480f3eb485ec680fca7451f936b9b0b3e05c116689a84694829e2750d5d7ca6c"} Nov 03 20:35:05.048216 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.048084 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="480f3eb485ec680fca7451f936b9b0b3e05c116689a84694829e2750d5d7ca6c" Nov 03 20:35:05.057422 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.057391 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:35:05.058860 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.058828 5059 generic.go:356] "Generic (PLEG): container finished" podID="68bc094a-84e2-41c2-9fb5-081682413f27" containerID="a96237ecf3f4902402f80b450ff7a22b4fcd8a6500b222e4be665f0e434b7f54" exitCode=0 Nov 03 20:35:05.058981 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.058962 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" event={"ID":"68bc094a-84e2-41c2-9fb5-081682413f27","Type":"ContainerDied","Data":"a96237ecf3f4902402f80b450ff7a22b4fcd8a6500b222e4be665f0e434b7f54"} Nov 03 20:35:05.064859 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.064810 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-envoy" pod "569b2103-147f-4ea1-9b55-0e859b1c5e52" (UID: "569b2103-147f-4ea1-9b55-0e859b1c5e52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.071797 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.070804 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82894901-9901-4dc4-948a-321782ef8288-kube-api-access-jcptm" pod "82894901-9901-4dc4-948a-321782ef8288" (UID: "82894901-9901-4dc4-948a-321782ef8288"). InnerVolumeSpecName "kube-api-access-jcptm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:05.071797 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.070891 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-istio-envoy" pod "82894901-9901-4dc4-948a-321782ef8288" (UID: "82894901-9901-4dc4-948a-321782ef8288"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.071797 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.071000 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82894901-9901-4dc4-948a-321782ef8288-istio-podinfo" pod "82894901-9901-4dc4-948a-321782ef8288" (UID: "82894901-9901-4dc4-948a-321782ef8288"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:05.071797 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.071104 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569b2103-147f-4ea1-9b55-0e859b1c5e52-kube-api-access-5hzr5" pod "569b2103-147f-4ea1-9b55-0e859b1c5e52" (UID: "569b2103-147f-4ea1-9b55-0e859b1c5e52"). InnerVolumeSpecName "kube-api-access-5hzr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:05.074388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.074361 5059 generic.go:356] "Generic (PLEG): container finished" podID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerID="a7892a98b6abfbe7e24a6a645e11e9dc24194d6039d51dac1b67f8117fb642b5" exitCode=0 Nov 03 20:35:05.074525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.074502 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" event={"ID":"19ab8a5f-fa5f-4548-bf5f-0465281d8677","Type":"ContainerDied","Data":"a7892a98b6abfbe7e24a6a645e11e9dc24194d6039d51dac1b67f8117fb642b5"} Nov 03 20:35:05.076650 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.076627 5059 generic.go:356] "Generic (PLEG): container finished" podID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerID="26d622763433e981c524cf0ddaec2006cc9ffcb67eb091acf6792a109cd90315" exitCode=0 Nov 03 20:35:05.076775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.076743 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" event={"ID":"ecab4d7a-4034-4b59-8500-8434baacafaf","Type":"ContainerDied","Data":"26d622763433e981c524cf0ddaec2006cc9ffcb67eb091acf6792a109cd90315"} Nov 03 20:35:05.076850 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.076788 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" event={"ID":"ecab4d7a-4034-4b59-8500-8434baacafaf","Type":"ContainerDied","Data":"4890a01b7c4abe7afbebae6c8bf2144e78a951720b341c57c7f308881ee37284"} Nov 03 20:35:05.076850 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.076813 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4890a01b7c4abe7afbebae6c8bf2144e78a951720b341c57c7f308881ee37284" Nov 03 20:35:05.077963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.077934 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:35:05.079166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.078606 5059 generic.go:356] "Generic (PLEG): container finished" podID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerID="1119a404faca34d41ecebf28afb622f4f839d6f1e5fd27a2976df8db31439851" exitCode=0 Nov 03 20:35:05.079166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.078777 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" event={"ID":"d75a00ce-f97c-431f-be9e-aac200bcd9d4","Type":"ContainerDied","Data":"1119a404faca34d41ecebf28afb622f4f839d6f1e5fd27a2976df8db31439851"} Nov 03 20:35:05.079166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.078821 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" event={"ID":"d75a00ce-f97c-431f-be9e-aac200bcd9d4","Type":"ContainerDied","Data":"515ce9f9064fcd016304283348824753b282aee6ca435c32656208c0c270345e"} Nov 03 20:35:05.079166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.078854 5059 scope.go:122] "RemoveContainer" containerID="1119a404faca34d41ecebf28afb622f4f839d6f1e5fd27a2976df8db31439851" Nov 03 20:35:05.079166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.079003 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn" Nov 03 20:35:05.082405 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.081519 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" event={"ID":"765cb609-3515-4d3d-8158-da836d306b6c","Type":"ContainerStarted","Data":"ad98b3a82780283d70af3272adec99dcd798b659e0ca38053ce40bf94a0d72eb"} Nov 03 20:35:05.082405 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.081551 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" event={"ID":"765cb609-3515-4d3d-8158-da836d306b6c","Type":"ContainerStarted","Data":"8b49fd665c473e908ea644249c843e4dc6beb1d996760a5818b09159babb5d73"} Nov 03 20:35:05.084894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.084870 5059 generic.go:356] "Generic (PLEG): container finished" podID="82894901-9901-4dc4-948a-321782ef8288" containerID="dfba132bc5182982715f89fef8a8bee5073dca359534056d29abfab70c0a5c2b" exitCode=0 Nov 03 20:35:05.085014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.084999 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" Nov 03 20:35:05.086853 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.086826 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" event={"ID":"82894901-9901-4dc4-948a-321782ef8288","Type":"ContainerDied","Data":"dfba132bc5182982715f89fef8a8bee5073dca359534056d29abfab70c0a5c2b"} Nov 03 20:35:05.086943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.086875 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf" event={"ID":"82894901-9901-4dc4-948a-321782ef8288","Type":"ContainerDied","Data":"75098182ba8d8b64866d00d509377e1d942976753e6779545667b12a2d2314dd"} Nov 03 20:35:05.087125 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.087100 5059 generic.go:356] "Generic (PLEG): container finished" podID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerID="52d9dfafb55bc5b6dad5352af2151b15f85f2c035a91e44f089c01883269dcbc" exitCode=0 Nov 03 20:35:05.087198 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.087147 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" Nov 03 20:35:05.087198 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.087183 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" event={"ID":"569b2103-147f-4ea1-9b55-0e859b1c5e52","Type":"ContainerDied","Data":"52d9dfafb55bc5b6dad5352af2151b15f85f2c035a91e44f089c01883269dcbc"} Nov 03 20:35:05.087306 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.087241 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk" event={"ID":"569b2103-147f-4ea1-9b55-0e859b1c5e52","Type":"ContainerDied","Data":"0e46c7c45209747169d06bd8d48946c023e60bcdc2467de6d71bf9b81bf509fd"} Nov 03 20:35:05.090927 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.090903 5059 generic.go:356] "Generic (PLEG): container finished" podID="d4fdd390-7fe2-406d-832a-7ecc545d97f4" containerID="c9577eca08e6a70c108edb2ac96c1b9d5f54df9956d0d1467ddc72fffdf59343" exitCode=0 Nov 03 20:35:05.091102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.091084 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" event={"ID":"d4fdd390-7fe2-406d-832a-7ecc545d97f4","Type":"ContainerDied","Data":"c9577eca08e6a70c108edb2ac96c1b9d5f54df9956d0d1467ddc72fffdf59343"} Nov 03 20:35:05.091223 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.091208 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" event={"ID":"d4fdd390-7fe2-406d-832a-7ecc545d97f4","Type":"ContainerDied","Data":"4255477aa69a37bca85ef94bde9dc14fba2598777f972bf06e1248d62af4c20d"} Nov 03 20:35:05.091369 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.091356 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr" Nov 03 20:35:05.121683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.119773 5059 scope.go:122] "RemoveContainer" containerID="1119a404faca34d41ecebf28afb622f4f839d6f1e5fd27a2976df8db31439851" Nov 03 20:35:05.121683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.120311 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:35:05.124298 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:35:05.124264 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1119a404faca34d41ecebf28afb622f4f839d6f1e5fd27a2976df8db31439851\": container with ID starting with 1119a404faca34d41ecebf28afb622f4f839d6f1e5fd27a2976df8db31439851 not found: ID does not exist" containerID="1119a404faca34d41ecebf28afb622f4f839d6f1e5fd27a2976df8db31439851" Nov 03 20:35:05.124402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.124310 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1119a404faca34d41ecebf28afb622f4f839d6f1e5fd27a2976df8db31439851"} err="failed to get container status \"1119a404faca34d41ecebf28afb622f4f839d6f1e5fd27a2976df8db31439851\": rpc error: code = NotFound desc = could not find container \"1119a404faca34d41ecebf28afb622f4f839d6f1e5fd27a2976df8db31439851\": container with ID starting with 1119a404faca34d41ecebf28afb622f4f839d6f1e5fd27a2976df8db31439851 not found: ID does not exist" Nov 03 20:35:05.124402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.124333 5059 scope.go:122] "RemoveContainer" containerID="dfba132bc5182982715f89fef8a8bee5073dca359534056d29abfab70c0a5c2b" Nov 03 20:35:05.124540 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.124411 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82894901-9901-4dc4-948a-321782ef8288-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.124540 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.124440 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hzr5\" (UniqueName: \"kubernetes.io/projected/569b2103-147f-4ea1-9b55-0e859b1c5e52-kube-api-access-5hzr5\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.124540 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.124458 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcptm\" (UniqueName: \"kubernetes.io/projected/82894901-9901-4dc4-948a-321782ef8288-kube-api-access-jcptm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.124540 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.124473 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82894901-9901-4dc4-948a-321782ef8288-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.124540 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.124486 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569b2103-147f-4ea1-9b55-0e859b1c5e52-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.124540 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.124498 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82894901-9901-4dc4-948a-321782ef8288-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.126552 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.126530 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:35:05.134558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.133941 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:35:05.138895 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.138873 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.232301 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-token\" (UniqueName: \"kubernetes.io/projected/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-token\") pod \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.232382 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-workload-certs\") pod \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.232413 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-data\") pod \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.232478 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4fdd390-7fe2-406d-832a-7ecc545d97f4-kube-api-access-mrhl7\" (UniqueName: \"kubernetes.io/projected/d4fdd390-7fe2-406d-832a-7ecc545d97f4-kube-api-access-mrhl7\") pod \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.232517 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-envoy\") pod \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.232548 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-data\") pod \"ecab4d7a-4034-4b59-8500-8434baacafaf\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.232591 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-workload-socket\") pod \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.232640 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d75a00ce-f97c-431f-be9e-aac200bcd9d4-kube-api-access-f8rw6\" (UniqueName: \"kubernetes.io/projected/d75a00ce-f97c-431f-be9e-aac200bcd9d4-kube-api-access-f8rw6\") pod \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.232693 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-credential-socket\") pod \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.232740 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-token\" (UniqueName: \"kubernetes.io/projected/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-token\") pod \"6b615028-a2db-4ad0-8a57-53efc040d6a6\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.232785 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-data\") pod \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.232828 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-token\" (UniqueName: \"kubernetes.io/projected/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-token\") pod \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.232856 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-envoy\") pod \"6b615028-a2db-4ad0-8a57-53efc040d6a6\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.232899 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-envoy\") pod \"ecab4d7a-4034-4b59-8500-8434baacafaf\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.232965 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-podinfo\") pod \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.233007 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-podinfo\") pod \"ecab4d7a-4034-4b59-8500-8434baacafaf\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.233091 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-workload-certs\") pod \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.233132 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecab4d7a-4034-4b59-8500-8434baacafaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecab4d7a-4034-4b59-8500-8434baacafaf-istiod-ca-cert\") pod \"ecab4d7a-4034-4b59-8500-8434baacafaf\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.233195 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-workload-certs\") pod \"ecab4d7a-4034-4b59-8500-8434baacafaf\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.233225 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-workload-socket\") pod \"6b615028-a2db-4ad0-8a57-53efc040d6a6\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.233270 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-credential-socket\") pod \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.233318 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecab4d7a-4034-4b59-8500-8434baacafaf-kube-api-access-5fzss\" (UniqueName: \"kubernetes.io/projected/ecab4d7a-4034-4b59-8500-8434baacafaf-kube-api-access-5fzss\") pod \"ecab4d7a-4034-4b59-8500-8434baacafaf\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.233382 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-data\") pod \"6b615028-a2db-4ad0-8a57-53efc040d6a6\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.233425 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-workload-certs\") pod \"6b615028-a2db-4ad0-8a57-53efc040d6a6\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.233480 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-podinfo\") pod \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.233514 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b615028-a2db-4ad0-8a57-53efc040d6a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b615028-a2db-4ad0-8a57-53efc040d6a6-istiod-ca-cert\") pod \"6b615028-a2db-4ad0-8a57-53efc040d6a6\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.233582 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-workload-socket\") pod \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.233626 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-podinfo\") pod \"6b615028-a2db-4ad0-8a57-53efc040d6a6\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.233679 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-credential-socket\") pod \"6b615028-a2db-4ad0-8a57-53efc040d6a6\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.233718 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-workload-socket\") pod \"ecab4d7a-4034-4b59-8500-8434baacafaf\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.233771 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-envoy\") pod \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.233799 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b615028-a2db-4ad0-8a57-53efc040d6a6-kube-api-access-xvx8m\" (UniqueName: \"kubernetes.io/projected/6b615028-a2db-4ad0-8a57-53efc040d6a6-kube-api-access-xvx8m\") pod \"6b615028-a2db-4ad0-8a57-53efc040d6a6\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.233827 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-token\" (UniqueName: \"kubernetes.io/projected/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-token\") pod \"ecab4d7a-4034-4b59-8500-8434baacafaf\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.233882 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istiod-ca-cert\") pod \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.233937 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-credential-socket\") pod \"ecab4d7a-4034-4b59-8500-8434baacafaf\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.234029 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istiod-ca-cert\") pod \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.246243 5059 scope.go:122] "RemoveContainer" containerID="dfba132bc5182982715f89fef8a8bee5073dca359534056d29abfab70c0a5c2b" Nov 03 20:35:05.364444 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:35:05.248035 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfba132bc5182982715f89fef8a8bee5073dca359534056d29abfab70c0a5c2b\": container with ID starting with dfba132bc5182982715f89fef8a8bee5073dca359534056d29abfab70c0a5c2b not found: ID does not exist" containerID="dfba132bc5182982715f89fef8a8bee5073dca359534056d29abfab70c0a5c2b" Nov 03 20:35:05.395920 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.248076 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfba132bc5182982715f89fef8a8bee5073dca359534056d29abfab70c0a5c2b"} err="failed to get container status \"dfba132bc5182982715f89fef8a8bee5073dca359534056d29abfab70c0a5c2b\": rpc error: code = NotFound desc = could not find container \"dfba132bc5182982715f89fef8a8bee5073dca359534056d29abfab70c0a5c2b\": container with ID starting with dfba132bc5182982715f89fef8a8bee5073dca359534056d29abfab70c0a5c2b not found: ID does not exist" Nov 03 20:35:05.395920 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.248113 5059 scope.go:122] "RemoveContainer" containerID="52d9dfafb55bc5b6dad5352af2151b15f85f2c035a91e44f089c01883269dcbc" Nov 03 20:35:05.395920 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.362837 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-workload-certs" pod "d4fdd390-7fe2-406d-832a-7ecc545d97f4" (UID: "d4fdd390-7fe2-406d-832a-7ecc545d97f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.395920 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.364051 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-data" pod "d75a00ce-f97c-431f-be9e-aac200bcd9d4" (UID: "d75a00ce-f97c-431f-be9e-aac200bcd9d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.395920 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.393993 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-workload-socket" pod "6b615028-a2db-4ad0-8a57-53efc040d6a6" (UID: "6b615028-a2db-4ad0-8a57-53efc040d6a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.395920 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.394244 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-credential-socket" pod "d75a00ce-f97c-431f-be9e-aac200bcd9d4" (UID: "d75a00ce-f97c-431f-be9e-aac200bcd9d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.397037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.396964 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" podStartSLOduration=2.396945318 podStartE2EDuration="2.396945318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:05.268423956 +0000 UTC m=+3499.526221215" watchObservedRunningTime="2026-11-03 20:35:05.396945318 +0000 UTC m=+3499.654742576" Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.397408 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-workload-socket" pod "d4fdd390-7fe2-406d-832a-7ecc545d97f4" (UID: "d4fdd390-7fe2-406d-832a-7ecc545d97f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.397601 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b615028-a2db-4ad0-8a57-53efc040d6a6-istiod-ca-cert" pod "6b615028-a2db-4ad0-8a57-53efc040d6a6" (UID: "6b615028-a2db-4ad0-8a57-53efc040d6a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.415833 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-workload-certs" pod "6b615028-a2db-4ad0-8a57-53efc040d6a6" (UID: "6b615028-a2db-4ad0-8a57-53efc040d6a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.422891 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-data" pod "6b615028-a2db-4ad0-8a57-53efc040d6a6" (UID: "6b615028-a2db-4ad0-8a57-53efc040d6a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.423636 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-podinfo" pod "6b615028-a2db-4ad0-8a57-53efc040d6a6" (UID: "6b615028-a2db-4ad0-8a57-53efc040d6a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.449488 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-credential-socket" pod "ecab4d7a-4034-4b59-8500-8434baacafaf" (UID: "ecab4d7a-4034-4b59-8500-8434baacafaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.464921 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istiod-ca-cert" pod "d75a00ce-f97c-431f-be9e-aac200bcd9d4" (UID: "d75a00ce-f97c-431f-be9e-aac200bcd9d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.492809 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-workload-socket" pod "ecab4d7a-4034-4b59-8500-8434baacafaf" (UID: "ecab4d7a-4034-4b59-8500-8434baacafaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.501957 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-credential-socket" pod "6b615028-a2db-4ad0-8a57-53efc040d6a6" (UID: "6b615028-a2db-4ad0-8a57-53efc040d6a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.527793 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-workload-certs" pod "68bc094a-84e2-41c2-9fb5-081682413f27" (UID: "68bc094a-84e2-41c2-9fb5-081682413f27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.552804 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecab4d7a-4034-4b59-8500-8434baacafaf-kube-api-access-5fzss" pod "ecab4d7a-4034-4b59-8500-8434baacafaf" (UID: "ecab4d7a-4034-4b59-8500-8434baacafaf"). InnerVolumeSpecName "kube-api-access-5fzss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.553344 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-workload-certs" pod "ecab4d7a-4034-4b59-8500-8434baacafaf" (UID: "ecab4d7a-4034-4b59-8500-8434baacafaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.502516 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-workload-certs\") pod \"68bc094a-84e2-41c2-9fb5-081682413f27\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.553568 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-workload-certs\") pod \"ecab4d7a-4034-4b59-8500-8434baacafaf\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.553621 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecab4d7a-4034-4b59-8500-8434baacafaf-kube-api-access-5fzss\" (UniqueName: \"kubernetes.io/projected/ecab4d7a-4034-4b59-8500-8434baacafaf-kube-api-access-5fzss\") pod \"ecab4d7a-4034-4b59-8500-8434baacafaf\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.553728 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-data\") pod \"6b615028-a2db-4ad0-8a57-53efc040d6a6\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.553759 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-workload-socket\") pod \"68bc094a-84e2-41c2-9fb5-081682413f27\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.553790 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-credential-socket\") pod \"68bc094a-84e2-41c2-9fb5-081682413f27\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.553822 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istiod-ca-cert\") pod \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.553851 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-workload-certs\") pod \"6b615028-a2db-4ad0-8a57-53efc040d6a6\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.553900 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b615028-a2db-4ad0-8a57-53efc040d6a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b615028-a2db-4ad0-8a57-53efc040d6a6-istiod-ca-cert\") pod \"6b615028-a2db-4ad0-8a57-53efc040d6a6\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.553942 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68bc094a-84e2-41c2-9fb5-081682413f27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68bc094a-84e2-41c2-9fb5-081682413f27-istiod-ca-cert\") pod \"68bc094a-84e2-41c2-9fb5-081682413f27\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.553982 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-workload-socket\") pod \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\" (UID: \"d4fdd390-7fe2-406d-832a-7ecc545d97f4\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.554034 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-podinfo\") pod \"6b615028-a2db-4ad0-8a57-53efc040d6a6\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.554074 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-credential-socket\") pod \"6b615028-a2db-4ad0-8a57-53efc040d6a6\" (UID: \"6b615028-a2db-4ad0-8a57-53efc040d6a6\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.554117 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-workload-socket\") pod \"ecab4d7a-4034-4b59-8500-8434baacafaf\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.554149 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-token\" (UniqueName: \"kubernetes.io/projected/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-token\") pod \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.554240 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-istio-data\") pod \"68bc094a-84e2-41c2-9fb5-081682413f27\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.554313 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68bc094a-84e2-41c2-9fb5-081682413f27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68bc094a-84e2-41c2-9fb5-081682413f27-istio-podinfo\") pod \"68bc094a-84e2-41c2-9fb5-081682413f27\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.554352 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-credential-socket\") pod \"ecab4d7a-4034-4b59-8500-8434baacafaf\" (UID: \"ecab4d7a-4034-4b59-8500-8434baacafaf\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.554406 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ab8a5f-fa5f-4548-bf5f-0465281d8677-kube-api-access-xj2v6\" (UniqueName: \"kubernetes.io/projected/19ab8a5f-fa5f-4548-bf5f-0465281d8677-kube-api-access-xj2v6\") pod \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.554475 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istiod-ca-cert\") pod \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\" (UID: \"d75a00ce-f97c-431f-be9e-aac200bcd9d4\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.554510 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-istio-envoy\") pod \"68bc094a-84e2-41c2-9fb5-081682413f27\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.554653 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-workload-certs\") pod \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.554780 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68bc094a-84e2-41c2-9fb5-081682413f27-kube-api-access-wwfgt\" (UniqueName: \"kubernetes.io/projected/68bc094a-84e2-41c2-9fb5-081682413f27-kube-api-access-wwfgt\") pod \"68bc094a-84e2-41c2-9fb5-081682413f27\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.554822 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68bc094a-84e2-41c2-9fb5-081682413f27-istio-token\" (UniqueName: \"kubernetes.io/projected/68bc094a-84e2-41c2-9fb5-081682413f27-istio-token\") pod \"68bc094a-84e2-41c2-9fb5-081682413f27\" (UID: \"68bc094a-84e2-41c2-9fb5-081682413f27\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.554938 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-credential-socket\") pod \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " Nov 03 20:35:05.587731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.555024 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-envoy\") pod \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.555079 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-podinfo\") pod \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.555190 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-workload-socket\") pod \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.555262 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-data\") pod \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\" (UID: \"19ab8a5f-fa5f-4548-bf5f-0465281d8677\") " Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.556403 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.556431 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.556449 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.556475 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.556490 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.556729 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-data" pod "19ab8a5f-fa5f-4548-bf5f-0465281d8677" (UID: "19ab8a5f-fa5f-4548-bf5f-0465281d8677"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:35:05.556802 5059 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/ecab4d7a-4034-4b59-8500-8434baacafaf/volumes/kubernetes.io~empty-dir/workload-certs Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.556814 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-workload-certs" pod "ecab4d7a-4034-4b59-8500-8434baacafaf" (UID: "ecab4d7a-4034-4b59-8500-8434baacafaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:35:05.556873 5059 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/ecab4d7a-4034-4b59-8500-8434baacafaf/volumes/kubernetes.io~projected/kube-api-access-5fzss Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.556914 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecab4d7a-4034-4b59-8500-8434baacafaf-kube-api-access-5fzss" pod "ecab4d7a-4034-4b59-8500-8434baacafaf" (UID: "ecab4d7a-4034-4b59-8500-8434baacafaf"). InnerVolumeSpecName "kube-api-access-5fzss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:35:05.556961 5059 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/6b615028-a2db-4ad0-8a57-53efc040d6a6/volumes/kubernetes.io~empty-dir/istio-data Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.556971 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-data" pod "6b615028-a2db-4ad0-8a57-53efc040d6a6" (UID: "6b615028-a2db-4ad0-8a57-53efc040d6a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.557193 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-workload-socket" pod "68bc094a-84e2-41c2-9fb5-081682413f27" (UID: "68bc094a-84e2-41c2-9fb5-081682413f27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.557377 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-credential-socket" pod "68bc094a-84e2-41c2-9fb5-081682413f27" (UID: "68bc094a-84e2-41c2-9fb5-081682413f27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.557694 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istiod-ca-cert" pod "19ab8a5f-fa5f-4548-bf5f-0465281d8677" (UID: "19ab8a5f-fa5f-4548-bf5f-0465281d8677"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:35:05.557755 5059 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/6b615028-a2db-4ad0-8a57-53efc040d6a6/volumes/kubernetes.io~empty-dir/workload-certs Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.557767 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-workload-certs" pod "6b615028-a2db-4ad0-8a57-53efc040d6a6" (UID: "6b615028-a2db-4ad0-8a57-53efc040d6a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:35:05.557859 5059 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/6b615028-a2db-4ad0-8a57-53efc040d6a6/volumes/kubernetes.io~configmap/istiod-ca-cert Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.557870 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b615028-a2db-4ad0-8a57-53efc040d6a6-istiod-ca-cert" pod "6b615028-a2db-4ad0-8a57-53efc040d6a6" (UID: "6b615028-a2db-4ad0-8a57-53efc040d6a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.558215 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68bc094a-84e2-41c2-9fb5-081682413f27-istiod-ca-cert" pod "68bc094a-84e2-41c2-9fb5-081682413f27" (UID: "68bc094a-84e2-41c2-9fb5-081682413f27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:35:05.558282 5059 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/d4fdd390-7fe2-406d-832a-7ecc545d97f4/volumes/kubernetes.io~empty-dir/workload-socket Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.558293 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-workload-socket" pod "d4fdd390-7fe2-406d-832a-7ecc545d97f4" (UID: "d4fdd390-7fe2-406d-832a-7ecc545d97f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:35:05.558346 5059 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/6b615028-a2db-4ad0-8a57-53efc040d6a6/volumes/kubernetes.io~downward-api/istio-podinfo Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.558357 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-podinfo" pod "6b615028-a2db-4ad0-8a57-53efc040d6a6" (UID: "6b615028-a2db-4ad0-8a57-53efc040d6a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:35:05.558400 5059 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/6b615028-a2db-4ad0-8a57-53efc040d6a6/volumes/kubernetes.io~empty-dir/credential-socket Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.558409 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-credential-socket" pod "6b615028-a2db-4ad0-8a57-53efc040d6a6" (UID: "6b615028-a2db-4ad0-8a57-53efc040d6a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:35:05.558451 5059 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/ecab4d7a-4034-4b59-8500-8434baacafaf/volumes/kubernetes.io~empty-dir/workload-socket Nov 03 20:35:05.589886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.558461 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-workload-socket" pod "ecab4d7a-4034-4b59-8500-8434baacafaf" (UID: "ecab4d7a-4034-4b59-8500-8434baacafaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.633744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.600941 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-token" pod "d4fdd390-7fe2-406d-832a-7ecc545d97f4" (UID: "d4fdd390-7fe2-406d-832a-7ecc545d97f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:05.633744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.601229 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-workload-certs" pod "19ab8a5f-fa5f-4548-bf5f-0465281d8677" (UID: "19ab8a5f-fa5f-4548-bf5f-0465281d8677"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.633744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.601446 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-istio-data" pod "68bc094a-84e2-41c2-9fb5-081682413f27" (UID: "68bc094a-84e2-41c2-9fb5-081682413f27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.633744 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:35:05.601830 5059 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/d75a00ce-f97c-431f-be9e-aac200bcd9d4/volumes/kubernetes.io~configmap/istiod-ca-cert Nov 03 20:35:05.633744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.601855 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istiod-ca-cert" pod "d75a00ce-f97c-431f-be9e-aac200bcd9d4" (UID: "d75a00ce-f97c-431f-be9e-aac200bcd9d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:05.633744 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:35:05.601902 5059 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/ecab4d7a-4034-4b59-8500-8434baacafaf/volumes/kubernetes.io~empty-dir/credential-socket Nov 03 20:35:05.633744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.601910 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-credential-socket" pod "ecab4d7a-4034-4b59-8500-8434baacafaf" (UID: "ecab4d7a-4034-4b59-8500-8434baacafaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.633744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.602169 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-credential-socket" pod "19ab8a5f-fa5f-4548-bf5f-0465281d8677" (UID: "19ab8a5f-fa5f-4548-bf5f-0465281d8677"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.633744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.602733 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-envoy" pod "d75a00ce-f97c-431f-be9e-aac200bcd9d4" (UID: "d75a00ce-f97c-431f-be9e-aac200bcd9d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.633744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.631943 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-data" pod "d4fdd390-7fe2-406d-832a-7ecc545d97f4" (UID: "d4fdd390-7fe2-406d-832a-7ecc545d97f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.633744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.632346 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-data" pod "ecab4d7a-4034-4b59-8500-8434baacafaf" (UID: "ecab4d7a-4034-4b59-8500-8434baacafaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.633744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.632569 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-workload-socket" pod "d75a00ce-f97c-431f-be9e-aac200bcd9d4" (UID: "d75a00ce-f97c-431f-be9e-aac200bcd9d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.652556 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.643114 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istiod-ca-cert" pod "d4fdd390-7fe2-406d-832a-7ecc545d97f4" (UID: "d4fdd390-7fe2-406d-832a-7ecc545d97f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:05.652556 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.643971 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b615028-a2db-4ad0-8a57-53efc040d6a6-kube-api-access-xvx8m" pod "6b615028-a2db-4ad0-8a57-53efc040d6a6" (UID: "6b615028-a2db-4ad0-8a57-53efc040d6a6"). InnerVolumeSpecName "kube-api-access-xvx8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:05.652556 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.645709 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-token" pod "ecab4d7a-4034-4b59-8500-8434baacafaf" (UID: "ecab4d7a-4034-4b59-8500-8434baacafaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:05.652556 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.649624 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-workload-socket" pod "19ab8a5f-fa5f-4548-bf5f-0465281d8677" (UID: "19ab8a5f-fa5f-4548-bf5f-0465281d8677"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.652980 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.652601 5059 scope.go:122] "RemoveContainer" containerID="52d9dfafb55bc5b6dad5352af2151b15f85f2c035a91e44f089c01883269dcbc" Nov 03 20:35:05.655012 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:35:05.654935 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52d9dfafb55bc5b6dad5352af2151b15f85f2c035a91e44f089c01883269dcbc\": container with ID starting with 52d9dfafb55bc5b6dad5352af2151b15f85f2c035a91e44f089c01883269dcbc not found: ID does not exist" containerID="52d9dfafb55bc5b6dad5352af2151b15f85f2c035a91e44f089c01883269dcbc" Nov 03 20:35:05.655012 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.654985 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52d9dfafb55bc5b6dad5352af2151b15f85f2c035a91e44f089c01883269dcbc"} err="failed to get container status \"52d9dfafb55bc5b6dad5352af2151b15f85f2c035a91e44f089c01883269dcbc\": rpc error: code = NotFound desc = could not find container \"52d9dfafb55bc5b6dad5352af2151b15f85f2c035a91e44f089c01883269dcbc\": container with ID starting with 52d9dfafb55bc5b6dad5352af2151b15f85f2c035a91e44f089c01883269dcbc not found: ID does not exist" Nov 03 20:35:05.655012 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.655011 5059 scope.go:122] "RemoveContainer" containerID="c9577eca08e6a70c108edb2ac96c1b9d5f54df9956d0d1467ddc72fffdf59343" Nov 03 20:35:05.658954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.658770 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-workload-certs" pod "d75a00ce-f97c-431f-be9e-aac200bcd9d4" (UID: "d75a00ce-f97c-431f-be9e-aac200bcd9d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.660466 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.659185 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecab4d7a-4034-4b59-8500-8434baacafaf-istiod-ca-cert" pod "ecab4d7a-4034-4b59-8500-8434baacafaf" (UID: "ecab4d7a-4034-4b59-8500-8434baacafaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:05.661094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.661074 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-credential-socket" pod "d4fdd390-7fe2-406d-832a-7ecc545d97f4" (UID: "d4fdd390-7fe2-406d-832a-7ecc545d97f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672489 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fzss\" (UniqueName: \"kubernetes.io/projected/ecab4d7a-4034-4b59-8500-8434baacafaf-kube-api-access-5fzss\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672526 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672546 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672565 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672582 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672597 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672613 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b615028-a2db-4ad0-8a57-53efc040d6a6-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672628 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68bc094a-84e2-41c2-9fb5-081682413f27-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672646 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672681 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672697 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672712 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672729 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672744 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvx8m\" (UniqueName: \"kubernetes.io/projected/6b615028-a2db-4ad0-8a57-53efc040d6a6-kube-api-access-xvx8m\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672760 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672775 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672790 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672808 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672823 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672840 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672857 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672871 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672884 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672897 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672911 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672924 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.672918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672936 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.674301 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672951 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.674301 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672964 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75a00ce-f97c-431f-be9e-aac200bcd9d4-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.674301 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672976 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecab4d7a-4034-4b59-8500-8434baacafaf-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.674301 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.672989 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.676435 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.674536 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4fdd390-7fe2-406d-832a-7ecc545d97f4-kube-api-access-mrhl7" pod "d4fdd390-7fe2-406d-832a-7ecc545d97f4" (UID: "d4fdd390-7fe2-406d-832a-7ecc545d97f4"). InnerVolumeSpecName "kube-api-access-mrhl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:05.676435 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.674584 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-podinfo" pod "d75a00ce-f97c-431f-be9e-aac200bcd9d4" (UID: "d75a00ce-f97c-431f-be9e-aac200bcd9d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:05.676435 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.674638 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68bc094a-84e2-41c2-9fb5-081682413f27-kube-api-access-wwfgt" pod "68bc094a-84e2-41c2-9fb5-081682413f27" (UID: "68bc094a-84e2-41c2-9fb5-081682413f27"). InnerVolumeSpecName "kube-api-access-wwfgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:05.685728 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.685365 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-token" pod "19ab8a5f-fa5f-4548-bf5f-0465281d8677" (UID: "19ab8a5f-fa5f-4548-bf5f-0465281d8677"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:05.687843 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.687775 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-envoy" pod "19ab8a5f-fa5f-4548-bf5f-0465281d8677" (UID: "19ab8a5f-fa5f-4548-bf5f-0465281d8677"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.687843 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.687815 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-token" pod "d75a00ce-f97c-431f-be9e-aac200bcd9d4" (UID: "d75a00ce-f97c-431f-be9e-aac200bcd9d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:05.688431 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.688352 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-envoy" pod "6b615028-a2db-4ad0-8a57-53efc040d6a6" (UID: "6b615028-a2db-4ad0-8a57-53efc040d6a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.688431 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.688396 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-podinfo" pod "d4fdd390-7fe2-406d-832a-7ecc545d97f4" (UID: "d4fdd390-7fe2-406d-832a-7ecc545d97f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:05.688823 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.688462 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-podinfo" pod "19ab8a5f-fa5f-4548-bf5f-0465281d8677" (UID: "19ab8a5f-fa5f-4548-bf5f-0465281d8677"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:05.688823 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.688529 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-podinfo" pod "ecab4d7a-4034-4b59-8500-8434baacafaf" (UID: "ecab4d7a-4034-4b59-8500-8434baacafaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:05.688823 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.688549 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ab8a5f-fa5f-4548-bf5f-0465281d8677-kube-api-access-xj2v6" pod "19ab8a5f-fa5f-4548-bf5f-0465281d8677" (UID: "19ab8a5f-fa5f-4548-bf5f-0465281d8677"). InnerVolumeSpecName "kube-api-access-xj2v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:05.689161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.688878 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75a00ce-f97c-431f-be9e-aac200bcd9d4-kube-api-access-f8rw6" pod "d75a00ce-f97c-431f-be9e-aac200bcd9d4" (UID: "d75a00ce-f97c-431f-be9e-aac200bcd9d4"). InnerVolumeSpecName "kube-api-access-f8rw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:05.689161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.688960 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68bc094a-84e2-41c2-9fb5-081682413f27-istio-podinfo" pod "68bc094a-84e2-41c2-9fb5-081682413f27" (UID: "68bc094a-84e2-41c2-9fb5-081682413f27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:05.689161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.689089 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-envoy" pod "d4fdd390-7fe2-406d-832a-7ecc545d97f4" (UID: "d4fdd390-7fe2-406d-832a-7ecc545d97f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.697250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.697222 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68bc094a-84e2-41c2-9fb5-081682413f27-istio-token" pod "68bc094a-84e2-41c2-9fb5-081682413f27" (UID: "68bc094a-84e2-41c2-9fb5-081682413f27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:05.697788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.697761 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-istio-envoy" pod "68bc094a-84e2-41c2-9fb5-081682413f27" (UID: "68bc094a-84e2-41c2-9fb5-081682413f27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.698102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.698071 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-token" pod "6b615028-a2db-4ad0-8a57-53efc040d6a6" (UID: "6b615028-a2db-4ad0-8a57-53efc040d6a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:05.698475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.698414 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-envoy" pod "ecab4d7a-4034-4b59-8500-8434baacafaf" (UID: "ecab4d7a-4034-4b59-8500-8434baacafaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.700759 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.700730 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf"] Nov 03 20:35:05.709754 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.706814 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-7hfqf"] Nov 03 20:35:05.711838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.711811 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk"] Nov 03 20:35:05.713960 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.713942 5059 scope.go:122] "RemoveContainer" containerID="c9577eca08e6a70c108edb2ac96c1b9d5f54df9956d0d1467ddc72fffdf59343" Nov 03 20:35:05.714785 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:35:05.714757 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9577eca08e6a70c108edb2ac96c1b9d5f54df9956d0d1467ddc72fffdf59343\": container with ID starting with c9577eca08e6a70c108edb2ac96c1b9d5f54df9956d0d1467ddc72fffdf59343 not found: ID does not exist" containerID="c9577eca08e6a70c108edb2ac96c1b9d5f54df9956d0d1467ddc72fffdf59343" Nov 03 20:35:05.714886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.714796 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9577eca08e6a70c108edb2ac96c1b9d5f54df9956d0d1467ddc72fffdf59343"} err="failed to get container status \"c9577eca08e6a70c108edb2ac96c1b9d5f54df9956d0d1467ddc72fffdf59343\": rpc error: code = NotFound desc = could not find container \"c9577eca08e6a70c108edb2ac96c1b9d5f54df9956d0d1467ddc72fffdf59343\": container with ID starting with c9577eca08e6a70c108edb2ac96c1b9d5f54df9956d0d1467ddc72fffdf59343 not found: ID does not exist" Nov 03 20:35:05.716578 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.716553 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-2qpzk"] Nov 03 20:35:05.742272 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.742241 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr"] Nov 03 20:35:05.748162 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.748134 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-vlnhr"] Nov 03 20:35:05.773937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.773907 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrhl7\" (UniqueName: \"kubernetes.io/projected/d4fdd390-7fe2-406d-832a-7ecc545d97f4-kube-api-access-mrhl7\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.774026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.773940 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.774026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.773951 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwfgt\" (UniqueName: \"kubernetes.io/projected/68bc094a-84e2-41c2-9fb5-081682413f27-kube-api-access-wwfgt\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.774026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.773962 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68bc094a-84e2-41c2-9fb5-081682413f27-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.774026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.773970 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8rw6\" (UniqueName: \"kubernetes.io/projected/d75a00ce-f97c-431f-be9e-aac200bcd9d4-kube-api-access-f8rw6\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.774026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.773979 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.774026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.773988 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.774026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.773996 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.774026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.774004 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.774026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.774012 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b615028-a2db-4ad0-8a57-53efc040d6a6-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.774026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.774020 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.774350 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.774030 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4fdd390-7fe2-406d-832a-7ecc545d97f4-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.774350 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.774039 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecab4d7a-4034-4b59-8500-8434baacafaf-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.774350 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.774047 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75a00ce-f97c-431f-be9e-aac200bcd9d4-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.774350 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.774055 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ab8a5f-fa5f-4548-bf5f-0465281d8677-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.774350 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.774063 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68bc094a-84e2-41c2-9fb5-081682413f27-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.774350 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.774072 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj2v6\" (UniqueName: \"kubernetes.io/projected/19ab8a5f-fa5f-4548-bf5f-0465281d8677-kube-api-access-xj2v6\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.774350 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.774081 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68bc094a-84e2-41c2-9fb5-081682413f27-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.814006 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.813953 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:35:05.857879 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.857848 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" path="/var/lib/kubelet/pods/569b2103-147f-4ea1-9b55-0e859b1c5e52/volumes" Nov 03 20:35:05.858413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.858397 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82894901-9901-4dc4-948a-321782ef8288" path="/var/lib/kubelet/pods/82894901-9901-4dc4-948a-321782ef8288/volumes" Nov 03 20:35:05.858917 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.858902 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4fdd390-7fe2-406d-832a-7ecc545d97f4" path="/var/lib/kubelet/pods/d4fdd390-7fe2-406d-832a-7ecc545d97f4/volumes" Nov 03 20:35:05.999420 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:05.999349 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn"] Nov 03 20:35:06.006063 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:06.006038 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-4pvqn"] Nov 03 20:35:06.104621 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:06.104590 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" Nov 03 20:35:06.104794 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:06.104590 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x" event={"ID":"68bc094a-84e2-41c2-9fb5-081682413f27","Type":"ContainerDied","Data":"fee83e63ae644329d8b09b1c0537ad9f0cc6c7bd1363ada3503d23316698ca6d"} Nov 03 20:35:06.104794 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:06.104730 5059 scope.go:122] "RemoveContainer" containerID="a96237ecf3f4902402f80b450ff7a22b4fcd8a6500b222e4be665f0e434b7f54" Nov 03 20:35:06.106106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:06.106030 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" Nov 03 20:35:06.106106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:06.106062 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws" event={"ID":"19ab8a5f-fa5f-4548-bf5f-0465281d8677","Type":"ContainerDied","Data":"ca46816d93d3647be27f1def6097d494f92d55d75742712c07da2fe44f6fc3cd"} Nov 03 20:35:06.106411 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:06.106385 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7" Nov 03 20:35:06.106547 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:06.106516 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w" Nov 03 20:35:06.117217 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:06.117187 5059 scope.go:122] "RemoveContainer" containerID="a7892a98b6abfbe7e24a6a645e11e9dc24194d6039d51dac1b67f8117fb642b5" Nov 03 20:35:06.138161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:06.138121 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws"] Nov 03 20:35:06.142517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:06.142494 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-f68ws"] Nov 03 20:35:06.153100 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:06.153072 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x"] Nov 03 20:35:06.157684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:06.157651 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-r4j9x"] Nov 03 20:35:06.170076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:06.170053 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w"] Nov 03 20:35:06.178102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:06.178082 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-8w76w"] Nov 03 20:35:06.190037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:06.190015 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7"] Nov 03 20:35:06.194610 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:06.194591 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-hw6z7"] Nov 03 20:35:07.264782 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.264745 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856"] Nov 03 20:35:07.265706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.265682 5059 cpu_manager.go:414] "containerMap: removing container" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" Nov 03 20:35:07.265706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.265705 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" Nov 03 20:35:07.265881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.265727 5059 cpu_manager.go:414] "containerMap: removing container" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" Nov 03 20:35:07.265881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.265735 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" Nov 03 20:35:07.265881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.265749 5059 cpu_manager.go:414] "containerMap: removing container" podUID="68bc094a-84e2-41c2-9fb5-081682413f27" containerName="istio-proxy" Nov 03 20:35:07.265881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.265757 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68bc094a-84e2-41c2-9fb5-081682413f27" containerName="istio-proxy" Nov 03 20:35:07.265881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.265770 5059 cpu_manager.go:414] "containerMap: removing container" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" Nov 03 20:35:07.265881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.265779 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" Nov 03 20:35:07.265881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.265793 5059 cpu_manager.go:414] "containerMap: removing container" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerName="istio-proxy" Nov 03 20:35:07.265881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.265800 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerName="istio-proxy" Nov 03 20:35:07.265881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.265811 5059 cpu_manager.go:414] "containerMap: removing container" podUID="d4fdd390-7fe2-406d-832a-7ecc545d97f4" containerName="istio-proxy" Nov 03 20:35:07.265881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.265818 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4fdd390-7fe2-406d-832a-7ecc545d97f4" containerName="istio-proxy" Nov 03 20:35:07.265881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.265843 5059 cpu_manager.go:414] "containerMap: removing container" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" Nov 03 20:35:07.265881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.265850 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" Nov 03 20:35:07.265881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.265863 5059 cpu_manager.go:414] "containerMap: removing container" podUID="82894901-9901-4dc4-948a-321782ef8288" containerName="istio-proxy" Nov 03 20:35:07.265881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.265870 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82894901-9901-4dc4-948a-321782ef8288" containerName="istio-proxy" Nov 03 20:35:07.266531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.266024 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" containerName="istio-proxy" Nov 03 20:35:07.266531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.266035 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="68bc094a-84e2-41c2-9fb5-081682413f27" containerName="istio-proxy" Nov 03 20:35:07.266531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.266048 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="569b2103-147f-4ea1-9b55-0e859b1c5e52" containerName="istio-proxy" Nov 03 20:35:07.266531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.266059 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" containerName="istio-proxy" Nov 03 20:35:07.266531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.266067 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4fdd390-7fe2-406d-832a-7ecc545d97f4" containerName="istio-proxy" Nov 03 20:35:07.266531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.266080 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" containerName="istio-proxy" Nov 03 20:35:07.266531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.266089 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="82894901-9901-4dc4-948a-321782ef8288" containerName="istio-proxy" Nov 03 20:35:07.266531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.266103 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" containerName="istio-proxy" Nov 03 20:35:07.275868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.275843 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:35:07.278256 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.278227 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856"] Nov 03 20:35:07.281793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.281767 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bcfde8\"/\"default-dockercfg-jwcxg\"" Nov 03 20:35:07.281909 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.281861 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bcfde8\"/\"sub-test-model-bcfde8-kserve-self-signed-certs\"" Nov 03 20:35:07.288491 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.288471 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bcfde8\"/\"kube-root-ca.crt\"" Nov 03 20:35:07.288581 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.288549 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bcfde8\"/\"openshift-service-ca.crt\"" Nov 03 20:35:07.400555 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.400516 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-model-cache\") pod \"sub-test-model-bcfde8-kserve-854897f565-47856\" (UID: \"9f2c7048-0565-4bd2-b05b-735757071fb6\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:35:07.400711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.400617 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-tmp-dir\") pod \"sub-test-model-bcfde8-kserve-854897f565-47856\" (UID: \"9f2c7048-0565-4bd2-b05b-735757071fb6\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:35:07.400753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.400719 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-dshm\") pod \"sub-test-model-bcfde8-kserve-854897f565-47856\" (UID: \"9f2c7048-0565-4bd2-b05b-735757071fb6\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:35:07.400827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.400808 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2c7048-0565-4bd2-b05b-735757071fb6-tls-certs\") pod \"sub-test-model-bcfde8-kserve-854897f565-47856\" (UID: \"9f2c7048-0565-4bd2-b05b-735757071fb6\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:35:07.400904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.400893 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-home\") pod \"sub-test-model-bcfde8-kserve-854897f565-47856\" (UID: \"9f2c7048-0565-4bd2-b05b-735757071fb6\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:35:07.400964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.400951 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqxcf\" (UniqueName: \"kubernetes.io/projected/9f2c7048-0565-4bd2-b05b-735757071fb6-kube-api-access-qqxcf\") pod \"sub-test-model-bcfde8-kserve-854897f565-47856\" (UID: \"9f2c7048-0565-4bd2-b05b-735757071fb6\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:35:07.509908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.509816 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-model-cache\") pod \"sub-test-model-bcfde8-kserve-854897f565-47856\" (UID: \"9f2c7048-0565-4bd2-b05b-735757071fb6\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:35:07.509908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.509908 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-tmp-dir\") pod \"sub-test-model-bcfde8-kserve-854897f565-47856\" (UID: \"9f2c7048-0565-4bd2-b05b-735757071fb6\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:35:07.510161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.509989 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-dshm\") pod \"sub-test-model-bcfde8-kserve-854897f565-47856\" (UID: \"9f2c7048-0565-4bd2-b05b-735757071fb6\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:35:07.510161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.510021 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2c7048-0565-4bd2-b05b-735757071fb6-tls-certs\") pod \"sub-test-model-bcfde8-kserve-854897f565-47856\" (UID: \"9f2c7048-0565-4bd2-b05b-735757071fb6\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:35:07.510161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.510129 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-home\") pod \"sub-test-model-bcfde8-kserve-854897f565-47856\" (UID: \"9f2c7048-0565-4bd2-b05b-735757071fb6\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:35:07.510309 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.510172 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqxcf\" (UniqueName: \"kubernetes.io/projected/9f2c7048-0565-4bd2-b05b-735757071fb6-kube-api-access-qqxcf\") pod \"sub-test-model-bcfde8-kserve-854897f565-47856\" (UID: \"9f2c7048-0565-4bd2-b05b-735757071fb6\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:35:07.513681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.511146 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-model-cache\") pod \"sub-test-model-bcfde8-kserve-854897f565-47856\" (UID: \"9f2c7048-0565-4bd2-b05b-735757071fb6\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:35:07.513681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.511398 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-tmp-dir\") pod \"sub-test-model-bcfde8-kserve-854897f565-47856\" (UID: \"9f2c7048-0565-4bd2-b05b-735757071fb6\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:35:07.531781 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.515010 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-home\") pod \"sub-test-model-bcfde8-kserve-854897f565-47856\" (UID: \"9f2c7048-0565-4bd2-b05b-735757071fb6\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:35:07.532104 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.532069 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-dshm\") pod \"sub-test-model-bcfde8-kserve-854897f565-47856\" (UID: \"9f2c7048-0565-4bd2-b05b-735757071fb6\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:35:07.535688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.532649 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2c7048-0565-4bd2-b05b-735757071fb6-tls-certs\") pod \"sub-test-model-bcfde8-kserve-854897f565-47856\" (UID: \"9f2c7048-0565-4bd2-b05b-735757071fb6\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:35:07.538037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.537219 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqxcf\" (UniqueName: \"kubernetes.io/projected/9f2c7048-0565-4bd2-b05b-735757071fb6-kube-api-access-qqxcf\") pod \"sub-test-model-bcfde8-kserve-854897f565-47856\" (UID: \"9f2c7048-0565-4bd2-b05b-735757071fb6\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:35:07.567696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.560408 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg"] Nov 03 20:35:07.567696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.560624 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg" podUID="aeb32457-81a5-4505-b189-2a98f5fda7da" containerName="maas-api" containerID="cri-o://fab37fbb13493385737d28f94387461758e3c2831d009aad282214e9aa7cd031" gracePeriod=30 Nov 03 20:35:07.590196 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.589934 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:35:07.893980 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.892674 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19ab8a5f-fa5f-4548-bf5f-0465281d8677" path="/var/lib/kubelet/pods/19ab8a5f-fa5f-4548-bf5f-0465281d8677/volumes" Nov 03 20:35:07.893980 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.893520 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68bc094a-84e2-41c2-9fb5-081682413f27" path="/var/lib/kubelet/pods/68bc094a-84e2-41c2-9fb5-081682413f27/volumes" Nov 03 20:35:07.894372 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.894308 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b615028-a2db-4ad0-8a57-53efc040d6a6" path="/var/lib/kubelet/pods/6b615028-a2db-4ad0-8a57-53efc040d6a6/volumes" Nov 03 20:35:07.895575 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.895542 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d75a00ce-f97c-431f-be9e-aac200bcd9d4" path="/var/lib/kubelet/pods/d75a00ce-f97c-431f-be9e-aac200bcd9d4/volumes" Nov 03 20:35:07.896390 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.896367 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecab4d7a-4034-4b59-8500-8434baacafaf" path="/var/lib/kubelet/pods/ecab4d7a-4034-4b59-8500-8434baacafaf/volumes" Nov 03 20:35:07.904929 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.904801 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856"] Nov 03 20:35:07.908750 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:35:07.908692 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f2c7048_0565_4bd2_b05b_735757071fb6.slice/crio-cb578c4d9633b9fb63d1621a269b7b333edc333d483e44c7eb8037dcd0e3ccb8 WatchSource:0}: Error finding container cb578c4d9633b9fb63d1621a269b7b333edc333d483e44c7eb8037dcd0e3ccb8: Status 404 returned error can't find the container with id cb578c4d9633b9fb63d1621a269b7b333edc333d483e44c7eb8037dcd0e3ccb8 Nov 03 20:35:07.988273 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:07.988164 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg" Nov 03 20:35:08.125456 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.125404 5059 generic.go:356] "Generic (PLEG): container finished" podID="aeb32457-81a5-4505-b189-2a98f5fda7da" containerID="fab37fbb13493385737d28f94387461758e3c2831d009aad282214e9aa7cd031" exitCode=0 Nov 03 20:35:08.125626 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.125466 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg" Nov 03 20:35:08.125626 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.125515 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg" event={"ID":"aeb32457-81a5-4505-b189-2a98f5fda7da","Type":"ContainerDied","Data":"fab37fbb13493385737d28f94387461758e3c2831d009aad282214e9aa7cd031"} Nov 03 20:35:08.125626 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.125566 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg" event={"ID":"aeb32457-81a5-4505-b189-2a98f5fda7da","Type":"ContainerDied","Data":"f9e9ea28bc7e11bb18a879b11400bbe3b69a5bbc6f66b5ffe248c798052afb31"} Nov 03 20:35:08.125626 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.125597 5059 scope.go:122] "RemoveContainer" containerID="fab37fbb13493385737d28f94387461758e3c2831d009aad282214e9aa7cd031" Nov 03 20:35:08.135211 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.135168 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" event={"ID":"9f2c7048-0565-4bd2-b05b-735757071fb6","Type":"ContainerStarted","Data":"aa189ee45c1a7aa60d8380219508c000c3dd82c301f41e2f5b8b4a73880c7105"} Nov 03 20:35:08.135338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.135244 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" event={"ID":"9f2c7048-0565-4bd2-b05b-735757071fb6","Type":"ContainerStarted","Data":"cb578c4d9633b9fb63d1621a269b7b333edc333d483e44c7eb8037dcd0e3ccb8"} Nov 03 20:35:08.143563 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.143543 5059 scope.go:122] "RemoveContainer" containerID="fab37fbb13493385737d28f94387461758e3c2831d009aad282214e9aa7cd031" Nov 03 20:35:08.143950 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:35:08.143925 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab37fbb13493385737d28f94387461758e3c2831d009aad282214e9aa7cd031\": container with ID starting with fab37fbb13493385737d28f94387461758e3c2831d009aad282214e9aa7cd031 not found: ID does not exist" containerID="fab37fbb13493385737d28f94387461758e3c2831d009aad282214e9aa7cd031" Nov 03 20:35:08.144071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.143960 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab37fbb13493385737d28f94387461758e3c2831d009aad282214e9aa7cd031"} err="failed to get container status \"fab37fbb13493385737d28f94387461758e3c2831d009aad282214e9aa7cd031\": rpc error: code = NotFound desc = could not find container \"fab37fbb13493385737d28f94387461758e3c2831d009aad282214e9aa7cd031\": container with ID starting with fab37fbb13493385737d28f94387461758e3c2831d009aad282214e9aa7cd031 not found: ID does not exist" Nov 03 20:35:08.158719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.158678 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" podStartSLOduration=1.158652909 podStartE2EDuration="1.158652909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:08.15441869 +0000 UTC m=+3502.412215971" watchObservedRunningTime="2026-11-03 20:35:08.158652909 +0000 UTC m=+3502.416450166" Nov 03 20:35:08.161197 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.161167 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aeb32457-81a5-4505-b189-2a98f5fda7da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeb32457-81a5-4505-b189-2a98f5fda7da-metrics-certs\") pod \"aeb32457-81a5-4505-b189-2a98f5fda7da\" (UID: \"aeb32457-81a5-4505-b189-2a98f5fda7da\") " Nov 03 20:35:08.161294 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.161218 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeb32457-81a5-4505-b189-2a98f5fda7da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb32457-81a5-4505-b189-2a98f5fda7da-openshift-service-ca\") pod \"aeb32457-81a5-4505-b189-2a98f5fda7da\" (UID: \"aeb32457-81a5-4505-b189-2a98f5fda7da\") " Nov 03 20:35:08.161340 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.161321 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aeb32457-81a5-4505-b189-2a98f5fda7da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeb32457-81a5-4505-b189-2a98f5fda7da-maas-api-tls\") pod \"aeb32457-81a5-4505-b189-2a98f5fda7da\" (UID: \"aeb32457-81a5-4505-b189-2a98f5fda7da\") " Nov 03 20:35:08.161527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.161511 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeb32457-81a5-4505-b189-2a98f5fda7da-kube-api-access-kxj5l\" (UniqueName: \"kubernetes.io/projected/aeb32457-81a5-4505-b189-2a98f5fda7da-kube-api-access-kxj5l\") pod \"aeb32457-81a5-4505-b189-2a98f5fda7da\" (UID: \"aeb32457-81a5-4505-b189-2a98f5fda7da\") " Nov 03 20:35:08.161618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.161595 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb32457-81a5-4505-b189-2a98f5fda7da-openshift-service-ca" pod "aeb32457-81a5-4505-b189-2a98f5fda7da" (UID: "aeb32457-81a5-4505-b189-2a98f5fda7da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:08.162186 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.162164 5059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb32457-81a5-4505-b189-2a98f5fda7da-openshift-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:08.163477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.163423 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeb32457-81a5-4505-b189-2a98f5fda7da-maas-api-tls" pod "aeb32457-81a5-4505-b189-2a98f5fda7da" (UID: "aeb32457-81a5-4505-b189-2a98f5fda7da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:08.163916 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.163896 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeb32457-81a5-4505-b189-2a98f5fda7da-metrics-certs" pod "aeb32457-81a5-4505-b189-2a98f5fda7da" (UID: "aeb32457-81a5-4505-b189-2a98f5fda7da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:08.164012 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.163988 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb32457-81a5-4505-b189-2a98f5fda7da-kube-api-access-kxj5l" pod "aeb32457-81a5-4505-b189-2a98f5fda7da" (UID: "aeb32457-81a5-4505-b189-2a98f5fda7da"). InnerVolumeSpecName "kube-api-access-kxj5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:08.263973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.263940 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeb32457-81a5-4505-b189-2a98f5fda7da-maas-api-tls\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:08.263973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.263970 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxj5l\" (UniqueName: \"kubernetes.io/projected/aeb32457-81a5-4505-b189-2a98f5fda7da-kube-api-access-kxj5l\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:08.263973 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.263980 5059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeb32457-81a5-4505-b189-2a98f5fda7da-metrics-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:08.460624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.460583 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg"] Nov 03 20:35:08.464562 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.464533 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-t4dxg"] Nov 03 20:35:08.595004 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:08.594972 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:35:09.298029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:09.297996 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28cac4fe-lrfxw"] Nov 03 20:35:09.298620 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:09.298605 5059 cpu_manager.go:414] "containerMap: removing container" podUID="aeb32457-81a5-4505-b189-2a98f5fda7da" containerName="maas-api" Nov 03 20:35:09.298699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:09.298622 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeb32457-81a5-4505-b189-2a98f5fda7da" containerName="maas-api" Nov 03 20:35:09.298769 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:09.298756 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeb32457-81a5-4505-b189-2a98f5fda7da" containerName="maas-api" Nov 03 20:35:09.303763 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:09.303745 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cac4fe-lrfxw" Nov 03 20:35:09.309299 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:09.308536 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:35:09.309419 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:09.308610 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:35:09.311045 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:09.311015 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28cac4fe-lrfxw"] Nov 03 20:35:09.481695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:09.481643 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqgsq\" (UniqueName: \"kubernetes.io/projected/acf514f7-7a99-44ca-b62c-058eb4a39d26-kube-api-access-bqgsq\") pod \"maas-api-revoke-sub-28cac4fe-lrfxw\" (UID: \"acf514f7-7a99-44ca-b62c-058eb4a39d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cac4fe-lrfxw" Nov 03 20:35:09.482041 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:09.481744 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf514f7-7a99-44ca-b62c-058eb4a39d26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28cac4fe-lrfxw\" (UID: \"acf514f7-7a99-44ca-b62c-058eb4a39d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cac4fe-lrfxw" Nov 03 20:35:09.482041 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:09.481950 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf514f7-7a99-44ca-b62c-058eb4a39d26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28cac4fe-lrfxw\" (UID: \"acf514f7-7a99-44ca-b62c-058eb4a39d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cac4fe-lrfxw" Nov 03 20:35:09.582885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:09.582862 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf514f7-7a99-44ca-b62c-058eb4a39d26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28cac4fe-lrfxw\" (UID: \"acf514f7-7a99-44ca-b62c-058eb4a39d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cac4fe-lrfxw" Nov 03 20:35:09.583010 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:09.582913 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqgsq\" (UniqueName: \"kubernetes.io/projected/acf514f7-7a99-44ca-b62c-058eb4a39d26-kube-api-access-bqgsq\") pod \"maas-api-revoke-sub-28cac4fe-lrfxw\" (UID: \"acf514f7-7a99-44ca-b62c-058eb4a39d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cac4fe-lrfxw" Nov 03 20:35:09.583010 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:09.582953 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf514f7-7a99-44ca-b62c-058eb4a39d26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28cac4fe-lrfxw\" (UID: \"acf514f7-7a99-44ca-b62c-058eb4a39d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cac4fe-lrfxw" Nov 03 20:35:09.583468 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:09.583446 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf514f7-7a99-44ca-b62c-058eb4a39d26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28cac4fe-lrfxw\" (UID: \"acf514f7-7a99-44ca-b62c-058eb4a39d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cac4fe-lrfxw" Nov 03 20:35:09.585422 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:09.585400 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf514f7-7a99-44ca-b62c-058eb4a39d26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28cac4fe-lrfxw\" (UID: \"acf514f7-7a99-44ca-b62c-058eb4a39d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cac4fe-lrfxw" Nov 03 20:35:09.595352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:09.595322 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqgsq\" (UniqueName: \"kubernetes.io/projected/acf514f7-7a99-44ca-b62c-058eb4a39d26-kube-api-access-bqgsq\") pod \"maas-api-revoke-sub-28cac4fe-lrfxw\" (UID: \"acf514f7-7a99-44ca-b62c-058eb4a39d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cac4fe-lrfxw" Nov 03 20:35:09.620577 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:09.620558 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cac4fe-lrfxw" Nov 03 20:35:09.857846 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:09.857769 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeb32457-81a5-4505-b189-2a98f5fda7da" path="/var/lib/kubelet/pods/aeb32457-81a5-4505-b189-2a98f5fda7da/volumes" Nov 03 20:35:09.919923 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:35:09.919879 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf514f7_7a99_44ca_b62c_058eb4a39d26.slice/crio-1026af8db30fa04c5e2a6b21265542e61ca531cd7f8157c838a2a4c12b179cd1 WatchSource:0}: Error finding container 1026af8db30fa04c5e2a6b21265542e61ca531cd7f8157c838a2a4c12b179cd1: Status 404 returned error can't find the container with id 1026af8db30fa04c5e2a6b21265542e61ca531cd7f8157c838a2a4c12b179cd1 Nov 03 20:35:09.920800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:09.920763 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28cac4fe-lrfxw"] Nov 03 20:35:10.031704 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:10.031633 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp" podUID="a9b40731-f01a-4209-af40-de342324d721" containerName="payload-processing" containerID="cri-o://19bd246d98f0088e731c3a855629700c012e09eaf4097b51292b87ae09515fee" gracePeriod=25 Nov 03 20:35:10.151746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:10.151718 5059 generic.go:356] "Generic (PLEG): container finished" podID="a9b40731-f01a-4209-af40-de342324d721" containerID="19bd246d98f0088e731c3a855629700c012e09eaf4097b51292b87ae09515fee" exitCode=0 Nov 03 20:35:10.151823 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:10.151794 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp" event={"ID":"a9b40731-f01a-4209-af40-de342324d721","Type":"ContainerDied","Data":"19bd246d98f0088e731c3a855629700c012e09eaf4097b51292b87ae09515fee"} Nov 03 20:35:10.153132 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:10.153106 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cac4fe-lrfxw" event={"ID":"acf514f7-7a99-44ca-b62c-058eb4a39d26","Type":"ContainerStarted","Data":"1026af8db30fa04c5e2a6b21265542e61ca531cd7f8157c838a2a4c12b179cd1"} Nov 03 20:35:10.277729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:10.277704 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp" Nov 03 20:35:10.301596 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:10.301569 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b40731-f01a-4209-af40-de342324d721-kube-api-access-g5twj\" (UniqueName: \"kubernetes.io/projected/a9b40731-f01a-4209-af40-de342324d721-kube-api-access-g5twj\") pod \"a9b40731-f01a-4209-af40-de342324d721\" (UID: \"a9b40731-f01a-4209-af40-de342324d721\") " Nov 03 20:35:10.301767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:10.301715 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9b40731-f01a-4209-af40-de342324d721-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b40731-f01a-4209-af40-de342324d721-plugins-config-volume\") pod \"a9b40731-f01a-4209-af40-de342324d721\" (UID: \"a9b40731-f01a-4209-af40-de342324d721\") " Nov 03 20:35:10.302110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:10.302082 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b40731-f01a-4209-af40-de342324d721-plugins-config-volume" pod "a9b40731-f01a-4209-af40-de342324d721" (UID: "a9b40731-f01a-4209-af40-de342324d721"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:10.302431 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:10.302395 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b40731-f01a-4209-af40-de342324d721-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:10.303845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:10.303821 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b40731-f01a-4209-af40-de342324d721-kube-api-access-g5twj" pod "a9b40731-f01a-4209-af40-de342324d721" (UID: "a9b40731-f01a-4209-af40-de342324d721"). InnerVolumeSpecName "kube-api-access-g5twj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:10.403951 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:10.403862 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5twj\" (UniqueName: \"kubernetes.io/projected/a9b40731-f01a-4209-af40-de342324d721-kube-api-access-g5twj\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:10.606337 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:35:10.606299 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:35:11.161926 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:11.161890 5059 generic.go:356] "Generic (PLEG): container finished" podID="acf514f7-7a99-44ca-b62c-058eb4a39d26" containerID="6f43ee19f2585664e349b56b9bab1c96bb6a783ac14fe8a9f0e4970b2f827788" exitCode=0 Nov 03 20:35:11.162085 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:11.161971 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cac4fe-lrfxw" event={"ID":"acf514f7-7a99-44ca-b62c-058eb4a39d26","Type":"ContainerDied","Data":"6f43ee19f2585664e349b56b9bab1c96bb6a783ac14fe8a9f0e4970b2f827788"} Nov 03 20:35:11.163442 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:11.163426 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp" Nov 03 20:35:11.163546 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:11.163456 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp" event={"ID":"a9b40731-f01a-4209-af40-de342324d721","Type":"ContainerDied","Data":"35fb4ca7ab225127f28dc977cdce50d38860e87ed78e7cbbb6e2dc8c57830d53"} Nov 03 20:35:11.163546 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:11.163504 5059 scope.go:122] "RemoveContainer" containerID="19bd246d98f0088e731c3a855629700c012e09eaf4097b51292b87ae09515fee" Nov 03 20:35:11.196962 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:11.196939 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp"] Nov 03 20:35:11.203249 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:11.203228 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-flcjp"] Nov 03 20:35:11.885356 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:11.885315 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9b40731-f01a-4209-af40-de342324d721" path="/var/lib/kubelet/pods/a9b40731-f01a-4209-af40-de342324d721/volumes" Nov 03 20:35:12.341862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:12.341839 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cac4fe-lrfxw" Nov 03 20:35:12.433391 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:12.433301 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acf514f7-7a99-44ca-b62c-058eb4a39d26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf514f7-7a99-44ca-b62c-058eb4a39d26-maas-api-xks-ca\") pod \"acf514f7-7a99-44ca-b62c-058eb4a39d26\" (UID: \"acf514f7-7a99-44ca-b62c-058eb4a39d26\") " Nov 03 20:35:12.433529 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:12.433411 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acf514f7-7a99-44ca-b62c-058eb4a39d26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf514f7-7a99-44ca-b62c-058eb4a39d26-maas-api-openshift-ca\") pod \"acf514f7-7a99-44ca-b62c-058eb4a39d26\" (UID: \"acf514f7-7a99-44ca-b62c-058eb4a39d26\") " Nov 03 20:35:12.433529 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:12.433494 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acf514f7-7a99-44ca-b62c-058eb4a39d26-kube-api-access-bqgsq\" (UniqueName: \"kubernetes.io/projected/acf514f7-7a99-44ca-b62c-058eb4a39d26-kube-api-access-bqgsq\") pod \"acf514f7-7a99-44ca-b62c-058eb4a39d26\" (UID: \"acf514f7-7a99-44ca-b62c-058eb4a39d26\") " Nov 03 20:35:12.433807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:12.433785 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acf514f7-7a99-44ca-b62c-058eb4a39d26-maas-api-openshift-ca" pod "acf514f7-7a99-44ca-b62c-058eb4a39d26" (UID: "acf514f7-7a99-44ca-b62c-058eb4a39d26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:12.434183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:12.434164 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf514f7-7a99-44ca-b62c-058eb4a39d26-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:12.435704 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:12.435648 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acf514f7-7a99-44ca-b62c-058eb4a39d26-maas-api-xks-ca" pod "acf514f7-7a99-44ca-b62c-058eb4a39d26" (UID: "acf514f7-7a99-44ca-b62c-058eb4a39d26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:12.435704 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:12.435649 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acf514f7-7a99-44ca-b62c-058eb4a39d26-kube-api-access-bqgsq" pod "acf514f7-7a99-44ca-b62c-058eb4a39d26" (UID: "acf514f7-7a99-44ca-b62c-058eb4a39d26"). InnerVolumeSpecName "kube-api-access-bqgsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:12.535495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:12.535457 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqgsq\" (UniqueName: \"kubernetes.io/projected/acf514f7-7a99-44ca-b62c-058eb4a39d26-kube-api-access-bqgsq\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:12.535495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:12.535489 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf514f7-7a99-44ca-b62c-058eb4a39d26-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:13.183175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:13.183143 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cac4fe-lrfxw" Nov 03 20:35:13.183542 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:13.183180 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cac4fe-lrfxw" event={"ID":"acf514f7-7a99-44ca-b62c-058eb4a39d26","Type":"ContainerDied","Data":"1026af8db30fa04c5e2a6b21265542e61ca531cd7f8157c838a2a4c12b179cd1"} Nov 03 20:35:13.183542 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:13.183224 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1026af8db30fa04c5e2a6b21265542e61ca531cd7f8157c838a2a4c12b179cd1" Nov 03 20:35:13.814309 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:13.814276 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:35:13.827432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:13.827405 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:35:14.201481 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:14.201446 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:35:17.594871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:17.594834 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:35:17.606778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:17.606752 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:35:18.231618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:18.231588 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:35:19.616543 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:19.616512 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8095d1bf-dtz7r"] Nov 03 20:35:19.617126 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:19.617111 5059 cpu_manager.go:414] "containerMap: removing container" podUID="a9b40731-f01a-4209-af40-de342324d721" containerName="payload-processing" Nov 03 20:35:19.617191 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:19.617127 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b40731-f01a-4209-af40-de342324d721" containerName="payload-processing" Nov 03 20:35:19.617191 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:19.617148 5059 cpu_manager.go:414] "containerMap: removing container" podUID="acf514f7-7a99-44ca-b62c-058eb4a39d26" containerName="revoke-keys" Nov 03 20:35:19.617191 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:19.617153 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acf514f7-7a99-44ca-b62c-058eb4a39d26" containerName="revoke-keys" Nov 03 20:35:19.617277 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:19.617237 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="acf514f7-7a99-44ca-b62c-058eb4a39d26" containerName="revoke-keys" Nov 03 20:35:19.617277 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:19.617253 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b40731-f01a-4209-af40-de342324d721" containerName="payload-processing" Nov 03 20:35:19.620756 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:19.620738 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8095d1bf-dtz7r" Nov 03 20:35:19.628182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:19.628157 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:35:19.629167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:19.629147 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:35:19.630334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:19.630308 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8095d1bf-dtz7r"] Nov 03 20:35:19.759939 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:19.759895 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a004055f-9ef2-4be0-b531-6a65b0caa7c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8095d1bf-dtz7r\" (UID: \"a004055f-9ef2-4be0-b531-6a65b0caa7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8095d1bf-dtz7r" Nov 03 20:35:19.760152 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:19.760133 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a004055f-9ef2-4be0-b531-6a65b0caa7c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8095d1bf-dtz7r\" (UID: \"a004055f-9ef2-4be0-b531-6a65b0caa7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8095d1bf-dtz7r" Nov 03 20:35:19.760334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:19.760310 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v58d\" (UniqueName: \"kubernetes.io/projected/a004055f-9ef2-4be0-b531-6a65b0caa7c2-kube-api-access-9v58d\") pod \"maas-api-revoke-sub-8095d1bf-dtz7r\" (UID: \"a004055f-9ef2-4be0-b531-6a65b0caa7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8095d1bf-dtz7r" Nov 03 20:35:19.773335 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:19.773303 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7"] Nov 03 20:35:19.773583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:19.773546 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" podUID="a2c58648-4022-4bfb-be9f-170a6222dfd8" containerName="istio-proxy" containerID="cri-o://fb118c660710eb7d73eb1decfc3f34428b1d8bf365ce4f36a506649974ba8750" gracePeriod=30 Nov 03 20:35:19.861275 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:19.861246 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a004055f-9ef2-4be0-b531-6a65b0caa7c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8095d1bf-dtz7r\" (UID: \"a004055f-9ef2-4be0-b531-6a65b0caa7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8095d1bf-dtz7r" Nov 03 20:35:19.861409 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:19.861395 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a004055f-9ef2-4be0-b531-6a65b0caa7c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8095d1bf-dtz7r\" (UID: \"a004055f-9ef2-4be0-b531-6a65b0caa7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8095d1bf-dtz7r" Nov 03 20:35:19.861503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:19.861491 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v58d\" (UniqueName: \"kubernetes.io/projected/a004055f-9ef2-4be0-b531-6a65b0caa7c2-kube-api-access-9v58d\") pod \"maas-api-revoke-sub-8095d1bf-dtz7r\" (UID: \"a004055f-9ef2-4be0-b531-6a65b0caa7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8095d1bf-dtz7r" Nov 03 20:35:19.862034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:19.862008 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a004055f-9ef2-4be0-b531-6a65b0caa7c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8095d1bf-dtz7r\" (UID: \"a004055f-9ef2-4be0-b531-6a65b0caa7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8095d1bf-dtz7r" Nov 03 20:35:19.863793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:19.863766 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a004055f-9ef2-4be0-b531-6a65b0caa7c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8095d1bf-dtz7r\" (UID: \"a004055f-9ef2-4be0-b531-6a65b0caa7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8095d1bf-dtz7r" Nov 03 20:35:19.874199 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:19.874136 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v58d\" (UniqueName: \"kubernetes.io/projected/a004055f-9ef2-4be0-b531-6a65b0caa7c2-kube-api-access-9v58d\") pod \"maas-api-revoke-sub-8095d1bf-dtz7r\" (UID: \"a004055f-9ef2-4be0-b531-6a65b0caa7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8095d1bf-dtz7r" Nov 03 20:35:19.940012 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:19.939978 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8095d1bf-dtz7r" Nov 03 20:35:20.350892 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:35:20.350849 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda004055f_9ef2_4be0_b531_6a65b0caa7c2.slice/crio-bfccb5d2fccc321c9cb9fbf74d72139af0f95f61b6de0e49b1de6f046cd14ca6 WatchSource:0}: Error finding container bfccb5d2fccc321c9cb9fbf74d72139af0f95f61b6de0e49b1de6f046cd14ca6: Status 404 returned error can't find the container with id bfccb5d2fccc321c9cb9fbf74d72139af0f95f61b6de0e49b1de6f046cd14ca6 Nov 03 20:35:20.351274 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.351244 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8095d1bf-dtz7r"] Nov 03 20:35:20.413046 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.413011 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn"] Nov 03 20:35:20.418487 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.418469 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.422494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.422474 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-836265-openshift-default-dockercfg-t9cxz\"" Nov 03 20:35:20.432009 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.431986 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn"] Nov 03 20:35:20.585774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.585736 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-podinfo\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.585903 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.585781 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-workload-socket\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.585903 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.585895 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-data\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.586013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.585941 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-token\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.586094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.586071 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-envoy\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.586129 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.586115 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-workload-certs\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.586231 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.586203 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istiod-ca-cert\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.586276 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.586264 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9wgn\" (UniqueName: \"kubernetes.io/projected/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-kube-api-access-c9wgn\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.586329 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.586317 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-credential-socket\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.687903 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.687878 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-workload-socket\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.688222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.687932 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-data\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.688222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.687956 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-token\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.688222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.687999 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-envoy\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.688222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.688028 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-workload-certs\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.688222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.688097 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istiod-ca-cert\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.688222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.688152 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9wgn\" (UniqueName: \"kubernetes.io/projected/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-kube-api-access-c9wgn\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.688222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.688179 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-credential-socket\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.688530 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.688284 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-podinfo\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.688530 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.688292 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-workload-socket\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.688530 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.688325 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-data\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.688530 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.688433 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-workload-certs\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.688684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.688592 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-credential-socket\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.699633 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.688901 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istiod-ca-cert\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.699633 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.690937 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-envoy\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.699633 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.691154 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-podinfo\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.703587 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.703562 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-token\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.703758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.703727 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9wgn\" (UniqueName: \"kubernetes.io/projected/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-kube-api-access-c9wgn\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:20.735311 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:20.735291 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:21.024741 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:35:21.024709 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04b50ed_38f6_4c24_b4ca_046ca2f9d170.slice/crio-35f501e7e614516b4b1d3a6e3904912ca4b78a1820a4dc30fd6522533bc077dd WatchSource:0}: Error finding container 35f501e7e614516b4b1d3a6e3904912ca4b78a1820a4dc30fd6522533bc077dd: Status 404 returned error can't find the container with id 35f501e7e614516b4b1d3a6e3904912ca4b78a1820a4dc30fd6522533bc077dd Nov 03 20:35:21.024863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.024738 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn"] Nov 03 20:35:21.027408 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.027375 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:35:21.027485 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.027441 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:35:21.027485 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.027471 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:35:21.230128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.230087 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:35:21.230324 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.230145 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" podUID="a2c58648-4022-4bfb-be9f-170a6222dfd8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:35:21.244764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.244724 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" event={"ID":"e04b50ed-38f6-4c24-b4ca-046ca2f9d170","Type":"ContainerStarted","Data":"23edab384e4fee0335fdc43dcc0e0ae8943aa7e398e33d03b809e83016b240a5"} Nov 03 20:35:21.244912 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.244786 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" event={"ID":"e04b50ed-38f6-4c24-b4ca-046ca2f9d170","Type":"ContainerStarted","Data":"35f501e7e614516b4b1d3a6e3904912ca4b78a1820a4dc30fd6522533bc077dd"} Nov 03 20:35:21.246592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.246564 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8095d1bf-dtz7r" event={"ID":"a004055f-9ef2-4be0-b531-6a65b0caa7c2","Type":"ContainerStarted","Data":"de718ea41d179627b84ca65669c1d61a191409b00fbc8e497064dc92a4410c03"} Nov 03 20:35:21.246731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.246607 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8095d1bf-dtz7r" event={"ID":"a004055f-9ef2-4be0-b531-6a65b0caa7c2","Type":"ContainerStarted","Data":"bfccb5d2fccc321c9cb9fbf74d72139af0f95f61b6de0e49b1de6f046cd14ca6"} Nov 03 20:35:21.306353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.306262 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" podStartSLOduration=1.306242804 podStartE2EDuration="1.306242804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:21.30288373 +0000 UTC m=+3515.560681194" watchObservedRunningTime="2026-11-03 20:35:21.306242804 +0000 UTC m=+3515.564040061" Nov 03 20:35:21.347251 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.347201 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8095d1bf-dtz7r" podStartSLOduration=2.347186952 podStartE2EDuration="2.347186952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:21.34620182 +0000 UTC m=+3515.603999078" watchObservedRunningTime="2026-11-03 20:35:21.347186952 +0000 UTC m=+3515.604984208" Nov 03 20:35:21.354637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.354611 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn"] Nov 03 20:35:21.359310 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.359291 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:35:21.373988 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.373965 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-43c674\"/\"openshift-service-ca.crt\"" Nov 03 20:35:21.374112 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.374018 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-43c674\"/\"kube-root-ca.crt\"" Nov 03 20:35:21.376754 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.376727 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn"] Nov 03 20:35:21.383115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.383089 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-43c674\"/\"default-dockercfg-jtdmn\"" Nov 03 20:35:21.383341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.383320 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-43c674\"/\"sub-test-model-43c674-kserve-self-signed-certs\"" Nov 03 20:35:21.503911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.503870 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-dshm\") pod \"sub-test-model-43c674-kserve-677ff676dc-t4xqn\" (UID: \"af50a359-40ae-481c-a707-44a349210f33\") " pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:35:21.504092 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.504060 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt72n\" (UniqueName: \"kubernetes.io/projected/af50a359-40ae-481c-a707-44a349210f33-kube-api-access-jt72n\") pod \"sub-test-model-43c674-kserve-677ff676dc-t4xqn\" (UID: \"af50a359-40ae-481c-a707-44a349210f33\") " pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:35:21.504156 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.504115 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-model-cache\") pod \"sub-test-model-43c674-kserve-677ff676dc-t4xqn\" (UID: \"af50a359-40ae-481c-a707-44a349210f33\") " pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:35:21.504206 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.504185 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-tmp-dir\") pod \"sub-test-model-43c674-kserve-677ff676dc-t4xqn\" (UID: \"af50a359-40ae-481c-a707-44a349210f33\") " pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:35:21.504315 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.504294 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-home\") pod \"sub-test-model-43c674-kserve-677ff676dc-t4xqn\" (UID: \"af50a359-40ae-481c-a707-44a349210f33\") " pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:35:21.504365 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.504349 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af50a359-40ae-481c-a707-44a349210f33-tls-certs\") pod \"sub-test-model-43c674-kserve-677ff676dc-t4xqn\" (UID: \"af50a359-40ae-481c-a707-44a349210f33\") " pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:35:21.605498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.605454 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt72n\" (UniqueName: \"kubernetes.io/projected/af50a359-40ae-481c-a707-44a349210f33-kube-api-access-jt72n\") pod \"sub-test-model-43c674-kserve-677ff676dc-t4xqn\" (UID: \"af50a359-40ae-481c-a707-44a349210f33\") " pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:35:21.605708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.605517 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-model-cache\") pod \"sub-test-model-43c674-kserve-677ff676dc-t4xqn\" (UID: \"af50a359-40ae-481c-a707-44a349210f33\") " pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:35:21.605708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.605536 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-tmp-dir\") pod \"sub-test-model-43c674-kserve-677ff676dc-t4xqn\" (UID: \"af50a359-40ae-481c-a707-44a349210f33\") " pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:35:21.605708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.605564 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-home\") pod \"sub-test-model-43c674-kserve-677ff676dc-t4xqn\" (UID: \"af50a359-40ae-481c-a707-44a349210f33\") " pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:35:21.605708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.605593 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af50a359-40ae-481c-a707-44a349210f33-tls-certs\") pod \"sub-test-model-43c674-kserve-677ff676dc-t4xqn\" (UID: \"af50a359-40ae-481c-a707-44a349210f33\") " pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:35:21.605921 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.605817 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-dshm\") pod \"sub-test-model-43c674-kserve-677ff676dc-t4xqn\" (UID: \"af50a359-40ae-481c-a707-44a349210f33\") " pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:35:21.609874 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.608183 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-tmp-dir\") pod \"sub-test-model-43c674-kserve-677ff676dc-t4xqn\" (UID: \"af50a359-40ae-481c-a707-44a349210f33\") " pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:35:21.609874 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.608830 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-model-cache\") pod \"sub-test-model-43c674-kserve-677ff676dc-t4xqn\" (UID: \"af50a359-40ae-481c-a707-44a349210f33\") " pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:35:21.610725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.610704 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-home\") pod \"sub-test-model-43c674-kserve-677ff676dc-t4xqn\" (UID: \"af50a359-40ae-481c-a707-44a349210f33\") " pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:35:21.614024 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.613979 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-dshm\") pod \"sub-test-model-43c674-kserve-677ff676dc-t4xqn\" (UID: \"af50a359-40ae-481c-a707-44a349210f33\") " pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:35:21.615588 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.615566 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af50a359-40ae-481c-a707-44a349210f33-tls-certs\") pod \"sub-test-model-43c674-kserve-677ff676dc-t4xqn\" (UID: \"af50a359-40ae-481c-a707-44a349210f33\") " pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:35:21.621408 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.621385 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt72n\" (UniqueName: \"kubernetes.io/projected/af50a359-40ae-481c-a707-44a349210f33-kube-api-access-jt72n\") pod \"sub-test-model-43c674-kserve-677ff676dc-t4xqn\" (UID: \"af50a359-40ae-481c-a707-44a349210f33\") " pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:35:21.674166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.674129 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:35:21.736420 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.736233 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:21.968955 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:35:21.968803 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf50a359_40ae_481c_a707_44a349210f33.slice/crio-a28bf82c879c8b6a87eaabc34e2102698cd449d638eaa71e899f6718a526a33f WatchSource:0}: Error finding container a28bf82c879c8b6a87eaabc34e2102698cd449d638eaa71e899f6718a526a33f: Status 404 returned error can't find the container with id a28bf82c879c8b6a87eaabc34e2102698cd449d638eaa71e899f6718a526a33f Nov 03 20:35:21.969260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:21.969234 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn"] Nov 03 20:35:22.226765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:22.226675 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9"] Nov 03 20:35:22.237815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:22.237780 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t"] Nov 03 20:35:22.300948 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:22.300898 5059 generic.go:356] "Generic (PLEG): container finished" podID="a004055f-9ef2-4be0-b531-6a65b0caa7c2" containerID="de718ea41d179627b84ca65669c1d61a191409b00fbc8e497064dc92a4410c03" exitCode=0 Nov 03 20:35:22.301506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:22.301471 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8095d1bf-dtz7r" event={"ID":"a004055f-9ef2-4be0-b531-6a65b0caa7c2","Type":"ContainerDied","Data":"de718ea41d179627b84ca65669c1d61a191409b00fbc8e497064dc92a4410c03"} Nov 03 20:35:22.304391 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:22.304358 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" event={"ID":"af50a359-40ae-481c-a707-44a349210f33","Type":"ContainerStarted","Data":"17071bf2df6f0005f3cc04c51576f8b88cbfd532f726b2bb03027f79acfd5bea"} Nov 03 20:35:22.304512 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:22.304402 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" event={"ID":"af50a359-40ae-481c-a707-44a349210f33","Type":"ContainerStarted","Data":"a28bf82c879c8b6a87eaabc34e2102698cd449d638eaa71e899f6718a526a33f"} Nov 03 20:35:22.344236 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:22.344185 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" podStartSLOduration=1.344171112 podStartE2EDuration="1.344171112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:22.341361966 +0000 UTC m=+3516.599159224" watchObservedRunningTime="2026-11-03 20:35:22.344171112 +0000 UTC m=+3516.601968384" Nov 03 20:35:22.680063 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:22.679992 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:35:22.756011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:22.750792 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body= Nov 03 20:35:22.756011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:22.750878 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" podUID="e04b50ed-38f6-4c24-b4ca-046ca2f9d170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" Nov 03 20:35:23.516821 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:23.516793 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8095d1bf-dtz7r" Nov 03 20:35:23.688294 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:23.688256 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a004055f-9ef2-4be0-b531-6a65b0caa7c2-kube-api-access-9v58d\" (UniqueName: \"kubernetes.io/projected/a004055f-9ef2-4be0-b531-6a65b0caa7c2-kube-api-access-9v58d\") pod \"a004055f-9ef2-4be0-b531-6a65b0caa7c2\" (UID: \"a004055f-9ef2-4be0-b531-6a65b0caa7c2\") " Nov 03 20:35:23.688473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:23.688312 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a004055f-9ef2-4be0-b531-6a65b0caa7c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a004055f-9ef2-4be0-b531-6a65b0caa7c2-maas-api-openshift-ca\") pod \"a004055f-9ef2-4be0-b531-6a65b0caa7c2\" (UID: \"a004055f-9ef2-4be0-b531-6a65b0caa7c2\") " Nov 03 20:35:23.688473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:23.688337 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a004055f-9ef2-4be0-b531-6a65b0caa7c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a004055f-9ef2-4be0-b531-6a65b0caa7c2-maas-api-xks-ca\") pod \"a004055f-9ef2-4be0-b531-6a65b0caa7c2\" (UID: \"a004055f-9ef2-4be0-b531-6a65b0caa7c2\") " Nov 03 20:35:23.688783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:23.688758 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a004055f-9ef2-4be0-b531-6a65b0caa7c2-maas-api-openshift-ca" pod "a004055f-9ef2-4be0-b531-6a65b0caa7c2" (UID: "a004055f-9ef2-4be0-b531-6a65b0caa7c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:23.688989 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:23.688971 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a004055f-9ef2-4be0-b531-6a65b0caa7c2-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:23.690489 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:23.690462 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a004055f-9ef2-4be0-b531-6a65b0caa7c2-kube-api-access-9v58d" pod "a004055f-9ef2-4be0-b531-6a65b0caa7c2" (UID: "a004055f-9ef2-4be0-b531-6a65b0caa7c2"). InnerVolumeSpecName "kube-api-access-9v58d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:23.690579 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:23.690489 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a004055f-9ef2-4be0-b531-6a65b0caa7c2-maas-api-xks-ca" pod "a004055f-9ef2-4be0-b531-6a65b0caa7c2" (UID: "a004055f-9ef2-4be0-b531-6a65b0caa7c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:23.735940 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:23.735910 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body= Nov 03 20:35:23.736039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:23.735960 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" podUID="e04b50ed-38f6-4c24-b4ca-046ca2f9d170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" Nov 03 20:35:23.790409 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:23.790375 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v58d\" (UniqueName: \"kubernetes.io/projected/a004055f-9ef2-4be0-b531-6a65b0caa7c2-kube-api-access-9v58d\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:23.790409 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:23.790410 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a004055f-9ef2-4be0-b531-6a65b0caa7c2-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:24.319653 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:24.319624 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8095d1bf-dtz7r" Nov 03 20:35:24.319852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:24.319698 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8095d1bf-dtz7r" event={"ID":"a004055f-9ef2-4be0-b531-6a65b0caa7c2","Type":"ContainerDied","Data":"bfccb5d2fccc321c9cb9fbf74d72139af0f95f61b6de0e49b1de6f046cd14ca6"} Nov 03 20:35:24.319852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:24.319755 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfccb5d2fccc321c9cb9fbf74d72139af0f95f61b6de0e49b1de6f046cd14ca6" Nov 03 20:35:24.736627 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:24.736541 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body= Nov 03 20:35:24.736627 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:24.736604 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" podUID="e04b50ed-38f6-4c24-b4ca-046ca2f9d170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" Nov 03 20:35:25.210084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.210056 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:35:25.270721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.253679 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-workload-certs\") pod \"a2c58648-4022-4bfb-be9f-170a6222dfd8\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " Nov 03 20:35:25.291817 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.279430 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-workload-certs" pod "a2c58648-4022-4bfb-be9f-170a6222dfd8" (UID: "a2c58648-4022-4bfb-be9f-170a6222dfd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:25.291979 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.291897 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-token\" (UniqueName: \"kubernetes.io/projected/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-token\") pod \"a2c58648-4022-4bfb-be9f-170a6222dfd8\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " Nov 03 20:35:25.291979 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.291951 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-data\") pod \"a2c58648-4022-4bfb-be9f-170a6222dfd8\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " Nov 03 20:35:25.292114 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.292038 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2c58648-4022-4bfb-be9f-170a6222dfd8-kube-api-access-ggjjf\" (UniqueName: \"kubernetes.io/projected/a2c58648-4022-4bfb-be9f-170a6222dfd8-kube-api-access-ggjjf\") pod \"a2c58648-4022-4bfb-be9f-170a6222dfd8\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " Nov 03 20:35:25.292178 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.292119 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2c58648-4022-4bfb-be9f-170a6222dfd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c58648-4022-4bfb-be9f-170a6222dfd8-istiod-ca-cert\") pod \"a2c58648-4022-4bfb-be9f-170a6222dfd8\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " Nov 03 20:35:25.292178 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.292147 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-envoy\") pod \"a2c58648-4022-4bfb-be9f-170a6222dfd8\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " Nov 03 20:35:25.292286 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.292228 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-workload-socket\") pod \"a2c58648-4022-4bfb-be9f-170a6222dfd8\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " Nov 03 20:35:25.292286 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.292271 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-credential-socket\") pod \"a2c58648-4022-4bfb-be9f-170a6222dfd8\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " Nov 03 20:35:25.292391 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.292332 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-podinfo\") pod \"a2c58648-4022-4bfb-be9f-170a6222dfd8\" (UID: \"a2c58648-4022-4bfb-be9f-170a6222dfd8\") " Nov 03 20:35:25.293288 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.293257 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:25.293410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.293323 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-workload-socket" pod "a2c58648-4022-4bfb-be9f-170a6222dfd8" (UID: "a2c58648-4022-4bfb-be9f-170a6222dfd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:25.293472 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.293425 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c58648-4022-4bfb-be9f-170a6222dfd8-istiod-ca-cert" pod "a2c58648-4022-4bfb-be9f-170a6222dfd8" (UID: "a2c58648-4022-4bfb-be9f-170a6222dfd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:25.293588 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.293564 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-credential-socket" pod "a2c58648-4022-4bfb-be9f-170a6222dfd8" (UID: "a2c58648-4022-4bfb-be9f-170a6222dfd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:25.294228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.294194 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-data" pod "a2c58648-4022-4bfb-be9f-170a6222dfd8" (UID: "a2c58648-4022-4bfb-be9f-170a6222dfd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:25.298704 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.295728 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-podinfo" pod "a2c58648-4022-4bfb-be9f-170a6222dfd8" (UID: "a2c58648-4022-4bfb-be9f-170a6222dfd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:25.298704 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.296057 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-envoy" pod "a2c58648-4022-4bfb-be9f-170a6222dfd8" (UID: "a2c58648-4022-4bfb-be9f-170a6222dfd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:25.298704 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.296072 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c58648-4022-4bfb-be9f-170a6222dfd8-kube-api-access-ggjjf" pod "a2c58648-4022-4bfb-be9f-170a6222dfd8" (UID: "a2c58648-4022-4bfb-be9f-170a6222dfd8"). InnerVolumeSpecName "kube-api-access-ggjjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:25.299435 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.299401 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-token" pod "a2c58648-4022-4bfb-be9f-170a6222dfd8" (UID: "a2c58648-4022-4bfb-be9f-170a6222dfd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:25.330570 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.330543 5059 generic.go:356] "Generic (PLEG): container finished" podID="a2c58648-4022-4bfb-be9f-170a6222dfd8" containerID="fb118c660710eb7d73eb1decfc3f34428b1d8bf365ce4f36a506649974ba8750" exitCode=0 Nov 03 20:35:25.330656 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.330587 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" Nov 03 20:35:25.330656 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.330625 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" event={"ID":"a2c58648-4022-4bfb-be9f-170a6222dfd8","Type":"ContainerDied","Data":"fb118c660710eb7d73eb1decfc3f34428b1d8bf365ce4f36a506649974ba8750"} Nov 03 20:35:25.330763 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.330711 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7" event={"ID":"a2c58648-4022-4bfb-be9f-170a6222dfd8","Type":"ContainerDied","Data":"cb8f0b2617626011a18289e66bb16c024dc66fe6709c48401aea2b0c398d900c"} Nov 03 20:35:25.330763 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.330736 5059 scope.go:122] "RemoveContainer" containerID="fb118c660710eb7d73eb1decfc3f34428b1d8bf365ce4f36a506649974ba8750" Nov 03 20:35:25.342140 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.342120 5059 scope.go:122] "RemoveContainer" containerID="fb118c660710eb7d73eb1decfc3f34428b1d8bf365ce4f36a506649974ba8750" Nov 03 20:35:25.342422 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:35:25.342385 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb118c660710eb7d73eb1decfc3f34428b1d8bf365ce4f36a506649974ba8750\": container with ID starting with fb118c660710eb7d73eb1decfc3f34428b1d8bf365ce4f36a506649974ba8750 not found: ID does not exist" containerID="fb118c660710eb7d73eb1decfc3f34428b1d8bf365ce4f36a506649974ba8750" Nov 03 20:35:25.342514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.342438 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb118c660710eb7d73eb1decfc3f34428b1d8bf365ce4f36a506649974ba8750"} err="failed to get container status \"fb118c660710eb7d73eb1decfc3f34428b1d8bf365ce4f36a506649974ba8750\": rpc error: code = NotFound desc = could not find container \"fb118c660710eb7d73eb1decfc3f34428b1d8bf365ce4f36a506649974ba8750\": container with ID starting with fb118c660710eb7d73eb1decfc3f34428b1d8bf365ce4f36a506649974ba8750 not found: ID does not exist" Nov 03 20:35:25.356479 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.356459 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7"] Nov 03 20:35:25.387380 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.387348 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-r5bq7"] Nov 03 20:35:25.395193 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.395171 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:25.395263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.395197 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:25.395263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.395210 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggjjf\" (UniqueName: \"kubernetes.io/projected/a2c58648-4022-4bfb-be9f-170a6222dfd8-kube-api-access-ggjjf\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:25.395263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.395220 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c58648-4022-4bfb-be9f-170a6222dfd8-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:25.395263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.395229 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:25.395263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.395238 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:25.395263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.395247 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c58648-4022-4bfb-be9f-170a6222dfd8-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:25.395263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.395255 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c58648-4022-4bfb-be9f-170a6222dfd8-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:25.736562 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.736523 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body= Nov 03 20:35:25.736784 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.736578 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" podUID="e04b50ed-38f6-4c24-b4ca-046ca2f9d170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" Nov 03 20:35:25.858710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:25.857878 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2c58648-4022-4bfb-be9f-170a6222dfd8" path="/var/lib/kubelet/pods/a2c58648-4022-4bfb-be9f-170a6222dfd8/volumes" Nov 03 20:35:26.735770 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:26.735733 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body= Nov 03 20:35:26.736144 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:26.735787 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" podUID="e04b50ed-38f6-4c24-b4ca-046ca2f9d170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" Nov 03 20:35:27.228459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:27.228054 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9" podUID="fd9d29fb-d764-4eac-a744-dbdecde12ff0" containerName="payload-processing" containerID="cri-o://69579c0e40da217f8ae4fdef8b052b316aff66db158b9fbdeef2495a9ac79ec8" gracePeriod=25 Nov 03 20:35:27.238806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:27.238772 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t" podUID="4b25c2d0-60bb-4df1-8193-5f7f853c0a73" containerName="payload-pre-processing" containerID="cri-o://bd910959fcbe106c88812aae26cf4e57d7855ae92a08f802492c9b663a34b390" gracePeriod=25 Nov 03 20:35:27.352193 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:27.352150 5059 generic.go:356] "Generic (PLEG): container finished" podID="fd9d29fb-d764-4eac-a744-dbdecde12ff0" containerID="69579c0e40da217f8ae4fdef8b052b316aff66db158b9fbdeef2495a9ac79ec8" exitCode=0 Nov 03 20:35:27.352292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:27.352234 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9" event={"ID":"fd9d29fb-d764-4eac-a744-dbdecde12ff0","Type":"ContainerDied","Data":"69579c0e40da217f8ae4fdef8b052b316aff66db158b9fbdeef2495a9ac79ec8"} Nov 03 20:35:27.599424 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:27.598239 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9" Nov 03 20:35:27.624360 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:27.624331 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t" Nov 03 20:35:27.657612 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:27.657580 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd9d29fb-d764-4eac-a744-dbdecde12ff0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd9d29fb-d764-4eac-a744-dbdecde12ff0-plugins-config-volume\") pod \"fd9d29fb-d764-4eac-a744-dbdecde12ff0\" (UID: \"fd9d29fb-d764-4eac-a744-dbdecde12ff0\") " Nov 03 20:35:27.657848 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:27.657831 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd9d29fb-d764-4eac-a744-dbdecde12ff0-kube-api-access-ccx9j\" (UniqueName: \"kubernetes.io/projected/fd9d29fb-d764-4eac-a744-dbdecde12ff0-kube-api-access-ccx9j\") pod \"fd9d29fb-d764-4eac-a744-dbdecde12ff0\" (UID: \"fd9d29fb-d764-4eac-a744-dbdecde12ff0\") " Nov 03 20:35:27.659918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:27.659890 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd9d29fb-d764-4eac-a744-dbdecde12ff0-plugins-config-volume" pod "fd9d29fb-d764-4eac-a744-dbdecde12ff0" (UID: "fd9d29fb-d764-4eac-a744-dbdecde12ff0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:27.660512 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:27.660490 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9d29fb-d764-4eac-a744-dbdecde12ff0-kube-api-access-ccx9j" pod "fd9d29fb-d764-4eac-a744-dbdecde12ff0" (UID: "fd9d29fb-d764-4eac-a744-dbdecde12ff0"). InnerVolumeSpecName "kube-api-access-ccx9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:27.735930 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:27.735889 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body= Nov 03 20:35:27.736429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:27.735948 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" podUID="e04b50ed-38f6-4c24-b4ca-046ca2f9d170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" Nov 03 20:35:27.759003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:27.758967 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b25c2d0-60bb-4df1-8193-5f7f853c0a73-kube-api-access-625k6\" (UniqueName: \"kubernetes.io/projected/4b25c2d0-60bb-4df1-8193-5f7f853c0a73-kube-api-access-625k6\") pod \"4b25c2d0-60bb-4df1-8193-5f7f853c0a73\" (UID: \"4b25c2d0-60bb-4df1-8193-5f7f853c0a73\") " Nov 03 20:35:27.759168 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:27.759153 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b25c2d0-60bb-4df1-8193-5f7f853c0a73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b25c2d0-60bb-4df1-8193-5f7f853c0a73-plugins-config-volume\") pod \"4b25c2d0-60bb-4df1-8193-5f7f853c0a73\" (UID: \"4b25c2d0-60bb-4df1-8193-5f7f853c0a73\") " Nov 03 20:35:27.759575 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:27.759549 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b25c2d0-60bb-4df1-8193-5f7f853c0a73-plugins-config-volume" pod "4b25c2d0-60bb-4df1-8193-5f7f853c0a73" (UID: "4b25c2d0-60bb-4df1-8193-5f7f853c0a73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:27.759699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:27.759681 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccx9j\" (UniqueName: \"kubernetes.io/projected/fd9d29fb-d764-4eac-a744-dbdecde12ff0-kube-api-access-ccx9j\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:27.759765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:27.759699 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd9d29fb-d764-4eac-a744-dbdecde12ff0-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:27.759765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:27.759708 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b25c2d0-60bb-4df1-8193-5f7f853c0a73-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:27.761153 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:27.761128 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b25c2d0-60bb-4df1-8193-5f7f853c0a73-kube-api-access-625k6" pod "4b25c2d0-60bb-4df1-8193-5f7f853c0a73" (UID: "4b25c2d0-60bb-4df1-8193-5f7f853c0a73"). InnerVolumeSpecName "kube-api-access-625k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:27.861095 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:27.861053 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-625k6\" (UniqueName: \"kubernetes.io/projected/4b25c2d0-60bb-4df1-8193-5f7f853c0a73-kube-api-access-625k6\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:28.359967 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:28.359940 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9" Nov 03 20:35:28.359967 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:28.359946 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9" event={"ID":"fd9d29fb-d764-4eac-a744-dbdecde12ff0","Type":"ContainerDied","Data":"6fea7c3735106edfc3123cf029de6ac4573a06a4c349292f2bbcb66df7a37986"} Nov 03 20:35:28.360204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:28.360008 5059 scope.go:122] "RemoveContainer" containerID="69579c0e40da217f8ae4fdef8b052b316aff66db158b9fbdeef2495a9ac79ec8" Nov 03 20:35:28.361441 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:28.361419 5059 generic.go:356] "Generic (PLEG): container finished" podID="4b25c2d0-60bb-4df1-8193-5f7f853c0a73" containerID="bd910959fcbe106c88812aae26cf4e57d7855ae92a08f802492c9b663a34b390" exitCode=0 Nov 03 20:35:28.361504 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:28.361468 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t" Nov 03 20:35:28.361504 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:28.361486 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t" event={"ID":"4b25c2d0-60bb-4df1-8193-5f7f853c0a73","Type":"ContainerDied","Data":"bd910959fcbe106c88812aae26cf4e57d7855ae92a08f802492c9b663a34b390"} Nov 03 20:35:28.361621 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:28.361515 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t" event={"ID":"4b25c2d0-60bb-4df1-8193-5f7f853c0a73","Type":"ContainerDied","Data":"d1aa1348af3cbccddd516d7cce315ccc0828f8681c485afd95e08448f2ce5bae"} Nov 03 20:35:28.370847 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:28.370825 5059 scope.go:122] "RemoveContainer" containerID="bd910959fcbe106c88812aae26cf4e57d7855ae92a08f802492c9b663a34b390" Nov 03 20:35:28.379813 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:28.379790 5059 scope.go:122] "RemoveContainer" containerID="bd910959fcbe106c88812aae26cf4e57d7855ae92a08f802492c9b663a34b390" Nov 03 20:35:28.380080 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:35:28.380058 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd910959fcbe106c88812aae26cf4e57d7855ae92a08f802492c9b663a34b390\": container with ID starting with bd910959fcbe106c88812aae26cf4e57d7855ae92a08f802492c9b663a34b390 not found: ID does not exist" containerID="bd910959fcbe106c88812aae26cf4e57d7855ae92a08f802492c9b663a34b390" Nov 03 20:35:28.380129 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:28.380090 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd910959fcbe106c88812aae26cf4e57d7855ae92a08f802492c9b663a34b390"} err="failed to get container status \"bd910959fcbe106c88812aae26cf4e57d7855ae92a08f802492c9b663a34b390\": rpc error: code = NotFound desc = could not find container \"bd910959fcbe106c88812aae26cf4e57d7855ae92a08f802492c9b663a34b390\": container with ID starting with bd910959fcbe106c88812aae26cf4e57d7855ae92a08f802492c9b663a34b390 not found: ID does not exist" Nov 03 20:35:28.386569 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:28.386547 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t"] Nov 03 20:35:28.390944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:28.390924 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a0d4ec-7db9f85699-9np8t"] Nov 03 20:35:28.404469 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:28.404444 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9"] Nov 03 20:35:28.405818 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:28.405799 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a0d4ec-576fcc686-zvkt9"] Nov 03 20:35:28.741178 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:28.741148 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:28.741627 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:28.741612 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:28.742296 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:28.742274 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:35:29.857138 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:29.857104 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b25c2d0-60bb-4df1-8193-5f7f853c0a73" path="/var/lib/kubelet/pods/4b25c2d0-60bb-4df1-8193-5f7f853c0a73/volumes" Nov 03 20:35:29.857590 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:29.857575 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd9d29fb-d764-4eac-a744-dbdecde12ff0" path="/var/lib/kubelet/pods/fd9d29fb-d764-4eac-a744-dbdecde12ff0/volumes" Nov 03 20:35:31.675363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:31.675329 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:35:31.687617 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:31.687589 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:35:32.411996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:32.411964 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:35:40.605604 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:35:40.605568 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:35:41.725007 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:41.724969 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62"] Nov 03 20:35:41.725482 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:41.725243 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" containerID="cri-o://2d9824c6fbca0342e936df0e333132fa3d4104bf9d7e7be14d38328d09ab2343" gracePeriod=30 Nov 03 20:35:46.459912 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:46.459873 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs"] Nov 03 20:35:46.460281 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:46.460142 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" podUID="4b3627ae-66c6-44a4-aa60-480bd54008f0" containerName="istio-proxy" containerID="cri-o://0a4e5fbcc2f971059f484ad26dce5294c486b94025b6ad0f0353b5190dfe9a11" gracePeriod=30 Nov 03 20:35:47.028414 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.028381 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:35:47.105415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.105364 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75efd589-b943-46c2-add0-d44ccdaff537-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75efd589-b943-46c2-add0-d44ccdaff537-istio-podinfo\") pod \"75efd589-b943-46c2-add0-d44ccdaff537\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " Nov 03 20:35:47.105592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.105531 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-workload-certs\") pod \"75efd589-b943-46c2-add0-d44ccdaff537\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " Nov 03 20:35:47.105642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.105602 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-credential-socket\") pod \"75efd589-b943-46c2-add0-d44ccdaff537\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " Nov 03 20:35:47.105718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.105694 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-istio-data\") pod \"75efd589-b943-46c2-add0-d44ccdaff537\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " Nov 03 20:35:47.105772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.105739 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75efd589-b943-46c2-add0-d44ccdaff537-istio-token\" (UniqueName: \"kubernetes.io/projected/75efd589-b943-46c2-add0-d44ccdaff537-istio-token\") pod \"75efd589-b943-46c2-add0-d44ccdaff537\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " Nov 03 20:35:47.105891 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.105869 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-istio-envoy\") pod \"75efd589-b943-46c2-add0-d44ccdaff537\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " Nov 03 20:35:47.105891 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.105857 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-workload-certs" pod "75efd589-b943-46c2-add0-d44ccdaff537" (UID: "75efd589-b943-46c2-add0-d44ccdaff537"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:47.106093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.105943 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-credential-socket" pod "75efd589-b943-46c2-add0-d44ccdaff537" (UID: "75efd589-b943-46c2-add0-d44ccdaff537"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:47.106093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.105960 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-istio-data" pod "75efd589-b943-46c2-add0-d44ccdaff537" (UID: "75efd589-b943-46c2-add0-d44ccdaff537"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:47.106093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.105967 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-workload-socket\") pod \"75efd589-b943-46c2-add0-d44ccdaff537\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " Nov 03 20:35:47.106093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.106017 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75efd589-b943-46c2-add0-d44ccdaff537-kube-api-access-zp7pg\" (UniqueName: \"kubernetes.io/projected/75efd589-b943-46c2-add0-d44ccdaff537-kube-api-access-zp7pg\") pod \"75efd589-b943-46c2-add0-d44ccdaff537\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " Nov 03 20:35:47.106093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.106087 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75efd589-b943-46c2-add0-d44ccdaff537-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75efd589-b943-46c2-add0-d44ccdaff537-istiod-ca-cert\") pod \"75efd589-b943-46c2-add0-d44ccdaff537\" (UID: \"75efd589-b943-46c2-add0-d44ccdaff537\") " Nov 03 20:35:47.106375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.106169 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-workload-socket" pod "75efd589-b943-46c2-add0-d44ccdaff537" (UID: "75efd589-b943-46c2-add0-d44ccdaff537"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:47.106557 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.106523 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75efd589-b943-46c2-add0-d44ccdaff537-istiod-ca-cert" pod "75efd589-b943-46c2-add0-d44ccdaff537" (UID: "75efd589-b943-46c2-add0-d44ccdaff537"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:47.106844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.106825 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:47.106916 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.106849 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75efd589-b943-46c2-add0-d44ccdaff537-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:47.106916 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.106862 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:47.106916 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.106876 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:47.106916 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.106889 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:47.108099 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.108072 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-istio-envoy" pod "75efd589-b943-46c2-add0-d44ccdaff537" (UID: "75efd589-b943-46c2-add0-d44ccdaff537"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:47.108230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.108210 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75efd589-b943-46c2-add0-d44ccdaff537-istio-podinfo" pod "75efd589-b943-46c2-add0-d44ccdaff537" (UID: "75efd589-b943-46c2-add0-d44ccdaff537"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:47.108305 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.108272 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75efd589-b943-46c2-add0-d44ccdaff537-istio-token" pod "75efd589-b943-46c2-add0-d44ccdaff537" (UID: "75efd589-b943-46c2-add0-d44ccdaff537"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:47.108432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.108412 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75efd589-b943-46c2-add0-d44ccdaff537-kube-api-access-zp7pg" pod "75efd589-b943-46c2-add0-d44ccdaff537" (UID: "75efd589-b943-46c2-add0-d44ccdaff537"). InnerVolumeSpecName "kube-api-access-zp7pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:47.208047 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.208025 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75efd589-b943-46c2-add0-d44ccdaff537-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:47.208047 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.208048 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75efd589-b943-46c2-add0-d44ccdaff537-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:47.208173 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.208058 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp7pg\" (UniqueName: \"kubernetes.io/projected/75efd589-b943-46c2-add0-d44ccdaff537-kube-api-access-zp7pg\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:47.208173 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.208069 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75efd589-b943-46c2-add0-d44ccdaff537-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:47.547615 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.547575 5059 generic.go:356] "Generic (PLEG): container finished" podID="75efd589-b943-46c2-add0-d44ccdaff537" containerID="2d9824c6fbca0342e936df0e333132fa3d4104bf9d7e7be14d38328d09ab2343" exitCode=0 Nov 03 20:35:47.548101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.547699 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" event={"ID":"75efd589-b943-46c2-add0-d44ccdaff537","Type":"ContainerDied","Data":"2d9824c6fbca0342e936df0e333132fa3d4104bf9d7e7be14d38328d09ab2343"} Nov 03 20:35:47.548101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.547762 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" event={"ID":"75efd589-b943-46c2-add0-d44ccdaff537","Type":"ContainerDied","Data":"7733b8a23d9b015a70236a5b6383a704b97047c69c34cafeea1d149c9885c552"} Nov 03 20:35:47.548101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.547786 5059 scope.go:122] "RemoveContainer" containerID="2d9824c6fbca0342e936df0e333132fa3d4104bf9d7e7be14d38328d09ab2343" Nov 03 20:35:47.548101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.547707 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62" Nov 03 20:35:47.559010 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.558987 5059 scope.go:122] "RemoveContainer" containerID="2d9824c6fbca0342e936df0e333132fa3d4104bf9d7e7be14d38328d09ab2343" Nov 03 20:35:47.559307 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:35:47.559278 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d9824c6fbca0342e936df0e333132fa3d4104bf9d7e7be14d38328d09ab2343\": container with ID starting with 2d9824c6fbca0342e936df0e333132fa3d4104bf9d7e7be14d38328d09ab2343 not found: ID does not exist" containerID="2d9824c6fbca0342e936df0e333132fa3d4104bf9d7e7be14d38328d09ab2343" Nov 03 20:35:47.559416 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.559316 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d9824c6fbca0342e936df0e333132fa3d4104bf9d7e7be14d38328d09ab2343"} err="failed to get container status \"2d9824c6fbca0342e936df0e333132fa3d4104bf9d7e7be14d38328d09ab2343\": rpc error: code = NotFound desc = could not find container \"2d9824c6fbca0342e936df0e333132fa3d4104bf9d7e7be14d38328d09ab2343\": container with ID starting with 2d9824c6fbca0342e936df0e333132fa3d4104bf9d7e7be14d38328d09ab2343 not found: ID does not exist" Nov 03 20:35:47.573533 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.573507 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62"] Nov 03 20:35:47.578521 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.578497 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-942930-openshift-default-548c4f9d89-rdm62"] Nov 03 20:35:47.857112 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:47.857083 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75efd589-b943-46c2-add0-d44ccdaff537" path="/var/lib/kubelet/pods/75efd589-b943-46c2-add0-d44ccdaff537/volumes" Nov 03 20:35:51.653999 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.653801 5059 generic.go:356] "Generic (PLEG): container finished" podID="4b3627ae-66c6-44a4-aa60-480bd54008f0" containerID="0a4e5fbcc2f971059f484ad26dce5294c486b94025b6ad0f0353b5190dfe9a11" exitCode=0 Nov 03 20:35:51.653999 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.653935 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" event={"ID":"4b3627ae-66c6-44a4-aa60-480bd54008f0","Type":"ContainerDied","Data":"0a4e5fbcc2f971059f484ad26dce5294c486b94025b6ad0f0353b5190dfe9a11"} Nov 03 20:35:51.767953 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.767892 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:35:51.922984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.922937 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-podinfo\") pod \"4b3627ae-66c6-44a4-aa60-480bd54008f0\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " Nov 03 20:35:51.923173 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.923084 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b3627ae-66c6-44a4-aa60-480bd54008f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3627ae-66c6-44a4-aa60-480bd54008f0-istiod-ca-cert\") pod \"4b3627ae-66c6-44a4-aa60-480bd54008f0\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " Nov 03 20:35:51.923238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.923175 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b3627ae-66c6-44a4-aa60-480bd54008f0-kube-api-access-s5bf8\" (UniqueName: \"kubernetes.io/projected/4b3627ae-66c6-44a4-aa60-480bd54008f0-kube-api-access-s5bf8\") pod \"4b3627ae-66c6-44a4-aa60-480bd54008f0\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " Nov 03 20:35:51.923311 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.923249 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-credential-socket\") pod \"4b3627ae-66c6-44a4-aa60-480bd54008f0\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " Nov 03 20:35:51.923363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.923353 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-workload-socket\") pod \"4b3627ae-66c6-44a4-aa60-480bd54008f0\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " Nov 03 20:35:51.923417 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.923395 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-data\") pod \"4b3627ae-66c6-44a4-aa60-480bd54008f0\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " Nov 03 20:35:51.923488 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.923468 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-workload-certs\") pod \"4b3627ae-66c6-44a4-aa60-480bd54008f0\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " Nov 03 20:35:51.923622 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.923589 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b3627ae-66c6-44a4-aa60-480bd54008f0-istiod-ca-cert" pod "4b3627ae-66c6-44a4-aa60-480bd54008f0" (UID: "4b3627ae-66c6-44a4-aa60-480bd54008f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:51.923701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.923679 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-token\" (UniqueName: \"kubernetes.io/projected/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-token\") pod \"4b3627ae-66c6-44a4-aa60-480bd54008f0\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " Nov 03 20:35:51.923763 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.923712 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-envoy\") pod \"4b3627ae-66c6-44a4-aa60-480bd54008f0\" (UID: \"4b3627ae-66c6-44a4-aa60-480bd54008f0\") " Nov 03 20:35:51.923824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.923785 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-workload-socket" pod "4b3627ae-66c6-44a4-aa60-480bd54008f0" (UID: "4b3627ae-66c6-44a4-aa60-480bd54008f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:51.923915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.923892 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-workload-certs" pod "4b3627ae-66c6-44a4-aa60-480bd54008f0" (UID: "4b3627ae-66c6-44a4-aa60-480bd54008f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:51.923980 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.923915 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-data" pod "4b3627ae-66c6-44a4-aa60-480bd54008f0" (UID: "4b3627ae-66c6-44a4-aa60-480bd54008f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:51.924216 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.924191 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-credential-socket" pod "4b3627ae-66c6-44a4-aa60-480bd54008f0" (UID: "4b3627ae-66c6-44a4-aa60-480bd54008f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:51.924371 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.924335 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3627ae-66c6-44a4-aa60-480bd54008f0-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:51.924463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.924369 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:51.924463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.924386 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:51.924463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.924417 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:51.924463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.924432 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:51.926981 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.926940 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-envoy" pod "4b3627ae-66c6-44a4-aa60-480bd54008f0" (UID: "4b3627ae-66c6-44a4-aa60-480bd54008f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:51.926981 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.926975 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-token" pod "4b3627ae-66c6-44a4-aa60-480bd54008f0" (UID: "4b3627ae-66c6-44a4-aa60-480bd54008f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:51.930081 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.930058 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-podinfo" pod "4b3627ae-66c6-44a4-aa60-480bd54008f0" (UID: "4b3627ae-66c6-44a4-aa60-480bd54008f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:51.930181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:51.930161 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b3627ae-66c6-44a4-aa60-480bd54008f0-kube-api-access-s5bf8" pod "4b3627ae-66c6-44a4-aa60-480bd54008f0" (UID: "4b3627ae-66c6-44a4-aa60-480bd54008f0"). InnerVolumeSpecName "kube-api-access-s5bf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:52.026056 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:52.025971 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:52.026056 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:52.026004 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:52.026056 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:52.026015 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3627ae-66c6-44a4-aa60-480bd54008f0-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:52.026056 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:52.026025 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5bf8\" (UniqueName: \"kubernetes.io/projected/4b3627ae-66c6-44a4-aa60-480bd54008f0-kube-api-access-s5bf8\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:52.664171 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:52.663649 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" Nov 03 20:35:52.664171 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:52.663699 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs" event={"ID":"4b3627ae-66c6-44a4-aa60-480bd54008f0","Type":"ContainerDied","Data":"56675e05c55dfbccd6b9e31541db45c3dccf7f2cfe09e69dcaadea37cd39f9bc"} Nov 03 20:35:52.664171 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:52.663786 5059 scope.go:122] "RemoveContainer" containerID="0a4e5fbcc2f971059f484ad26dce5294c486b94025b6ad0f0353b5190dfe9a11" Nov 03 20:35:52.703542 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:52.703506 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs"] Nov 03 20:35:52.703767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:52.703557 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a0d4ec-openshift-default-698f57dbd9-bcfqs"] Nov 03 20:35:53.856675 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:53.856629 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b3627ae-66c6-44a4-aa60-480bd54008f0" path="/var/lib/kubelet/pods/4b3627ae-66c6-44a4-aa60-480bd54008f0/volumes" Nov 03 20:35:54.034143 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:54.034107 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm"] Nov 03 20:35:55.052160 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:55.052131 5059 kubelet.go:1689] "Image garbage collection succeeded" Nov 03 20:35:59.035499 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:59.035429 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm" podUID="7dc48f99-393e-4af4-b942-1ed63909b588" containerName="payload-pre-processing" containerID="cri-o://9cda458c63e04efc7e5496fc503f63daafb5eed92eca9ae85fff848227a806aa" gracePeriod=25 Nov 03 20:35:59.304013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:59.303958 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm" Nov 03 20:35:59.430036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:59.430006 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dc48f99-393e-4af4-b942-1ed63909b588-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc48f99-393e-4af4-b942-1ed63909b588-plugins-config-volume\") pod \"7dc48f99-393e-4af4-b942-1ed63909b588\" (UID: \"7dc48f99-393e-4af4-b942-1ed63909b588\") " Nov 03 20:35:59.430202 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:59.430179 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc48f99-393e-4af4-b942-1ed63909b588-kube-api-access-5kvn9\" (UniqueName: \"kubernetes.io/projected/7dc48f99-393e-4af4-b942-1ed63909b588-kube-api-access-5kvn9\") pod \"7dc48f99-393e-4af4-b942-1ed63909b588\" (UID: \"7dc48f99-393e-4af4-b942-1ed63909b588\") " Nov 03 20:35:59.430402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:59.430381 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc48f99-393e-4af4-b942-1ed63909b588-plugins-config-volume" pod "7dc48f99-393e-4af4-b942-1ed63909b588" (UID: "7dc48f99-393e-4af4-b942-1ed63909b588"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:59.430649 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:59.430631 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc48f99-393e-4af4-b942-1ed63909b588-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:59.432379 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:59.432350 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc48f99-393e-4af4-b942-1ed63909b588-kube-api-access-5kvn9" pod "7dc48f99-393e-4af4-b942-1ed63909b588" (UID: "7dc48f99-393e-4af4-b942-1ed63909b588"). InnerVolumeSpecName "kube-api-access-5kvn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:59.531570 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:59.531537 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kvn9\" (UniqueName: \"kubernetes.io/projected/7dc48f99-393e-4af4-b942-1ed63909b588-kube-api-access-5kvn9\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:59.718675 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:59.718621 5059 generic.go:356] "Generic (PLEG): container finished" podID="7dc48f99-393e-4af4-b942-1ed63909b588" containerID="9cda458c63e04efc7e5496fc503f63daafb5eed92eca9ae85fff848227a806aa" exitCode=0 Nov 03 20:35:59.718849 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:59.718706 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm" Nov 03 20:35:59.718849 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:59.718709 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm" event={"ID":"7dc48f99-393e-4af4-b942-1ed63909b588","Type":"ContainerDied","Data":"9cda458c63e04efc7e5496fc503f63daafb5eed92eca9ae85fff848227a806aa"} Nov 03 20:35:59.718849 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:59.718806 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm" event={"ID":"7dc48f99-393e-4af4-b942-1ed63909b588","Type":"ContainerDied","Data":"5b5c31d092c9ba53079cea073aa8897c3126023324d7fbd6bbe70935eae34a3b"} Nov 03 20:35:59.718849 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:59.718833 5059 scope.go:122] "RemoveContainer" containerID="9cda458c63e04efc7e5496fc503f63daafb5eed92eca9ae85fff848227a806aa" Nov 03 20:35:59.729261 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:59.729240 5059 scope.go:122] "RemoveContainer" containerID="9cda458c63e04efc7e5496fc503f63daafb5eed92eca9ae85fff848227a806aa" Nov 03 20:35:59.729515 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:35:59.729495 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cda458c63e04efc7e5496fc503f63daafb5eed92eca9ae85fff848227a806aa\": container with ID starting with 9cda458c63e04efc7e5496fc503f63daafb5eed92eca9ae85fff848227a806aa not found: ID does not exist" containerID="9cda458c63e04efc7e5496fc503f63daafb5eed92eca9ae85fff848227a806aa" Nov 03 20:35:59.729557 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:59.729528 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cda458c63e04efc7e5496fc503f63daafb5eed92eca9ae85fff848227a806aa"} err="failed to get container status \"9cda458c63e04efc7e5496fc503f63daafb5eed92eca9ae85fff848227a806aa\": rpc error: code = NotFound desc = could not find container \"9cda458c63e04efc7e5496fc503f63daafb5eed92eca9ae85fff848227a806aa\": container with ID starting with 9cda458c63e04efc7e5496fc503f63daafb5eed92eca9ae85fff848227a806aa not found: ID does not exist" Nov 03 20:35:59.744178 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:59.744155 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm"] Nov 03 20:35:59.748460 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:59.748437 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e1546-76fc58fc67-hqpgm"] Nov 03 20:35:59.856246 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:35:59.856219 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dc48f99-393e-4af4-b942-1ed63909b588" path="/var/lib/kubelet/pods/7dc48f99-393e-4af4-b942-1ed63909b588/volumes" Nov 03 20:36:02.882094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.877857 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm"] Nov 03 20:36:02.882094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.879083 5059 cpu_manager.go:414] "containerMap: removing container" podUID="a2c58648-4022-4bfb-be9f-170a6222dfd8" containerName="istio-proxy" Nov 03 20:36:02.882094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.879108 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2c58648-4022-4bfb-be9f-170a6222dfd8" containerName="istio-proxy" Nov 03 20:36:02.882094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.879195 5059 cpu_manager.go:414] "containerMap: removing container" podUID="fd9d29fb-d764-4eac-a744-dbdecde12ff0" containerName="payload-processing" Nov 03 20:36:02.882094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.879203 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd9d29fb-d764-4eac-a744-dbdecde12ff0" containerName="payload-processing" Nov 03 20:36:02.882094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.879226 5059 cpu_manager.go:414] "containerMap: removing container" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" Nov 03 20:36:02.882094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.879235 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" Nov 03 20:36:02.882094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.879243 5059 cpu_manager.go:414] "containerMap: removing container" podUID="4b3627ae-66c6-44a4-aa60-480bd54008f0" containerName="istio-proxy" Nov 03 20:36:02.882094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.879250 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b3627ae-66c6-44a4-aa60-480bd54008f0" containerName="istio-proxy" Nov 03 20:36:02.882094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.879260 5059 cpu_manager.go:414] "containerMap: removing container" podUID="4b25c2d0-60bb-4df1-8193-5f7f853c0a73" containerName="payload-pre-processing" Nov 03 20:36:02.882094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.879267 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b25c2d0-60bb-4df1-8193-5f7f853c0a73" containerName="payload-pre-processing" Nov 03 20:36:02.882094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.879288 5059 cpu_manager.go:414] "containerMap: removing container" podUID="7dc48f99-393e-4af4-b942-1ed63909b588" containerName="payload-pre-processing" Nov 03 20:36:02.882094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.879296 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dc48f99-393e-4af4-b942-1ed63909b588" containerName="payload-pre-processing" Nov 03 20:36:02.882094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.879368 5059 cpu_manager.go:414] "containerMap: removing container" podUID="a004055f-9ef2-4be0-b531-6a65b0caa7c2" containerName="revoke-keys" Nov 03 20:36:02.882094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.879377 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a004055f-9ef2-4be0-b531-6a65b0caa7c2" containerName="revoke-keys" Nov 03 20:36:02.882094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.879638 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd9d29fb-d764-4eac-a744-dbdecde12ff0" containerName="payload-processing" Nov 03 20:36:02.882094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.879672 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b3627ae-66c6-44a4-aa60-480bd54008f0" containerName="istio-proxy" Nov 03 20:36:02.882094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.879688 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="75efd589-b943-46c2-add0-d44ccdaff537" containerName="istio-proxy" Nov 03 20:36:02.882094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.879698 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b25c2d0-60bb-4df1-8193-5f7f853c0a73" containerName="payload-pre-processing" Nov 03 20:36:02.882094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.879716 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dc48f99-393e-4af4-b942-1ed63909b588" containerName="payload-pre-processing" Nov 03 20:36:02.882094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.879729 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a004055f-9ef2-4be0-b531-6a65b0caa7c2" containerName="revoke-keys" Nov 03 20:36:02.882094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.879745 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2c58648-4022-4bfb-be9f-170a6222dfd8" containerName="istio-proxy" Nov 03 20:36:02.885331 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.885304 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:02.894929 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.894520 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm"] Nov 03 20:36:02.978959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.978917 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-workload-certs\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:02.979140 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.979081 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896c119d-c988-4954-8606-72f0de8944cf-istiod-ca-cert\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:02.979217 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.979136 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-workload-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:02.979217 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.979159 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896c119d-c988-4954-8606-72f0de8944cf-istio-podinfo\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:02.979217 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.979187 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/896c119d-c988-4954-8606-72f0de8944cf-istio-token\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:02.979327 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.979278 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzn96\" (UniqueName: \"kubernetes.io/projected/896c119d-c988-4954-8606-72f0de8944cf-kube-api-access-nzn96\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:02.979368 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.979347 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-istio-data\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:02.979447 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.979422 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-istio-envoy\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:02.979527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:02.979514 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-credential-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:03.081104 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.081071 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-istio-envoy\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:03.081235 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.081156 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-credential-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:03.081235 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.081186 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-workload-certs\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:03.081316 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.081243 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896c119d-c988-4954-8606-72f0de8944cf-istiod-ca-cert\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:03.081316 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.081264 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-workload-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:03.081316 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.081281 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896c119d-c988-4954-8606-72f0de8944cf-istio-podinfo\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:03.081460 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.081315 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/896c119d-c988-4954-8606-72f0de8944cf-istio-token\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:03.081616 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.081591 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-credential-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:03.081819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.081620 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzn96\" (UniqueName: \"kubernetes.io/projected/896c119d-c988-4954-8606-72f0de8944cf-kube-api-access-nzn96\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:03.081936 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.081891 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-istio-data\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:03.081936 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.081638 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-workload-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:03.082042 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.081723 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-workload-certs\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:03.082042 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.082030 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896c119d-c988-4954-8606-72f0de8944cf-istiod-ca-cert\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:03.082307 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.082282 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-istio-data\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:03.083556 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.083534 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-istio-envoy\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:03.083758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.083736 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896c119d-c988-4954-8606-72f0de8944cf-istio-podinfo\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:03.093158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.093138 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzn96\" (UniqueName: \"kubernetes.io/projected/896c119d-c988-4954-8606-72f0de8944cf-kube-api-access-nzn96\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:03.093272 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.093207 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/896c119d-c988-4954-8606-72f0de8944cf-istio-token\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:03.201877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.201795 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:03.595760 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.576994 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm"] Nov 03 20:36:03.595951 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:36:03.595805 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896c119d_c988_4954_8606_72f0de8944cf.slice/crio-4081a1dd2fe8268ac5c31286bc25269fc5b237bcdc5b2ba48213f729201a653d WatchSource:0}: Error finding container 4081a1dd2fe8268ac5c31286bc25269fc5b237bcdc5b2ba48213f729201a653d: Status 404 returned error can't find the container with id 4081a1dd2fe8268ac5c31286bc25269fc5b237bcdc5b2ba48213f729201a653d Nov 03 20:36:03.617275 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.615020 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:36:03.617275 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.615135 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:36:03.617275 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.615193 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:36:03.747607 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.747569 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" event={"ID":"896c119d-c988-4954-8606-72f0de8944cf","Type":"ContainerStarted","Data":"8eadfd624b310d22690f66e7e83531449a5166ef3ff1887788a0daf43a33c2bb"} Nov 03 20:36:03.747950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.747631 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" event={"ID":"896c119d-c988-4954-8606-72f0de8944cf","Type":"ContainerStarted","Data":"4081a1dd2fe8268ac5c31286bc25269fc5b237bcdc5b2ba48213f729201a653d"} Nov 03 20:36:03.776124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:03.775580 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" podStartSLOduration=1.7755671259999999 podStartE2EDuration="1.775567126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:03.770931677 +0000 UTC m=+3558.028728946" watchObservedRunningTime="2026-11-03 20:36:03.775567126 +0000 UTC m=+3558.033364387" Nov 03 20:36:04.201996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:04.201965 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:04.202919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:04.202895 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused" start-of-body= Nov 03 20:36:04.202978 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:04.202948 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" podUID="896c119d-c988-4954-8606-72f0de8944cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused" Nov 03 20:36:05.202810 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:05.202776 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused" start-of-body= Nov 03 20:36:05.203196 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:05.202832 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" podUID="896c119d-c988-4954-8606-72f0de8944cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused" Nov 03 20:36:06.206020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:06.205984 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused" start-of-body= Nov 03 20:36:06.206640 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:06.206589 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" podUID="896c119d-c988-4954-8606-72f0de8944cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused" Nov 03 20:36:07.202529 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:07.202484 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused" start-of-body= Nov 03 20:36:07.202733 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:07.202549 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" podUID="896c119d-c988-4954-8606-72f0de8944cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused" Nov 03 20:36:08.202458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:08.202410 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused" start-of-body= Nov 03 20:36:08.202895 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:08.202484 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" podUID="896c119d-c988-4954-8606-72f0de8944cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused" Nov 03 20:36:08.330142 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:08.330104 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf"] Nov 03 20:36:08.330363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:08.330339 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" containerID="cri-o://838b620ee70f5ced5fecf537721d400be59c4cfe985b56a65503fe7e0a658ced" gracePeriod=30 Nov 03 20:36:09.205684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:09.204019 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused" start-of-body= Nov 03 20:36:09.205684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:09.204079 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" podUID="896c119d-c988-4954-8606-72f0de8944cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused" Nov 03 20:36:10.202698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:10.202647 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused" start-of-body= Nov 03 20:36:10.202908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:10.202742 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" podUID="896c119d-c988-4954-8606-72f0de8944cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused" Nov 03 20:36:10.627876 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:36:10.627833 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:36:11.207420 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:11.207384 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:11.207889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:11.207870 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:11.208529 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:11.208508 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:36:11.958266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:11.958227 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4"] Nov 03 20:36:11.958634 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:11.958532 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" podUID="6752fea6-4adc-47e6-858c-101dcb6d6a84" containerName="istio-proxy" containerID="cri-o://aa92feed8292653704478e650971480d864ee8d63ab7879692555ed5a5ac1b97" gracePeriod=30 Nov 03 20:36:12.402964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:12.402926 5059 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:36:12.403121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:12.402984 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:36:13.600766 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.600712 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:36:13.674393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.674352 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-workload-socket\") pod \"3eda42a5-5b61-4b34-8447-8767ed097200\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " Nov 03 20:36:13.674559 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.674412 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-workload-certs\") pod \"3eda42a5-5b61-4b34-8447-8767ed097200\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " Nov 03 20:36:13.674559 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.674443 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eda42a5-5b61-4b34-8447-8767ed097200-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eda42a5-5b61-4b34-8447-8767ed097200-istiod-ca-cert\") pod \"3eda42a5-5b61-4b34-8447-8767ed097200\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " Nov 03 20:36:13.674559 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.674462 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eda42a5-5b61-4b34-8447-8767ed097200-istio-token\" (UniqueName: \"kubernetes.io/projected/3eda42a5-5b61-4b34-8447-8767ed097200-istio-token\") pod \"3eda42a5-5b61-4b34-8447-8767ed097200\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " Nov 03 20:36:13.674559 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.674495 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3eda42a5-5b61-4b34-8447-8767ed097200-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eda42a5-5b61-4b34-8447-8767ed097200-istio-podinfo\") pod \"3eda42a5-5b61-4b34-8447-8767ed097200\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " Nov 03 20:36:13.674819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.674590 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-istio-data\") pod \"3eda42a5-5b61-4b34-8447-8767ed097200\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " Nov 03 20:36:13.674819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.674625 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-istio-envoy\") pod \"3eda42a5-5b61-4b34-8447-8767ed097200\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " Nov 03 20:36:13.674819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.674706 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-workload-socket" pod "3eda42a5-5b61-4b34-8447-8767ed097200" (UID: "3eda42a5-5b61-4b34-8447-8767ed097200"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:13.674819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.674739 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eda42a5-5b61-4b34-8447-8767ed097200-kube-api-access-n7slk\" (UniqueName: \"kubernetes.io/projected/3eda42a5-5b61-4b34-8447-8767ed097200-kube-api-access-n7slk\") pod \"3eda42a5-5b61-4b34-8447-8767ed097200\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " Nov 03 20:36:13.674819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.674794 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-credential-socket\") pod \"3eda42a5-5b61-4b34-8447-8767ed097200\" (UID: \"3eda42a5-5b61-4b34-8447-8767ed097200\") " Nov 03 20:36:13.675074 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.674949 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-workload-certs" pod "3eda42a5-5b61-4b34-8447-8767ed097200" (UID: "3eda42a5-5b61-4b34-8447-8767ed097200"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:13.675074 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.675031 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eda42a5-5b61-4b34-8447-8767ed097200-istiod-ca-cert" pod "3eda42a5-5b61-4b34-8447-8767ed097200" (UID: "3eda42a5-5b61-4b34-8447-8767ed097200"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:13.675265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.675244 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-credential-socket" pod "3eda42a5-5b61-4b34-8447-8767ed097200" (UID: "3eda42a5-5b61-4b34-8447-8767ed097200"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:13.675528 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.675507 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-istio-data" pod "3eda42a5-5b61-4b34-8447-8767ed097200" (UID: "3eda42a5-5b61-4b34-8447-8767ed097200"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:13.675603 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.675572 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:13.675603 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.675593 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:13.675757 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.675608 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:13.675757 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.675623 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eda42a5-5b61-4b34-8447-8767ed097200-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:13.677596 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.677559 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3eda42a5-5b61-4b34-8447-8767ed097200-istio-podinfo" pod "3eda42a5-5b61-4b34-8447-8767ed097200" (UID: "3eda42a5-5b61-4b34-8447-8767ed097200"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:36:13.677785 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.677760 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eda42a5-5b61-4b34-8447-8767ed097200-kube-api-access-n7slk" pod "3eda42a5-5b61-4b34-8447-8767ed097200" (UID: "3eda42a5-5b61-4b34-8447-8767ed097200"). InnerVolumeSpecName "kube-api-access-n7slk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:13.677926 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.677901 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-istio-envoy" pod "3eda42a5-5b61-4b34-8447-8767ed097200" (UID: "3eda42a5-5b61-4b34-8447-8767ed097200"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:13.677993 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.677934 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eda42a5-5b61-4b34-8447-8767ed097200-istio-token" pod "3eda42a5-5b61-4b34-8447-8767ed097200" (UID: "3eda42a5-5b61-4b34-8447-8767ed097200"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:13.777341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.777312 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eda42a5-5b61-4b34-8447-8767ed097200-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:13.777341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.777339 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eda42a5-5b61-4b34-8447-8767ed097200-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:13.777503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.777349 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:13.777503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.777358 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eda42a5-5b61-4b34-8447-8767ed097200-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:13.777503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.777367 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7slk\" (UniqueName: \"kubernetes.io/projected/3eda42a5-5b61-4b34-8447-8767ed097200-kube-api-access-n7slk\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:13.880628 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.880517 5059 generic.go:356] "Generic (PLEG): container finished" podID="3eda42a5-5b61-4b34-8447-8767ed097200" containerID="838b620ee70f5ced5fecf537721d400be59c4cfe985b56a65503fe7e0a658ced" exitCode=0 Nov 03 20:36:13.880838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.880765 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" Nov 03 20:36:13.881836 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.881801 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" event={"ID":"3eda42a5-5b61-4b34-8447-8767ed097200","Type":"ContainerDied","Data":"838b620ee70f5ced5fecf537721d400be59c4cfe985b56a65503fe7e0a658ced"} Nov 03 20:36:13.881976 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.881860 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf" event={"ID":"3eda42a5-5b61-4b34-8447-8767ed097200","Type":"ContainerDied","Data":"5b8c83ec24a2946aa3e8001a3b660835c9b3e7c2ec1dfc3bc8aa2c6464424b71"} Nov 03 20:36:13.881976 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.881881 5059 scope.go:122] "RemoveContainer" containerID="838b620ee70f5ced5fecf537721d400be59c4cfe985b56a65503fe7e0a658ced" Nov 03 20:36:13.894771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.894753 5059 scope.go:122] "RemoveContainer" containerID="838b620ee70f5ced5fecf537721d400be59c4cfe985b56a65503fe7e0a658ced" Nov 03 20:36:13.895048 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:36:13.895026 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"838b620ee70f5ced5fecf537721d400be59c4cfe985b56a65503fe7e0a658ced\": container with ID starting with 838b620ee70f5ced5fecf537721d400be59c4cfe985b56a65503fe7e0a658ced not found: ID does not exist" containerID="838b620ee70f5ced5fecf537721d400be59c4cfe985b56a65503fe7e0a658ced" Nov 03 20:36:13.895096 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.895059 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838b620ee70f5ced5fecf537721d400be59c4cfe985b56a65503fe7e0a658ced"} err="failed to get container status \"838b620ee70f5ced5fecf537721d400be59c4cfe985b56a65503fe7e0a658ced\": rpc error: code = NotFound desc = could not find container \"838b620ee70f5ced5fecf537721d400be59c4cfe985b56a65503fe7e0a658ced\": container with ID starting with 838b620ee70f5ced5fecf537721d400be59c4cfe985b56a65503fe7e0a658ced not found: ID does not exist" Nov 03 20:36:13.911621 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.911582 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf"] Nov 03 20:36:13.915538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:13.915512 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-qznmf"] Nov 03 20:36:15.857846 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:15.857758 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" path="/var/lib/kubelet/pods/3eda42a5-5b61-4b34-8447-8767ed097200/volumes" Nov 03 20:36:17.252025 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.252001 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:36:17.342035 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.342007 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-podinfo\") pod \"6752fea6-4adc-47e6-858c-101dcb6d6a84\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " Nov 03 20:36:17.342200 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.342089 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-envoy\") pod \"6752fea6-4adc-47e6-858c-101dcb6d6a84\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " Nov 03 20:36:17.342200 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.342182 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6752fea6-4adc-47e6-858c-101dcb6d6a84-kube-api-access-qq8k2\" (UniqueName: \"kubernetes.io/projected/6752fea6-4adc-47e6-858c-101dcb6d6a84-kube-api-access-qq8k2\") pod \"6752fea6-4adc-47e6-858c-101dcb6d6a84\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " Nov 03 20:36:17.342337 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.342227 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-data\") pod \"6752fea6-4adc-47e6-858c-101dcb6d6a84\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " Nov 03 20:36:17.342396 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.342358 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-workload-socket\") pod \"6752fea6-4adc-47e6-858c-101dcb6d6a84\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " Nov 03 20:36:17.342449 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.342437 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6752fea6-4adc-47e6-858c-101dcb6d6a84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6752fea6-4adc-47e6-858c-101dcb6d6a84-istiod-ca-cert\") pod \"6752fea6-4adc-47e6-858c-101dcb6d6a84\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " Nov 03 20:36:17.342508 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.342479 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-data" pod "6752fea6-4adc-47e6-858c-101dcb6d6a84" (UID: "6752fea6-4adc-47e6-858c-101dcb6d6a84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:17.342574 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.342503 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-token\" (UniqueName: \"kubernetes.io/projected/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-token\") pod \"6752fea6-4adc-47e6-858c-101dcb6d6a84\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " Nov 03 20:36:17.342648 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.342616 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-workload-socket" pod "6752fea6-4adc-47e6-858c-101dcb6d6a84" (UID: "6752fea6-4adc-47e6-858c-101dcb6d6a84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:17.342755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.342649 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-credential-socket\") pod \"6752fea6-4adc-47e6-858c-101dcb6d6a84\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " Nov 03 20:36:17.342755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.342723 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-workload-certs\") pod \"6752fea6-4adc-47e6-858c-101dcb6d6a84\" (UID: \"6752fea6-4adc-47e6-858c-101dcb6d6a84\") " Nov 03 20:36:17.342935 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.342910 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-credential-socket" pod "6752fea6-4adc-47e6-858c-101dcb6d6a84" (UID: "6752fea6-4adc-47e6-858c-101dcb6d6a84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:17.343007 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.342930 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6752fea6-4adc-47e6-858c-101dcb6d6a84-istiod-ca-cert" pod "6752fea6-4adc-47e6-858c-101dcb6d6a84" (UID: "6752fea6-4adc-47e6-858c-101dcb6d6a84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:17.343007 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.342928 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-workload-certs" pod "6752fea6-4adc-47e6-858c-101dcb6d6a84" (UID: "6752fea6-4adc-47e6-858c-101dcb6d6a84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:17.343437 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.343410 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6752fea6-4adc-47e6-858c-101dcb6d6a84-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:17.343519 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.343445 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:17.343519 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.343461 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:17.343519 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.343475 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:17.343519 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.343489 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:17.344459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.344431 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-envoy" pod "6752fea6-4adc-47e6-858c-101dcb6d6a84" (UID: "6752fea6-4adc-47e6-858c-101dcb6d6a84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:17.344572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.344546 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-podinfo" pod "6752fea6-4adc-47e6-858c-101dcb6d6a84" (UID: "6752fea6-4adc-47e6-858c-101dcb6d6a84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:36:17.344633 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.344551 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6752fea6-4adc-47e6-858c-101dcb6d6a84-kube-api-access-qq8k2" pod "6752fea6-4adc-47e6-858c-101dcb6d6a84" (UID: "6752fea6-4adc-47e6-858c-101dcb6d6a84"). InnerVolumeSpecName "kube-api-access-qq8k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:17.344756 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.344737 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-token" pod "6752fea6-4adc-47e6-858c-101dcb6d6a84" (UID: "6752fea6-4adc-47e6-858c-101dcb6d6a84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:17.444915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.444874 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:17.444915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.444915 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq8k2\" (UniqueName: \"kubernetes.io/projected/6752fea6-4adc-47e6-858c-101dcb6d6a84-kube-api-access-qq8k2\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:17.445109 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.444932 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:17.445109 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.444946 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6752fea6-4adc-47e6-858c-101dcb6d6a84-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:17.936686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.934230 5059 generic.go:356] "Generic (PLEG): container finished" podID="6752fea6-4adc-47e6-858c-101dcb6d6a84" containerID="aa92feed8292653704478e650971480d864ee8d63ab7879692555ed5a5ac1b97" exitCode=0 Nov 03 20:36:17.936686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.934325 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" event={"ID":"6752fea6-4adc-47e6-858c-101dcb6d6a84","Type":"ContainerDied","Data":"aa92feed8292653704478e650971480d864ee8d63ab7879692555ed5a5ac1b97"} Nov 03 20:36:17.936686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.934378 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" event={"ID":"6752fea6-4adc-47e6-858c-101dcb6d6a84","Type":"ContainerDied","Data":"19ab4abf491c7bd264adde0c19aaf80256930441969660b13f973717c3f387f6"} Nov 03 20:36:17.936686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.934406 5059 scope.go:122] "RemoveContainer" containerID="aa92feed8292653704478e650971480d864ee8d63ab7879692555ed5a5ac1b97" Nov 03 20:36:17.936686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.935173 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4" Nov 03 20:36:17.949758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.949734 5059 scope.go:122] "RemoveContainer" containerID="aa92feed8292653704478e650971480d864ee8d63ab7879692555ed5a5ac1b97" Nov 03 20:36:17.950106 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:36:17.950077 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa92feed8292653704478e650971480d864ee8d63ab7879692555ed5a5ac1b97\": container with ID starting with aa92feed8292653704478e650971480d864ee8d63ab7879692555ed5a5ac1b97 not found: ID does not exist" containerID="aa92feed8292653704478e650971480d864ee8d63ab7879692555ed5a5ac1b97" Nov 03 20:36:17.950192 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.950122 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa92feed8292653704478e650971480d864ee8d63ab7879692555ed5a5ac1b97"} err="failed to get container status \"aa92feed8292653704478e650971480d864ee8d63ab7879692555ed5a5ac1b97\": rpc error: code = NotFound desc = could not find container \"aa92feed8292653704478e650971480d864ee8d63ab7879692555ed5a5ac1b97\": container with ID starting with aa92feed8292653704478e650971480d864ee8d63ab7879692555ed5a5ac1b97 not found: ID does not exist" Nov 03 20:36:17.959771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.959747 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4"] Nov 03 20:36:17.964778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:17.964750 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-8j7v4"] Nov 03 20:36:19.858619 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:19.858584 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6752fea6-4adc-47e6-858c-101dcb6d6a84" path="/var/lib/kubelet/pods/6752fea6-4adc-47e6-858c-101dcb6d6a84/volumes" Nov 03 20:36:22.832020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:22.831984 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b792ade-c6v6j"] Nov 03 20:36:22.832533 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:22.832520 5059 cpu_manager.go:414] "containerMap: removing container" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" Nov 03 20:36:22.832587 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:22.832535 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" Nov 03 20:36:22.832587 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:22.832571 5059 cpu_manager.go:414] "containerMap: removing container" podUID="6752fea6-4adc-47e6-858c-101dcb6d6a84" containerName="istio-proxy" Nov 03 20:36:22.832587 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:22.832578 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6752fea6-4adc-47e6-858c-101dcb6d6a84" containerName="istio-proxy" Nov 03 20:36:22.832697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:22.832684 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eda42a5-5b61-4b34-8447-8767ed097200" containerName="istio-proxy" Nov 03 20:36:22.832697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:22.832697 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6752fea6-4adc-47e6-858c-101dcb6d6a84" containerName="istio-proxy" Nov 03 20:36:22.836172 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:22.836153 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b792ade-c6v6j" Nov 03 20:36:22.845698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:22.842177 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:36:22.845698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:22.842205 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:36:22.845698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:22.845272 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b792ade-c6v6j"] Nov 03 20:36:22.916337 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:22.916298 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd77x\" (UniqueName: \"kubernetes.io/projected/e531663b-ff8f-426c-b4c4-362049ae9294-kube-api-access-zd77x\") pod \"maas-api-revoke-sub-2b792ade-c6v6j\" (UID: \"e531663b-ff8f-426c-b4c4-362049ae9294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b792ade-c6v6j" Nov 03 20:36:22.916497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:22.916371 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e531663b-ff8f-426c-b4c4-362049ae9294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b792ade-c6v6j\" (UID: \"e531663b-ff8f-426c-b4c4-362049ae9294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b792ade-c6v6j" Nov 03 20:36:22.916539 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:22.916505 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e531663b-ff8f-426c-b4c4-362049ae9294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b792ade-c6v6j\" (UID: \"e531663b-ff8f-426c-b4c4-362049ae9294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b792ade-c6v6j" Nov 03 20:36:23.018010 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:23.017975 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd77x\" (UniqueName: \"kubernetes.io/projected/e531663b-ff8f-426c-b4c4-362049ae9294-kube-api-access-zd77x\") pod \"maas-api-revoke-sub-2b792ade-c6v6j\" (UID: \"e531663b-ff8f-426c-b4c4-362049ae9294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b792ade-c6v6j" Nov 03 20:36:23.018171 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:23.018040 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e531663b-ff8f-426c-b4c4-362049ae9294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b792ade-c6v6j\" (UID: \"e531663b-ff8f-426c-b4c4-362049ae9294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b792ade-c6v6j" Nov 03 20:36:23.018171 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:23.018068 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e531663b-ff8f-426c-b4c4-362049ae9294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b792ade-c6v6j\" (UID: \"e531663b-ff8f-426c-b4c4-362049ae9294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b792ade-c6v6j" Nov 03 20:36:23.018743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:23.018710 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e531663b-ff8f-426c-b4c4-362049ae9294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b792ade-c6v6j\" (UID: \"e531663b-ff8f-426c-b4c4-362049ae9294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b792ade-c6v6j" Nov 03 20:36:23.020507 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:23.020480 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e531663b-ff8f-426c-b4c4-362049ae9294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b792ade-c6v6j\" (UID: \"e531663b-ff8f-426c-b4c4-362049ae9294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b792ade-c6v6j" Nov 03 20:36:23.031166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:23.031142 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd77x\" (UniqueName: \"kubernetes.io/projected/e531663b-ff8f-426c-b4c4-362049ae9294-kube-api-access-zd77x\") pod \"maas-api-revoke-sub-2b792ade-c6v6j\" (UID: \"e531663b-ff8f-426c-b4c4-362049ae9294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b792ade-c6v6j" Nov 03 20:36:23.151247 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:23.151159 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b792ade-c6v6j" Nov 03 20:36:23.435388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:23.435360 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b792ade-c6v6j"] Nov 03 20:36:23.435860 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:36:23.435832 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode531663b_ff8f_426c_b4c4_362049ae9294.slice/crio-06a2411b7830277f460dc80c91f0a51fd2bbf45ef921358303470bc87db5a096 WatchSource:0}: Error finding container 06a2411b7830277f460dc80c91f0a51fd2bbf45ef921358303470bc87db5a096: Status 404 returned error can't find the container with id 06a2411b7830277f460dc80c91f0a51fd2bbf45ef921358303470bc87db5a096 Nov 03 20:36:23.972438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:23.972405 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b792ade-c6v6j" event={"ID":"e531663b-ff8f-426c-b4c4-362049ae9294","Type":"ContainerStarted","Data":"2d95d4d5964dcdf538400f2dd8cdba6b546459884cd613156a703f0e2e134293"} Nov 03 20:36:23.972813 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:23.972465 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b792ade-c6v6j" event={"ID":"e531663b-ff8f-426c-b4c4-362049ae9294","Type":"ContainerStarted","Data":"06a2411b7830277f460dc80c91f0a51fd2bbf45ef921358303470bc87db5a096"} Nov 03 20:36:23.992346 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:23.992302 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b792ade-c6v6j" podStartSLOduration=1.99229067 podStartE2EDuration="1.99229067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:23.989975704 +0000 UTC m=+3578.247772962" watchObservedRunningTime="2026-11-03 20:36:23.99229067 +0000 UTC m=+3578.250087960" Nov 03 20:36:24.979972 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:24.979935 5059 generic.go:356] "Generic (PLEG): container finished" podID="e531663b-ff8f-426c-b4c4-362049ae9294" containerID="2d95d4d5964dcdf538400f2dd8cdba6b546459884cd613156a703f0e2e134293" exitCode=0 Nov 03 20:36:24.980427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:24.979968 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b792ade-c6v6j" event={"ID":"e531663b-ff8f-426c-b4c4-362049ae9294","Type":"ContainerDied","Data":"2d95d4d5964dcdf538400f2dd8cdba6b546459884cd613156a703f0e2e134293"} Nov 03 20:36:26.179328 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:26.179304 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b792ade-c6v6j" Nov 03 20:36:26.269063 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:26.268981 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e531663b-ff8f-426c-b4c4-362049ae9294-kube-api-access-zd77x\" (UniqueName: \"kubernetes.io/projected/e531663b-ff8f-426c-b4c4-362049ae9294-kube-api-access-zd77x\") pod \"e531663b-ff8f-426c-b4c4-362049ae9294\" (UID: \"e531663b-ff8f-426c-b4c4-362049ae9294\") " Nov 03 20:36:26.269219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:26.269082 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e531663b-ff8f-426c-b4c4-362049ae9294-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e531663b-ff8f-426c-b4c4-362049ae9294-maas-api-xks-ca\") pod \"e531663b-ff8f-426c-b4c4-362049ae9294\" (UID: \"e531663b-ff8f-426c-b4c4-362049ae9294\") " Nov 03 20:36:26.269219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:26.269176 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e531663b-ff8f-426c-b4c4-362049ae9294-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e531663b-ff8f-426c-b4c4-362049ae9294-maas-api-openshift-ca\") pod \"e531663b-ff8f-426c-b4c4-362049ae9294\" (UID: \"e531663b-ff8f-426c-b4c4-362049ae9294\") " Nov 03 20:36:26.269582 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:26.269551 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e531663b-ff8f-426c-b4c4-362049ae9294-maas-api-openshift-ca" pod "e531663b-ff8f-426c-b4c4-362049ae9294" (UID: "e531663b-ff8f-426c-b4c4-362049ae9294"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:26.269849 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:26.269830 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e531663b-ff8f-426c-b4c4-362049ae9294-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.271288 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:26.271266 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e531663b-ff8f-426c-b4c4-362049ae9294-kube-api-access-zd77x" pod "e531663b-ff8f-426c-b4c4-362049ae9294" (UID: "e531663b-ff8f-426c-b4c4-362049ae9294"). InnerVolumeSpecName "kube-api-access-zd77x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:26.271383 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:26.271269 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e531663b-ff8f-426c-b4c4-362049ae9294-maas-api-xks-ca" pod "e531663b-ff8f-426c-b4c4-362049ae9294" (UID: "e531663b-ff8f-426c-b4c4-362049ae9294"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:36:26.371123 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:26.371088 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e531663b-ff8f-426c-b4c4-362049ae9294-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.371123 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:26.371119 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd77x\" (UniqueName: \"kubernetes.io/projected/e531663b-ff8f-426c-b4c4-362049ae9294-kube-api-access-zd77x\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.993480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:26.993448 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b792ade-c6v6j" Nov 03 20:36:26.993691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:26.993456 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b792ade-c6v6j" event={"ID":"e531663b-ff8f-426c-b4c4-362049ae9294","Type":"ContainerDied","Data":"06a2411b7830277f460dc80c91f0a51fd2bbf45ef921358303470bc87db5a096"} Nov 03 20:36:26.993691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:26.993570 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06a2411b7830277f460dc80c91f0a51fd2bbf45ef921358303470bc87db5a096" Nov 03 20:36:33.104093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:33.104055 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8"] Nov 03 20:36:33.117610 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:33.117578 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h"] Nov 03 20:36:33.878616 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:33.878584 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddfa3748-d2b62"] Nov 03 20:36:33.879145 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:33.879127 5059 cpu_manager.go:414] "containerMap: removing container" podUID="e531663b-ff8f-426c-b4c4-362049ae9294" containerName="revoke-keys" Nov 03 20:36:33.879192 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:33.879147 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e531663b-ff8f-426c-b4c4-362049ae9294" containerName="revoke-keys" Nov 03 20:36:33.879264 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:33.879254 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e531663b-ff8f-426c-b4c4-362049ae9294" containerName="revoke-keys" Nov 03 20:36:33.882704 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:33.882688 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfa3748-d2b62" Nov 03 20:36:33.888290 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:33.888253 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:36:33.888415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:33.888402 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:36:33.891818 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:33.891796 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddfa3748-d2b62"] Nov 03 20:36:33.897154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:33.897129 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e87c74-5443-4f7e-b782-d46308c58b9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddfa3748-d2b62\" (UID: \"c9e87c74-5443-4f7e-b782-d46308c58b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfa3748-d2b62" Nov 03 20:36:33.897463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:33.897442 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58nw9\" (UniqueName: \"kubernetes.io/projected/c9e87c74-5443-4f7e-b782-d46308c58b9c-kube-api-access-58nw9\") pod \"maas-api-revoke-sub-ddfa3748-d2b62\" (UID: \"c9e87c74-5443-4f7e-b782-d46308c58b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfa3748-d2b62" Nov 03 20:36:33.897568 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:33.897539 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e87c74-5443-4f7e-b782-d46308c58b9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddfa3748-d2b62\" (UID: \"c9e87c74-5443-4f7e-b782-d46308c58b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfa3748-d2b62" Nov 03 20:36:33.998731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:33.998697 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e87c74-5443-4f7e-b782-d46308c58b9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddfa3748-d2b62\" (UID: \"c9e87c74-5443-4f7e-b782-d46308c58b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfa3748-d2b62" Nov 03 20:36:33.998884 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:33.998858 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58nw9\" (UniqueName: \"kubernetes.io/projected/c9e87c74-5443-4f7e-b782-d46308c58b9c-kube-api-access-58nw9\") pod \"maas-api-revoke-sub-ddfa3748-d2b62\" (UID: \"c9e87c74-5443-4f7e-b782-d46308c58b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfa3748-d2b62" Nov 03 20:36:33.998951 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:33.998889 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e87c74-5443-4f7e-b782-d46308c58b9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddfa3748-d2b62\" (UID: \"c9e87c74-5443-4f7e-b782-d46308c58b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfa3748-d2b62" Nov 03 20:36:33.999576 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:33.999551 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e87c74-5443-4f7e-b782-d46308c58b9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddfa3748-d2b62\" (UID: \"c9e87c74-5443-4f7e-b782-d46308c58b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfa3748-d2b62" Nov 03 20:36:34.001097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:34.001074 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e87c74-5443-4f7e-b782-d46308c58b9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddfa3748-d2b62\" (UID: \"c9e87c74-5443-4f7e-b782-d46308c58b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfa3748-d2b62" Nov 03 20:36:34.011677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:34.011641 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58nw9\" (UniqueName: \"kubernetes.io/projected/c9e87c74-5443-4f7e-b782-d46308c58b9c-kube-api-access-58nw9\") pod \"maas-api-revoke-sub-ddfa3748-d2b62\" (UID: \"c9e87c74-5443-4f7e-b782-d46308c58b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfa3748-d2b62" Nov 03 20:36:34.197678 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:34.197591 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfa3748-d2b62" Nov 03 20:36:34.476456 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:34.476428 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddfa3748-d2b62"] Nov 03 20:36:34.477171 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:36:34.477145 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e87c74_5443_4f7e_b782_d46308c58b9c.slice/crio-51c74478791fdd743add262928c6b911b60a6fd3f6421420916c217983cb4f62 WatchSource:0}: Error finding container 51c74478791fdd743add262928c6b911b60a6fd3f6421420916c217983cb4f62: Status 404 returned error can't find the container with id 51c74478791fdd743add262928c6b911b60a6fd3f6421420916c217983cb4f62 Nov 03 20:36:35.043625 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:35.043539 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfa3748-d2b62" event={"ID":"c9e87c74-5443-4f7e-b782-d46308c58b9c","Type":"ContainerStarted","Data":"ffc657b6b1ccc18cb02ede62e23f821eed2b5cedb21fa8048d678ad100ea5dae"} Nov 03 20:36:35.043625 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:35.043587 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfa3748-d2b62" event={"ID":"c9e87c74-5443-4f7e-b782-d46308c58b9c","Type":"ContainerStarted","Data":"51c74478791fdd743add262928c6b911b60a6fd3f6421420916c217983cb4f62"} Nov 03 20:36:35.062796 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:35.062742 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfa3748-d2b62" podStartSLOduration=2.062728893 podStartE2EDuration="2.062728893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:35.06111948 +0000 UTC m=+3589.318916742" watchObservedRunningTime="2026-11-03 20:36:35.062728893 +0000 UTC m=+3589.320526149" Nov 03 20:36:36.051500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:36.051458 5059 generic.go:356] "Generic (PLEG): container finished" podID="c9e87c74-5443-4f7e-b782-d46308c58b9c" containerID="ffc657b6b1ccc18cb02ede62e23f821eed2b5cedb21fa8048d678ad100ea5dae" exitCode=0 Nov 03 20:36:36.051968 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:36.051534 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfa3748-d2b62" event={"ID":"c9e87c74-5443-4f7e-b782-d46308c58b9c","Type":"ContainerDied","Data":"ffc657b6b1ccc18cb02ede62e23f821eed2b5cedb21fa8048d678ad100ea5dae"} Nov 03 20:36:37.195962 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:37.195935 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfa3748-d2b62" Nov 03 20:36:37.246084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:37.246047 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9e87c74-5443-4f7e-b782-d46308c58b9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e87c74-5443-4f7e-b782-d46308c58b9c-maas-api-openshift-ca\") pod \"c9e87c74-5443-4f7e-b782-d46308c58b9c\" (UID: \"c9e87c74-5443-4f7e-b782-d46308c58b9c\") " Nov 03 20:36:37.246248 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:37.246109 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9e87c74-5443-4f7e-b782-d46308c58b9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e87c74-5443-4f7e-b782-d46308c58b9c-maas-api-xks-ca\") pod \"c9e87c74-5443-4f7e-b782-d46308c58b9c\" (UID: \"c9e87c74-5443-4f7e-b782-d46308c58b9c\") " Nov 03 20:36:37.246368 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:37.246344 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e87c74-5443-4f7e-b782-d46308c58b9c-kube-api-access-58nw9\" (UniqueName: \"kubernetes.io/projected/c9e87c74-5443-4f7e-b782-d46308c58b9c-kube-api-access-58nw9\") pod \"c9e87c74-5443-4f7e-b782-d46308c58b9c\" (UID: \"c9e87c74-5443-4f7e-b782-d46308c58b9c\") " Nov 03 20:36:37.246475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:37.246448 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e87c74-5443-4f7e-b782-d46308c58b9c-maas-api-openshift-ca" pod "c9e87c74-5443-4f7e-b782-d46308c58b9c" (UID: "c9e87c74-5443-4f7e-b782-d46308c58b9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:37.246974 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:37.246954 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e87c74-5443-4f7e-b782-d46308c58b9c-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:37.248376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:37.248354 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e87c74-5443-4f7e-b782-d46308c58b9c-kube-api-access-58nw9" pod "c9e87c74-5443-4f7e-b782-d46308c58b9c" (UID: "c9e87c74-5443-4f7e-b782-d46308c58b9c"). InnerVolumeSpecName "kube-api-access-58nw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:37.248620 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:37.248605 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9e87c74-5443-4f7e-b782-d46308c58b9c-maas-api-xks-ca" pod "c9e87c74-5443-4f7e-b782-d46308c58b9c" (UID: "c9e87c74-5443-4f7e-b782-d46308c58b9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:36:37.348333 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:37.348306 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e87c74-5443-4f7e-b782-d46308c58b9c-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:37.348333 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:37.348333 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58nw9\" (UniqueName: \"kubernetes.io/projected/c9e87c74-5443-4f7e-b782-d46308c58b9c-kube-api-access-58nw9\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:38.065620 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:38.065591 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfa3748-d2b62" Nov 03 20:36:38.065792 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:38.065641 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfa3748-d2b62" event={"ID":"c9e87c74-5443-4f7e-b782-d46308c58b9c","Type":"ContainerDied","Data":"51c74478791fdd743add262928c6b911b60a6fd3f6421420916c217983cb4f62"} Nov 03 20:36:38.065792 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:38.065708 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51c74478791fdd743add262928c6b911b60a6fd3f6421420916c217983cb4f62" Nov 03 20:36:38.104343 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:38.104308 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8" podUID="ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988" containerName="payload-processing" containerID="cri-o://314d646649bd50357a3898f7794abc5ffb6d6c3c1a641d460a6c9fc3147b1d2c" gracePeriod=25 Nov 03 20:36:38.118393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:38.118335 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h" podUID="2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1" containerName="payload-pre-processing" containerID="cri-o://d40d4645400fbdd61191d0029f2cb502ccdd5240fb067cb03ff331ea2b1ebb45" gracePeriod=25 Nov 03 20:36:38.426759 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:38.426730 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8" Nov 03 20:36:38.455995 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:38.455967 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h" Nov 03 20:36:38.483508 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:38.483098 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988-kube-api-access-4s7n4\" (UniqueName: \"kubernetes.io/projected/ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988-kube-api-access-4s7n4\") pod \"ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988\" (UID: \"ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988\") " Nov 03 20:36:38.484242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:38.483755 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988-plugins-config-volume\") pod \"ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988\" (UID: \"ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988\") " Nov 03 20:36:38.484624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:38.484444 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988-plugins-config-volume" pod "ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988" (UID: "ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:38.484624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:38.484592 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:38.485983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:38.485956 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988-kube-api-access-4s7n4" pod "ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988" (UID: "ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988"). InnerVolumeSpecName "kube-api-access-4s7n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:38.585285 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:38.585251 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1-plugins-config-volume\") pod \"2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1\" (UID: \"2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1\") " Nov 03 20:36:38.585428 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:38.585307 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1-kube-api-access-nskbm\" (UniqueName: \"kubernetes.io/projected/2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1-kube-api-access-nskbm\") pod \"2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1\" (UID: \"2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1\") " Nov 03 20:36:38.585708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:38.585694 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s7n4\" (UniqueName: \"kubernetes.io/projected/ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988-kube-api-access-4s7n4\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:38.585765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:38.585712 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1-plugins-config-volume" pod "2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1" (UID: "2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:38.587436 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:38.587417 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1-kube-api-access-nskbm" pod "2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1" (UID: "2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1"). InnerVolumeSpecName "kube-api-access-nskbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:38.686771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:38.686655 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:38.686771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:38.686718 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nskbm\" (UniqueName: \"kubernetes.io/projected/2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1-kube-api-access-nskbm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:39.073693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:39.073639 5059 generic.go:356] "Generic (PLEG): container finished" podID="ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988" containerID="314d646649bd50357a3898f7794abc5ffb6d6c3c1a641d460a6c9fc3147b1d2c" exitCode=0 Nov 03 20:36:39.073861 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:39.073695 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8" Nov 03 20:36:39.073861 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:39.073729 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8" event={"ID":"ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988","Type":"ContainerDied","Data":"314d646649bd50357a3898f7794abc5ffb6d6c3c1a641d460a6c9fc3147b1d2c"} Nov 03 20:36:39.073861 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:39.073785 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8" event={"ID":"ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988","Type":"ContainerDied","Data":"0aaedbc7dc2a65872890219273178402f55efd0bc2d6336437d7741bab1dffcb"} Nov 03 20:36:39.073861 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:39.073817 5059 scope.go:122] "RemoveContainer" containerID="314d646649bd50357a3898f7794abc5ffb6d6c3c1a641d460a6c9fc3147b1d2c" Nov 03 20:36:39.075233 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:39.075205 5059 generic.go:356] "Generic (PLEG): container finished" podID="2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1" containerID="d40d4645400fbdd61191d0029f2cb502ccdd5240fb067cb03ff331ea2b1ebb45" exitCode=0 Nov 03 20:36:39.075364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:39.075263 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h" Nov 03 20:36:39.075364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:39.075292 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h" event={"ID":"2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1","Type":"ContainerDied","Data":"d40d4645400fbdd61191d0029f2cb502ccdd5240fb067cb03ff331ea2b1ebb45"} Nov 03 20:36:39.075364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:39.075344 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h" event={"ID":"2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1","Type":"ContainerDied","Data":"b3c1602d6af6a906f6869be53ebabaedf926eb70cc1fd5c92e659b441752317a"} Nov 03 20:36:39.086119 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:39.086095 5059 scope.go:122] "RemoveContainer" containerID="314d646649bd50357a3898f7794abc5ffb6d6c3c1a641d460a6c9fc3147b1d2c" Nov 03 20:36:39.086507 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:36:39.086479 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"314d646649bd50357a3898f7794abc5ffb6d6c3c1a641d460a6c9fc3147b1d2c\": container with ID starting with 314d646649bd50357a3898f7794abc5ffb6d6c3c1a641d460a6c9fc3147b1d2c not found: ID does not exist" containerID="314d646649bd50357a3898f7794abc5ffb6d6c3c1a641d460a6c9fc3147b1d2c" Nov 03 20:36:39.086592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:39.086516 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314d646649bd50357a3898f7794abc5ffb6d6c3c1a641d460a6c9fc3147b1d2c"} err="failed to get container status \"314d646649bd50357a3898f7794abc5ffb6d6c3c1a641d460a6c9fc3147b1d2c\": rpc error: code = NotFound desc = could not find container \"314d646649bd50357a3898f7794abc5ffb6d6c3c1a641d460a6c9fc3147b1d2c\": container with ID starting with 314d646649bd50357a3898f7794abc5ffb6d6c3c1a641d460a6c9fc3147b1d2c not found: ID does not exist" Nov 03 20:36:39.086592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:39.086543 5059 scope.go:122] "RemoveContainer" containerID="d40d4645400fbdd61191d0029f2cb502ccdd5240fb067cb03ff331ea2b1ebb45" Nov 03 20:36:39.096027 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:39.096002 5059 scope.go:122] "RemoveContainer" containerID="d40d4645400fbdd61191d0029f2cb502ccdd5240fb067cb03ff331ea2b1ebb45" Nov 03 20:36:39.096313 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:36:39.096294 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d40d4645400fbdd61191d0029f2cb502ccdd5240fb067cb03ff331ea2b1ebb45\": container with ID starting with d40d4645400fbdd61191d0029f2cb502ccdd5240fb067cb03ff331ea2b1ebb45 not found: ID does not exist" containerID="d40d4645400fbdd61191d0029f2cb502ccdd5240fb067cb03ff331ea2b1ebb45" Nov 03 20:36:39.096377 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:39.096321 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d40d4645400fbdd61191d0029f2cb502ccdd5240fb067cb03ff331ea2b1ebb45"} err="failed to get container status \"d40d4645400fbdd61191d0029f2cb502ccdd5240fb067cb03ff331ea2b1ebb45\": rpc error: code = NotFound desc = could not find container \"d40d4645400fbdd61191d0029f2cb502ccdd5240fb067cb03ff331ea2b1ebb45\": container with ID starting with d40d4645400fbdd61191d0029f2cb502ccdd5240fb067cb03ff331ea2b1ebb45 not found: ID does not exist" Nov 03 20:36:39.106645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:39.106621 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8"] Nov 03 20:36:39.113852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:39.113830 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3fbb1a-755f448d4-7dhl8"] Nov 03 20:36:39.127498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:39.127468 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h"] Nov 03 20:36:39.130860 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:39.130836 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3fbb1a-66d698c4cf-r7j7h"] Nov 03 20:36:39.856862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:39.856828 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1" path="/var/lib/kubelet/pods/2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1/volumes" Nov 03 20:36:39.857264 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:39.857249 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988" path="/var/lib/kubelet/pods/ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988/volumes" Nov 03 20:36:40.604771 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:36:40.604729 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:36:47.183204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:47.183155 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj"] Nov 03 20:36:47.183762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:47.183382 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" containerID="cri-o://535c97f981aaf14f23e0940aa42d33bed0f295ee3f35cefd0fb15098b2c7793f" gracePeriod=30 Nov 03 20:36:48.037946 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.037905 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce2e52c1-nh59n"] Nov 03 20:36:48.038530 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.038509 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c9e87c74-5443-4f7e-b782-d46308c58b9c" containerName="revoke-keys" Nov 03 20:36:48.038642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.038531 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e87c74-5443-4f7e-b782-d46308c58b9c" containerName="revoke-keys" Nov 03 20:36:48.038642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.038567 5059 cpu_manager.go:414] "containerMap: removing container" podUID="2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1" containerName="payload-pre-processing" Nov 03 20:36:48.038642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.038576 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1" containerName="payload-pre-processing" Nov 03 20:36:48.038642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.038598 5059 cpu_manager.go:414] "containerMap: removing container" podUID="ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988" containerName="payload-processing" Nov 03 20:36:48.038642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.038606 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988" containerName="payload-processing" Nov 03 20:36:48.038943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.038746 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceaafbb6-31d6-4a8e-b971-8a4bd5e0b988" containerName="payload-processing" Nov 03 20:36:48.038943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.038767 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9e87c74-5443-4f7e-b782-d46308c58b9c" containerName="revoke-keys" Nov 03 20:36:48.038943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.038782 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ffc63d9-248c-4fe9-ba40-bb2f8e2f6eb1" containerName="payload-pre-processing" Nov 03 20:36:48.043716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.043695 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2e52c1-nh59n" Nov 03 20:36:48.047970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.047944 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:36:48.048362 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.048337 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:36:48.050869 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.050842 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xn98\" (UniqueName: \"kubernetes.io/projected/eb070666-da11-410c-a670-46da774feef2-kube-api-access-2xn98\") pod \"maas-api-revoke-sub-ce2e52c1-nh59n\" (UID: \"eb070666-da11-410c-a670-46da774feef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2e52c1-nh59n" Nov 03 20:36:48.051023 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.051005 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb070666-da11-410c-a670-46da774feef2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce2e52c1-nh59n\" (UID: \"eb070666-da11-410c-a670-46da774feef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2e52c1-nh59n" Nov 03 20:36:48.051086 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.051035 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb070666-da11-410c-a670-46da774feef2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce2e52c1-nh59n\" (UID: \"eb070666-da11-410c-a670-46da774feef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2e52c1-nh59n" Nov 03 20:36:48.051086 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.051044 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce2e52c1-nh59n"] Nov 03 20:36:48.152797 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.152766 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xn98\" (UniqueName: \"kubernetes.io/projected/eb070666-da11-410c-a670-46da774feef2-kube-api-access-2xn98\") pod \"maas-api-revoke-sub-ce2e52c1-nh59n\" (UID: \"eb070666-da11-410c-a670-46da774feef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2e52c1-nh59n" Nov 03 20:36:48.152926 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.152875 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb070666-da11-410c-a670-46da774feef2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce2e52c1-nh59n\" (UID: \"eb070666-da11-410c-a670-46da774feef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2e52c1-nh59n" Nov 03 20:36:48.152926 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.152902 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb070666-da11-410c-a670-46da774feef2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce2e52c1-nh59n\" (UID: \"eb070666-da11-410c-a670-46da774feef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2e52c1-nh59n" Nov 03 20:36:48.153480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.153460 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb070666-da11-410c-a670-46da774feef2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce2e52c1-nh59n\" (UID: \"eb070666-da11-410c-a670-46da774feef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2e52c1-nh59n" Nov 03 20:36:48.155342 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.155314 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb070666-da11-410c-a670-46da774feef2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce2e52c1-nh59n\" (UID: \"eb070666-da11-410c-a670-46da774feef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2e52c1-nh59n" Nov 03 20:36:48.165459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.165434 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xn98\" (UniqueName: \"kubernetes.io/projected/eb070666-da11-410c-a670-46da774feef2-kube-api-access-2xn98\") pod \"maas-api-revoke-sub-ce2e52c1-nh59n\" (UID: \"eb070666-da11-410c-a670-46da774feef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2e52c1-nh59n" Nov 03 20:36:48.360574 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.360543 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2e52c1-nh59n" Nov 03 20:36:48.644977 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:48.644942 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce2e52c1-nh59n"] Nov 03 20:36:48.646105 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:36:48.646078 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb070666_da11_410c_a670_46da774feef2.slice/crio-045deed17343e5237ec80a0bcb47557cdc1afee40bd0b8fea4f8907a94c8f531 WatchSource:0}: Error finding container 045deed17343e5237ec80a0bcb47557cdc1afee40bd0b8fea4f8907a94c8f531: Status 404 returned error can't find the container with id 045deed17343e5237ec80a0bcb47557cdc1afee40bd0b8fea4f8907a94c8f531 Nov 03 20:36:49.178092 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:49.178060 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2e52c1-nh59n" event={"ID":"eb070666-da11-410c-a670-46da774feef2","Type":"ContainerStarted","Data":"ed7c56678aaead66b76d36977ee5bbcff501f82edcc806f55e8ece2a056414d6"} Nov 03 20:36:49.178251 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:49.178113 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2e52c1-nh59n" event={"ID":"eb070666-da11-410c-a670-46da774feef2","Type":"ContainerStarted","Data":"045deed17343e5237ec80a0bcb47557cdc1afee40bd0b8fea4f8907a94c8f531"} Nov 03 20:36:49.201586 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:49.201536 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2e52c1-nh59n" podStartSLOduration=1.201521425 podStartE2EDuration="1.201521425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:49.197170081 +0000 UTC m=+3603.454967340" watchObservedRunningTime="2026-11-03 20:36:49.201521425 +0000 UTC m=+3603.459318682" Nov 03 20:36:50.186040 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:50.186001 5059 generic.go:356] "Generic (PLEG): container finished" podID="eb070666-da11-410c-a670-46da774feef2" containerID="ed7c56678aaead66b76d36977ee5bbcff501f82edcc806f55e8ece2a056414d6" exitCode=0 Nov 03 20:36:50.186806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:50.186039 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2e52c1-nh59n" event={"ID":"eb070666-da11-410c-a670-46da774feef2","Type":"ContainerDied","Data":"ed7c56678aaead66b76d36977ee5bbcff501f82edcc806f55e8ece2a056414d6"} Nov 03 20:36:51.329489 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:51.329458 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2e52c1-nh59n" Nov 03 20:36:51.401429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:51.401396 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb070666-da11-410c-a670-46da774feef2-kube-api-access-2xn98\" (UniqueName: \"kubernetes.io/projected/eb070666-da11-410c-a670-46da774feef2-kube-api-access-2xn98\") pod \"eb070666-da11-410c-a670-46da774feef2\" (UID: \"eb070666-da11-410c-a670-46da774feef2\") " Nov 03 20:36:51.401932 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:51.401527 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb070666-da11-410c-a670-46da774feef2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb070666-da11-410c-a670-46da774feef2-maas-api-xks-ca\") pod \"eb070666-da11-410c-a670-46da774feef2\" (UID: \"eb070666-da11-410c-a670-46da774feef2\") " Nov 03 20:36:51.401932 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:51.401633 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb070666-da11-410c-a670-46da774feef2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb070666-da11-410c-a670-46da774feef2-maas-api-openshift-ca\") pod \"eb070666-da11-410c-a670-46da774feef2\" (UID: \"eb070666-da11-410c-a670-46da774feef2\") " Nov 03 20:36:51.402470 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:51.402434 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb070666-da11-410c-a670-46da774feef2-maas-api-openshift-ca" pod "eb070666-da11-410c-a670-46da774feef2" (UID: "eb070666-da11-410c-a670-46da774feef2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:51.403794 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:51.403768 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb070666-da11-410c-a670-46da774feef2-kube-api-access-2xn98" pod "eb070666-da11-410c-a670-46da774feef2" (UID: "eb070666-da11-410c-a670-46da774feef2"). InnerVolumeSpecName "kube-api-access-2xn98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:51.404185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:51.404164 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb070666-da11-410c-a670-46da774feef2-maas-api-xks-ca" pod "eb070666-da11-410c-a670-46da774feef2" (UID: "eb070666-da11-410c-a670-46da774feef2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:36:51.506694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:51.504076 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb070666-da11-410c-a670-46da774feef2-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:51.506694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:51.504121 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb070666-da11-410c-a670-46da774feef2-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:51.506694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:51.504139 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xn98\" (UniqueName: \"kubernetes.io/projected/eb070666-da11-410c-a670-46da774feef2-kube-api-access-2xn98\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:52.202030 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.202004 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2e52c1-nh59n" Nov 03 20:36:52.202157 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.202028 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2e52c1-nh59n" event={"ID":"eb070666-da11-410c-a670-46da774feef2","Type":"ContainerDied","Data":"045deed17343e5237ec80a0bcb47557cdc1afee40bd0b8fea4f8907a94c8f531"} Nov 03 20:36:52.202157 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.202088 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="045deed17343e5237ec80a0bcb47557cdc1afee40bd0b8fea4f8907a94c8f531" Nov 03 20:36:52.441832 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.441809 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:36:52.513586 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.513493 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-workload-certs\") pod \"7f438514-df13-4871-9c2a-ef8e56a570c9\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " Nov 03 20:36:52.513586 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.513550 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-token\" (UniqueName: \"kubernetes.io/projected/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-token\") pod \"7f438514-df13-4871-9c2a-ef8e56a570c9\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " Nov 03 20:36:52.513834 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.513592 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-data\") pod \"7f438514-df13-4871-9c2a-ef8e56a570c9\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " Nov 03 20:36:52.513834 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.513638 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-credential-socket\") pod \"7f438514-df13-4871-9c2a-ef8e56a570c9\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " Nov 03 20:36:52.513834 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.513692 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-workload-socket\") pod \"7f438514-df13-4871-9c2a-ef8e56a570c9\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " Nov 03 20:36:52.514002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.513828 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-workload-certs" pod "7f438514-df13-4871-9c2a-ef8e56a570c9" (UID: "7f438514-df13-4871-9c2a-ef8e56a570c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:52.514002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.513847 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-podinfo\") pod \"7f438514-df13-4871-9c2a-ef8e56a570c9\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " Nov 03 20:36:52.514002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.513881 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-envoy\") pod \"7f438514-df13-4871-9c2a-ef8e56a570c9\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " Nov 03 20:36:52.514002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.513914 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f438514-df13-4871-9c2a-ef8e56a570c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f438514-df13-4871-9c2a-ef8e56a570c9-istiod-ca-cert\") pod \"7f438514-df13-4871-9c2a-ef8e56a570c9\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " Nov 03 20:36:52.514002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.513976 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f438514-df13-4871-9c2a-ef8e56a570c9-kube-api-access-t2j7f\" (UniqueName: \"kubernetes.io/projected/7f438514-df13-4871-9c2a-ef8e56a570c9-kube-api-access-t2j7f\") pod \"7f438514-df13-4871-9c2a-ef8e56a570c9\" (UID: \"7f438514-df13-4871-9c2a-ef8e56a570c9\") " Nov 03 20:36:52.514263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.514047 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-data" pod "7f438514-df13-4871-9c2a-ef8e56a570c9" (UID: "7f438514-df13-4871-9c2a-ef8e56a570c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:52.514263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.514071 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-workload-socket" pod "7f438514-df13-4871-9c2a-ef8e56a570c9" (UID: "7f438514-df13-4871-9c2a-ef8e56a570c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:52.514436 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.514412 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-credential-socket" pod "7f438514-df13-4871-9c2a-ef8e56a570c9" (UID: "7f438514-df13-4871-9c2a-ef8e56a570c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:52.514436 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.514422 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f438514-df13-4871-9c2a-ef8e56a570c9-istiod-ca-cert" pod "7f438514-df13-4871-9c2a-ef8e56a570c9" (UID: "7f438514-df13-4871-9c2a-ef8e56a570c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:52.514631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.514607 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:52.514631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.514633 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:52.514849 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.514647 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:52.514849 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.514697 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:52.514849 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.514712 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f438514-df13-4871-9c2a-ef8e56a570c9-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:52.518709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.516458 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-podinfo" pod "7f438514-df13-4871-9c2a-ef8e56a570c9" (UID: "7f438514-df13-4871-9c2a-ef8e56a570c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:36:52.518709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.516752 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-envoy" pod "7f438514-df13-4871-9c2a-ef8e56a570c9" (UID: "7f438514-df13-4871-9c2a-ef8e56a570c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:52.518709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.516791 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-token" pod "7f438514-df13-4871-9c2a-ef8e56a570c9" (UID: "7f438514-df13-4871-9c2a-ef8e56a570c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:52.518709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.516911 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f438514-df13-4871-9c2a-ef8e56a570c9-kube-api-access-t2j7f" pod "7f438514-df13-4871-9c2a-ef8e56a570c9" (UID: "7f438514-df13-4871-9c2a-ef8e56a570c9"). InnerVolumeSpecName "kube-api-access-t2j7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:52.615987 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.615957 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:52.615987 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.615985 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:52.616149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.615995 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2j7f\" (UniqueName: \"kubernetes.io/projected/7f438514-df13-4871-9c2a-ef8e56a570c9-kube-api-access-t2j7f\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:52.616149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:52.616006 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f438514-df13-4871-9c2a-ef8e56a570c9-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:53.209638 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:53.209603 5059 generic.go:356] "Generic (PLEG): container finished" podID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerID="535c97f981aaf14f23e0940aa42d33bed0f295ee3f35cefd0fb15098b2c7793f" exitCode=0 Nov 03 20:36:53.209851 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:53.209648 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" Nov 03 20:36:53.209851 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:53.209693 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" event={"ID":"7f438514-df13-4871-9c2a-ef8e56a570c9","Type":"ContainerDied","Data":"535c97f981aaf14f23e0940aa42d33bed0f295ee3f35cefd0fb15098b2c7793f"} Nov 03 20:36:53.209851 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:53.209752 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj" event={"ID":"7f438514-df13-4871-9c2a-ef8e56a570c9","Type":"ContainerDied","Data":"554b28fb563761ef01e05125a4f730f1a30ede2e8f438cb7d70a6f254e87ef31"} Nov 03 20:36:53.209851 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:53.209774 5059 scope.go:122] "RemoveContainer" containerID="535c97f981aaf14f23e0940aa42d33bed0f295ee3f35cefd0fb15098b2c7793f" Nov 03 20:36:53.220980 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:53.220959 5059 scope.go:122] "RemoveContainer" containerID="535c97f981aaf14f23e0940aa42d33bed0f295ee3f35cefd0fb15098b2c7793f" Nov 03 20:36:53.221275 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:36:53.221256 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"535c97f981aaf14f23e0940aa42d33bed0f295ee3f35cefd0fb15098b2c7793f\": container with ID starting with 535c97f981aaf14f23e0940aa42d33bed0f295ee3f35cefd0fb15098b2c7793f not found: ID does not exist" containerID="535c97f981aaf14f23e0940aa42d33bed0f295ee3f35cefd0fb15098b2c7793f" Nov 03 20:36:53.221340 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:53.221285 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535c97f981aaf14f23e0940aa42d33bed0f295ee3f35cefd0fb15098b2c7793f"} err="failed to get container status \"535c97f981aaf14f23e0940aa42d33bed0f295ee3f35cefd0fb15098b2c7793f\": rpc error: code = NotFound desc = could not find container \"535c97f981aaf14f23e0940aa42d33bed0f295ee3f35cefd0fb15098b2c7793f\": container with ID starting with 535c97f981aaf14f23e0940aa42d33bed0f295ee3f35cefd0fb15098b2c7793f not found: ID does not exist" Nov 03 20:36:53.236871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:53.236763 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj"] Nov 03 20:36:53.240735 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:53.240710 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-5xlqj"] Nov 03 20:36:53.857458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:53.857424 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" path="/var/lib/kubelet/pods/7f438514-df13-4871-9c2a-ef8e56a570c9/volumes" Nov 03 20:36:58.748026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:58.747993 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2390e8d3-6b6b5"] Nov 03 20:36:58.748536 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:58.748522 5059 cpu_manager.go:414] "containerMap: removing container" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" Nov 03 20:36:58.748585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:58.748537 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" Nov 03 20:36:58.748585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:58.748557 5059 cpu_manager.go:414] "containerMap: removing container" podUID="eb070666-da11-410c-a670-46da774feef2" containerName="revoke-keys" Nov 03 20:36:58.748585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:58.748563 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb070666-da11-410c-a670-46da774feef2" containerName="revoke-keys" Nov 03 20:36:58.748701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:58.748642 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb070666-da11-410c-a670-46da774feef2" containerName="revoke-keys" Nov 03 20:36:58.748701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:58.748657 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f438514-df13-4871-9c2a-ef8e56a570c9" containerName="istio-proxy" Nov 03 20:36:58.753503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:58.753484 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2390e8d3-6b6b5" Nov 03 20:36:58.758091 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:58.758062 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:36:58.758222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:58.758125 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:36:58.761480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:58.761456 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2390e8d3-6b6b5"] Nov 03 20:36:58.891591 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:58.891558 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c43a0d3c-3581-4b3b-8e77-9758d17fd710-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2390e8d3-6b6b5\" (UID: \"c43a0d3c-3581-4b3b-8e77-9758d17fd710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2390e8d3-6b6b5" Nov 03 20:36:58.891777 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:58.891601 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c43a0d3c-3581-4b3b-8e77-9758d17fd710-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2390e8d3-6b6b5\" (UID: \"c43a0d3c-3581-4b3b-8e77-9758d17fd710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2390e8d3-6b6b5" Nov 03 20:36:58.891777 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:58.891731 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghjl6\" (UniqueName: \"kubernetes.io/projected/c43a0d3c-3581-4b3b-8e77-9758d17fd710-kube-api-access-ghjl6\") pod \"maas-api-revoke-sub-2390e8d3-6b6b5\" (UID: \"c43a0d3c-3581-4b3b-8e77-9758d17fd710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2390e8d3-6b6b5" Nov 03 20:36:58.993401 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:58.993368 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c43a0d3c-3581-4b3b-8e77-9758d17fd710-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2390e8d3-6b6b5\" (UID: \"c43a0d3c-3581-4b3b-8e77-9758d17fd710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2390e8d3-6b6b5" Nov 03 20:36:58.993401 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:58.993414 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c43a0d3c-3581-4b3b-8e77-9758d17fd710-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2390e8d3-6b6b5\" (UID: \"c43a0d3c-3581-4b3b-8e77-9758d17fd710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2390e8d3-6b6b5" Nov 03 20:36:58.993630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:58.993462 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghjl6\" (UniqueName: \"kubernetes.io/projected/c43a0d3c-3581-4b3b-8e77-9758d17fd710-kube-api-access-ghjl6\") pod \"maas-api-revoke-sub-2390e8d3-6b6b5\" (UID: \"c43a0d3c-3581-4b3b-8e77-9758d17fd710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2390e8d3-6b6b5" Nov 03 20:36:58.994120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:58.994091 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c43a0d3c-3581-4b3b-8e77-9758d17fd710-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2390e8d3-6b6b5\" (UID: \"c43a0d3c-3581-4b3b-8e77-9758d17fd710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2390e8d3-6b6b5" Nov 03 20:36:58.996160 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:58.996138 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c43a0d3c-3581-4b3b-8e77-9758d17fd710-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2390e8d3-6b6b5\" (UID: \"c43a0d3c-3581-4b3b-8e77-9758d17fd710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2390e8d3-6b6b5" Nov 03 20:36:59.006355 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:59.006304 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghjl6\" (UniqueName: \"kubernetes.io/projected/c43a0d3c-3581-4b3b-8e77-9758d17fd710-kube-api-access-ghjl6\") pod \"maas-api-revoke-sub-2390e8d3-6b6b5\" (UID: \"c43a0d3c-3581-4b3b-8e77-9758d17fd710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2390e8d3-6b6b5" Nov 03 20:36:59.069535 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:59.069511 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2390e8d3-6b6b5" Nov 03 20:36:59.349912 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:36:59.349477 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2390e8d3-6b6b5"] Nov 03 20:36:59.350246 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:36:59.350213 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43a0d3c_3581_4b3b_8e77_9758d17fd710.slice/crio-e432d2a1b619b0539a38e124a41937f57a1924f8054c203236eb6bf81fbb9ed9 WatchSource:0}: Error finding container e432d2a1b619b0539a38e124a41937f57a1924f8054c203236eb6bf81fbb9ed9: Status 404 returned error can't find the container with id e432d2a1b619b0539a38e124a41937f57a1924f8054c203236eb6bf81fbb9ed9 Nov 03 20:37:00.257899 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:00.257843 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2390e8d3-6b6b5" event={"ID":"c43a0d3c-3581-4b3b-8e77-9758d17fd710","Type":"ContainerStarted","Data":"d803c02bb6490bea1157b932cae2178c788af8b08840683537bc731d34499fcc"} Nov 03 20:37:00.258365 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:00.257923 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2390e8d3-6b6b5" event={"ID":"c43a0d3c-3581-4b3b-8e77-9758d17fd710","Type":"ContainerStarted","Data":"e432d2a1b619b0539a38e124a41937f57a1924f8054c203236eb6bf81fbb9ed9"} Nov 03 20:37:00.277324 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:00.277260 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2390e8d3-6b6b5" podStartSLOduration=2.2772450539999998 podStartE2EDuration="2.277245054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:00.276234496 +0000 UTC m=+3614.534031755" watchObservedRunningTime="2026-11-03 20:37:00.277245054 +0000 UTC m=+3614.535042311" Nov 03 20:37:01.297404 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:01.296904 5059 generic.go:356] "Generic (PLEG): container finished" podID="c43a0d3c-3581-4b3b-8e77-9758d17fd710" containerID="d803c02bb6490bea1157b932cae2178c788af8b08840683537bc731d34499fcc" exitCode=0 Nov 03 20:37:01.297404 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:01.296977 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2390e8d3-6b6b5" event={"ID":"c43a0d3c-3581-4b3b-8e77-9758d17fd710","Type":"ContainerDied","Data":"d803c02bb6490bea1157b932cae2178c788af8b08840683537bc731d34499fcc"} Nov 03 20:37:02.440131 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:02.440108 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2390e8d3-6b6b5" Nov 03 20:37:02.541729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:02.541692 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c43a0d3c-3581-4b3b-8e77-9758d17fd710-kube-api-access-ghjl6\" (UniqueName: \"kubernetes.io/projected/c43a0d3c-3581-4b3b-8e77-9758d17fd710-kube-api-access-ghjl6\") pod \"c43a0d3c-3581-4b3b-8e77-9758d17fd710\" (UID: \"c43a0d3c-3581-4b3b-8e77-9758d17fd710\") " Nov 03 20:37:02.541889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:02.541747 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c43a0d3c-3581-4b3b-8e77-9758d17fd710-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c43a0d3c-3581-4b3b-8e77-9758d17fd710-maas-api-openshift-ca\") pod \"c43a0d3c-3581-4b3b-8e77-9758d17fd710\" (UID: \"c43a0d3c-3581-4b3b-8e77-9758d17fd710\") " Nov 03 20:37:02.541889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:02.541830 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c43a0d3c-3581-4b3b-8e77-9758d17fd710-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c43a0d3c-3581-4b3b-8e77-9758d17fd710-maas-api-xks-ca\") pod \"c43a0d3c-3581-4b3b-8e77-9758d17fd710\" (UID: \"c43a0d3c-3581-4b3b-8e77-9758d17fd710\") " Nov 03 20:37:02.542457 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:02.542423 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c43a0d3c-3581-4b3b-8e77-9758d17fd710-maas-api-openshift-ca" pod "c43a0d3c-3581-4b3b-8e77-9758d17fd710" (UID: "c43a0d3c-3581-4b3b-8e77-9758d17fd710"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:02.544141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:02.544111 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c43a0d3c-3581-4b3b-8e77-9758d17fd710-kube-api-access-ghjl6" pod "c43a0d3c-3581-4b3b-8e77-9758d17fd710" (UID: "c43a0d3c-3581-4b3b-8e77-9758d17fd710"). InnerVolumeSpecName "kube-api-access-ghjl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:02.544450 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:02.544427 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c43a0d3c-3581-4b3b-8e77-9758d17fd710-maas-api-xks-ca" pod "c43a0d3c-3581-4b3b-8e77-9758d17fd710" (UID: "c43a0d3c-3581-4b3b-8e77-9758d17fd710"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:37:02.644006 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:02.643974 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghjl6\" (UniqueName: \"kubernetes.io/projected/c43a0d3c-3581-4b3b-8e77-9758d17fd710-kube-api-access-ghjl6\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:02.644006 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:02.644005 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c43a0d3c-3581-4b3b-8e77-9758d17fd710-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:02.644166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:02.644018 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c43a0d3c-3581-4b3b-8e77-9758d17fd710-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:03.310872 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:03.310828 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2390e8d3-6b6b5" Nov 03 20:37:03.311049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:03.310875 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2390e8d3-6b6b5" event={"ID":"c43a0d3c-3581-4b3b-8e77-9758d17fd710","Type":"ContainerDied","Data":"e432d2a1b619b0539a38e124a41937f57a1924f8054c203236eb6bf81fbb9ed9"} Nov 03 20:37:03.311049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:03.310929 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e432d2a1b619b0539a38e124a41937f57a1924f8054c203236eb6bf81fbb9ed9" Nov 03 20:37:10.604769 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:37:10.604728 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:37:21.714338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.714237 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz"] Nov 03 20:37:21.715008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.714986 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c43a0d3c-3581-4b3b-8e77-9758d17fd710" containerName="revoke-keys" Nov 03 20:37:21.715100 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.715009 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c43a0d3c-3581-4b3b-8e77-9758d17fd710" containerName="revoke-keys" Nov 03 20:37:21.715172 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.715157 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c43a0d3c-3581-4b3b-8e77-9758d17fd710" containerName="revoke-keys" Nov 03 20:37:21.718468 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.718447 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.731174 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.731139 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz"] Nov 03 20:37:21.821577 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.821530 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.821773 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.821592 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.821773 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.821741 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.821851 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.821775 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-279j5\" (UniqueName: \"kubernetes.io/projected/7f06aace-29a2-4e89-b84c-b5898aa29355-kube-api-access-279j5\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.821851 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.821802 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f06aace-29a2-4e89-b84c-b5898aa29355-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.821918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.821875 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.821918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.821907 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.822045 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.822030 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.822087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.822063 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.923726 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.923689 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.923879 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.923737 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-279j5\" (UniqueName: \"kubernetes.io/projected/7f06aace-29a2-4e89-b84c-b5898aa29355-kube-api-access-279j5\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.923879 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.923759 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f06aace-29a2-4e89-b84c-b5898aa29355-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.923976 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.923913 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.924020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.923973 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.924158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.924135 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.924226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.924196 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.924285 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.924269 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.924343 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.924282 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.924405 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.924341 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.924405 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.924356 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.924567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.924541 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.924628 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.924600 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f06aace-29a2-4e89-b84c-b5898aa29355-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.924700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.924606 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.926198 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.926173 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.926607 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.926589 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.938476 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.938452 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:21.938476 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:21.938470 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-279j5\" (UniqueName: \"kubernetes.io/projected/7f06aace-29a2-4e89-b84c-b5898aa29355-kube-api-access-279j5\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:22.033280 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:22.033194 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:22.318438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:22.318409 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz"] Nov 03 20:37:22.319001 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:37:22.318977 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f06aace_29a2_4e89_b84c_b5898aa29355.slice/crio-807230085a1bddafd01ac1e87421a4ac4efdba94fe81cb02ccd02cbc0806291c WatchSource:0}: Error finding container 807230085a1bddafd01ac1e87421a4ac4efdba94fe81cb02ccd02cbc0806291c: Status 404 returned error can't find the container with id 807230085a1bddafd01ac1e87421a4ac4efdba94fe81cb02ccd02cbc0806291c Nov 03 20:37:22.321522 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:22.321489 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:37:22.321587 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:22.321569 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:37:22.321623 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:22.321597 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:37:22.441876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:22.441841 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" event={"ID":"7f06aace-29a2-4e89-b84c-b5898aa29355","Type":"ContainerStarted","Data":"930b88a761fd1cff644a8c8caf01cad9323f84c95ae7484e5318c95da5b0dbd5"} Nov 03 20:37:22.442061 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:22.441901 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" event={"ID":"7f06aace-29a2-4e89-b84c-b5898aa29355","Type":"ContainerStarted","Data":"807230085a1bddafd01ac1e87421a4ac4efdba94fe81cb02ccd02cbc0806291c"} Nov 03 20:37:22.467712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:22.467608 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" podStartSLOduration=1.467593693 podStartE2EDuration="1.467593693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:22.465451904 +0000 UTC m=+3636.723249157" watchObservedRunningTime="2026-11-03 20:37:22.467593693 +0000 UTC m=+3636.725390975" Nov 03 20:37:23.034048 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:23.034004 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:23.035863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:23.035834 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" start-of-body= Nov 03 20:37:23.035994 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:23.035885 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" podUID="7f06aace-29a2-4e89-b84c-b5898aa29355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" Nov 03 20:37:24.033850 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:24.033807 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" start-of-body= Nov 03 20:37:24.034065 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:24.033862 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" podUID="7f06aace-29a2-4e89-b84c-b5898aa29355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" Nov 03 20:37:25.033790 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:25.033753 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" start-of-body= Nov 03 20:37:25.033987 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:25.033826 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" podUID="7f06aace-29a2-4e89-b84c-b5898aa29355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" Nov 03 20:37:26.034594 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:26.034560 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" start-of-body= Nov 03 20:37:26.034992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:26.034616 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" podUID="7f06aace-29a2-4e89-b84c-b5898aa29355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" Nov 03 20:37:27.034233 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:27.034199 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" start-of-body= Nov 03 20:37:27.034445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:27.034263 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" podUID="7f06aace-29a2-4e89-b84c-b5898aa29355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" Nov 03 20:37:28.033953 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:28.033915 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" start-of-body= Nov 03 20:37:28.034416 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:28.033972 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" podUID="7f06aace-29a2-4e89-b84c-b5898aa29355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.156:15021/healthz/ready\": dial tcp 10.132.1.156:15021: connect: connection refused" Nov 03 20:37:29.241311 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:29.241277 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:29.241954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:29.241758 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:29.242730 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:29.242707 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:40.016076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:40.016037 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl"] Nov 03 20:37:40.017049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:40.017010 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" podUID="9c4b9c59-a980-4ec8-af7b-fc2c08133f19" containerName="istio-proxy" containerID="cri-o://3e8c7d9a4d245a077d7d5d91682130d61c0ca287bcd31fe821677ff75b460f32" gracePeriod=30 Nov 03 20:37:40.022912 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:40.022878 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg"] Nov 03 20:37:40.023536 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:40.023382 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" containerID="cri-o://b7cacec852af6354300c159e158582fcff3f1d2e974a9d742f79b212daaff852" gracePeriod=30 Nov 03 20:37:40.030650 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:40.030622 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz"] Nov 03 20:37:40.030928 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:40.030877 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" podUID="7f06aace-29a2-4e89-b84c-b5898aa29355" containerName="istio-proxy" containerID="cri-o://930b88a761fd1cff644a8c8caf01cad9323f84c95ae7484e5318c95da5b0dbd5" gracePeriod=30 Nov 03 20:37:40.607932 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:37:40.607896 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:37:44.244418 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:44.244379 5059 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:37:44.244833 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:44.244444 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" podUID="7f06aace-29a2-4e89-b84c-b5898aa29355" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:37:45.305860 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.305830 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:37:45.333141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.333110 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-credential-socket\") pod \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " Nov 03 20:37:45.333376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.333354 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-podinfo\") pod \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " Nov 03 20:37:45.333529 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.333512 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-kube-api-access-4h9kv\" (UniqueName: \"kubernetes.io/projected/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-kube-api-access-4h9kv\") pod \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " Nov 03 20:37:45.333759 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.333738 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-data\") pod \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " Nov 03 20:37:45.333905 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.333888 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-workload-certs\") pod \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " Nov 03 20:37:45.334045 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.334029 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-token\" (UniqueName: \"kubernetes.io/projected/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-token\") pod \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " Nov 03 20:37:45.334175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.334160 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istiod-ca-cert\") pod \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " Nov 03 20:37:45.334317 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.334301 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-envoy\") pod \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " Nov 03 20:37:45.334497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.334480 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-workload-socket\") pod \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\" (UID: \"9c4b9c59-a980-4ec8-af7b-fc2c08133f19\") " Nov 03 20:37:45.336475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.333438 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-credential-socket" pod "9c4b9c59-a980-4ec8-af7b-fc2c08133f19" (UID: "9c4b9c59-a980-4ec8-af7b-fc2c08133f19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:45.336475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.335813 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-data" pod "9c4b9c59-a980-4ec8-af7b-fc2c08133f19" (UID: "9c4b9c59-a980-4ec8-af7b-fc2c08133f19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:45.341602 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.341333 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istiod-ca-cert" pod "9c4b9c59-a980-4ec8-af7b-fc2c08133f19" (UID: "9c4b9c59-a980-4ec8-af7b-fc2c08133f19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:45.342908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.342880 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-workload-socket" pod "9c4b9c59-a980-4ec8-af7b-fc2c08133f19" (UID: "9c4b9c59-a980-4ec8-af7b-fc2c08133f19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:45.343070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.343049 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-workload-certs" pod "9c4b9c59-a980-4ec8-af7b-fc2c08133f19" (UID: "9c4b9c59-a980-4ec8-af7b-fc2c08133f19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:45.345304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.345275 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-token" pod "9c4b9c59-a980-4ec8-af7b-fc2c08133f19" (UID: "9c4b9c59-a980-4ec8-af7b-fc2c08133f19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:45.352141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.352007 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-podinfo" pod "9c4b9c59-a980-4ec8-af7b-fc2c08133f19" (UID: "9c4b9c59-a980-4ec8-af7b-fc2c08133f19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:37:45.352141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.352106 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-kube-api-access-4h9kv" pod "9c4b9c59-a980-4ec8-af7b-fc2c08133f19" (UID: "9c4b9c59-a980-4ec8-af7b-fc2c08133f19"). InnerVolumeSpecName "kube-api-access-4h9kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:45.353559 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.353537 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-envoy" pod "9c4b9c59-a980-4ec8-af7b-fc2c08133f19" (UID: "9c4b9c59-a980-4ec8-af7b-fc2c08133f19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:45.377600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.377574 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:45.381800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.381776 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:37:45.442008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.441966 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-credential-socket\") pod \"e2fe4277-3acc-4378-86d9-374b69918ec9\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " Nov 03 20:37:45.442184 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.442075 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-workload-socket\") pod \"7f06aace-29a2-4e89-b84c-b5898aa29355\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " Nov 03 20:37:45.442184 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.442122 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-podinfo\") pod \"e2fe4277-3acc-4378-86d9-374b69918ec9\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " Nov 03 20:37:45.442184 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.442159 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-workload-certs\") pod \"7f06aace-29a2-4e89-b84c-b5898aa29355\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " Nov 03 20:37:45.442352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.442194 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f06aace-29a2-4e89-b84c-b5898aa29355-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f06aace-29a2-4e89-b84c-b5898aa29355-istiod-ca-cert\") pod \"7f06aace-29a2-4e89-b84c-b5898aa29355\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " Nov 03 20:37:45.442352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.442246 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-podinfo\") pod \"7f06aace-29a2-4e89-b84c-b5898aa29355\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " Nov 03 20:37:45.442352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.442295 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-data\") pod \"e2fe4277-3acc-4378-86d9-374b69918ec9\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " Nov 03 20:37:45.442352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.442330 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-envoy\") pod \"e2fe4277-3acc-4378-86d9-374b69918ec9\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " Nov 03 20:37:45.442549 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.442373 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-envoy\") pod \"7f06aace-29a2-4e89-b84c-b5898aa29355\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " Nov 03 20:37:45.442549 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.442418 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2fe4277-3acc-4378-86d9-374b69918ec9-kube-api-access-bhmg9\" (UniqueName: \"kubernetes.io/projected/e2fe4277-3acc-4378-86d9-374b69918ec9-kube-api-access-bhmg9\") pod \"e2fe4277-3acc-4378-86d9-374b69918ec9\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " Nov 03 20:37:45.442549 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.442466 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-workload-socket\") pod \"e2fe4277-3acc-4378-86d9-374b69918ec9\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " Nov 03 20:37:45.442549 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.442516 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f06aace-29a2-4e89-b84c-b5898aa29355-kube-api-access-279j5\" (UniqueName: \"kubernetes.io/projected/7f06aace-29a2-4e89-b84c-b5898aa29355-kube-api-access-279j5\") pod \"7f06aace-29a2-4e89-b84c-b5898aa29355\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " Nov 03 20:37:45.442776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.442558 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2fe4277-3acc-4378-86d9-374b69918ec9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2fe4277-3acc-4378-86d9-374b69918ec9-istiod-ca-cert\") pod \"e2fe4277-3acc-4378-86d9-374b69918ec9\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " Nov 03 20:37:45.442776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.442595 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-workload-certs\") pod \"e2fe4277-3acc-4378-86d9-374b69918ec9\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " Nov 03 20:37:45.442776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.442645 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-data\") pod \"7f06aace-29a2-4e89-b84c-b5898aa29355\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " Nov 03 20:37:45.442776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.442765 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-token\" (UniqueName: \"kubernetes.io/projected/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-token\") pod \"7f06aace-29a2-4e89-b84c-b5898aa29355\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " Nov 03 20:37:45.443056 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.442803 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-credential-socket\") pod \"7f06aace-29a2-4e89-b84c-b5898aa29355\" (UID: \"7f06aace-29a2-4e89-b84c-b5898aa29355\") " Nov 03 20:37:45.443056 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.442864 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-token\" (UniqueName: \"kubernetes.io/projected/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-token\") pod \"e2fe4277-3acc-4378-86d9-374b69918ec9\" (UID: \"e2fe4277-3acc-4378-86d9-374b69918ec9\") " Nov 03 20:37:45.446686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.446639 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-credential-socket" pod "e2fe4277-3acc-4378-86d9-374b69918ec9" (UID: "e2fe4277-3acc-4378-86d9-374b69918ec9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:45.447104 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.447082 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.447598 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.447141 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-workload-socket" pod "7f06aace-29a2-4e89-b84c-b5898aa29355" (UID: "7f06aace-29a2-4e89-b84c-b5898aa29355"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:45.447598 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.447178 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f06aace-29a2-4e89-b84c-b5898aa29355-istiod-ca-cert" pod "7f06aace-29a2-4e89-b84c-b5898aa29355" (UID: "7f06aace-29a2-4e89-b84c-b5898aa29355"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:45.447598 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.447473 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-workload-socket" pod "e2fe4277-3acc-4378-86d9-374b69918ec9" (UID: "e2fe4277-3acc-4378-86d9-374b69918ec9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:45.447598 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.447562 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2fe4277-3acc-4378-86d9-374b69918ec9-istiod-ca-cert" pod "e2fe4277-3acc-4378-86d9-374b69918ec9" (UID: "e2fe4277-3acc-4378-86d9-374b69918ec9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:45.447598 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.447568 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.447950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.447605 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.447950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.447761 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-workload-certs" pod "7f06aace-29a2-4e89-b84c-b5898aa29355" (UID: "7f06aace-29a2-4e89-b84c-b5898aa29355"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:45.447950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.447857 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.447950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.447886 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h9kv\" (UniqueName: \"kubernetes.io/projected/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-kube-api-access-4h9kv\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.447950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.447905 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.447950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.447922 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.447950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.447940 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.448279 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.447955 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.448279 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.447970 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c4b9c59-a980-4ec8-af7b-fc2c08133f19-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.448411 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.448194 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-data" pod "7f06aace-29a2-4e89-b84c-b5898aa29355" (UID: "7f06aace-29a2-4e89-b84c-b5898aa29355"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:45.448479 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.448395 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-workload-certs" pod "e2fe4277-3acc-4378-86d9-374b69918ec9" (UID: "e2fe4277-3acc-4378-86d9-374b69918ec9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:45.450385 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.450092 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-data" pod "e2fe4277-3acc-4378-86d9-374b69918ec9" (UID: "e2fe4277-3acc-4378-86d9-374b69918ec9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:45.450385 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.450233 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-credential-socket" pod "7f06aace-29a2-4e89-b84c-b5898aa29355" (UID: "7f06aace-29a2-4e89-b84c-b5898aa29355"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:45.450385 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.450335 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-envoy" pod "7f06aace-29a2-4e89-b84c-b5898aa29355" (UID: "7f06aace-29a2-4e89-b84c-b5898aa29355"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:45.450793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.450771 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-token" pod "e2fe4277-3acc-4378-86d9-374b69918ec9" (UID: "e2fe4277-3acc-4378-86d9-374b69918ec9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:45.452236 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.452209 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-envoy" pod "e2fe4277-3acc-4378-86d9-374b69918ec9" (UID: "e2fe4277-3acc-4378-86d9-374b69918ec9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:45.452930 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.452895 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-podinfo" pod "7f06aace-29a2-4e89-b84c-b5898aa29355" (UID: "7f06aace-29a2-4e89-b84c-b5898aa29355"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:37:45.453034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.452925 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-podinfo" pod "e2fe4277-3acc-4378-86d9-374b69918ec9" (UID: "e2fe4277-3acc-4378-86d9-374b69918ec9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:37:45.453034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.453014 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2fe4277-3acc-4378-86d9-374b69918ec9-kube-api-access-bhmg9" pod "e2fe4277-3acc-4378-86d9-374b69918ec9" (UID: "e2fe4277-3acc-4378-86d9-374b69918ec9"). InnerVolumeSpecName "kube-api-access-bhmg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:45.453453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.453432 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f06aace-29a2-4e89-b84c-b5898aa29355-kube-api-access-279j5" pod "7f06aace-29a2-4e89-b84c-b5898aa29355" (UID: "7f06aace-29a2-4e89-b84c-b5898aa29355"). InnerVolumeSpecName "kube-api-access-279j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:45.453574 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.453557 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-token" pod "7f06aace-29a2-4e89-b84c-b5898aa29355" (UID: "7f06aace-29a2-4e89-b84c-b5898aa29355"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:45.548823 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.548751 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.548823 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.548780 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.548823 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.548790 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.548823 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.548799 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f06aace-29a2-4e89-b84c-b5898aa29355-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.548823 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.548808 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.548823 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.548816 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.548823 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.548825 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.549128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.548834 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.549128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.548862 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhmg9\" (UniqueName: \"kubernetes.io/projected/e2fe4277-3acc-4378-86d9-374b69918ec9-kube-api-access-bhmg9\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.549128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.548873 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.549128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.548882 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-279j5\" (UniqueName: \"kubernetes.io/projected/7f06aace-29a2-4e89-b84c-b5898aa29355-kube-api-access-279j5\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.549128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.548898 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2fe4277-3acc-4378-86d9-374b69918ec9-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.549128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.548906 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2fe4277-3acc-4378-86d9-374b69918ec9-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.549128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.548914 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.549128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.548923 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f06aace-29a2-4e89-b84c-b5898aa29355-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.549128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.548931 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f06aace-29a2-4e89-b84c-b5898aa29355-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.549128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.548939 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2fe4277-3acc-4378-86d9-374b69918ec9-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:45.626323 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.626285 5059 generic.go:356] "Generic (PLEG): container finished" podID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerID="b7cacec852af6354300c159e158582fcff3f1d2e974a9d742f79b212daaff852" exitCode=0 Nov 03 20:37:45.626493 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.626364 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" event={"ID":"e2fe4277-3acc-4378-86d9-374b69918ec9","Type":"ContainerDied","Data":"b7cacec852af6354300c159e158582fcff3f1d2e974a9d742f79b212daaff852"} Nov 03 20:37:45.626493 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.626389 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" Nov 03 20:37:45.626493 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.626452 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg" event={"ID":"e2fe4277-3acc-4378-86d9-374b69918ec9","Type":"ContainerDied","Data":"ea085d2ffafa8646c924f9e8777a5c0a04f6ac65639f507368d7fc95cd47e35d"} Nov 03 20:37:45.626493 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.626487 5059 scope.go:122] "RemoveContainer" containerID="b7cacec852af6354300c159e158582fcff3f1d2e974a9d742f79b212daaff852" Nov 03 20:37:45.627929 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.627906 5059 generic.go:356] "Generic (PLEG): container finished" podID="9c4b9c59-a980-4ec8-af7b-fc2c08133f19" containerID="3e8c7d9a4d245a077d7d5d91682130d61c0ca287bcd31fe821677ff75b460f32" exitCode=0 Nov 03 20:37:45.628079 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.627973 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" Nov 03 20:37:45.628079 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.627994 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" event={"ID":"9c4b9c59-a980-4ec8-af7b-fc2c08133f19","Type":"ContainerDied","Data":"3e8c7d9a4d245a077d7d5d91682130d61c0ca287bcd31fe821677ff75b460f32"} Nov 03 20:37:45.628079 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.628044 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl" event={"ID":"9c4b9c59-a980-4ec8-af7b-fc2c08133f19","Type":"ContainerDied","Data":"4bbb12ec9bb4aa5fd6ab26e02ad9fcff691d47daa518d4e66b7d56093ec2c10d"} Nov 03 20:37:45.629596 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.629575 5059 generic.go:356] "Generic (PLEG): container finished" podID="7f06aace-29a2-4e89-b84c-b5898aa29355" containerID="930b88a761fd1cff644a8c8caf01cad9323f84c95ae7484e5318c95da5b0dbd5" exitCode=0 Nov 03 20:37:45.629731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.629622 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" event={"ID":"7f06aace-29a2-4e89-b84c-b5898aa29355","Type":"ContainerDied","Data":"930b88a761fd1cff644a8c8caf01cad9323f84c95ae7484e5318c95da5b0dbd5"} Nov 03 20:37:45.629731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.629642 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" Nov 03 20:37:45.629731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.629675 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz" event={"ID":"7f06aace-29a2-4e89-b84c-b5898aa29355","Type":"ContainerDied","Data":"807230085a1bddafd01ac1e87421a4ac4efdba94fe81cb02ccd02cbc0806291c"} Nov 03 20:37:45.637798 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.637779 5059 scope.go:122] "RemoveContainer" containerID="b7cacec852af6354300c159e158582fcff3f1d2e974a9d742f79b212daaff852" Nov 03 20:37:45.638054 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:37:45.638034 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7cacec852af6354300c159e158582fcff3f1d2e974a9d742f79b212daaff852\": container with ID starting with b7cacec852af6354300c159e158582fcff3f1d2e974a9d742f79b212daaff852 not found: ID does not exist" containerID="b7cacec852af6354300c159e158582fcff3f1d2e974a9d742f79b212daaff852" Nov 03 20:37:45.638121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.638067 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7cacec852af6354300c159e158582fcff3f1d2e974a9d742f79b212daaff852"} err="failed to get container status \"b7cacec852af6354300c159e158582fcff3f1d2e974a9d742f79b212daaff852\": rpc error: code = NotFound desc = could not find container \"b7cacec852af6354300c159e158582fcff3f1d2e974a9d742f79b212daaff852\": container with ID starting with b7cacec852af6354300c159e158582fcff3f1d2e974a9d742f79b212daaff852 not found: ID does not exist" Nov 03 20:37:45.638121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.638086 5059 scope.go:122] "RemoveContainer" containerID="3e8c7d9a4d245a077d7d5d91682130d61c0ca287bcd31fe821677ff75b460f32" Nov 03 20:37:45.648392 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.648371 5059 scope.go:122] "RemoveContainer" containerID="3e8c7d9a4d245a077d7d5d91682130d61c0ca287bcd31fe821677ff75b460f32" Nov 03 20:37:45.648639 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:37:45.648619 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e8c7d9a4d245a077d7d5d91682130d61c0ca287bcd31fe821677ff75b460f32\": container with ID starting with 3e8c7d9a4d245a077d7d5d91682130d61c0ca287bcd31fe821677ff75b460f32 not found: ID does not exist" containerID="3e8c7d9a4d245a077d7d5d91682130d61c0ca287bcd31fe821677ff75b460f32" Nov 03 20:37:45.648714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.648645 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e8c7d9a4d245a077d7d5d91682130d61c0ca287bcd31fe821677ff75b460f32"} err="failed to get container status \"3e8c7d9a4d245a077d7d5d91682130d61c0ca287bcd31fe821677ff75b460f32\": rpc error: code = NotFound desc = could not find container \"3e8c7d9a4d245a077d7d5d91682130d61c0ca287bcd31fe821677ff75b460f32\": container with ID starting with 3e8c7d9a4d245a077d7d5d91682130d61c0ca287bcd31fe821677ff75b460f32 not found: ID does not exist" Nov 03 20:37:45.648714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.648710 5059 scope.go:122] "RemoveContainer" containerID="930b88a761fd1cff644a8c8caf01cad9323f84c95ae7484e5318c95da5b0dbd5" Nov 03 20:37:45.657847 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.657828 5059 scope.go:122] "RemoveContainer" containerID="930b88a761fd1cff644a8c8caf01cad9323f84c95ae7484e5318c95da5b0dbd5" Nov 03 20:37:45.658126 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:37:45.658107 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"930b88a761fd1cff644a8c8caf01cad9323f84c95ae7484e5318c95da5b0dbd5\": container with ID starting with 930b88a761fd1cff644a8c8caf01cad9323f84c95ae7484e5318c95da5b0dbd5 not found: ID does not exist" containerID="930b88a761fd1cff644a8c8caf01cad9323f84c95ae7484e5318c95da5b0dbd5" Nov 03 20:37:45.658202 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.658133 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"930b88a761fd1cff644a8c8caf01cad9323f84c95ae7484e5318c95da5b0dbd5"} err="failed to get container status \"930b88a761fd1cff644a8c8caf01cad9323f84c95ae7484e5318c95da5b0dbd5\": rpc error: code = NotFound desc = could not find container \"930b88a761fd1cff644a8c8caf01cad9323f84c95ae7484e5318c95da5b0dbd5\": container with ID starting with 930b88a761fd1cff644a8c8caf01cad9323f84c95ae7484e5318c95da5b0dbd5 not found: ID does not exist" Nov 03 20:37:45.659097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.659075 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl"] Nov 03 20:37:45.677912 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.677890 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-cm8xl"] Nov 03 20:37:45.735318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.735286 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg"] Nov 03 20:37:45.744547 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.744522 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-nzqgg"] Nov 03 20:37:45.805087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.805012 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz"] Nov 03 20:37:45.808511 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.808485 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-fs8mz"] Nov 03 20:37:45.857317 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.857289 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f06aace-29a2-4e89-b84c-b5898aa29355" path="/var/lib/kubelet/pods/7f06aace-29a2-4e89-b84c-b5898aa29355/volumes" Nov 03 20:37:45.857835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.857817 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c4b9c59-a980-4ec8-af7b-fc2c08133f19" path="/var/lib/kubelet/pods/9c4b9c59-a980-4ec8-af7b-fc2c08133f19/volumes" Nov 03 20:37:45.858283 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:45.858265 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" path="/var/lib/kubelet/pods/e2fe4277-3acc-4378-86d9-374b69918ec9/volumes" Nov 03 20:37:52.248164 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.248130 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f20dd15-mw284"] Nov 03 20:37:52.248683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.248645 5059 cpu_manager.go:414] "containerMap: removing container" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" Nov 03 20:37:52.248683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.248672 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" Nov 03 20:37:52.248772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.248696 5059 cpu_manager.go:414] "containerMap: removing container" podUID="7f06aace-29a2-4e89-b84c-b5898aa29355" containerName="istio-proxy" Nov 03 20:37:52.248772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.248701 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f06aace-29a2-4e89-b84c-b5898aa29355" containerName="istio-proxy" Nov 03 20:37:52.248772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.248710 5059 cpu_manager.go:414] "containerMap: removing container" podUID="9c4b9c59-a980-4ec8-af7b-fc2c08133f19" containerName="istio-proxy" Nov 03 20:37:52.248772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.248716 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4b9c59-a980-4ec8-af7b-fc2c08133f19" containerName="istio-proxy" Nov 03 20:37:52.249377 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.248792 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c4b9c59-a980-4ec8-af7b-fc2c08133f19" containerName="istio-proxy" Nov 03 20:37:52.249377 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.248806 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2fe4277-3acc-4378-86d9-374b69918ec9" containerName="istio-proxy" Nov 03 20:37:52.249377 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.248815 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f06aace-29a2-4e89-b84c-b5898aa29355" containerName="istio-proxy" Nov 03 20:37:52.251965 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.251948 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f20dd15-mw284" Nov 03 20:37:52.260400 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.260378 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:37:52.260526 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.260417 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:37:52.263908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.263877 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f20dd15-mw284"] Nov 03 20:37:52.350028 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.349996 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4116cd80-c522-4b8b-9b3e-6229592443e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f20dd15-mw284\" (UID: \"4116cd80-c522-4b8b-9b3e-6229592443e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f20dd15-mw284" Nov 03 20:37:52.350174 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.350059 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4116cd80-c522-4b8b-9b3e-6229592443e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f20dd15-mw284\" (UID: \"4116cd80-c522-4b8b-9b3e-6229592443e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f20dd15-mw284" Nov 03 20:37:52.350174 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.350144 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbkcj\" (UniqueName: \"kubernetes.io/projected/4116cd80-c522-4b8b-9b3e-6229592443e1-kube-api-access-qbkcj\") pod \"maas-api-revoke-sub-7f20dd15-mw284\" (UID: \"4116cd80-c522-4b8b-9b3e-6229592443e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f20dd15-mw284" Nov 03 20:37:52.451706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.451646 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4116cd80-c522-4b8b-9b3e-6229592443e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f20dd15-mw284\" (UID: \"4116cd80-c522-4b8b-9b3e-6229592443e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f20dd15-mw284" Nov 03 20:37:52.451861 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.451731 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4116cd80-c522-4b8b-9b3e-6229592443e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f20dd15-mw284\" (UID: \"4116cd80-c522-4b8b-9b3e-6229592443e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f20dd15-mw284" Nov 03 20:37:52.451861 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.451761 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbkcj\" (UniqueName: \"kubernetes.io/projected/4116cd80-c522-4b8b-9b3e-6229592443e1-kube-api-access-qbkcj\") pod \"maas-api-revoke-sub-7f20dd15-mw284\" (UID: \"4116cd80-c522-4b8b-9b3e-6229592443e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f20dd15-mw284" Nov 03 20:37:52.452415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.452391 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4116cd80-c522-4b8b-9b3e-6229592443e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f20dd15-mw284\" (UID: \"4116cd80-c522-4b8b-9b3e-6229592443e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f20dd15-mw284" Nov 03 20:37:52.454129 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.454105 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4116cd80-c522-4b8b-9b3e-6229592443e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f20dd15-mw284\" (UID: \"4116cd80-c522-4b8b-9b3e-6229592443e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f20dd15-mw284" Nov 03 20:37:52.464096 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.464075 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbkcj\" (UniqueName: \"kubernetes.io/projected/4116cd80-c522-4b8b-9b3e-6229592443e1-kube-api-access-qbkcj\") pod \"maas-api-revoke-sub-7f20dd15-mw284\" (UID: \"4116cd80-c522-4b8b-9b3e-6229592443e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f20dd15-mw284" Nov 03 20:37:52.567379 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.567072 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f20dd15-mw284" Nov 03 20:37:52.855579 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:52.854417 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f20dd15-mw284"] Nov 03 20:37:53.685259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:53.685215 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f20dd15-mw284" event={"ID":"4116cd80-c522-4b8b-9b3e-6229592443e1","Type":"ContainerStarted","Data":"0ed100d4d9dd9a93af0fb92d61429fe51c8e89a379a6a74ce59ae9100b66a5ea"} Nov 03 20:37:53.685735 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:53.685319 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f20dd15-mw284" event={"ID":"4116cd80-c522-4b8b-9b3e-6229592443e1","Type":"ContainerStarted","Data":"a60a9f8e9c98ebc7ab99782a09feaead183d5f128296c383413957dfa3dc1f63"} Nov 03 20:37:53.704606 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:53.704556 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f20dd15-mw284" podStartSLOduration=1.704543141 podStartE2EDuration="1.704543141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:53.703473452 +0000 UTC m=+3667.961270711" watchObservedRunningTime="2026-11-03 20:37:53.704543141 +0000 UTC m=+3667.962340430" Nov 03 20:37:54.692877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:54.692835 5059 generic.go:356] "Generic (PLEG): container finished" podID="4116cd80-c522-4b8b-9b3e-6229592443e1" containerID="0ed100d4d9dd9a93af0fb92d61429fe51c8e89a379a6a74ce59ae9100b66a5ea" exitCode=0 Nov 03 20:37:54.693349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:54.692914 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f20dd15-mw284" event={"ID":"4116cd80-c522-4b8b-9b3e-6229592443e1","Type":"ContainerDied","Data":"0ed100d4d9dd9a93af0fb92d61429fe51c8e89a379a6a74ce59ae9100b66a5ea"} Nov 03 20:37:55.853383 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:55.853359 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f20dd15-mw284" Nov 03 20:37:55.996633 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:55.996550 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4116cd80-c522-4b8b-9b3e-6229592443e1-kube-api-access-qbkcj\" (UniqueName: \"kubernetes.io/projected/4116cd80-c522-4b8b-9b3e-6229592443e1-kube-api-access-qbkcj\") pod \"4116cd80-c522-4b8b-9b3e-6229592443e1\" (UID: \"4116cd80-c522-4b8b-9b3e-6229592443e1\") " Nov 03 20:37:55.996869 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:55.996657 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4116cd80-c522-4b8b-9b3e-6229592443e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4116cd80-c522-4b8b-9b3e-6229592443e1-maas-api-xks-ca\") pod \"4116cd80-c522-4b8b-9b3e-6229592443e1\" (UID: \"4116cd80-c522-4b8b-9b3e-6229592443e1\") " Nov 03 20:37:55.996869 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:55.996743 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4116cd80-c522-4b8b-9b3e-6229592443e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4116cd80-c522-4b8b-9b3e-6229592443e1-maas-api-openshift-ca\") pod \"4116cd80-c522-4b8b-9b3e-6229592443e1\" (UID: \"4116cd80-c522-4b8b-9b3e-6229592443e1\") " Nov 03 20:37:55.997180 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:55.997147 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4116cd80-c522-4b8b-9b3e-6229592443e1-maas-api-openshift-ca" pod "4116cd80-c522-4b8b-9b3e-6229592443e1" (UID: "4116cd80-c522-4b8b-9b3e-6229592443e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:55.998843 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:55.998810 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4116cd80-c522-4b8b-9b3e-6229592443e1-maas-api-xks-ca" pod "4116cd80-c522-4b8b-9b3e-6229592443e1" (UID: "4116cd80-c522-4b8b-9b3e-6229592443e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:37:55.998928 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:55.998895 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4116cd80-c522-4b8b-9b3e-6229592443e1-kube-api-access-qbkcj" pod "4116cd80-c522-4b8b-9b3e-6229592443e1" (UID: "4116cd80-c522-4b8b-9b3e-6229592443e1"). InnerVolumeSpecName "kube-api-access-qbkcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:56.098231 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:56.098181 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbkcj\" (UniqueName: \"kubernetes.io/projected/4116cd80-c522-4b8b-9b3e-6229592443e1-kube-api-access-qbkcj\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:56.098231 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:56.098226 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4116cd80-c522-4b8b-9b3e-6229592443e1-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:56.098507 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:56.098251 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4116cd80-c522-4b8b-9b3e-6229592443e1-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:56.707823 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:56.707785 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f20dd15-mw284" event={"ID":"4116cd80-c522-4b8b-9b3e-6229592443e1","Type":"ContainerDied","Data":"a60a9f8e9c98ebc7ab99782a09feaead183d5f128296c383413957dfa3dc1f63"} Nov 03 20:37:56.708001 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:56.707823 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f20dd15-mw284" Nov 03 20:37:56.708001 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:37:56.707844 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a60a9f8e9c98ebc7ab99782a09feaead183d5f128296c383413957dfa3dc1f63" Nov 03 20:38:02.443931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:02.443892 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4df82f-f6w99"] Nov 03 20:38:02.444415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:02.444402 5059 cpu_manager.go:414] "containerMap: removing container" podUID="4116cd80-c522-4b8b-9b3e-6229592443e1" containerName="revoke-keys" Nov 03 20:38:02.444463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:02.444416 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4116cd80-c522-4b8b-9b3e-6229592443e1" containerName="revoke-keys" Nov 03 20:38:02.444506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:02.444496 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4116cd80-c522-4b8b-9b3e-6229592443e1" containerName="revoke-keys" Nov 03 20:38:02.447881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:02.447862 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4df82f-f6w99" Nov 03 20:38:02.454674 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:02.454638 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:38:02.456906 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:02.456883 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:38:02.458747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:02.458725 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4df82f-f6w99"] Nov 03 20:38:02.583063 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:02.583020 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d9aa74-8169-4e28-9811-c3befaa47f6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e4df82f-f6w99\" (UID: \"85d9aa74-8169-4e28-9811-c3befaa47f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4df82f-f6w99" Nov 03 20:38:02.583261 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:02.583235 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vhdm\" (UniqueName: \"kubernetes.io/projected/85d9aa74-8169-4e28-9811-c3befaa47f6d-kube-api-access-4vhdm\") pod \"maas-api-revoke-sub-7e4df82f-f6w99\" (UID: \"85d9aa74-8169-4e28-9811-c3befaa47f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4df82f-f6w99" Nov 03 20:38:02.583332 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:02.583308 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d9aa74-8169-4e28-9811-c3befaa47f6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e4df82f-f6w99\" (UID: \"85d9aa74-8169-4e28-9811-c3befaa47f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4df82f-f6w99" Nov 03 20:38:02.685071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:02.685037 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vhdm\" (UniqueName: \"kubernetes.io/projected/85d9aa74-8169-4e28-9811-c3befaa47f6d-kube-api-access-4vhdm\") pod \"maas-api-revoke-sub-7e4df82f-f6w99\" (UID: \"85d9aa74-8169-4e28-9811-c3befaa47f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4df82f-f6w99" Nov 03 20:38:02.685071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:02.685075 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d9aa74-8169-4e28-9811-c3befaa47f6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e4df82f-f6w99\" (UID: \"85d9aa74-8169-4e28-9811-c3befaa47f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4df82f-f6w99" Nov 03 20:38:02.685316 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:02.685258 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d9aa74-8169-4e28-9811-c3befaa47f6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e4df82f-f6w99\" (UID: \"85d9aa74-8169-4e28-9811-c3befaa47f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4df82f-f6w99" Nov 03 20:38:02.685851 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:02.685827 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d9aa74-8169-4e28-9811-c3befaa47f6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e4df82f-f6w99\" (UID: \"85d9aa74-8169-4e28-9811-c3befaa47f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4df82f-f6w99" Nov 03 20:38:02.687595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:02.687576 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d9aa74-8169-4e28-9811-c3befaa47f6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e4df82f-f6w99\" (UID: \"85d9aa74-8169-4e28-9811-c3befaa47f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4df82f-f6w99" Nov 03 20:38:02.697806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:02.697749 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vhdm\" (UniqueName: \"kubernetes.io/projected/85d9aa74-8169-4e28-9811-c3befaa47f6d-kube-api-access-4vhdm\") pod \"maas-api-revoke-sub-7e4df82f-f6w99\" (UID: \"85d9aa74-8169-4e28-9811-c3befaa47f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4df82f-f6w99" Nov 03 20:38:02.765282 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:02.765255 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4df82f-f6w99" Nov 03 20:38:03.045144 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:03.045111 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4df82f-f6w99"] Nov 03 20:38:03.045644 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:38:03.045619 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d9aa74_8169_4e28_9811_c3befaa47f6d.slice/crio-941753a47f2215e74119c4398f481f363f5d5cc99b7b73e4e39434ccd19ac6db WatchSource:0}: Error finding container 941753a47f2215e74119c4398f481f363f5d5cc99b7b73e4e39434ccd19ac6db: Status 404 returned error can't find the container with id 941753a47f2215e74119c4398f481f363f5d5cc99b7b73e4e39434ccd19ac6db Nov 03 20:38:03.759529 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:03.759429 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4df82f-f6w99" event={"ID":"85d9aa74-8169-4e28-9811-c3befaa47f6d","Type":"ContainerStarted","Data":"a73d1f17cbb151aa3c795ae2e9fb3a6caf3cd036fdad2181a72441d23d729288"} Nov 03 20:38:03.759529 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:03.759521 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4df82f-f6w99" event={"ID":"85d9aa74-8169-4e28-9811-c3befaa47f6d","Type":"ContainerStarted","Data":"941753a47f2215e74119c4398f481f363f5d5cc99b7b73e4e39434ccd19ac6db"} Nov 03 20:38:03.779555 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:03.779483 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4df82f-f6w99" podStartSLOduration=1.779470233 podStartE2EDuration="1.779470233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:03.778485961 +0000 UTC m=+3678.036283218" watchObservedRunningTime="2026-11-03 20:38:03.779470233 +0000 UTC m=+3678.037267489" Nov 03 20:38:04.768143 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:04.768104 5059 generic.go:356] "Generic (PLEG): container finished" podID="85d9aa74-8169-4e28-9811-c3befaa47f6d" containerID="a73d1f17cbb151aa3c795ae2e9fb3a6caf3cd036fdad2181a72441d23d729288" exitCode=0 Nov 03 20:38:04.768508 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:04.768195 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4df82f-f6w99" event={"ID":"85d9aa74-8169-4e28-9811-c3befaa47f6d","Type":"ContainerDied","Data":"a73d1f17cbb151aa3c795ae2e9fb3a6caf3cd036fdad2181a72441d23d729288"} Nov 03 20:38:05.929405 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:05.929379 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4df82f-f6w99" Nov 03 20:38:06.126494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:06.126463 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85d9aa74-8169-4e28-9811-c3befaa47f6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d9aa74-8169-4e28-9811-c3befaa47f6d-maas-api-openshift-ca\") pod \"85d9aa74-8169-4e28-9811-c3befaa47f6d\" (UID: \"85d9aa74-8169-4e28-9811-c3befaa47f6d\") " Nov 03 20:38:06.126741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:06.126593 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85d9aa74-8169-4e28-9811-c3befaa47f6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d9aa74-8169-4e28-9811-c3befaa47f6d-maas-api-xks-ca\") pod \"85d9aa74-8169-4e28-9811-c3befaa47f6d\" (UID: \"85d9aa74-8169-4e28-9811-c3befaa47f6d\") " Nov 03 20:38:06.126741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:06.126617 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85d9aa74-8169-4e28-9811-c3befaa47f6d-kube-api-access-4vhdm\" (UniqueName: \"kubernetes.io/projected/85d9aa74-8169-4e28-9811-c3befaa47f6d-kube-api-access-4vhdm\") pod \"85d9aa74-8169-4e28-9811-c3befaa47f6d\" (UID: \"85d9aa74-8169-4e28-9811-c3befaa47f6d\") " Nov 03 20:38:06.126939 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:06.126911 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d9aa74-8169-4e28-9811-c3befaa47f6d-maas-api-openshift-ca" pod "85d9aa74-8169-4e28-9811-c3befaa47f6d" (UID: "85d9aa74-8169-4e28-9811-c3befaa47f6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:06.127101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:06.127083 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d9aa74-8169-4e28-9811-c3befaa47f6d-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:06.128746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:06.128721 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85d9aa74-8169-4e28-9811-c3befaa47f6d-maas-api-xks-ca" pod "85d9aa74-8169-4e28-9811-c3befaa47f6d" (UID: "85d9aa74-8169-4e28-9811-c3befaa47f6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:38:06.128836 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:06.128811 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d9aa74-8169-4e28-9811-c3befaa47f6d-kube-api-access-4vhdm" pod "85d9aa74-8169-4e28-9811-c3befaa47f6d" (UID: "85d9aa74-8169-4e28-9811-c3befaa47f6d"). InnerVolumeSpecName "kube-api-access-4vhdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:06.228304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:06.228276 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d9aa74-8169-4e28-9811-c3befaa47f6d-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:06.228304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:06.228305 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vhdm\" (UniqueName: \"kubernetes.io/projected/85d9aa74-8169-4e28-9811-c3befaa47f6d-kube-api-access-4vhdm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:06.784677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:06.784638 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4df82f-f6w99" Nov 03 20:38:06.784883 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:06.784678 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4df82f-f6w99" event={"ID":"85d9aa74-8169-4e28-9811-c3befaa47f6d","Type":"ContainerDied","Data":"941753a47f2215e74119c4398f481f363f5d5cc99b7b73e4e39434ccd19ac6db"} Nov 03 20:38:06.784883 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:06.784764 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="941753a47f2215e74119c4398f481f363f5d5cc99b7b73e4e39434ccd19ac6db" Nov 03 20:38:10.604317 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:38:10.604271 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:38:14.797243 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:14.797210 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afbce252-5p2lc"] Nov 03 20:38:14.797732 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:14.797716 5059 cpu_manager.go:414] "containerMap: removing container" podUID="85d9aa74-8169-4e28-9811-c3befaa47f6d" containerName="revoke-keys" Nov 03 20:38:14.797786 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:14.797734 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85d9aa74-8169-4e28-9811-c3befaa47f6d" containerName="revoke-keys" Nov 03 20:38:14.797862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:14.797852 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="85d9aa74-8169-4e28-9811-c3befaa47f6d" containerName="revoke-keys" Nov 03 20:38:14.801241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:14.801219 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbce252-5p2lc" Nov 03 20:38:14.806456 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:14.806431 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:38:14.806631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:14.806430 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:38:14.810601 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:14.810577 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afbce252-5p2lc"] Nov 03 20:38:14.885700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:14.885650 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e8799b-adbe-467e-9d24-e023836ea491-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afbce252-5p2lc\" (UID: \"63e8799b-adbe-467e-9d24-e023836ea491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbce252-5p2lc" Nov 03 20:38:14.885823 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:14.885799 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbqtt\" (UniqueName: \"kubernetes.io/projected/63e8799b-adbe-467e-9d24-e023836ea491-kube-api-access-pbqtt\") pod \"maas-api-revoke-sub-afbce252-5p2lc\" (UID: \"63e8799b-adbe-467e-9d24-e023836ea491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbce252-5p2lc" Nov 03 20:38:14.886098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:14.886065 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e8799b-adbe-467e-9d24-e023836ea491-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afbce252-5p2lc\" (UID: \"63e8799b-adbe-467e-9d24-e023836ea491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbce252-5p2lc" Nov 03 20:38:14.987254 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:14.987221 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e8799b-adbe-467e-9d24-e023836ea491-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afbce252-5p2lc\" (UID: \"63e8799b-adbe-467e-9d24-e023836ea491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbce252-5p2lc" Nov 03 20:38:14.987412 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:14.987304 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e8799b-adbe-467e-9d24-e023836ea491-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afbce252-5p2lc\" (UID: \"63e8799b-adbe-467e-9d24-e023836ea491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbce252-5p2lc" Nov 03 20:38:14.987412 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:14.987374 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbqtt\" (UniqueName: \"kubernetes.io/projected/63e8799b-adbe-467e-9d24-e023836ea491-kube-api-access-pbqtt\") pod \"maas-api-revoke-sub-afbce252-5p2lc\" (UID: \"63e8799b-adbe-467e-9d24-e023836ea491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbce252-5p2lc" Nov 03 20:38:14.988014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:14.987991 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e8799b-adbe-467e-9d24-e023836ea491-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afbce252-5p2lc\" (UID: \"63e8799b-adbe-467e-9d24-e023836ea491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbce252-5p2lc" Nov 03 20:38:14.989915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:14.989893 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e8799b-adbe-467e-9d24-e023836ea491-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afbce252-5p2lc\" (UID: \"63e8799b-adbe-467e-9d24-e023836ea491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbce252-5p2lc" Nov 03 20:38:15.000089 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:15.000067 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbqtt\" (UniqueName: \"kubernetes.io/projected/63e8799b-adbe-467e-9d24-e023836ea491-kube-api-access-pbqtt\") pod \"maas-api-revoke-sub-afbce252-5p2lc\" (UID: \"63e8799b-adbe-467e-9d24-e023836ea491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbce252-5p2lc" Nov 03 20:38:15.116680 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:15.116632 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbce252-5p2lc" Nov 03 20:38:15.367952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:15.367871 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7243-900536"] Nov 03 20:38:15.372919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:15.372896 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7243-900536" Nov 03 20:38:15.376926 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:15.376893 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-tz7np\"" Nov 03 20:38:15.384258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:15.384228 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7243-900536"] Nov 03 20:38:15.400055 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:15.399975 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgzmb\" (UniqueName: \"kubernetes.io/projected/f38bac0d-bcd6-480b-9773-8b7530c60ee3-kube-api-access-pgzmb\") pod \"test-curl-7243-900536\" (UID: \"f38bac0d-bcd6-480b-9773-8b7530c60ee3\") " pod="opendatahub/test-curl-7243-900536" Nov 03 20:38:15.417071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:15.417048 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afbce252-5p2lc"] Nov 03 20:38:15.419595 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:38:15.419564 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e8799b_adbe_467e_9d24_e023836ea491.slice/crio-9e36b54f70648ccfd938cb643df6b67d1578adf5b27db6fdba7b6cf6a6304535 WatchSource:0}: Error finding container 9e36b54f70648ccfd938cb643df6b67d1578adf5b27db6fdba7b6cf6a6304535: Status 404 returned error can't find the container with id 9e36b54f70648ccfd938cb643df6b67d1578adf5b27db6fdba7b6cf6a6304535 Nov 03 20:38:15.502024 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:15.501998 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgzmb\" (UniqueName: \"kubernetes.io/projected/f38bac0d-bcd6-480b-9773-8b7530c60ee3-kube-api-access-pgzmb\") pod \"test-curl-7243-900536\" (UID: \"f38bac0d-bcd6-480b-9773-8b7530c60ee3\") " pod="opendatahub/test-curl-7243-900536" Nov 03 20:38:15.513857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:15.513830 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgzmb\" (UniqueName: \"kubernetes.io/projected/f38bac0d-bcd6-480b-9773-8b7530c60ee3-kube-api-access-pgzmb\") pod \"test-curl-7243-900536\" (UID: \"f38bac0d-bcd6-480b-9773-8b7530c60ee3\") " pod="opendatahub/test-curl-7243-900536" Nov 03 20:38:15.699565 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:15.699536 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7243-900536" Nov 03 20:38:15.858635 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:15.858598 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbce252-5p2lc" event={"ID":"63e8799b-adbe-467e-9d24-e023836ea491","Type":"ContainerStarted","Data":"c90537b876b3c98c55cfbbfdebb402a668ecd00a9ef5329a9c55ef34f54464bc"} Nov 03 20:38:15.859109 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:15.858649 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbce252-5p2lc" event={"ID":"63e8799b-adbe-467e-9d24-e023836ea491","Type":"ContainerStarted","Data":"9e36b54f70648ccfd938cb643df6b67d1578adf5b27db6fdba7b6cf6a6304535"} Nov 03 20:38:15.878330 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:15.878280 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbce252-5p2lc" podStartSLOduration=1.878266763 podStartE2EDuration="1.878266763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:15.876007046 +0000 UTC m=+3690.133804304" watchObservedRunningTime="2026-11-03 20:38:15.878266763 +0000 UTC m=+3690.136064021" Nov 03 20:38:15.998037 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:38:15.997995 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf38bac0d_bcd6_480b_9773_8b7530c60ee3.slice/crio-337fe169370f93692ff1fec7e4167e098d38d26b7181e0822257f31d15c75421 WatchSource:0}: Error finding container 337fe169370f93692ff1fec7e4167e098d38d26b7181e0822257f31d15c75421: Status 404 returned error can't find the container with id 337fe169370f93692ff1fec7e4167e098d38d26b7181e0822257f31d15c75421 Nov 03 20:38:16.005682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:16.005628 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7243-900536"] Nov 03 20:38:16.866265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:16.866221 5059 generic.go:356] "Generic (PLEG): container finished" podID="63e8799b-adbe-467e-9d24-e023836ea491" containerID="c90537b876b3c98c55cfbbfdebb402a668ecd00a9ef5329a9c55ef34f54464bc" exitCode=0 Nov 03 20:38:16.866717 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:16.866351 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbce252-5p2lc" event={"ID":"63e8799b-adbe-467e-9d24-e023836ea491","Type":"ContainerDied","Data":"c90537b876b3c98c55cfbbfdebb402a668ecd00a9ef5329a9c55ef34f54464bc"} Nov 03 20:38:16.868144 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:16.868115 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7243-900536" event={"ID":"f38bac0d-bcd6-480b-9773-8b7530c60ee3","Type":"ContainerStarted","Data":"8cdc5a414d633b080770b4f8a61e1bf4b4a015d5e8e69899741b8cfb7fc4ae47"} Nov 03 20:38:16.868261 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:16.868149 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7243-900536" event={"ID":"f38bac0d-bcd6-480b-9773-8b7530c60ee3","Type":"ContainerStarted","Data":"337fe169370f93692ff1fec7e4167e098d38d26b7181e0822257f31d15c75421"} Nov 03 20:38:16.905827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:16.905770 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7243-900536" podStartSLOduration=1.9057541580000001 podStartE2EDuration="1.905754158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:16.901338556 +0000 UTC m=+3691.159135813" watchObservedRunningTime="2026-11-03 20:38:16.905754158 +0000 UTC m=+3691.163551415" Nov 03 20:38:17.662734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:17.662693 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7243-900536"] Nov 03 20:38:17.666554 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:17.666523 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7243-900536"] Nov 03 20:38:17.817434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:17.817399 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7243-3ef9d3"] Nov 03 20:38:17.818052 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:17.818030 5059 cpu_manager.go:414] "containerMap: removing container" podUID="f38bac0d-bcd6-480b-9773-8b7530c60ee3" containerName="test-curl-7243-900536" Nov 03 20:38:17.818170 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:17.818053 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f38bac0d-bcd6-480b-9773-8b7530c60ee3" containerName="test-curl-7243-900536" Nov 03 20:38:17.818234 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:17.818166 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f38bac0d-bcd6-480b-9773-8b7530c60ee3" containerName="test-curl-7243-900536" Nov 03 20:38:17.821652 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:17.821630 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7243-3ef9d3" Nov 03 20:38:17.826234 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:17.826202 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7243-3ef9d3"] Nov 03 20:38:17.939628 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:17.939589 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh4sm\" (UniqueName: \"kubernetes.io/projected/d5f5d3fc-b391-480f-b0c3-d679d5faaf7a-kube-api-access-vh4sm\") pod \"test-curl-7243-3ef9d3\" (UID: \"d5f5d3fc-b391-480f-b0c3-d679d5faaf7a\") " pod="opendatahub/test-curl-7243-3ef9d3" Nov 03 20:38:18.025514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:18.025486 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbce252-5p2lc" Nov 03 20:38:18.041095 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:18.041067 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh4sm\" (UniqueName: \"kubernetes.io/projected/d5f5d3fc-b391-480f-b0c3-d679d5faaf7a-kube-api-access-vh4sm\") pod \"test-curl-7243-3ef9d3\" (UID: \"d5f5d3fc-b391-480f-b0c3-d679d5faaf7a\") " pod="opendatahub/test-curl-7243-3ef9d3" Nov 03 20:38:18.053877 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:18.053850 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh4sm\" (UniqueName: \"kubernetes.io/projected/d5f5d3fc-b391-480f-b0c3-d679d5faaf7a-kube-api-access-vh4sm\") pod \"test-curl-7243-3ef9d3\" (UID: \"d5f5d3fc-b391-480f-b0c3-d679d5faaf7a\") " pod="opendatahub/test-curl-7243-3ef9d3" Nov 03 20:38:18.141307 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:18.141277 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7243-3ef9d3" Nov 03 20:38:18.142565 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:18.142540 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63e8799b-adbe-467e-9d24-e023836ea491-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e8799b-adbe-467e-9d24-e023836ea491-maas-api-xks-ca\") pod \"63e8799b-adbe-467e-9d24-e023836ea491\" (UID: \"63e8799b-adbe-467e-9d24-e023836ea491\") " Nov 03 20:38:18.142740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:18.142718 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63e8799b-adbe-467e-9d24-e023836ea491-kube-api-access-pbqtt\" (UniqueName: \"kubernetes.io/projected/63e8799b-adbe-467e-9d24-e023836ea491-kube-api-access-pbqtt\") pod \"63e8799b-adbe-467e-9d24-e023836ea491\" (UID: \"63e8799b-adbe-467e-9d24-e023836ea491\") " Nov 03 20:38:18.142850 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:18.142839 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63e8799b-adbe-467e-9d24-e023836ea491-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e8799b-adbe-467e-9d24-e023836ea491-maas-api-openshift-ca\") pod \"63e8799b-adbe-467e-9d24-e023836ea491\" (UID: \"63e8799b-adbe-467e-9d24-e023836ea491\") " Nov 03 20:38:18.143176 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:18.143149 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e8799b-adbe-467e-9d24-e023836ea491-maas-api-openshift-ca" pod "63e8799b-adbe-467e-9d24-e023836ea491" (UID: "63e8799b-adbe-467e-9d24-e023836ea491"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:18.143319 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:18.143288 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e8799b-adbe-467e-9d24-e023836ea491-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:18.144643 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:18.144620 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63e8799b-adbe-467e-9d24-e023836ea491-maas-api-xks-ca" pod "63e8799b-adbe-467e-9d24-e023836ea491" (UID: "63e8799b-adbe-467e-9d24-e023836ea491"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:38:18.144854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:18.144782 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e8799b-adbe-467e-9d24-e023836ea491-kube-api-access-pbqtt" pod "63e8799b-adbe-467e-9d24-e023836ea491" (UID: "63e8799b-adbe-467e-9d24-e023836ea491"). InnerVolumeSpecName "kube-api-access-pbqtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:18.245093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:18.245060 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e8799b-adbe-467e-9d24-e023836ea491-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:18.245093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:18.245091 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbqtt\" (UniqueName: \"kubernetes.io/projected/63e8799b-adbe-467e-9d24-e023836ea491-kube-api-access-pbqtt\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:18.438165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:18.438131 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7243-3ef9d3"] Nov 03 20:38:18.438861 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:38:18.438831 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5f5d3fc_b391_480f_b0c3_d679d5faaf7a.slice/crio-9155db08123af8413afed4f62f7d1815b8d6e9f373bbd028b8d6ce9448fd34ee WatchSource:0}: Error finding container 9155db08123af8413afed4f62f7d1815b8d6e9f373bbd028b8d6ce9448fd34ee: Status 404 returned error can't find the container with id 9155db08123af8413afed4f62f7d1815b8d6e9f373bbd028b8d6ce9448fd34ee Nov 03 20:38:18.898294 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:18.898246 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7243-3ef9d3" event={"ID":"d5f5d3fc-b391-480f-b0c3-d679d5faaf7a","Type":"ContainerStarted","Data":"9b82e23f91f0349018c3802e958c8374ea5760b1d885a411c088652c61462a83"} Nov 03 20:38:18.898701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:18.898643 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7243-3ef9d3" event={"ID":"d5f5d3fc-b391-480f-b0c3-d679d5faaf7a","Type":"ContainerStarted","Data":"9155db08123af8413afed4f62f7d1815b8d6e9f373bbd028b8d6ce9448fd34ee"} Nov 03 20:38:18.901162 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:18.901137 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbce252-5p2lc" Nov 03 20:38:18.901268 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:18.901202 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7243-900536" podUID="f38bac0d-bcd6-480b-9773-8b7530c60ee3" containerName="test-curl-7243-900536" containerID="cri-o://8cdc5a414d633b080770b4f8a61e1bf4b4a015d5e8e69899741b8cfb7fc4ae47" gracePeriod=2 Nov 03 20:38:18.901319 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:18.901299 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbce252-5p2lc" event={"ID":"63e8799b-adbe-467e-9d24-e023836ea491","Type":"ContainerDied","Data":"9e36b54f70648ccfd938cb643df6b67d1578adf5b27db6fdba7b6cf6a6304535"} Nov 03 20:38:18.901367 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:18.901336 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e36b54f70648ccfd938cb643df6b67d1578adf5b27db6fdba7b6cf6a6304535" Nov 03 20:38:18.916641 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:18.916602 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7243-3ef9d3" podStartSLOduration=1.916590136 podStartE2EDuration="1.916590136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:18.915963308 +0000 UTC m=+3693.173760566" watchObservedRunningTime="2026-11-03 20:38:18.916590136 +0000 UTC m=+3693.174387392" Nov 03 20:38:19.613178 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:19.613141 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7243-3ef9d3"] Nov 03 20:38:19.617396 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:19.617362 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7243-3ef9d3"] Nov 03 20:38:20.061901 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:20.061823 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef84e730-nwk7x"] Nov 03 20:38:20.062328 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:20.062314 5059 cpu_manager.go:414] "containerMap: removing container" podUID="63e8799b-adbe-467e-9d24-e023836ea491" containerName="revoke-keys" Nov 03 20:38:20.062375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:20.062329 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e8799b-adbe-467e-9d24-e023836ea491" containerName="revoke-keys" Nov 03 20:38:20.062375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:20.062348 5059 cpu_manager.go:414] "containerMap: removing container" podUID="d5f5d3fc-b391-480f-b0c3-d679d5faaf7a" containerName="test-curl-7243-3ef9d3" Nov 03 20:38:20.062375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:20.062354 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5f5d3fc-b391-480f-b0c3-d679d5faaf7a" containerName="test-curl-7243-3ef9d3" Nov 03 20:38:20.062466 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:20.062439 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e8799b-adbe-467e-9d24-e023836ea491" containerName="revoke-keys" Nov 03 20:38:20.062466 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:20.062449 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5f5d3fc-b391-480f-b0c3-d679d5faaf7a" containerName="test-curl-7243-3ef9d3" Nov 03 20:38:20.065808 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:20.065791 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef84e730-nwk7x" Nov 03 20:38:20.070500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:20.070474 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:38:20.071642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:20.071617 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:38:20.075300 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:20.075275 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef84e730-nwk7x"] Nov 03 20:38:20.169214 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:20.169170 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9838d42f-c011-4397-b083-22844a12a935-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef84e730-nwk7x\" (UID: \"9838d42f-c011-4397-b083-22844a12a935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef84e730-nwk7x" Nov 03 20:38:20.169358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:20.169305 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df8xs\" (UniqueName: \"kubernetes.io/projected/9838d42f-c011-4397-b083-22844a12a935-kube-api-access-df8xs\") pod \"maas-api-revoke-sub-ef84e730-nwk7x\" (UID: \"9838d42f-c011-4397-b083-22844a12a935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef84e730-nwk7x" Nov 03 20:38:20.169396 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:20.169385 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9838d42f-c011-4397-b083-22844a12a935-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef84e730-nwk7x\" (UID: \"9838d42f-c011-4397-b083-22844a12a935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef84e730-nwk7x" Nov 03 20:38:20.270204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:20.270169 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df8xs\" (UniqueName: \"kubernetes.io/projected/9838d42f-c011-4397-b083-22844a12a935-kube-api-access-df8xs\") pod \"maas-api-revoke-sub-ef84e730-nwk7x\" (UID: \"9838d42f-c011-4397-b083-22844a12a935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef84e730-nwk7x" Nov 03 20:38:20.270343 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:20.270264 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9838d42f-c011-4397-b083-22844a12a935-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef84e730-nwk7x\" (UID: \"9838d42f-c011-4397-b083-22844a12a935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef84e730-nwk7x" Nov 03 20:38:20.270387 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:20.270377 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9838d42f-c011-4397-b083-22844a12a935-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef84e730-nwk7x\" (UID: \"9838d42f-c011-4397-b083-22844a12a935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef84e730-nwk7x" Nov 03 20:38:20.271061 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:20.271040 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9838d42f-c011-4397-b083-22844a12a935-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef84e730-nwk7x\" (UID: \"9838d42f-c011-4397-b083-22844a12a935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef84e730-nwk7x" Nov 03 20:38:20.272692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:20.272658 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9838d42f-c011-4397-b083-22844a12a935-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef84e730-nwk7x\" (UID: \"9838d42f-c011-4397-b083-22844a12a935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef84e730-nwk7x" Nov 03 20:38:20.282838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:20.282819 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df8xs\" (UniqueName: \"kubernetes.io/projected/9838d42f-c011-4397-b083-22844a12a935-kube-api-access-df8xs\") pod \"maas-api-revoke-sub-ef84e730-nwk7x\" (UID: \"9838d42f-c011-4397-b083-22844a12a935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef84e730-nwk7x" Nov 03 20:38:20.381541 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:20.381507 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef84e730-nwk7x" Nov 03 20:38:20.666314 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:20.666279 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef84e730-nwk7x"] Nov 03 20:38:20.666749 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:38:20.666719 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9838d42f_c011_4397_b083_22844a12a935.slice/crio-c7263c2819c6e7ab5931dda827959b256a5b06c27e71ec8496b32448d48f951c WatchSource:0}: Error finding container c7263c2819c6e7ab5931dda827959b256a5b06c27e71ec8496b32448d48f951c: Status 404 returned error can't find the container with id c7263c2819c6e7ab5931dda827959b256a5b06c27e71ec8496b32448d48f951c Nov 03 20:38:20.914625 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:20.914569 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef84e730-nwk7x" event={"ID":"9838d42f-c011-4397-b083-22844a12a935","Type":"ContainerStarted","Data":"c7263c2819c6e7ab5931dda827959b256a5b06c27e71ec8496b32448d48f951c"} Nov 03 20:38:20.915401 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:20.914777 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7243-3ef9d3" podUID="d5f5d3fc-b391-480f-b0c3-d679d5faaf7a" containerName="test-curl-7243-3ef9d3" containerID="cri-o://9b82e23f91f0349018c3802e958c8374ea5760b1d885a411c088652c61462a83" gracePeriod=2 Nov 03 20:38:21.098355 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:21.098330 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7243-900536" Nov 03 20:38:21.183471 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:21.183407 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f38bac0d-bcd6-480b-9773-8b7530c60ee3-kube-api-access-pgzmb\" (UniqueName: \"kubernetes.io/projected/f38bac0d-bcd6-480b-9773-8b7530c60ee3-kube-api-access-pgzmb\") pod \"f38bac0d-bcd6-480b-9773-8b7530c60ee3\" (UID: \"f38bac0d-bcd6-480b-9773-8b7530c60ee3\") " Nov 03 20:38:21.185454 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:21.185430 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38bac0d-bcd6-480b-9773-8b7530c60ee3-kube-api-access-pgzmb" pod "f38bac0d-bcd6-480b-9773-8b7530c60ee3" (UID: "f38bac0d-bcd6-480b-9773-8b7530c60ee3"). InnerVolumeSpecName "kube-api-access-pgzmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:21.285494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:21.285462 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgzmb\" (UniqueName: \"kubernetes.io/projected/f38bac0d-bcd6-480b-9773-8b7530c60ee3-kube-api-access-pgzmb\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:21.857232 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:21.857192 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f38bac0d-bcd6-480b-9773-8b7530c60ee3" path="/var/lib/kubelet/pods/f38bac0d-bcd6-480b-9773-8b7530c60ee3/volumes" Nov 03 20:38:21.925728 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:21.925640 5059 generic.go:356] "Generic (PLEG): container finished" podID="f38bac0d-bcd6-480b-9773-8b7530c60ee3" containerID="8cdc5a414d633b080770b4f8a61e1bf4b4a015d5e8e69899741b8cfb7fc4ae47" exitCode=137 Nov 03 20:38:21.925728 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:21.925726 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7243-900536" Nov 03 20:38:21.925983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:21.925750 5059 scope.go:122] "RemoveContainer" containerID="8cdc5a414d633b080770b4f8a61e1bf4b4a015d5e8e69899741b8cfb7fc4ae47" Nov 03 20:38:21.927927 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:21.927894 5059 generic.go:356] "Generic (PLEG): container finished" podID="9838d42f-c011-4397-b083-22844a12a935" containerID="fa1149da688477f85a9f5ad2c8fec97d2e9933c65e460445477663d5bc18db84" exitCode=0 Nov 03 20:38:21.928062 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:21.927980 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef84e730-nwk7x" event={"ID":"9838d42f-c011-4397-b083-22844a12a935","Type":"ContainerDied","Data":"fa1149da688477f85a9f5ad2c8fec97d2e9933c65e460445477663d5bc18db84"} Nov 03 20:38:21.935572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:21.935551 5059 scope.go:122] "RemoveContainer" containerID="8cdc5a414d633b080770b4f8a61e1bf4b4a015d5e8e69899741b8cfb7fc4ae47" Nov 03 20:38:21.935861 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:38:21.935841 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cdc5a414d633b080770b4f8a61e1bf4b4a015d5e8e69899741b8cfb7fc4ae47\": container with ID starting with 8cdc5a414d633b080770b4f8a61e1bf4b4a015d5e8e69899741b8cfb7fc4ae47 not found: ID does not exist" containerID="8cdc5a414d633b080770b4f8a61e1bf4b4a015d5e8e69899741b8cfb7fc4ae47" Nov 03 20:38:21.935944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:21.935868 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cdc5a414d633b080770b4f8a61e1bf4b4a015d5e8e69899741b8cfb7fc4ae47"} err="failed to get container status \"8cdc5a414d633b080770b4f8a61e1bf4b4a015d5e8e69899741b8cfb7fc4ae47\": rpc error: code = NotFound desc = could not find container \"8cdc5a414d633b080770b4f8a61e1bf4b4a015d5e8e69899741b8cfb7fc4ae47\": container with ID starting with 8cdc5a414d633b080770b4f8a61e1bf4b4a015d5e8e69899741b8cfb7fc4ae47 not found: ID does not exist" Nov 03 20:38:23.093813 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:23.093786 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef84e730-nwk7x" Nov 03 20:38:23.126187 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:23.126159 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7243-3ef9d3" Nov 03 20:38:23.217430 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:23.217361 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9838d42f-c011-4397-b083-22844a12a935-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9838d42f-c011-4397-b083-22844a12a935-maas-api-openshift-ca\") pod \"9838d42f-c011-4397-b083-22844a12a935\" (UID: \"9838d42f-c011-4397-b083-22844a12a935\") " Nov 03 20:38:23.217430 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:23.217416 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9838d42f-c011-4397-b083-22844a12a935-kube-api-access-df8xs\" (UniqueName: \"kubernetes.io/projected/9838d42f-c011-4397-b083-22844a12a935-kube-api-access-df8xs\") pod \"9838d42f-c011-4397-b083-22844a12a935\" (UID: \"9838d42f-c011-4397-b083-22844a12a935\") " Nov 03 20:38:23.217586 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:23.217574 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9838d42f-c011-4397-b083-22844a12a935-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9838d42f-c011-4397-b083-22844a12a935-maas-api-xks-ca\") pod \"9838d42f-c011-4397-b083-22844a12a935\" (UID: \"9838d42f-c011-4397-b083-22844a12a935\") " Nov 03 20:38:23.217634 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:23.217621 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5f5d3fc-b391-480f-b0c3-d679d5faaf7a-kube-api-access-vh4sm\" (UniqueName: \"kubernetes.io/projected/d5f5d3fc-b391-480f-b0c3-d679d5faaf7a-kube-api-access-vh4sm\") pod \"d5f5d3fc-b391-480f-b0c3-d679d5faaf7a\" (UID: \"d5f5d3fc-b391-480f-b0c3-d679d5faaf7a\") " Nov 03 20:38:23.217741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:23.217718 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9838d42f-c011-4397-b083-22844a12a935-maas-api-openshift-ca" pod "9838d42f-c011-4397-b083-22844a12a935" (UID: "9838d42f-c011-4397-b083-22844a12a935"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:23.218180 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:23.218155 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9838d42f-c011-4397-b083-22844a12a935-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:23.219657 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:23.219634 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9838d42f-c011-4397-b083-22844a12a935-kube-api-access-df8xs" pod "9838d42f-c011-4397-b083-22844a12a935" (UID: "9838d42f-c011-4397-b083-22844a12a935"). InnerVolumeSpecName "kube-api-access-df8xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:23.219771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:23.219687 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f5d3fc-b391-480f-b0c3-d679d5faaf7a-kube-api-access-vh4sm" pod "d5f5d3fc-b391-480f-b0c3-d679d5faaf7a" (UID: "d5f5d3fc-b391-480f-b0c3-d679d5faaf7a"). InnerVolumeSpecName "kube-api-access-vh4sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:23.219771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:23.219736 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9838d42f-c011-4397-b083-22844a12a935-maas-api-xks-ca" pod "9838d42f-c011-4397-b083-22844a12a935" (UID: "9838d42f-c011-4397-b083-22844a12a935"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:38:23.319545 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:23.319509 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9838d42f-c011-4397-b083-22844a12a935-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:23.319545 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:23.319541 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh4sm\" (UniqueName: \"kubernetes.io/projected/d5f5d3fc-b391-480f-b0c3-d679d5faaf7a-kube-api-access-vh4sm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:23.319725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:23.319552 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df8xs\" (UniqueName: \"kubernetes.io/projected/9838d42f-c011-4397-b083-22844a12a935-kube-api-access-df8xs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:23.856904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:23.856868 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5f5d3fc-b391-480f-b0c3-d679d5faaf7a" path="/var/lib/kubelet/pods/d5f5d3fc-b391-480f-b0c3-d679d5faaf7a/volumes" Nov 03 20:38:23.941879 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:23.941851 5059 generic.go:356] "Generic (PLEG): container finished" podID="d5f5d3fc-b391-480f-b0c3-d679d5faaf7a" containerID="9b82e23f91f0349018c3802e958c8374ea5760b1d885a411c088652c61462a83" exitCode=137 Nov 03 20:38:23.941986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:23.941896 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7243-3ef9d3" Nov 03 20:38:23.941986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:23.941948 5059 scope.go:122] "RemoveContainer" containerID="9b82e23f91f0349018c3802e958c8374ea5760b1d885a411c088652c61462a83" Nov 03 20:38:23.943631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:23.943608 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef84e730-nwk7x" Nov 03 20:38:23.943918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:23.943610 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef84e730-nwk7x" event={"ID":"9838d42f-c011-4397-b083-22844a12a935","Type":"ContainerDied","Data":"c7263c2819c6e7ab5931dda827959b256a5b06c27e71ec8496b32448d48f951c"} Nov 03 20:38:23.944065 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:23.943979 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7263c2819c6e7ab5931dda827959b256a5b06c27e71ec8496b32448d48f951c" Nov 03 20:38:23.952570 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:23.952362 5059 scope.go:122] "RemoveContainer" containerID="9b82e23f91f0349018c3802e958c8374ea5760b1d885a411c088652c61462a83" Nov 03 20:38:23.954800 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:38:23.954769 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b82e23f91f0349018c3802e958c8374ea5760b1d885a411c088652c61462a83\": container with ID starting with 9b82e23f91f0349018c3802e958c8374ea5760b1d885a411c088652c61462a83 not found: ID does not exist" containerID="9b82e23f91f0349018c3802e958c8374ea5760b1d885a411c088652c61462a83" Nov 03 20:38:23.954920 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:23.954809 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b82e23f91f0349018c3802e958c8374ea5760b1d885a411c088652c61462a83"} err="failed to get container status \"9b82e23f91f0349018c3802e958c8374ea5760b1d885a411c088652c61462a83\": rpc error: code = NotFound desc = could not find container \"9b82e23f91f0349018c3802e958c8374ea5760b1d885a411c088652c61462a83\": container with ID starting with 9b82e23f91f0349018c3802e958c8374ea5760b1d885a411c088652c61462a83 not found: ID does not exist" Nov 03 20:38:25.157769 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:25.157733 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1cffd0a-jw76s"] Nov 03 20:38:25.158280 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:25.158266 5059 cpu_manager.go:414] "containerMap: removing container" podUID="9838d42f-c011-4397-b083-22844a12a935" containerName="revoke-keys" Nov 03 20:38:25.158332 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:25.158280 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9838d42f-c011-4397-b083-22844a12a935" containerName="revoke-keys" Nov 03 20:38:25.158373 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:25.158363 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9838d42f-c011-4397-b083-22844a12a935" containerName="revoke-keys" Nov 03 20:38:25.162881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:25.162863 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cffd0a-jw76s" Nov 03 20:38:25.167204 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:25.167177 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:38:25.167354 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:25.167192 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:38:25.170907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:25.170884 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1cffd0a-jw76s"] Nov 03 20:38:25.342545 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:25.342507 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f78637-cb8a-476f-a6cb-40db782c9796-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1cffd0a-jw76s\" (UID: \"94f78637-cb8a-476f-a6cb-40db782c9796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cffd0a-jw76s" Nov 03 20:38:25.342739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:25.342574 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxztq\" (UniqueName: \"kubernetes.io/projected/94f78637-cb8a-476f-a6cb-40db782c9796-kube-api-access-dxztq\") pod \"maas-api-revoke-sub-c1cffd0a-jw76s\" (UID: \"94f78637-cb8a-476f-a6cb-40db782c9796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cffd0a-jw76s" Nov 03 20:38:25.342857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:25.342835 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f78637-cb8a-476f-a6cb-40db782c9796-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1cffd0a-jw76s\" (UID: \"94f78637-cb8a-476f-a6cb-40db782c9796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cffd0a-jw76s" Nov 03 20:38:25.443483 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:25.443412 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f78637-cb8a-476f-a6cb-40db782c9796-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1cffd0a-jw76s\" (UID: \"94f78637-cb8a-476f-a6cb-40db782c9796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cffd0a-jw76s" Nov 03 20:38:25.443483 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:25.443453 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f78637-cb8a-476f-a6cb-40db782c9796-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1cffd0a-jw76s\" (UID: \"94f78637-cb8a-476f-a6cb-40db782c9796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cffd0a-jw76s" Nov 03 20:38:25.443691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:25.443490 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxztq\" (UniqueName: \"kubernetes.io/projected/94f78637-cb8a-476f-a6cb-40db782c9796-kube-api-access-dxztq\") pod \"maas-api-revoke-sub-c1cffd0a-jw76s\" (UID: \"94f78637-cb8a-476f-a6cb-40db782c9796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cffd0a-jw76s" Nov 03 20:38:25.444186 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:25.444162 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f78637-cb8a-476f-a6cb-40db782c9796-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1cffd0a-jw76s\" (UID: \"94f78637-cb8a-476f-a6cb-40db782c9796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cffd0a-jw76s" Nov 03 20:38:25.445813 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:25.445783 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f78637-cb8a-476f-a6cb-40db782c9796-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1cffd0a-jw76s\" (UID: \"94f78637-cb8a-476f-a6cb-40db782c9796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cffd0a-jw76s" Nov 03 20:38:25.455474 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:25.455450 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxztq\" (UniqueName: \"kubernetes.io/projected/94f78637-cb8a-476f-a6cb-40db782c9796-kube-api-access-dxztq\") pod \"maas-api-revoke-sub-c1cffd0a-jw76s\" (UID: \"94f78637-cb8a-476f-a6cb-40db782c9796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cffd0a-jw76s" Nov 03 20:38:25.479243 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:25.479218 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cffd0a-jw76s" Nov 03 20:38:25.762604 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:25.762567 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1cffd0a-jw76s"] Nov 03 20:38:25.762932 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:38:25.762907 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f78637_cb8a_476f_a6cb_40db782c9796.slice/crio-cfa1ce4702dda3a912cbc9aabf54b10ccf88138f10250ca5d56f4a562492f62a WatchSource:0}: Error finding container cfa1ce4702dda3a912cbc9aabf54b10ccf88138f10250ca5d56f4a562492f62a: Status 404 returned error can't find the container with id cfa1ce4702dda3a912cbc9aabf54b10ccf88138f10250ca5d56f4a562492f62a Nov 03 20:38:25.957969 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:25.957924 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cffd0a-jw76s" event={"ID":"94f78637-cb8a-476f-a6cb-40db782c9796","Type":"ContainerStarted","Data":"cfa1ce4702dda3a912cbc9aabf54b10ccf88138f10250ca5d56f4a562492f62a"} Nov 03 20:38:26.977533 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:26.977446 5059 generic.go:356] "Generic (PLEG): container finished" podID="94f78637-cb8a-476f-a6cb-40db782c9796" containerID="4dfc1c488968df6a3c6e2090c8ab8f4cda5e9e8aa654dc23b5dd6a2512bfb770" exitCode=0 Nov 03 20:38:26.977533 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:26.977492 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cffd0a-jw76s" event={"ID":"94f78637-cb8a-476f-a6cb-40db782c9796","Type":"ContainerDied","Data":"4dfc1c488968df6a3c6e2090c8ab8f4cda5e9e8aa654dc23b5dd6a2512bfb770"} Nov 03 20:38:28.162039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:28.162005 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cffd0a-jw76s" Nov 03 20:38:28.200215 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:28.200185 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94f78637-cb8a-476f-a6cb-40db782c9796-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f78637-cb8a-476f-a6cb-40db782c9796-maas-api-xks-ca\") pod \"94f78637-cb8a-476f-a6cb-40db782c9796\" (UID: \"94f78637-cb8a-476f-a6cb-40db782c9796\") " Nov 03 20:38:28.200388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:28.200227 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f78637-cb8a-476f-a6cb-40db782c9796-kube-api-access-dxztq\" (UniqueName: \"kubernetes.io/projected/94f78637-cb8a-476f-a6cb-40db782c9796-kube-api-access-dxztq\") pod \"94f78637-cb8a-476f-a6cb-40db782c9796\" (UID: \"94f78637-cb8a-476f-a6cb-40db782c9796\") " Nov 03 20:38:28.200388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:28.200250 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94f78637-cb8a-476f-a6cb-40db782c9796-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f78637-cb8a-476f-a6cb-40db782c9796-maas-api-openshift-ca\") pod \"94f78637-cb8a-476f-a6cb-40db782c9796\" (UID: \"94f78637-cb8a-476f-a6cb-40db782c9796\") " Nov 03 20:38:28.200838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:28.200796 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f78637-cb8a-476f-a6cb-40db782c9796-maas-api-openshift-ca" pod "94f78637-cb8a-476f-a6cb-40db782c9796" (UID: "94f78637-cb8a-476f-a6cb-40db782c9796"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:28.200984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:28.200900 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f78637-cb8a-476f-a6cb-40db782c9796-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:28.202495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:28.202472 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94f78637-cb8a-476f-a6cb-40db782c9796-maas-api-xks-ca" pod "94f78637-cb8a-476f-a6cb-40db782c9796" (UID: "94f78637-cb8a-476f-a6cb-40db782c9796"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:38:28.202635 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:28.202618 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f78637-cb8a-476f-a6cb-40db782c9796-kube-api-access-dxztq" pod "94f78637-cb8a-476f-a6cb-40db782c9796" (UID: "94f78637-cb8a-476f-a6cb-40db782c9796"). InnerVolumeSpecName "kube-api-access-dxztq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:28.302312 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:28.302217 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f78637-cb8a-476f-a6cb-40db782c9796-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:28.302312 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:28.302259 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxztq\" (UniqueName: \"kubernetes.io/projected/94f78637-cb8a-476f-a6cb-40db782c9796-kube-api-access-dxztq\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:28.991897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:28.991865 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cffd0a-jw76s" Nov 03 20:38:28.992089 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:28.991904 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cffd0a-jw76s" event={"ID":"94f78637-cb8a-476f-a6cb-40db782c9796","Type":"ContainerDied","Data":"cfa1ce4702dda3a912cbc9aabf54b10ccf88138f10250ca5d56f4a562492f62a"} Nov 03 20:38:28.992089 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:28.991964 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfa1ce4702dda3a912cbc9aabf54b10ccf88138f10250ca5d56f4a562492f62a" Nov 03 20:38:30.286920 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:30.286887 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91eb51b2-ffrxf"] Nov 03 20:38:30.287430 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:30.287415 5059 cpu_manager.go:414] "containerMap: removing container" podUID="94f78637-cb8a-476f-a6cb-40db782c9796" containerName="revoke-keys" Nov 03 20:38:30.287484 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:30.287432 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f78637-cb8a-476f-a6cb-40db782c9796" containerName="revoke-keys" Nov 03 20:38:30.287517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:30.287500 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f78637-cb8a-476f-a6cb-40db782c9796" containerName="revoke-keys" Nov 03 20:38:30.290862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:30.290845 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91eb51b2-ffrxf" Nov 03 20:38:30.296354 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:30.296328 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:38:30.296561 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:30.296333 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:38:30.301190 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:30.301167 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91eb51b2-ffrxf"] Nov 03 20:38:30.336685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:30.329564 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698bcd39-e520-42bc-90d2-04a26edd1226-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91eb51b2-ffrxf\" (UID: \"698bcd39-e520-42bc-90d2-04a26edd1226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91eb51b2-ffrxf" Nov 03 20:38:30.336685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:30.329677 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698bcd39-e520-42bc-90d2-04a26edd1226-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91eb51b2-ffrxf\" (UID: \"698bcd39-e520-42bc-90d2-04a26edd1226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91eb51b2-ffrxf" Nov 03 20:38:30.336685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:30.329717 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9md7n\" (UniqueName: \"kubernetes.io/projected/698bcd39-e520-42bc-90d2-04a26edd1226-kube-api-access-9md7n\") pod \"maas-api-revoke-sub-91eb51b2-ffrxf\" (UID: \"698bcd39-e520-42bc-90d2-04a26edd1226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91eb51b2-ffrxf" Nov 03 20:38:30.451559 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:30.451508 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698bcd39-e520-42bc-90d2-04a26edd1226-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91eb51b2-ffrxf\" (UID: \"698bcd39-e520-42bc-90d2-04a26edd1226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91eb51b2-ffrxf" Nov 03 20:38:30.451819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:30.451608 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698bcd39-e520-42bc-90d2-04a26edd1226-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91eb51b2-ffrxf\" (UID: \"698bcd39-e520-42bc-90d2-04a26edd1226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91eb51b2-ffrxf" Nov 03 20:38:30.451819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:30.451641 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9md7n\" (UniqueName: \"kubernetes.io/projected/698bcd39-e520-42bc-90d2-04a26edd1226-kube-api-access-9md7n\") pod \"maas-api-revoke-sub-91eb51b2-ffrxf\" (UID: \"698bcd39-e520-42bc-90d2-04a26edd1226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91eb51b2-ffrxf" Nov 03 20:38:30.454468 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:30.454434 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698bcd39-e520-42bc-90d2-04a26edd1226-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91eb51b2-ffrxf\" (UID: \"698bcd39-e520-42bc-90d2-04a26edd1226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91eb51b2-ffrxf" Nov 03 20:38:30.459711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:30.456179 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698bcd39-e520-42bc-90d2-04a26edd1226-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91eb51b2-ffrxf\" (UID: \"698bcd39-e520-42bc-90d2-04a26edd1226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91eb51b2-ffrxf" Nov 03 20:38:30.465211 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:30.465184 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9md7n\" (UniqueName: \"kubernetes.io/projected/698bcd39-e520-42bc-90d2-04a26edd1226-kube-api-access-9md7n\") pod \"maas-api-revoke-sub-91eb51b2-ffrxf\" (UID: \"698bcd39-e520-42bc-90d2-04a26edd1226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91eb51b2-ffrxf" Nov 03 20:38:30.606172 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:30.606133 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91eb51b2-ffrxf" Nov 03 20:38:30.973807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:30.973768 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91eb51b2-ffrxf"] Nov 03 20:38:31.004808 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:31.004780 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91eb51b2-ffrxf" event={"ID":"698bcd39-e520-42bc-90d2-04a26edd1226","Type":"ContainerStarted","Data":"8d02d1866148f177303d1d8f1cc8bbb2c32c47381d92c22420c4952499547237"} Nov 03 20:38:32.012894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:32.012861 5059 generic.go:356] "Generic (PLEG): container finished" podID="698bcd39-e520-42bc-90d2-04a26edd1226" containerID="41e6f3ac25965c349009bee82654ba6d7bfbd2293c37b04e17673cefb35a9a22" exitCode=0 Nov 03 20:38:32.013244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:32.012962 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91eb51b2-ffrxf" event={"ID":"698bcd39-e520-42bc-90d2-04a26edd1226","Type":"ContainerDied","Data":"41e6f3ac25965c349009bee82654ba6d7bfbd2293c37b04e17673cefb35a9a22"} Nov 03 20:38:33.314089 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:33.314058 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91eb51b2-ffrxf" Nov 03 20:38:33.430923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:33.430885 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/698bcd39-e520-42bc-90d2-04a26edd1226-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698bcd39-e520-42bc-90d2-04a26edd1226-maas-api-xks-ca\") pod \"698bcd39-e520-42bc-90d2-04a26edd1226\" (UID: \"698bcd39-e520-42bc-90d2-04a26edd1226\") " Nov 03 20:38:33.431110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:33.430961 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/698bcd39-e520-42bc-90d2-04a26edd1226-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698bcd39-e520-42bc-90d2-04a26edd1226-maas-api-openshift-ca\") pod \"698bcd39-e520-42bc-90d2-04a26edd1226\" (UID: \"698bcd39-e520-42bc-90d2-04a26edd1226\") " Nov 03 20:38:33.431110 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:33.431082 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/698bcd39-e520-42bc-90d2-04a26edd1226-kube-api-access-9md7n\" (UniqueName: \"kubernetes.io/projected/698bcd39-e520-42bc-90d2-04a26edd1226-kube-api-access-9md7n\") pod \"698bcd39-e520-42bc-90d2-04a26edd1226\" (UID: \"698bcd39-e520-42bc-90d2-04a26edd1226\") " Nov 03 20:38:33.431370 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:33.431342 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/698bcd39-e520-42bc-90d2-04a26edd1226-maas-api-openshift-ca" pod "698bcd39-e520-42bc-90d2-04a26edd1226" (UID: "698bcd39-e520-42bc-90d2-04a26edd1226"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:33.431676 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:33.431641 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698bcd39-e520-42bc-90d2-04a26edd1226-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:33.433237 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:33.433208 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/698bcd39-e520-42bc-90d2-04a26edd1226-maas-api-xks-ca" pod "698bcd39-e520-42bc-90d2-04a26edd1226" (UID: "698bcd39-e520-42bc-90d2-04a26edd1226"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:38:33.433340 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:33.433316 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698bcd39-e520-42bc-90d2-04a26edd1226-kube-api-access-9md7n" pod "698bcd39-e520-42bc-90d2-04a26edd1226" (UID: "698bcd39-e520-42bc-90d2-04a26edd1226"). InnerVolumeSpecName "kube-api-access-9md7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:33.536686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:33.535797 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9md7n\" (UniqueName: \"kubernetes.io/projected/698bcd39-e520-42bc-90d2-04a26edd1226-kube-api-access-9md7n\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:33.536686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:33.535840 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698bcd39-e520-42bc-90d2-04a26edd1226-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:34.027631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:34.027600 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91eb51b2-ffrxf" Nov 03 20:38:34.027837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:34.027606 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91eb51b2-ffrxf" event={"ID":"698bcd39-e520-42bc-90d2-04a26edd1226","Type":"ContainerDied","Data":"8d02d1866148f177303d1d8f1cc8bbb2c32c47381d92c22420c4952499547237"} Nov 03 20:38:34.027837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:34.027735 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d02d1866148f177303d1d8f1cc8bbb2c32c47381d92c22420c4952499547237" Nov 03 20:38:35.754971 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:35.754937 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a2057e-ttfc8"] Nov 03 20:38:35.755474 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:35.755460 5059 cpu_manager.go:414] "containerMap: removing container" podUID="698bcd39-e520-42bc-90d2-04a26edd1226" containerName="revoke-keys" Nov 03 20:38:35.755517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:35.755475 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="698bcd39-e520-42bc-90d2-04a26edd1226" containerName="revoke-keys" Nov 03 20:38:35.755594 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:35.755582 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="698bcd39-e520-42bc-90d2-04a26edd1226" containerName="revoke-keys" Nov 03 20:38:35.759463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:35.759445 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a2057e-ttfc8" Nov 03 20:38:35.765072 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:35.765049 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:38:35.765217 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:35.765093 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:38:35.769218 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:35.769196 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a2057e-ttfc8"] Nov 03 20:38:35.873645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:35.873606 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d16ebeb-8fdc-4f51-9e3e-5242e112979a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94a2057e-ttfc8\" (UID: \"9d16ebeb-8fdc-4f51-9e3e-5242e112979a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a2057e-ttfc8" Nov 03 20:38:35.873825 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:35.873654 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkrp7\" (UniqueName: \"kubernetes.io/projected/9d16ebeb-8fdc-4f51-9e3e-5242e112979a-kube-api-access-zkrp7\") pod \"maas-api-revoke-sub-94a2057e-ttfc8\" (UID: \"9d16ebeb-8fdc-4f51-9e3e-5242e112979a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a2057e-ttfc8" Nov 03 20:38:35.873825 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:35.873794 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d16ebeb-8fdc-4f51-9e3e-5242e112979a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94a2057e-ttfc8\" (UID: \"9d16ebeb-8fdc-4f51-9e3e-5242e112979a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a2057e-ttfc8" Nov 03 20:38:35.975497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:35.975464 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d16ebeb-8fdc-4f51-9e3e-5242e112979a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94a2057e-ttfc8\" (UID: \"9d16ebeb-8fdc-4f51-9e3e-5242e112979a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a2057e-ttfc8" Nov 03 20:38:35.975497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:35.975503 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkrp7\" (UniqueName: \"kubernetes.io/projected/9d16ebeb-8fdc-4f51-9e3e-5242e112979a-kube-api-access-zkrp7\") pod \"maas-api-revoke-sub-94a2057e-ttfc8\" (UID: \"9d16ebeb-8fdc-4f51-9e3e-5242e112979a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a2057e-ttfc8" Nov 03 20:38:35.975761 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:35.975567 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d16ebeb-8fdc-4f51-9e3e-5242e112979a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94a2057e-ttfc8\" (UID: \"9d16ebeb-8fdc-4f51-9e3e-5242e112979a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a2057e-ttfc8" Nov 03 20:38:35.976381 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:35.976355 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d16ebeb-8fdc-4f51-9e3e-5242e112979a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94a2057e-ttfc8\" (UID: \"9d16ebeb-8fdc-4f51-9e3e-5242e112979a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a2057e-ttfc8" Nov 03 20:38:35.978096 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:35.978068 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d16ebeb-8fdc-4f51-9e3e-5242e112979a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94a2057e-ttfc8\" (UID: \"9d16ebeb-8fdc-4f51-9e3e-5242e112979a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a2057e-ttfc8" Nov 03 20:38:35.987872 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:35.987851 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkrp7\" (UniqueName: \"kubernetes.io/projected/9d16ebeb-8fdc-4f51-9e3e-5242e112979a-kube-api-access-zkrp7\") pod \"maas-api-revoke-sub-94a2057e-ttfc8\" (UID: \"9d16ebeb-8fdc-4f51-9e3e-5242e112979a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a2057e-ttfc8" Nov 03 20:38:36.075667 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:36.075628 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a2057e-ttfc8" Nov 03 20:38:36.361926 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:36.361892 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a2057e-ttfc8"] Nov 03 20:38:36.362056 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:38:36.361906 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d16ebeb_8fdc_4f51_9e3e_5242e112979a.slice/crio-89ebd9c7e425f3180606493e581210e3fc3a8f84c01fc03b86191d5231037998 WatchSource:0}: Error finding container 89ebd9c7e425f3180606493e581210e3fc3a8f84c01fc03b86191d5231037998: Status 404 returned error can't find the container with id 89ebd9c7e425f3180606493e581210e3fc3a8f84c01fc03b86191d5231037998 Nov 03 20:38:37.049219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:37.049123 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a2057e-ttfc8" event={"ID":"9d16ebeb-8fdc-4f51-9e3e-5242e112979a","Type":"ContainerStarted","Data":"613590ff856112cf9b392125922ee05136abf2b331b26654dd09d54fa11ca2c7"} Nov 03 20:38:37.049219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:37.049178 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a2057e-ttfc8" event={"ID":"9d16ebeb-8fdc-4f51-9e3e-5242e112979a","Type":"ContainerStarted","Data":"89ebd9c7e425f3180606493e581210e3fc3a8f84c01fc03b86191d5231037998"} Nov 03 20:38:37.067737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:37.067682 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a2057e-ttfc8" podStartSLOduration=2.067652566 podStartE2EDuration="2.067652566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:37.066994851 +0000 UTC m=+3711.324792110" watchObservedRunningTime="2026-11-03 20:38:37.067652566 +0000 UTC m=+3711.325449822" Nov 03 20:38:38.058140 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:38.058103 5059 generic.go:356] "Generic (PLEG): container finished" podID="9d16ebeb-8fdc-4f51-9e3e-5242e112979a" containerID="613590ff856112cf9b392125922ee05136abf2b331b26654dd09d54fa11ca2c7" exitCode=0 Nov 03 20:38:38.058555 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:38.058180 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a2057e-ttfc8" event={"ID":"9d16ebeb-8fdc-4f51-9e3e-5242e112979a","Type":"ContainerDied","Data":"613590ff856112cf9b392125922ee05136abf2b331b26654dd09d54fa11ca2c7"} Nov 03 20:38:39.203183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:39.203155 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a2057e-ttfc8" Nov 03 20:38:39.319207 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:39.319182 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d16ebeb-8fdc-4f51-9e3e-5242e112979a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d16ebeb-8fdc-4f51-9e3e-5242e112979a-maas-api-openshift-ca\") pod \"9d16ebeb-8fdc-4f51-9e3e-5242e112979a\" (UID: \"9d16ebeb-8fdc-4f51-9e3e-5242e112979a\") " Nov 03 20:38:39.319341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:39.319329 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d16ebeb-8fdc-4f51-9e3e-5242e112979a-kube-api-access-zkrp7\" (UniqueName: \"kubernetes.io/projected/9d16ebeb-8fdc-4f51-9e3e-5242e112979a-kube-api-access-zkrp7\") pod \"9d16ebeb-8fdc-4f51-9e3e-5242e112979a\" (UID: \"9d16ebeb-8fdc-4f51-9e3e-5242e112979a\") " Nov 03 20:38:39.319381 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:39.319357 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d16ebeb-8fdc-4f51-9e3e-5242e112979a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d16ebeb-8fdc-4f51-9e3e-5242e112979a-maas-api-xks-ca\") pod \"9d16ebeb-8fdc-4f51-9e3e-5242e112979a\" (UID: \"9d16ebeb-8fdc-4f51-9e3e-5242e112979a\") " Nov 03 20:38:39.319581 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:39.319555 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d16ebeb-8fdc-4f51-9e3e-5242e112979a-maas-api-openshift-ca" pod "9d16ebeb-8fdc-4f51-9e3e-5242e112979a" (UID: "9d16ebeb-8fdc-4f51-9e3e-5242e112979a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:39.319854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:39.319829 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d16ebeb-8fdc-4f51-9e3e-5242e112979a-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:39.321480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:39.321458 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d16ebeb-8fdc-4f51-9e3e-5242e112979a-maas-api-xks-ca" pod "9d16ebeb-8fdc-4f51-9e3e-5242e112979a" (UID: "9d16ebeb-8fdc-4f51-9e3e-5242e112979a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:38:39.321587 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:39.321519 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d16ebeb-8fdc-4f51-9e3e-5242e112979a-kube-api-access-zkrp7" pod "9d16ebeb-8fdc-4f51-9e3e-5242e112979a" (UID: "9d16ebeb-8fdc-4f51-9e3e-5242e112979a"). InnerVolumeSpecName "kube-api-access-zkrp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:39.421168 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:39.421136 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkrp7\" (UniqueName: \"kubernetes.io/projected/9d16ebeb-8fdc-4f51-9e3e-5242e112979a-kube-api-access-zkrp7\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:39.421168 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:39.421165 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d16ebeb-8fdc-4f51-9e3e-5242e112979a-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:40.072063 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:40.072028 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a2057e-ttfc8" Nov 03 20:38:40.072237 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:40.072067 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a2057e-ttfc8" event={"ID":"9d16ebeb-8fdc-4f51-9e3e-5242e112979a","Type":"ContainerDied","Data":"89ebd9c7e425f3180606493e581210e3fc3a8f84c01fc03b86191d5231037998"} Nov 03 20:38:40.072237 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:40.072123 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89ebd9c7e425f3180606493e581210e3fc3a8f84c01fc03b86191d5231037998" Nov 03 20:38:40.603988 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:38:40.603952 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:38:41.070752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:41.070721 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6e55f39-pk65m"] Nov 03 20:38:41.071244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:41.071230 5059 cpu_manager.go:414] "containerMap: removing container" podUID="9d16ebeb-8fdc-4f51-9e3e-5242e112979a" containerName="revoke-keys" Nov 03 20:38:41.071282 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:41.071246 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d16ebeb-8fdc-4f51-9e3e-5242e112979a" containerName="revoke-keys" Nov 03 20:38:41.071354 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:41.071344 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d16ebeb-8fdc-4f51-9e3e-5242e112979a" containerName="revoke-keys" Nov 03 20:38:41.074851 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:41.074830 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e55f39-pk65m" Nov 03 20:38:41.079587 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:41.079562 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:38:41.079803 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:41.079784 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:38:41.085713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:41.085686 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6e55f39-pk65m"] Nov 03 20:38:41.140201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:41.140173 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218d1b82-84c1-4985-a260-20625bae7705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6e55f39-pk65m\" (UID: \"218d1b82-84c1-4985-a260-20625bae7705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e55f39-pk65m" Nov 03 20:38:41.140335 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:41.140242 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlnwd\" (UniqueName: \"kubernetes.io/projected/218d1b82-84c1-4985-a260-20625bae7705-kube-api-access-jlnwd\") pod \"maas-api-revoke-sub-f6e55f39-pk65m\" (UID: \"218d1b82-84c1-4985-a260-20625bae7705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e55f39-pk65m" Nov 03 20:38:41.140429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:41.140407 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218d1b82-84c1-4985-a260-20625bae7705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6e55f39-pk65m\" (UID: \"218d1b82-84c1-4985-a260-20625bae7705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e55f39-pk65m" Nov 03 20:38:41.241061 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:41.241031 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218d1b82-84c1-4985-a260-20625bae7705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6e55f39-pk65m\" (UID: \"218d1b82-84c1-4985-a260-20625bae7705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e55f39-pk65m" Nov 03 20:38:41.241180 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:41.241146 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218d1b82-84c1-4985-a260-20625bae7705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6e55f39-pk65m\" (UID: \"218d1b82-84c1-4985-a260-20625bae7705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e55f39-pk65m" Nov 03 20:38:41.241222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:41.241203 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlnwd\" (UniqueName: \"kubernetes.io/projected/218d1b82-84c1-4985-a260-20625bae7705-kube-api-access-jlnwd\") pod \"maas-api-revoke-sub-f6e55f39-pk65m\" (UID: \"218d1b82-84c1-4985-a260-20625bae7705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e55f39-pk65m" Nov 03 20:38:41.241644 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:41.241619 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218d1b82-84c1-4985-a260-20625bae7705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6e55f39-pk65m\" (UID: \"218d1b82-84c1-4985-a260-20625bae7705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e55f39-pk65m" Nov 03 20:38:41.243583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:41.243565 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218d1b82-84c1-4985-a260-20625bae7705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6e55f39-pk65m\" (UID: \"218d1b82-84c1-4985-a260-20625bae7705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e55f39-pk65m" Nov 03 20:38:41.253830 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:41.253806 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlnwd\" (UniqueName: \"kubernetes.io/projected/218d1b82-84c1-4985-a260-20625bae7705-kube-api-access-jlnwd\") pod \"maas-api-revoke-sub-f6e55f39-pk65m\" (UID: \"218d1b82-84c1-4985-a260-20625bae7705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e55f39-pk65m" Nov 03 20:38:41.398121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:41.398059 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e55f39-pk65m" Nov 03 20:38:41.680788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:41.680756 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6e55f39-pk65m"] Nov 03 20:38:41.681185 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:38:41.681161 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218d1b82_84c1_4985_a260_20625bae7705.slice/crio-5de352d90d7d41895963c05d1fbf032b342391a95403f4713a022734ad360d9b WatchSource:0}: Error finding container 5de352d90d7d41895963c05d1fbf032b342391a95403f4713a022734ad360d9b: Status 404 returned error can't find the container with id 5de352d90d7d41895963c05d1fbf032b342391a95403f4713a022734ad360d9b Nov 03 20:38:42.085485 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:42.085439 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e55f39-pk65m" event={"ID":"218d1b82-84c1-4985-a260-20625bae7705","Type":"ContainerStarted","Data":"7c2c89a7d43a8afbeca773d9e3555c337c87948f42cef32e61ef1975070cbc13"} Nov 03 20:38:42.085623 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:42.085506 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e55f39-pk65m" event={"ID":"218d1b82-84c1-4985-a260-20625bae7705","Type":"ContainerStarted","Data":"5de352d90d7d41895963c05d1fbf032b342391a95403f4713a022734ad360d9b"} Nov 03 20:38:42.105415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:42.105363 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e55f39-pk65m" podStartSLOduration=1.105349612 podStartE2EDuration="1.105349612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:42.10344873 +0000 UTC m=+3716.361245990" watchObservedRunningTime="2026-11-03 20:38:42.105349612 +0000 UTC m=+3716.363146869" Nov 03 20:38:43.093445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:43.093407 5059 generic.go:356] "Generic (PLEG): container finished" podID="218d1b82-84c1-4985-a260-20625bae7705" containerID="7c2c89a7d43a8afbeca773d9e3555c337c87948f42cef32e61ef1975070cbc13" exitCode=0 Nov 03 20:38:43.093914 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:43.093493 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e55f39-pk65m" event={"ID":"218d1b82-84c1-4985-a260-20625bae7705","Type":"ContainerDied","Data":"7c2c89a7d43a8afbeca773d9e3555c337c87948f42cef32e61ef1975070cbc13"} Nov 03 20:38:44.251003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:44.250973 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e55f39-pk65m" Nov 03 20:38:44.367644 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:44.367571 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc"] Nov 03 20:38:44.367798 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:44.367781 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" containerID="cri-o://fb62268796fa9f889a9bbeab400464d27f27b14dca4f08044b07d6c2d0ba642b" gracePeriod=30 Nov 03 20:38:44.371724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:44.371699 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297"] Nov 03 20:38:44.371887 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:44.371863 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" containerID="cri-o://47973061462d7b92d288da1a98e63008b76ad7ea55f5bbc305750cfd1fd35ce0" gracePeriod=30 Nov 03 20:38:44.380615 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:44.380578 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/218d1b82-84c1-4985-a260-20625bae7705-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218d1b82-84c1-4985-a260-20625bae7705-maas-api-openshift-ca\") pod \"218d1b82-84c1-4985-a260-20625bae7705\" (UID: \"218d1b82-84c1-4985-a260-20625bae7705\") " Nov 03 20:38:44.380751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:44.380712 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/218d1b82-84c1-4985-a260-20625bae7705-kube-api-access-jlnwd\" (UniqueName: \"kubernetes.io/projected/218d1b82-84c1-4985-a260-20625bae7705-kube-api-access-jlnwd\") pod \"218d1b82-84c1-4985-a260-20625bae7705\" (UID: \"218d1b82-84c1-4985-a260-20625bae7705\") " Nov 03 20:38:44.380829 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:44.380763 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/218d1b82-84c1-4985-a260-20625bae7705-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218d1b82-84c1-4985-a260-20625bae7705-maas-api-xks-ca\") pod \"218d1b82-84c1-4985-a260-20625bae7705\" (UID: \"218d1b82-84c1-4985-a260-20625bae7705\") " Nov 03 20:38:44.381657 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:44.381627 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/218d1b82-84c1-4985-a260-20625bae7705-maas-api-openshift-ca" pod "218d1b82-84c1-4985-a260-20625bae7705" (UID: "218d1b82-84c1-4985-a260-20625bae7705"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:44.383101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:44.383079 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218d1b82-84c1-4985-a260-20625bae7705-kube-api-access-jlnwd" pod "218d1b82-84c1-4985-a260-20625bae7705" (UID: "218d1b82-84c1-4985-a260-20625bae7705"). InnerVolumeSpecName "kube-api-access-jlnwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:44.383775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:44.383748 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/218d1b82-84c1-4985-a260-20625bae7705-maas-api-xks-ca" pod "218d1b82-84c1-4985-a260-20625bae7705" (UID: "218d1b82-84c1-4985-a260-20625bae7705"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:38:44.483087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:44.483049 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218d1b82-84c1-4985-a260-20625bae7705-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:44.483087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:44.483080 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlnwd\" (UniqueName: \"kubernetes.io/projected/218d1b82-84c1-4985-a260-20625bae7705-kube-api-access-jlnwd\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:44.483087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:44.483090 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218d1b82-84c1-4985-a260-20625bae7705-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:45.116623 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:45.116586 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e55f39-pk65m" Nov 03 20:38:45.116841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:45.116580 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e55f39-pk65m" event={"ID":"218d1b82-84c1-4985-a260-20625bae7705","Type":"ContainerDied","Data":"5de352d90d7d41895963c05d1fbf032b342391a95403f4713a022734ad360d9b"} Nov 03 20:38:45.116841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:45.116740 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5de352d90d7d41895963c05d1fbf032b342391a95403f4713a022734ad360d9b" Nov 03 20:38:46.140375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:46.140344 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7e090c0-g6dld"] Nov 03 20:38:46.140903 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:46.140889 5059 cpu_manager.go:414] "containerMap: removing container" podUID="218d1b82-84c1-4985-a260-20625bae7705" containerName="revoke-keys" Nov 03 20:38:46.140958 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:46.140904 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="218d1b82-84c1-4985-a260-20625bae7705" containerName="revoke-keys" Nov 03 20:38:46.141011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:46.141001 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="218d1b82-84c1-4985-a260-20625bae7705" containerName="revoke-keys" Nov 03 20:38:46.144558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:46.144533 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e090c0-g6dld" Nov 03 20:38:46.151175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:46.151154 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:38:46.151472 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:46.151457 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:38:46.158459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:46.158435 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7e090c0-g6dld"] Nov 03 20:38:46.304011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:46.303972 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d66618e-ab4a-47f8-bfc6-1fcafe20214d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7e090c0-g6dld\" (UID: \"2d66618e-ab4a-47f8-bfc6-1fcafe20214d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e090c0-g6dld" Nov 03 20:38:46.304212 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:46.304018 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4zf2\" (UniqueName: \"kubernetes.io/projected/2d66618e-ab4a-47f8-bfc6-1fcafe20214d-kube-api-access-h4zf2\") pod \"maas-api-revoke-sub-b7e090c0-g6dld\" (UID: \"2d66618e-ab4a-47f8-bfc6-1fcafe20214d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e090c0-g6dld" Nov 03 20:38:46.304277 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:46.304222 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d66618e-ab4a-47f8-bfc6-1fcafe20214d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7e090c0-g6dld\" (UID: \"2d66618e-ab4a-47f8-bfc6-1fcafe20214d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e090c0-g6dld" Nov 03 20:38:46.405964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:46.405889 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d66618e-ab4a-47f8-bfc6-1fcafe20214d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7e090c0-g6dld\" (UID: \"2d66618e-ab4a-47f8-bfc6-1fcafe20214d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e090c0-g6dld" Nov 03 20:38:46.406096 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:46.406013 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d66618e-ab4a-47f8-bfc6-1fcafe20214d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7e090c0-g6dld\" (UID: \"2d66618e-ab4a-47f8-bfc6-1fcafe20214d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e090c0-g6dld" Nov 03 20:38:46.406096 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:46.406039 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4zf2\" (UniqueName: \"kubernetes.io/projected/2d66618e-ab4a-47f8-bfc6-1fcafe20214d-kube-api-access-h4zf2\") pod \"maas-api-revoke-sub-b7e090c0-g6dld\" (UID: \"2d66618e-ab4a-47f8-bfc6-1fcafe20214d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e090c0-g6dld" Nov 03 20:38:46.406695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:46.406652 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d66618e-ab4a-47f8-bfc6-1fcafe20214d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7e090c0-g6dld\" (UID: \"2d66618e-ab4a-47f8-bfc6-1fcafe20214d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e090c0-g6dld" Nov 03 20:38:46.408273 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:46.408253 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d66618e-ab4a-47f8-bfc6-1fcafe20214d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7e090c0-g6dld\" (UID: \"2d66618e-ab4a-47f8-bfc6-1fcafe20214d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e090c0-g6dld" Nov 03 20:38:46.417749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:46.417721 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4zf2\" (UniqueName: \"kubernetes.io/projected/2d66618e-ab4a-47f8-bfc6-1fcafe20214d-kube-api-access-h4zf2\") pod \"maas-api-revoke-sub-b7e090c0-g6dld\" (UID: \"2d66618e-ab4a-47f8-bfc6-1fcafe20214d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e090c0-g6dld" Nov 03 20:38:46.463783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:46.463760 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e090c0-g6dld" Nov 03 20:38:46.743176 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:46.743148 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7e090c0-g6dld"] Nov 03 20:38:46.744001 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:38:46.743975 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d66618e_ab4a_47f8_bfc6_1fcafe20214d.slice/crio-87afc4a5d055ba60f3759f615eb02c4cbb52ec9a36ba145778a721ad6a46d210 WatchSource:0}: Error finding container 87afc4a5d055ba60f3759f615eb02c4cbb52ec9a36ba145778a721ad6a46d210: Status 404 returned error can't find the container with id 87afc4a5d055ba60f3759f615eb02c4cbb52ec9a36ba145778a721ad6a46d210 Nov 03 20:38:47.131734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:47.131699 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e090c0-g6dld" event={"ID":"2d66618e-ab4a-47f8-bfc6-1fcafe20214d","Type":"ContainerStarted","Data":"5c408396e866fe620db1812eac6559783162f40c16b03f771dea78b617cb35bc"} Nov 03 20:38:47.131845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:47.131758 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e090c0-g6dld" event={"ID":"2d66618e-ab4a-47f8-bfc6-1fcafe20214d","Type":"ContainerStarted","Data":"87afc4a5d055ba60f3759f615eb02c4cbb52ec9a36ba145778a721ad6a46d210"} Nov 03 20:38:47.153879 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:47.153835 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e090c0-g6dld" podStartSLOduration=1.153820979 podStartE2EDuration="1.153820979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:47.150904766 +0000 UTC m=+3721.408702024" watchObservedRunningTime="2026-11-03 20:38:47.153820979 +0000 UTC m=+3721.411618235" Nov 03 20:38:48.140310 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:48.140270 5059 generic.go:356] "Generic (PLEG): container finished" podID="2d66618e-ab4a-47f8-bfc6-1fcafe20214d" containerID="5c408396e866fe620db1812eac6559783162f40c16b03f771dea78b617cb35bc" exitCode=0 Nov 03 20:38:48.140455 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:48.140352 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e090c0-g6dld" event={"ID":"2d66618e-ab4a-47f8-bfc6-1fcafe20214d","Type":"ContainerDied","Data":"5c408396e866fe620db1812eac6559783162f40c16b03f771dea78b617cb35bc"} Nov 03 20:38:49.293171 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.293145 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e090c0-g6dld" Nov 03 20:38:49.378610 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.378571 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d66618e-ab4a-47f8-bfc6-1fcafe20214d-kube-api-access-h4zf2\" (UniqueName: \"kubernetes.io/projected/2d66618e-ab4a-47f8-bfc6-1fcafe20214d-kube-api-access-h4zf2\") pod \"2d66618e-ab4a-47f8-bfc6-1fcafe20214d\" (UID: \"2d66618e-ab4a-47f8-bfc6-1fcafe20214d\") " Nov 03 20:38:49.379003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.378982 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d66618e-ab4a-47f8-bfc6-1fcafe20214d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d66618e-ab4a-47f8-bfc6-1fcafe20214d-maas-api-openshift-ca\") pod \"2d66618e-ab4a-47f8-bfc6-1fcafe20214d\" (UID: \"2d66618e-ab4a-47f8-bfc6-1fcafe20214d\") " Nov 03 20:38:49.379167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.379152 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d66618e-ab4a-47f8-bfc6-1fcafe20214d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d66618e-ab4a-47f8-bfc6-1fcafe20214d-maas-api-xks-ca\") pod \"2d66618e-ab4a-47f8-bfc6-1fcafe20214d\" (UID: \"2d66618e-ab4a-47f8-bfc6-1fcafe20214d\") " Nov 03 20:38:49.381041 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.379514 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d66618e-ab4a-47f8-bfc6-1fcafe20214d-maas-api-openshift-ca" pod "2d66618e-ab4a-47f8-bfc6-1fcafe20214d" (UID: "2d66618e-ab4a-47f8-bfc6-1fcafe20214d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:49.381788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.381752 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d66618e-ab4a-47f8-bfc6-1fcafe20214d-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:49.383383 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.383351 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d66618e-ab4a-47f8-bfc6-1fcafe20214d-maas-api-xks-ca" pod "2d66618e-ab4a-47f8-bfc6-1fcafe20214d" (UID: "2d66618e-ab4a-47f8-bfc6-1fcafe20214d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:38:49.384857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.384833 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d66618e-ab4a-47f8-bfc6-1fcafe20214d-kube-api-access-h4zf2" pod "2d66618e-ab4a-47f8-bfc6-1fcafe20214d" (UID: "2d66618e-ab4a-47f8-bfc6-1fcafe20214d"). InnerVolumeSpecName "kube-api-access-h4zf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:49.483402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.483372 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d66618e-ab4a-47f8-bfc6-1fcafe20214d-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:49.483508 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.483407 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4zf2\" (UniqueName: \"kubernetes.io/projected/2d66618e-ab4a-47f8-bfc6-1fcafe20214d-kube-api-access-h4zf2\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:49.657187 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.657125 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:38:49.678305 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.678282 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:38:49.685509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.685480 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7101a242-11ce-4b8d-9a74-dae3baf6444b-kube-api-access-nsnvd\" (UniqueName: \"kubernetes.io/projected/7101a242-11ce-4b8d-9a74-dae3baf6444b-kube-api-access-nsnvd\") pod \"7101a242-11ce-4b8d-9a74-dae3baf6444b\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " Nov 03 20:38:49.685633 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.685585 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-podinfo\") pod \"7101a242-11ce-4b8d-9a74-dae3baf6444b\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " Nov 03 20:38:49.685705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.685634 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-workload-certs\") pod \"7101a242-11ce-4b8d-9a74-dae3baf6444b\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " Nov 03 20:38:49.685819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.685794 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7101a242-11ce-4b8d-9a74-dae3baf6444b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7101a242-11ce-4b8d-9a74-dae3baf6444b-istiod-ca-cert\") pod \"7101a242-11ce-4b8d-9a74-dae3baf6444b\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " Nov 03 20:38:49.686009 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.685918 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-credential-socket\") pod \"7101a242-11ce-4b8d-9a74-dae3baf6444b\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " Nov 03 20:38:49.686009 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.685943 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-workload-certs" pod "7101a242-11ce-4b8d-9a74-dae3baf6444b" (UID: "7101a242-11ce-4b8d-9a74-dae3baf6444b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:49.686009 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.685978 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-token\" (UniqueName: \"kubernetes.io/projected/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-token\") pod \"7101a242-11ce-4b8d-9a74-dae3baf6444b\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " Nov 03 20:38:49.686230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.686192 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-workload-socket\") pod \"7101a242-11ce-4b8d-9a74-dae3baf6444b\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " Nov 03 20:38:49.686284 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.686268 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-data\") pod \"7101a242-11ce-4b8d-9a74-dae3baf6444b\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " Nov 03 20:38:49.686337 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.686304 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-envoy\") pod \"7101a242-11ce-4b8d-9a74-dae3baf6444b\" (UID: \"7101a242-11ce-4b8d-9a74-dae3baf6444b\") " Nov 03 20:38:49.687696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.686383 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7101a242-11ce-4b8d-9a74-dae3baf6444b-istiod-ca-cert" pod "7101a242-11ce-4b8d-9a74-dae3baf6444b" (UID: "7101a242-11ce-4b8d-9a74-dae3baf6444b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:49.687696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.686624 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-workload-socket" pod "7101a242-11ce-4b8d-9a74-dae3baf6444b" (UID: "7101a242-11ce-4b8d-9a74-dae3baf6444b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:49.687696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.686962 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7101a242-11ce-4b8d-9a74-dae3baf6444b-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:49.687696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.686987 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:49.687696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.687003 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:49.687696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.687213 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-data" pod "7101a242-11ce-4b8d-9a74-dae3baf6444b" (UID: "7101a242-11ce-4b8d-9a74-dae3baf6444b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:49.688654 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.688606 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7101a242-11ce-4b8d-9a74-dae3baf6444b-kube-api-access-nsnvd" pod "7101a242-11ce-4b8d-9a74-dae3baf6444b" (UID: "7101a242-11ce-4b8d-9a74-dae3baf6444b"). InnerVolumeSpecName "kube-api-access-nsnvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:49.688798 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.688772 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-credential-socket" pod "7101a242-11ce-4b8d-9a74-dae3baf6444b" (UID: "7101a242-11ce-4b8d-9a74-dae3baf6444b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:49.689215 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.689186 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-podinfo" pod "7101a242-11ce-4b8d-9a74-dae3baf6444b" (UID: "7101a242-11ce-4b8d-9a74-dae3baf6444b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:38:49.689507 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.689481 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-envoy" pod "7101a242-11ce-4b8d-9a74-dae3baf6444b" (UID: "7101a242-11ce-4b8d-9a74-dae3baf6444b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:49.689625 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.689502 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-token" pod "7101a242-11ce-4b8d-9a74-dae3baf6444b" (UID: "7101a242-11ce-4b8d-9a74-dae3baf6444b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:49.787500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.787466 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-workload-socket\") pod \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " Nov 03 20:38:49.787500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.787506 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-envoy\") pod \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " Nov 03 20:38:49.787755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.787536 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istiod-ca-cert\") pod \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " Nov 03 20:38:49.787755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.787606 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-credential-socket\") pod \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " Nov 03 20:38:49.787755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.787639 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-podinfo\") pod \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " Nov 03 20:38:49.787755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.787710 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-token\" (UniqueName: \"kubernetes.io/projected/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-token\") pod \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " Nov 03 20:38:49.787960 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.787791 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-workload-socket" pod "b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" (UID: "b6f0cdd8-43f1-4408-b5ad-2429b2dd8507"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:49.787960 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.787810 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-data\") pod \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " Nov 03 20:38:49.787960 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.787842 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-kube-api-access-2m4fg\" (UniqueName: \"kubernetes.io/projected/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-kube-api-access-2m4fg\") pod \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " Nov 03 20:38:49.787960 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.787879 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-workload-certs\") pod \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\" (UID: \"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507\") " Nov 03 20:38:49.788169 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.787985 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-credential-socket" pod "b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" (UID: "b6f0cdd8-43f1-4408-b5ad-2429b2dd8507"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:49.788169 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.788100 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istiod-ca-cert" pod "b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" (UID: "b6f0cdd8-43f1-4408-b5ad-2429b2dd8507"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:49.788169 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.788122 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-data" pod "b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" (UID: "b6f0cdd8-43f1-4408-b5ad-2429b2dd8507"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:49.788346 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.788325 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-workload-certs" pod "b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" (UID: "b6f0cdd8-43f1-4408-b5ad-2429b2dd8507"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:49.788470 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.788449 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:49.788571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.788475 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:49.788571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.788490 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsnvd\" (UniqueName: \"kubernetes.io/projected/7101a242-11ce-4b8d-9a74-dae3baf6444b-kube-api-access-nsnvd\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:49.788571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.788505 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:49.788571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.788519 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:49.788571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.788533 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:49.788571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.788547 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:49.788571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.788561 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:49.788965 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.788575 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:49.788965 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.788591 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7101a242-11ce-4b8d-9a74-dae3baf6444b-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:49.788965 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.788605 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:49.789809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.789781 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-envoy" pod "b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" (UID: "b6f0cdd8-43f1-4408-b5ad-2429b2dd8507"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:38:49.789905 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.789886 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-podinfo" pod "b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" (UID: "b6f0cdd8-43f1-4408-b5ad-2429b2dd8507"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:38:49.789954 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.789936 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-token" pod "b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" (UID: "b6f0cdd8-43f1-4408-b5ad-2429b2dd8507"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:49.790214 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.790191 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-kube-api-access-2m4fg" pod "b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" (UID: "b6f0cdd8-43f1-4408-b5ad-2429b2dd8507"). InnerVolumeSpecName "kube-api-access-2m4fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:49.890153 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.890126 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:49.890153 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.890152 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:49.890286 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.890163 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:49.890286 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:49.890173 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m4fg\" (UniqueName: \"kubernetes.io/projected/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507-kube-api-access-2m4fg\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:50.154316 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:50.154281 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e090c0-g6dld" Nov 03 20:38:50.154496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:50.154286 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e090c0-g6dld" event={"ID":"2d66618e-ab4a-47f8-bfc6-1fcafe20214d","Type":"ContainerDied","Data":"87afc4a5d055ba60f3759f615eb02c4cbb52ec9a36ba145778a721ad6a46d210"} Nov 03 20:38:50.154496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:50.154424 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87afc4a5d055ba60f3759f615eb02c4cbb52ec9a36ba145778a721ad6a46d210" Nov 03 20:38:50.155735 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:50.155704 5059 generic.go:356] "Generic (PLEG): container finished" podID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerID="47973061462d7b92d288da1a98e63008b76ad7ea55f5bbc305750cfd1fd35ce0" exitCode=0 Nov 03 20:38:50.155852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:50.155746 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" Nov 03 20:38:50.155852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:50.155783 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" event={"ID":"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507","Type":"ContainerDied","Data":"47973061462d7b92d288da1a98e63008b76ad7ea55f5bbc305750cfd1fd35ce0"} Nov 03 20:38:50.155852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:50.155834 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297" event={"ID":"b6f0cdd8-43f1-4408-b5ad-2429b2dd8507","Type":"ContainerDied","Data":"34a6b94a9819202c88c3017986397f0bc71ff236ace7b870b4731c004cf9c609"} Nov 03 20:38:50.156027 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:50.155862 5059 scope.go:122] "RemoveContainer" containerID="47973061462d7b92d288da1a98e63008b76ad7ea55f5bbc305750cfd1fd35ce0" Nov 03 20:38:50.157539 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:50.157520 5059 generic.go:356] "Generic (PLEG): container finished" podID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerID="fb62268796fa9f889a9bbeab400464d27f27b14dca4f08044b07d6c2d0ba642b" exitCode=0 Nov 03 20:38:50.157607 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:50.157559 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" Nov 03 20:38:50.157607 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:50.157591 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" event={"ID":"7101a242-11ce-4b8d-9a74-dae3baf6444b","Type":"ContainerDied","Data":"fb62268796fa9f889a9bbeab400464d27f27b14dca4f08044b07d6c2d0ba642b"} Nov 03 20:38:50.157758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:50.157630 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc" event={"ID":"7101a242-11ce-4b8d-9a74-dae3baf6444b","Type":"ContainerDied","Data":"2b33a04c0eadd1f9ac6712cc7a89380f25b0d7ef362dd8f03c2107365dee1e3b"} Nov 03 20:38:50.166935 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:50.166918 5059 scope.go:122] "RemoveContainer" containerID="47973061462d7b92d288da1a98e63008b76ad7ea55f5bbc305750cfd1fd35ce0" Nov 03 20:38:50.167221 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:38:50.167201 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47973061462d7b92d288da1a98e63008b76ad7ea55f5bbc305750cfd1fd35ce0\": container with ID starting with 47973061462d7b92d288da1a98e63008b76ad7ea55f5bbc305750cfd1fd35ce0 not found: ID does not exist" containerID="47973061462d7b92d288da1a98e63008b76ad7ea55f5bbc305750cfd1fd35ce0" Nov 03 20:38:50.167292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:50.167234 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47973061462d7b92d288da1a98e63008b76ad7ea55f5bbc305750cfd1fd35ce0"} err="failed to get container status \"47973061462d7b92d288da1a98e63008b76ad7ea55f5bbc305750cfd1fd35ce0\": rpc error: code = NotFound desc = could not find container \"47973061462d7b92d288da1a98e63008b76ad7ea55f5bbc305750cfd1fd35ce0\": container with ID starting with 47973061462d7b92d288da1a98e63008b76ad7ea55f5bbc305750cfd1fd35ce0 not found: ID does not exist" Nov 03 20:38:50.167292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:50.167258 5059 scope.go:122] "RemoveContainer" containerID="fb62268796fa9f889a9bbeab400464d27f27b14dca4f08044b07d6c2d0ba642b" Nov 03 20:38:50.177811 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:50.177786 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297"] Nov 03 20:38:50.179301 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:50.179267 5059 scope.go:122] "RemoveContainer" containerID="fb62268796fa9f889a9bbeab400464d27f27b14dca4f08044b07d6c2d0ba642b" Nov 03 20:38:50.179816 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:38:50.179785 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb62268796fa9f889a9bbeab400464d27f27b14dca4f08044b07d6c2d0ba642b\": container with ID starting with fb62268796fa9f889a9bbeab400464d27f27b14dca4f08044b07d6c2d0ba642b not found: ID does not exist" containerID="fb62268796fa9f889a9bbeab400464d27f27b14dca4f08044b07d6c2d0ba642b" Nov 03 20:38:50.179911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:50.179826 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb62268796fa9f889a9bbeab400464d27f27b14dca4f08044b07d6c2d0ba642b"} err="failed to get container status \"fb62268796fa9f889a9bbeab400464d27f27b14dca4f08044b07d6c2d0ba642b\": rpc error: code = NotFound desc = could not find container \"fb62268796fa9f889a9bbeab400464d27f27b14dca4f08044b07d6c2d0ba642b\": container with ID starting with fb62268796fa9f889a9bbeab400464d27f27b14dca4f08044b07d6c2d0ba642b not found: ID does not exist" Nov 03 20:38:50.184629 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:50.184602 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-nt297"] Nov 03 20:38:50.205311 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:50.205288 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc"] Nov 03 20:38:50.211912 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:50.211890 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-7dmwc"] Nov 03 20:38:51.448342 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.448306 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a0ef22-4nk2n"] Nov 03 20:38:51.449087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.449062 5059 cpu_manager.go:414] "containerMap: removing container" podUID="2d66618e-ab4a-47f8-bfc6-1fcafe20214d" containerName="revoke-keys" Nov 03 20:38:51.449087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.449087 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d66618e-ab4a-47f8-bfc6-1fcafe20214d" containerName="revoke-keys" Nov 03 20:38:51.449231 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.449114 5059 cpu_manager.go:414] "containerMap: removing container" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" Nov 03 20:38:51.449231 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.449122 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" Nov 03 20:38:51.449231 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.449164 5059 cpu_manager.go:414] "containerMap: removing container" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" Nov 03 20:38:51.449231 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.449173 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" Nov 03 20:38:51.449577 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.449297 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" containerName="istio-proxy" Nov 03 20:38:51.449577 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.449311 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d66618e-ab4a-47f8-bfc6-1fcafe20214d" containerName="revoke-keys" Nov 03 20:38:51.449577 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.449323 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" containerName="istio-proxy" Nov 03 20:38:51.454252 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.454232 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a0ef22-4nk2n" Nov 03 20:38:51.458553 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.458528 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:38:51.459363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.459340 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:38:51.461242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.461215 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a0ef22-4nk2n"] Nov 03 20:38:51.507700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.507649 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4a0ec2-017b-4272-a8e6-a95122b3e845-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2a0ef22-4nk2n\" (UID: \"0f4a0ec2-017b-4272-a8e6-a95122b3e845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a0ef22-4nk2n" Nov 03 20:38:51.507815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.507754 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4a0ec2-017b-4272-a8e6-a95122b3e845-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2a0ef22-4nk2n\" (UID: \"0f4a0ec2-017b-4272-a8e6-a95122b3e845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a0ef22-4nk2n" Nov 03 20:38:51.507859 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.507814 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blv6l\" (UniqueName: \"kubernetes.io/projected/0f4a0ec2-017b-4272-a8e6-a95122b3e845-kube-api-access-blv6l\") pod \"maas-api-revoke-sub-c2a0ef22-4nk2n\" (UID: \"0f4a0ec2-017b-4272-a8e6-a95122b3e845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a0ef22-4nk2n" Nov 03 20:38:51.609186 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.609154 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4a0ec2-017b-4272-a8e6-a95122b3e845-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2a0ef22-4nk2n\" (UID: \"0f4a0ec2-017b-4272-a8e6-a95122b3e845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a0ef22-4nk2n" Nov 03 20:38:51.609278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.609253 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blv6l\" (UniqueName: \"kubernetes.io/projected/0f4a0ec2-017b-4272-a8e6-a95122b3e845-kube-api-access-blv6l\") pod \"maas-api-revoke-sub-c2a0ef22-4nk2n\" (UID: \"0f4a0ec2-017b-4272-a8e6-a95122b3e845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a0ef22-4nk2n" Nov 03 20:38:51.609381 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.609366 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4a0ec2-017b-4272-a8e6-a95122b3e845-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2a0ef22-4nk2n\" (UID: \"0f4a0ec2-017b-4272-a8e6-a95122b3e845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a0ef22-4nk2n" Nov 03 20:38:51.610043 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.610020 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4a0ec2-017b-4272-a8e6-a95122b3e845-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2a0ef22-4nk2n\" (UID: \"0f4a0ec2-017b-4272-a8e6-a95122b3e845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a0ef22-4nk2n" Nov 03 20:38:51.611458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.611440 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4a0ec2-017b-4272-a8e6-a95122b3e845-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2a0ef22-4nk2n\" (UID: \"0f4a0ec2-017b-4272-a8e6-a95122b3e845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a0ef22-4nk2n" Nov 03 20:38:51.624217 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.624190 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blv6l\" (UniqueName: \"kubernetes.io/projected/0f4a0ec2-017b-4272-a8e6-a95122b3e845-kube-api-access-blv6l\") pod \"maas-api-revoke-sub-c2a0ef22-4nk2n\" (UID: \"0f4a0ec2-017b-4272-a8e6-a95122b3e845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a0ef22-4nk2n" Nov 03 20:38:51.771589 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.771516 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a0ef22-4nk2n" Nov 03 20:38:51.856536 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.856503 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7101a242-11ce-4b8d-9a74-dae3baf6444b" path="/var/lib/kubelet/pods/7101a242-11ce-4b8d-9a74-dae3baf6444b/volumes" Nov 03 20:38:51.857120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:51.857100 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6f0cdd8-43f1-4408-b5ad-2429b2dd8507" path="/var/lib/kubelet/pods/b6f0cdd8-43f1-4408-b5ad-2429b2dd8507/volumes" Nov 03 20:38:52.053813 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:52.053776 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a0ef22-4nk2n"] Nov 03 20:38:52.053938 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:38:52.053860 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4a0ec2_017b_4272_a8e6_a95122b3e845.slice/crio-e583bb4feb99d36f052781a815ba306a4f247888d1989f92e24c9777789bca54 WatchSource:0}: Error finding container e583bb4feb99d36f052781a815ba306a4f247888d1989f92e24c9777789bca54: Status 404 returned error can't find the container with id e583bb4feb99d36f052781a815ba306a4f247888d1989f92e24c9777789bca54 Nov 03 20:38:52.174286 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:52.174247 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a0ef22-4nk2n" event={"ID":"0f4a0ec2-017b-4272-a8e6-a95122b3e845","Type":"ContainerStarted","Data":"e583bb4feb99d36f052781a815ba306a4f247888d1989f92e24c9777789bca54"} Nov 03 20:38:53.182318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:53.182235 5059 generic.go:356] "Generic (PLEG): container finished" podID="0f4a0ec2-017b-4272-a8e6-a95122b3e845" containerID="d69aefb2211ad25e5f23b2f7114fb99097dde9e8d402059bcfd5a7ae5c2dbc94" exitCode=0 Nov 03 20:38:53.182717 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:53.182331 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a0ef22-4nk2n" event={"ID":"0f4a0ec2-017b-4272-a8e6-a95122b3e845","Type":"ContainerDied","Data":"d69aefb2211ad25e5f23b2f7114fb99097dde9e8d402059bcfd5a7ae5c2dbc94"} Nov 03 20:38:54.347239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:54.347213 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a0ef22-4nk2n" Nov 03 20:38:54.444315 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:54.444248 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f4a0ec2-017b-4272-a8e6-a95122b3e845-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4a0ec2-017b-4272-a8e6-a95122b3e845-maas-api-xks-ca\") pod \"0f4a0ec2-017b-4272-a8e6-a95122b3e845\" (UID: \"0f4a0ec2-017b-4272-a8e6-a95122b3e845\") " Nov 03 20:38:54.444315 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:54.444303 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f4a0ec2-017b-4272-a8e6-a95122b3e845-kube-api-access-blv6l\" (UniqueName: \"kubernetes.io/projected/0f4a0ec2-017b-4272-a8e6-a95122b3e845-kube-api-access-blv6l\") pod \"0f4a0ec2-017b-4272-a8e6-a95122b3e845\" (UID: \"0f4a0ec2-017b-4272-a8e6-a95122b3e845\") " Nov 03 20:38:54.444474 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:54.444375 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f4a0ec2-017b-4272-a8e6-a95122b3e845-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4a0ec2-017b-4272-a8e6-a95122b3e845-maas-api-openshift-ca\") pod \"0f4a0ec2-017b-4272-a8e6-a95122b3e845\" (UID: \"0f4a0ec2-017b-4272-a8e6-a95122b3e845\") " Nov 03 20:38:54.444781 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:54.444757 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f4a0ec2-017b-4272-a8e6-a95122b3e845-maas-api-openshift-ca" pod "0f4a0ec2-017b-4272-a8e6-a95122b3e845" (UID: "0f4a0ec2-017b-4272-a8e6-a95122b3e845"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:54.446402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:54.446380 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f4a0ec2-017b-4272-a8e6-a95122b3e845-maas-api-xks-ca" pod "0f4a0ec2-017b-4272-a8e6-a95122b3e845" (UID: "0f4a0ec2-017b-4272-a8e6-a95122b3e845"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:38:54.446466 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:54.446443 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4a0ec2-017b-4272-a8e6-a95122b3e845-kube-api-access-blv6l" pod "0f4a0ec2-017b-4272-a8e6-a95122b3e845" (UID: "0f4a0ec2-017b-4272-a8e6-a95122b3e845"). InnerVolumeSpecName "kube-api-access-blv6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:54.545233 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:54.545204 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4a0ec2-017b-4272-a8e6-a95122b3e845-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:54.545347 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:54.545253 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blv6l\" (UniqueName: \"kubernetes.io/projected/0f4a0ec2-017b-4272-a8e6-a95122b3e845-kube-api-access-blv6l\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:54.545347 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:54.545266 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4a0ec2-017b-4272-a8e6-a95122b3e845-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:55.197461 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:55.197424 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a0ef22-4nk2n" Nov 03 20:38:55.197686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:55.197457 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a0ef22-4nk2n" event={"ID":"0f4a0ec2-017b-4272-a8e6-a95122b3e845","Type":"ContainerDied","Data":"e583bb4feb99d36f052781a815ba306a4f247888d1989f92e24c9777789bca54"} Nov 03 20:38:55.197686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:55.197519 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e583bb4feb99d36f052781a815ba306a4f247888d1989f92e24c9777789bca54" Nov 03 20:38:56.915928 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:56.915887 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx"] Nov 03 20:38:58.958694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:38:58.958630 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn"] Nov 03 20:39:03.486641 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:03.486604 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l"] Nov 03 20:39:03.487045 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:03.486802 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" podUID="9bb4f711-0bcc-46f4-86b9-795c66613832" containerName="istio-proxy" containerID="cri-o://15e2cc84c3e225d2ad2495aa2c827011b729fdaac93a952c40e3cc3bde3f078d" gracePeriod=30 Nov 03 20:39:04.419590 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:04.419557 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk"] Nov 03 20:39:08.756509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.756448 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:39:08.870475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.870445 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-data\") pod \"9bb4f711-0bcc-46f4-86b9-795c66613832\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " Nov 03 20:39:08.870630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.870500 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-envoy\") pod \"9bb4f711-0bcc-46f4-86b9-795c66613832\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " Nov 03 20:39:08.870630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.870546 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-podinfo\") pod \"9bb4f711-0bcc-46f4-86b9-795c66613832\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " Nov 03 20:39:08.870630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.870578 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-credential-socket\") pod \"9bb4f711-0bcc-46f4-86b9-795c66613832\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " Nov 03 20:39:08.870630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.870609 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb4f711-0bcc-46f4-86b9-795c66613832-kube-api-access-n26xp\" (UniqueName: \"kubernetes.io/projected/9bb4f711-0bcc-46f4-86b9-795c66613832-kube-api-access-n26xp\") pod \"9bb4f711-0bcc-46f4-86b9-795c66613832\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " Nov 03 20:39:08.870885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.870773 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-data" pod "9bb4f711-0bcc-46f4-86b9-795c66613832" (UID: "9bb4f711-0bcc-46f4-86b9-795c66613832"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:08.870885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.870779 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-workload-socket\") pod \"9bb4f711-0bcc-46f4-86b9-795c66613832\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " Nov 03 20:39:08.870885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.870871 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bb4f711-0bcc-46f4-86b9-795c66613832-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb4f711-0bcc-46f4-86b9-795c66613832-istiod-ca-cert\") pod \"9bb4f711-0bcc-46f4-86b9-795c66613832\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " Nov 03 20:39:08.871050 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.870927 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-workload-certs\") pod \"9bb4f711-0bcc-46f4-86b9-795c66613832\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " Nov 03 20:39:08.871050 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.870958 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-token\" (UniqueName: \"kubernetes.io/projected/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-token\") pod \"9bb4f711-0bcc-46f4-86b9-795c66613832\" (UID: \"9bb4f711-0bcc-46f4-86b9-795c66613832\") " Nov 03 20:39:08.871273 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.870930 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-credential-socket" pod "9bb4f711-0bcc-46f4-86b9-795c66613832" (UID: "9bb4f711-0bcc-46f4-86b9-795c66613832"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:08.871273 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.870997 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-workload-socket" pod "9bb4f711-0bcc-46f4-86b9-795c66613832" (UID: "9bb4f711-0bcc-46f4-86b9-795c66613832"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:08.871408 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.871161 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-workload-certs" pod "9bb4f711-0bcc-46f4-86b9-795c66613832" (UID: "9bb4f711-0bcc-46f4-86b9-795c66613832"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:08.871408 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.871270 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb4f711-0bcc-46f4-86b9-795c66613832-istiod-ca-cert" pod "9bb4f711-0bcc-46f4-86b9-795c66613832" (UID: "9bb4f711-0bcc-46f4-86b9-795c66613832"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:08.871498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.871428 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:08.871498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.871453 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:08.871498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.871468 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb4f711-0bcc-46f4-86b9-795c66613832-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:08.871498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.871483 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:08.871498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.871496 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:08.872922 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.872901 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb4f711-0bcc-46f4-86b9-795c66613832-kube-api-access-n26xp" pod "9bb4f711-0bcc-46f4-86b9-795c66613832" (UID: "9bb4f711-0bcc-46f4-86b9-795c66613832"). InnerVolumeSpecName "kube-api-access-n26xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:08.873035 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.873019 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-token" pod "9bb4f711-0bcc-46f4-86b9-795c66613832" (UID: "9bb4f711-0bcc-46f4-86b9-795c66613832"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:08.873202 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.873177 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-envoy" pod "9bb4f711-0bcc-46f4-86b9-795c66613832" (UID: "9bb4f711-0bcc-46f4-86b9-795c66613832"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:08.873297 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.873199 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-podinfo" pod "9bb4f711-0bcc-46f4-86b9-795c66613832" (UID: "9bb4f711-0bcc-46f4-86b9-795c66613832"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:39:08.973420 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.973377 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:08.973420 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.973411 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n26xp\" (UniqueName: \"kubernetes.io/projected/9bb4f711-0bcc-46f4-86b9-795c66613832-kube-api-access-n26xp\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:08.973420 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.973426 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:08.973639 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:08.973438 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb4f711-0bcc-46f4-86b9-795c66613832-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:09.305722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:09.305688 5059 generic.go:356] "Generic (PLEG): container finished" podID="9bb4f711-0bcc-46f4-86b9-795c66613832" containerID="15e2cc84c3e225d2ad2495aa2c827011b729fdaac93a952c40e3cc3bde3f078d" exitCode=0 Nov 03 20:39:09.305893 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:09.305794 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" Nov 03 20:39:09.305893 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:09.305817 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" event={"ID":"9bb4f711-0bcc-46f4-86b9-795c66613832","Type":"ContainerDied","Data":"15e2cc84c3e225d2ad2495aa2c827011b729fdaac93a952c40e3cc3bde3f078d"} Nov 03 20:39:09.305893 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:09.305883 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l" event={"ID":"9bb4f711-0bcc-46f4-86b9-795c66613832","Type":"ContainerDied","Data":"89548cf27e0936cd588dd9c9835fdc9577a746519416c028395390c730ad8fd5"} Nov 03 20:39:09.306015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:09.305905 5059 scope.go:122] "RemoveContainer" containerID="15e2cc84c3e225d2ad2495aa2c827011b729fdaac93a952c40e3cc3bde3f078d" Nov 03 20:39:09.317558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:09.317539 5059 scope.go:122] "RemoveContainer" containerID="15e2cc84c3e225d2ad2495aa2c827011b729fdaac93a952c40e3cc3bde3f078d" Nov 03 20:39:09.317850 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:39:09.317832 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15e2cc84c3e225d2ad2495aa2c827011b729fdaac93a952c40e3cc3bde3f078d\": container with ID starting with 15e2cc84c3e225d2ad2495aa2c827011b729fdaac93a952c40e3cc3bde3f078d not found: ID does not exist" containerID="15e2cc84c3e225d2ad2495aa2c827011b729fdaac93a952c40e3cc3bde3f078d" Nov 03 20:39:09.317921 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:09.317857 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e2cc84c3e225d2ad2495aa2c827011b729fdaac93a952c40e3cc3bde3f078d"} err="failed to get container status \"15e2cc84c3e225d2ad2495aa2c827011b729fdaac93a952c40e3cc3bde3f078d\": rpc error: code = NotFound desc = could not find container \"15e2cc84c3e225d2ad2495aa2c827011b729fdaac93a952c40e3cc3bde3f078d\": container with ID starting with 15e2cc84c3e225d2ad2495aa2c827011b729fdaac93a952c40e3cc3bde3f078d not found: ID does not exist" Nov 03 20:39:09.342399 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:09.342369 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l"] Nov 03 20:39:09.346773 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:09.346748 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-t2d5l"] Nov 03 20:39:09.625095 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:09.625057 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856"] Nov 03 20:39:09.857414 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:09.857377 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bb4f711-0bcc-46f4-86b9-795c66613832" path="/var/lib/kubelet/pods/9bb4f711-0bcc-46f4-86b9-795c66613832/volumes" Nov 03 20:39:10.603964 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:39:10.603926 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:39:11.939321 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:11.939275 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" podUID="765cb609-3515-4d3d-8158-da836d306b6c" containerName="main" containerID="cri-o://ad98b3a82780283d70af3272adec99dcd798b659e0ca38053ce40bf94a0d72eb" gracePeriod=45 Nov 03 20:39:12.227099 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.227071 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:39:12.313955 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.313923 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/765cb609-3515-4d3d-8158-da836d306b6c-kube-api-access-2rxmm\" (UniqueName: \"kubernetes.io/projected/765cb609-3515-4d3d-8158-da836d306b6c-kube-api-access-2rxmm\") pod \"765cb609-3515-4d3d-8158-da836d306b6c\" (UID: \"765cb609-3515-4d3d-8158-da836d306b6c\") " Nov 03 20:39:12.314146 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.313984 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-home\" (UniqueName: \"kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-home\") pod \"765cb609-3515-4d3d-8158-da836d306b6c\" (UID: \"765cb609-3515-4d3d-8158-da836d306b6c\") " Nov 03 20:39:12.314146 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.314116 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-dshm\") pod \"765cb609-3515-4d3d-8158-da836d306b6c\" (UID: \"765cb609-3515-4d3d-8158-da836d306b6c\") " Nov 03 20:39:12.314507 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.314146 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-model-cache\") pod \"765cb609-3515-4d3d-8158-da836d306b6c\" (UID: \"765cb609-3515-4d3d-8158-da836d306b6c\") " Nov 03 20:39:12.314621 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.314554 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/765cb609-3515-4d3d-8158-da836d306b6c-tls-certs\" (UniqueName: \"kubernetes.io/secret/765cb609-3515-4d3d-8158-da836d306b6c-tls-certs\") pod \"765cb609-3515-4d3d-8158-da836d306b6c\" (UID: \"765cb609-3515-4d3d-8158-da836d306b6c\") " Nov 03 20:39:12.314621 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.314585 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-tmp-dir\") pod \"765cb609-3515-4d3d-8158-da836d306b6c\" (UID: \"765cb609-3515-4d3d-8158-da836d306b6c\") " Nov 03 20:39:12.315271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.314255 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-home" pod "765cb609-3515-4d3d-8158-da836d306b6c" (UID: "765cb609-3515-4d3d-8158-da836d306b6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:12.315430 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.314428 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-model-cache" pod "765cb609-3515-4d3d-8158-da836d306b6c" (UID: "765cb609-3515-4d3d-8158-da836d306b6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:12.315586 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.315569 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-tmp-dir" pod "765cb609-3515-4d3d-8158-da836d306b6c" (UID: "765cb609-3515-4d3d-8158-da836d306b6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:12.316822 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.316795 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/765cb609-3515-4d3d-8158-da836d306b6c-kube-api-access-2rxmm" pod "765cb609-3515-4d3d-8158-da836d306b6c" (UID: "765cb609-3515-4d3d-8158-da836d306b6c"). InnerVolumeSpecName "kube-api-access-2rxmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:12.317497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.317467 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/765cb609-3515-4d3d-8158-da836d306b6c-tls-certs" pod "765cb609-3515-4d3d-8158-da836d306b6c" (UID: "765cb609-3515-4d3d-8158-da836d306b6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:39:12.318136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.318115 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-dshm" pod "765cb609-3515-4d3d-8158-da836d306b6c" (UID: "765cb609-3515-4d3d-8158-da836d306b6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:12.331144 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.331118 5059 generic.go:356] "Generic (PLEG): container finished" podID="765cb609-3515-4d3d-8158-da836d306b6c" containerID="ad98b3a82780283d70af3272adec99dcd798b659e0ca38053ce40bf94a0d72eb" exitCode=0 Nov 03 20:39:12.331267 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.331165 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" Nov 03 20:39:12.331267 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.331204 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" event={"ID":"765cb609-3515-4d3d-8158-da836d306b6c","Type":"ContainerDied","Data":"ad98b3a82780283d70af3272adec99dcd798b659e0ca38053ce40bf94a0d72eb"} Nov 03 20:39:12.331267 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.331254 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" event={"ID":"765cb609-3515-4d3d-8158-da836d306b6c","Type":"ContainerDied","Data":"8b49fd665c473e908ea644249c843e4dc6beb1d996760a5818b09159babb5d73"} Nov 03 20:39:12.331433 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.331276 5059 scope.go:122] "RemoveContainer" containerID="ad98b3a82780283d70af3272adec99dcd798b659e0ca38053ce40bf94a0d72eb" Nov 03 20:39:12.347642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.347625 5059 scope.go:122] "RemoveContainer" containerID="ad98b3a82780283d70af3272adec99dcd798b659e0ca38053ce40bf94a0d72eb" Nov 03 20:39:12.347925 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:39:12.347900 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad98b3a82780283d70af3272adec99dcd798b659e0ca38053ce40bf94a0d72eb\": container with ID starting with ad98b3a82780283d70af3272adec99dcd798b659e0ca38053ce40bf94a0d72eb not found: ID does not exist" containerID="ad98b3a82780283d70af3272adec99dcd798b659e0ca38053ce40bf94a0d72eb" Nov 03 20:39:12.348028 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.347937 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad98b3a82780283d70af3272adec99dcd798b659e0ca38053ce40bf94a0d72eb"} err="failed to get container status \"ad98b3a82780283d70af3272adec99dcd798b659e0ca38053ce40bf94a0d72eb\": rpc error: code = NotFound desc = could not find container \"ad98b3a82780283d70af3272adec99dcd798b659e0ca38053ce40bf94a0d72eb\": container with ID starting with ad98b3a82780283d70af3272adec99dcd798b659e0ca38053ce40bf94a0d72eb not found: ID does not exist" Nov 03 20:39:12.358893 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.358869 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx"] Nov 03 20:39:12.362651 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.362630 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx"] Nov 03 20:39:12.415981 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.415959 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:12.415981 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.415982 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:12.416108 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.415993 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/765cb609-3515-4d3d-8158-da836d306b6c-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:12.416108 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.416002 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:12.416108 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.416011 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rxmm\" (UniqueName: \"kubernetes.io/projected/765cb609-3515-4d3d-8158-da836d306b6c-kube-api-access-2rxmm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:12.416108 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:12.416021 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/765cb609-3515-4d3d-8158-da836d306b6c-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:13.198694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:13.193737 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-f154ac/sub-test-model-f154ac-kserve-5846d75657-jrrlx" podUID="765cb609-3515-4d3d-8158-da836d306b6c" containerName="main" probeResult="failure" output="Get \"https://10.132.1.145:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:39:13.868696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:13.868645 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="765cb609-3515-4d3d-8158-da836d306b6c" path="/var/lib/kubelet/pods/765cb609-3515-4d3d-8158-da836d306b6c/volumes" Nov 03 20:39:13.979242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:13.979195 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" podUID="af50a359-40ae-481c-a707-44a349210f33" containerName="main" containerID="cri-o://17071bf2df6f0005f3cc04c51576f8b88cbfd532f726b2bb03027f79acfd5bea" gracePeriod=45 Nov 03 20:39:14.133337 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.126427 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2"] Nov 03 20:39:14.133337 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.126795 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" podUID="426d1445-9e48-4516-a920-fca04d918f22" containerName="istio-proxy" containerID="cri-o://3f775196318978b57c4c708d101e83f38ee80fd8c5d6c5eee6f5841b39c93535" gracePeriod=30 Nov 03 20:39:14.334338 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.334306 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:39:14.350193 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.350151 5059 generic.go:356] "Generic (PLEG): container finished" podID="af50a359-40ae-481c-a707-44a349210f33" containerID="17071bf2df6f0005f3cc04c51576f8b88cbfd532f726b2bb03027f79acfd5bea" exitCode=0 Nov 03 20:39:14.350374 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.350193 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" Nov 03 20:39:14.350374 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.350232 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" event={"ID":"af50a359-40ae-481c-a707-44a349210f33","Type":"ContainerDied","Data":"17071bf2df6f0005f3cc04c51576f8b88cbfd532f726b2bb03027f79acfd5bea"} Nov 03 20:39:14.350374 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.350280 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn" event={"ID":"af50a359-40ae-481c-a707-44a349210f33","Type":"ContainerDied","Data":"a28bf82c879c8b6a87eaabc34e2102698cd449d638eaa71e899f6718a526a33f"} Nov 03 20:39:14.350374 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.350298 5059 scope.go:122] "RemoveContainer" containerID="17071bf2df6f0005f3cc04c51576f8b88cbfd532f726b2bb03027f79acfd5bea" Nov 03 20:39:14.364352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.364324 5059 scope.go:122] "RemoveContainer" containerID="17071bf2df6f0005f3cc04c51576f8b88cbfd532f726b2bb03027f79acfd5bea" Nov 03 20:39:14.364777 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:39:14.364748 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17071bf2df6f0005f3cc04c51576f8b88cbfd532f726b2bb03027f79acfd5bea\": container with ID starting with 17071bf2df6f0005f3cc04c51576f8b88cbfd532f726b2bb03027f79acfd5bea not found: ID does not exist" containerID="17071bf2df6f0005f3cc04c51576f8b88cbfd532f726b2bb03027f79acfd5bea" Nov 03 20:39:14.364899 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.364790 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17071bf2df6f0005f3cc04c51576f8b88cbfd532f726b2bb03027f79acfd5bea"} err="failed to get container status \"17071bf2df6f0005f3cc04c51576f8b88cbfd532f726b2bb03027f79acfd5bea\": rpc error: code = NotFound desc = could not find container \"17071bf2df6f0005f3cc04c51576f8b88cbfd532f726b2bb03027f79acfd5bea\": container with ID starting with 17071bf2df6f0005f3cc04c51576f8b88cbfd532f726b2bb03027f79acfd5bea not found: ID does not exist" Nov 03 20:39:14.447264 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.447205 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af50a359-40ae-481c-a707-44a349210f33-kube-api-access-jt72n\" (UniqueName: \"kubernetes.io/projected/af50a359-40ae-481c-a707-44a349210f33-kube-api-access-jt72n\") pod \"af50a359-40ae-481c-a707-44a349210f33\" (UID: \"af50a359-40ae-481c-a707-44a349210f33\") " Nov 03 20:39:14.447264 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.447242 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-dshm\") pod \"af50a359-40ae-481c-a707-44a349210f33\" (UID: \"af50a359-40ae-481c-a707-44a349210f33\") " Nov 03 20:39:14.447415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.447291 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-model-cache\") pod \"af50a359-40ae-481c-a707-44a349210f33\" (UID: \"af50a359-40ae-481c-a707-44a349210f33\") " Nov 03 20:39:14.447415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.447343 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-tmp-dir\") pod \"af50a359-40ae-481c-a707-44a349210f33\" (UID: \"af50a359-40ae-481c-a707-44a349210f33\") " Nov 03 20:39:14.447415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.447376 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-home\" (UniqueName: \"kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-home\") pod \"af50a359-40ae-481c-a707-44a349210f33\" (UID: \"af50a359-40ae-481c-a707-44a349210f33\") " Nov 03 20:39:14.447555 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.447516 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af50a359-40ae-481c-a707-44a349210f33-tls-certs\" (UniqueName: \"kubernetes.io/secret/af50a359-40ae-481c-a707-44a349210f33-tls-certs\") pod \"af50a359-40ae-481c-a707-44a349210f33\" (UID: \"af50a359-40ae-481c-a707-44a349210f33\") " Nov 03 20:39:14.447640 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.447601 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-model-cache" pod "af50a359-40ae-481c-a707-44a349210f33" (UID: "af50a359-40ae-481c-a707-44a349210f33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:14.447640 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.447619 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-tmp-dir" pod "af50a359-40ae-481c-a707-44a349210f33" (UID: "af50a359-40ae-481c-a707-44a349210f33"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:14.447781 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.447737 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-home" pod "af50a359-40ae-481c-a707-44a349210f33" (UID: "af50a359-40ae-481c-a707-44a349210f33"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:14.448101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.448075 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:14.448190 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.448133 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:14.448190 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.448149 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:14.449589 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.449561 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-dshm" pod "af50a359-40ae-481c-a707-44a349210f33" (UID: "af50a359-40ae-481c-a707-44a349210f33"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:14.449718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.449695 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af50a359-40ae-481c-a707-44a349210f33-tls-certs" pod "af50a359-40ae-481c-a707-44a349210f33" (UID: "af50a359-40ae-481c-a707-44a349210f33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:39:14.449762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.449700 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af50a359-40ae-481c-a707-44a349210f33-kube-api-access-jt72n" pod "af50a359-40ae-481c-a707-44a349210f33" (UID: "af50a359-40ae-481c-a707-44a349210f33"). InnerVolumeSpecName "kube-api-access-jt72n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:14.549052 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.549026 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af50a359-40ae-481c-a707-44a349210f33-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:14.549052 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.549052 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt72n\" (UniqueName: \"kubernetes.io/projected/af50a359-40ae-481c-a707-44a349210f33-kube-api-access-jt72n\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:14.549189 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.549061 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af50a359-40ae-481c-a707-44a349210f33-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:14.677623 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.677593 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn"] Nov 03 20:39:14.681143 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:14.681120 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-43c674/sub-test-model-43c674-kserve-677ff676dc-t4xqn"] Nov 03 20:39:15.857037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:15.857002 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af50a359-40ae-481c-a707-44a349210f33" path="/var/lib/kubelet/pods/af50a359-40ae-481c-a707-44a349210f33/volumes" Nov 03 20:39:16.026028 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:16.025975 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-4vl4x"] Nov 03 20:39:16.031239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:16.031210 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-4vl4x"] Nov 03 20:39:17.863493 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:17.863043 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94e83c10-58b0-4bc7-9abd-3e144ba6e258" path="/var/lib/kubelet/pods/94e83c10-58b0-4bc7-9abd-3e144ba6e258/volumes" Nov 03 20:39:19.385403 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.385370 5059 generic.go:356] "Generic (PLEG): container finished" podID="426d1445-9e48-4516-a920-fca04d918f22" containerID="3f775196318978b57c4c708d101e83f38ee80fd8c5d6c5eee6f5841b39c93535" exitCode=0 Nov 03 20:39:19.385739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.385450 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" event={"ID":"426d1445-9e48-4516-a920-fca04d918f22","Type":"ContainerDied","Data":"3f775196318978b57c4c708d101e83f38ee80fd8c5d6c5eee6f5841b39c93535"} Nov 03 20:39:19.385739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.385553 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" event={"ID":"426d1445-9e48-4516-a920-fca04d918f22","Type":"ContainerDied","Data":"449a5120f0d2ba152b610add4e451094682460cd3022ae0152f8fdb0df868c9a"} Nov 03 20:39:19.385739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.385578 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="449a5120f0d2ba152b610add4e451094682460cd3022ae0152f8fdb0df868c9a" Nov 03 20:39:19.397248 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.397227 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:39:19.440534 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.440457 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" podUID="bb66cb90-2ccf-4325-80aa-d8e43ac54a22" containerName="main" containerID="cri-o://7a8879573a59af72b0c503748f3635c55ffb0f90934ffc77fc4ef4f8f957e740" gracePeriod=45 Nov 03 20:39:19.526271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.526238 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/426d1445-9e48-4516-a920-fca04d918f22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426d1445-9e48-4516-a920-fca04d918f22-istiod-ca-cert\") pod \"426d1445-9e48-4516-a920-fca04d918f22\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " Nov 03 20:39:19.526438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.526278 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-workload-certs\") pod \"426d1445-9e48-4516-a920-fca04d918f22\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " Nov 03 20:39:19.526438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.526329 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-workload-socket\") pod \"426d1445-9e48-4516-a920-fca04d918f22\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " Nov 03 20:39:19.526438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.526357 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-credential-socket\") pod \"426d1445-9e48-4516-a920-fca04d918f22\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " Nov 03 20:39:19.526438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.526384 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/426d1445-9e48-4516-a920-fca04d918f22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426d1445-9e48-4516-a920-fca04d918f22-istio-podinfo\") pod \"426d1445-9e48-4516-a920-fca04d918f22\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " Nov 03 20:39:19.526438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.526414 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/426d1445-9e48-4516-a920-fca04d918f22-istio-token\" (UniqueName: \"kubernetes.io/projected/426d1445-9e48-4516-a920-fca04d918f22-istio-token\") pod \"426d1445-9e48-4516-a920-fca04d918f22\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " Nov 03 20:39:19.526728 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.526494 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/426d1445-9e48-4516-a920-fca04d918f22-kube-api-access-6jqnx\" (UniqueName: \"kubernetes.io/projected/426d1445-9e48-4516-a920-fca04d918f22-kube-api-access-6jqnx\") pod \"426d1445-9e48-4516-a920-fca04d918f22\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " Nov 03 20:39:19.526728 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.526530 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-istio-data\") pod \"426d1445-9e48-4516-a920-fca04d918f22\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " Nov 03 20:39:19.526728 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.526629 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-istio-envoy\") pod \"426d1445-9e48-4516-a920-fca04d918f22\" (UID: \"426d1445-9e48-4516-a920-fca04d918f22\") " Nov 03 20:39:19.526728 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.526676 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-workload-socket" pod "426d1445-9e48-4516-a920-fca04d918f22" (UID: "426d1445-9e48-4516-a920-fca04d918f22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:19.526953 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.526787 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-credential-socket" pod "426d1445-9e48-4516-a920-fca04d918f22" (UID: "426d1445-9e48-4516-a920-fca04d918f22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:19.526953 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.526829 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426d1445-9e48-4516-a920-fca04d918f22-istiod-ca-cert" pod "426d1445-9e48-4516-a920-fca04d918f22" (UID: "426d1445-9e48-4516-a920-fca04d918f22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:19.526953 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.526899 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-workload-certs" pod "426d1445-9e48-4516-a920-fca04d918f22" (UID: "426d1445-9e48-4516-a920-fca04d918f22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:19.526953 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.526915 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-istio-data" pod "426d1445-9e48-4516-a920-fca04d918f22" (UID: "426d1445-9e48-4516-a920-fca04d918f22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:19.527289 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.527259 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426d1445-9e48-4516-a920-fca04d918f22-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:19.527405 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.527292 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:19.527405 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.527307 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:19.527405 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.527324 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:19.527405 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.527339 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:19.528931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.528905 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-istio-envoy" pod "426d1445-9e48-4516-a920-fca04d918f22" (UID: "426d1445-9e48-4516-a920-fca04d918f22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:19.528931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.528920 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/426d1445-9e48-4516-a920-fca04d918f22-istio-podinfo" pod "426d1445-9e48-4516-a920-fca04d918f22" (UID: "426d1445-9e48-4516-a920-fca04d918f22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:39:19.529113 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.529095 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426d1445-9e48-4516-a920-fca04d918f22-kube-api-access-6jqnx" pod "426d1445-9e48-4516-a920-fca04d918f22" (UID: "426d1445-9e48-4516-a920-fca04d918f22"). InnerVolumeSpecName "kube-api-access-6jqnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:19.529214 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.529195 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426d1445-9e48-4516-a920-fca04d918f22-istio-token" pod "426d1445-9e48-4516-a920-fca04d918f22" (UID: "426d1445-9e48-4516-a920-fca04d918f22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:19.628351 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.628322 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426d1445-9e48-4516-a920-fca04d918f22-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:19.628496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.628353 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426d1445-9e48-4516-a920-fca04d918f22-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:19.628496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.628368 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jqnx\" (UniqueName: \"kubernetes.io/projected/426d1445-9e48-4516-a920-fca04d918f22-kube-api-access-6jqnx\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:19.628496 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.628384 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426d1445-9e48-4516-a920-fca04d918f22-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:19.685344 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.685318 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:39:19.830190 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.830160 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-model-cache\") pod \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\" (UID: \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\") " Nov 03 20:39:19.830327 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.830219 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-tls-certs\" (UniqueName: \"kubernetes.io/secret/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-tls-certs\") pod \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\" (UID: \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\") " Nov 03 20:39:19.830327 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.830265 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-kube-api-access-l7cbh\" (UniqueName: \"kubernetes.io/projected/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-kube-api-access-l7cbh\") pod \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\" (UID: \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\") " Nov 03 20:39:19.830447 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.830341 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-dshm\") pod \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\" (UID: \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\") " Nov 03 20:39:19.830447 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.830366 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-tmp-dir\") pod \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\" (UID: \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\") " Nov 03 20:39:19.830447 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.830400 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-home\" (UniqueName: \"kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-home\") pod \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\" (UID: \"bb66cb90-2ccf-4325-80aa-d8e43ac54a22\") " Nov 03 20:39:19.830601 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.830471 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-model-cache" pod "bb66cb90-2ccf-4325-80aa-d8e43ac54a22" (UID: "bb66cb90-2ccf-4325-80aa-d8e43ac54a22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:19.830693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.830647 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-tmp-dir" pod "bb66cb90-2ccf-4325-80aa-d8e43ac54a22" (UID: "bb66cb90-2ccf-4325-80aa-d8e43ac54a22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:19.830754 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.830698 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-home" pod "bb66cb90-2ccf-4325-80aa-d8e43ac54a22" (UID: "bb66cb90-2ccf-4325-80aa-d8e43ac54a22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:19.830854 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.830836 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:19.830890 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.830861 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:19.830890 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.830876 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:19.832490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.832470 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-tls-certs" pod "bb66cb90-2ccf-4325-80aa-d8e43ac54a22" (UID: "bb66cb90-2ccf-4325-80aa-d8e43ac54a22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:39:19.832560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.832511 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-dshm" pod "bb66cb90-2ccf-4325-80aa-d8e43ac54a22" (UID: "bb66cb90-2ccf-4325-80aa-d8e43ac54a22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:19.832603 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.832558 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-kube-api-access-l7cbh" pod "bb66cb90-2ccf-4325-80aa-d8e43ac54a22" (UID: "bb66cb90-2ccf-4325-80aa-d8e43ac54a22"). InnerVolumeSpecName "kube-api-access-l7cbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:19.932116 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.932092 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:19.932116 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.932114 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:19.932242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:19.932124 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7cbh\" (UniqueName: \"kubernetes.io/projected/bb66cb90-2ccf-4325-80aa-d8e43ac54a22-kube-api-access-l7cbh\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:20.392613 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:20.392578 5059 generic.go:356] "Generic (PLEG): container finished" podID="bb66cb90-2ccf-4325-80aa-d8e43ac54a22" containerID="7a8879573a59af72b0c503748f3635c55ffb0f90934ffc77fc4ef4f8f957e740" exitCode=0 Nov 03 20:39:20.393044 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:20.392624 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" Nov 03 20:39:20.393044 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:20.392688 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" event={"ID":"bb66cb90-2ccf-4325-80aa-d8e43ac54a22","Type":"ContainerDied","Data":"7a8879573a59af72b0c503748f3635c55ffb0f90934ffc77fc4ef4f8f957e740"} Nov 03 20:39:20.393044 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:20.392729 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk" event={"ID":"bb66cb90-2ccf-4325-80aa-d8e43ac54a22","Type":"ContainerDied","Data":"5d31881ffd1668800f9e00f1343ca4f9463f990eb1ee3878260c553cc9f7433b"} Nov 03 20:39:20.393044 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:20.392746 5059 scope.go:122] "RemoveContainer" containerID="7a8879573a59af72b0c503748f3635c55ffb0f90934ffc77fc4ef4f8f957e740" Nov 03 20:39:20.393044 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:20.392856 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2" Nov 03 20:39:20.402797 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:20.402780 5059 scope.go:122] "RemoveContainer" containerID="7a8879573a59af72b0c503748f3635c55ffb0f90934ffc77fc4ef4f8f957e740" Nov 03 20:39:20.403065 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:39:20.403040 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a8879573a59af72b0c503748f3635c55ffb0f90934ffc77fc4ef4f8f957e740\": container with ID starting with 7a8879573a59af72b0c503748f3635c55ffb0f90934ffc77fc4ef4f8f957e740 not found: ID does not exist" containerID="7a8879573a59af72b0c503748f3635c55ffb0f90934ffc77fc4ef4f8f957e740" Nov 03 20:39:20.403118 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:20.403069 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a8879573a59af72b0c503748f3635c55ffb0f90934ffc77fc4ef4f8f957e740"} err="failed to get container status \"7a8879573a59af72b0c503748f3635c55ffb0f90934ffc77fc4ef4f8f957e740\": rpc error: code = NotFound desc = could not find container \"7a8879573a59af72b0c503748f3635c55ffb0f90934ffc77fc4ef4f8f957e740\": container with ID starting with 7a8879573a59af72b0c503748f3635c55ffb0f90934ffc77fc4ef4f8f957e740 not found: ID does not exist" Nov 03 20:39:20.416807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:20.416781 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk"] Nov 03 20:39:20.423379 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:20.423327 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bfc240/sub-test-model-bfc240-kserve-7d5997f4f9-9f6zk"] Nov 03 20:39:20.435211 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:20.435188 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2"] Nov 03 20:39:20.439282 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:20.439259 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-33e267-openshift-default-85f8d58cb5-vhbf2"] Nov 03 20:39:21.597500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:21.597461 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn"] Nov 03 20:39:21.598020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:21.597722 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" podUID="e04b50ed-38f6-4c24-b4ca-046ca2f9d170" containerName="istio-proxy" containerID="cri-o://23edab384e4fee0335fdc43dcc0e0ae8943aa7e398e33d03b809e83016b240a5" gracePeriod=30 Nov 03 20:39:21.856905 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:21.856828 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="426d1445-9e48-4516-a920-fca04d918f22" path="/var/lib/kubelet/pods/426d1445-9e48-4516-a920-fca04d918f22/volumes" Nov 03 20:39:21.857267 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:21.857252 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb66cb90-2ccf-4325-80aa-d8e43ac54a22" path="/var/lib/kubelet/pods/bb66cb90-2ccf-4325-80aa-d8e43ac54a22/volumes" Nov 03 20:39:22.033205 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:22.033162 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls"] Nov 03 20:39:23.028209 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:23.028167 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-nw2ph"] Nov 03 20:39:23.035463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:23.035425 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx"] Nov 03 20:39:23.040741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:23.040710 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-nw2ph"] Nov 03 20:39:23.044171 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:23.044149 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-2nlxx"] Nov 03 20:39:23.856741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:23.856709 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31725d6d-9a91-44ba-a08b-1520f5853190" path="/var/lib/kubelet/pods/31725d6d-9a91-44ba-a08b-1520f5853190/volumes" Nov 03 20:39:23.857189 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:23.857172 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="847e7b38-8fb0-4c19-8943-f43e364f989b" path="/var/lib/kubelet/pods/847e7b38-8fb0-4c19-8943-f43e364f989b/volumes" Nov 03 20:39:24.026322 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:24.026287 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-7w5r9"] Nov 03 20:39:24.029976 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:24.029947 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-8sgnl"] Nov 03 20:39:24.033037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:24.033009 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-7w5r9"] Nov 03 20:39:24.035584 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:24.035561 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-8sgnl"] Nov 03 20:39:24.645196 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:24.645158 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" podUID="9f2c7048-0565-4bd2-b05b-735757071fb6" containerName="main" containerID="cri-o://aa189ee45c1a7aa60d8380219508c000c3dd82c301f41e2f5b8b4a73880c7105" gracePeriod=45 Nov 03 20:39:24.897445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:24.897385 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:39:24.914482 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:24.914457 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-dshm\") pod \"9f2c7048-0565-4bd2-b05b-735757071fb6\" (UID: \"9f2c7048-0565-4bd2-b05b-735757071fb6\") " Nov 03 20:39:24.914551 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:24.914521 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-home\" (UniqueName: \"kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-home\") pod \"9f2c7048-0565-4bd2-b05b-735757071fb6\" (UID: \"9f2c7048-0565-4bd2-b05b-735757071fb6\") " Nov 03 20:39:24.914584 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:24.914553 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f2c7048-0565-4bd2-b05b-735757071fb6-kube-api-access-qqxcf\" (UniqueName: \"kubernetes.io/projected/9f2c7048-0565-4bd2-b05b-735757071fb6-kube-api-access-qqxcf\") pod \"9f2c7048-0565-4bd2-b05b-735757071fb6\" (UID: \"9f2c7048-0565-4bd2-b05b-735757071fb6\") " Nov 03 20:39:24.914631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:24.914605 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-tmp-dir\") pod \"9f2c7048-0565-4bd2-b05b-735757071fb6\" (UID: \"9f2c7048-0565-4bd2-b05b-735757071fb6\") " Nov 03 20:39:24.914780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:24.914759 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f2c7048-0565-4bd2-b05b-735757071fb6-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2c7048-0565-4bd2-b05b-735757071fb6-tls-certs\") pod \"9f2c7048-0565-4bd2-b05b-735757071fb6\" (UID: \"9f2c7048-0565-4bd2-b05b-735757071fb6\") " Nov 03 20:39:24.914868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:24.914832 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-home" pod "9f2c7048-0565-4bd2-b05b-735757071fb6" (UID: "9f2c7048-0565-4bd2-b05b-735757071fb6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:24.914913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:24.914877 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-tmp-dir" pod "9f2c7048-0565-4bd2-b05b-735757071fb6" (UID: "9f2c7048-0565-4bd2-b05b-735757071fb6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:24.914913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:24.914887 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-model-cache\") pod \"9f2c7048-0565-4bd2-b05b-735757071fb6\" (UID: \"9f2c7048-0565-4bd2-b05b-735757071fb6\") " Nov 03 20:39:24.915080 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:24.915062 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-model-cache" pod "9f2c7048-0565-4bd2-b05b-735757071fb6" (UID: "9f2c7048-0565-4bd2-b05b-735757071fb6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:24.915348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:24.915329 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:24.915348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:24.915348 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:24.915474 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:24.915357 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:24.917165 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:24.917141 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f2c7048-0565-4bd2-b05b-735757071fb6-tls-certs" pod "9f2c7048-0565-4bd2-b05b-735757071fb6" (UID: "9f2c7048-0565-4bd2-b05b-735757071fb6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:39:24.917346 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:24.917321 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-dshm" pod "9f2c7048-0565-4bd2-b05b-735757071fb6" (UID: "9f2c7048-0565-4bd2-b05b-735757071fb6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:24.917479 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:24.917454 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2c7048-0565-4bd2-b05b-735757071fb6-kube-api-access-qqxcf" pod "9f2c7048-0565-4bd2-b05b-735757071fb6" (UID: "9f2c7048-0565-4bd2-b05b-735757071fb6"). InnerVolumeSpecName "kube-api-access-qqxcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:25.016673 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:25.016644 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2c7048-0565-4bd2-b05b-735757071fb6-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:25.016787 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:25.016696 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2c7048-0565-4bd2-b05b-735757071fb6-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:25.016787 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:25.016708 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqxcf\" (UniqueName: \"kubernetes.io/projected/9f2c7048-0565-4bd2-b05b-735757071fb6-kube-api-access-qqxcf\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:25.427491 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:25.427457 5059 generic.go:356] "Generic (PLEG): container finished" podID="9f2c7048-0565-4bd2-b05b-735757071fb6" containerID="aa189ee45c1a7aa60d8380219508c000c3dd82c301f41e2f5b8b4a73880c7105" exitCode=0 Nov 03 20:39:25.428157 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:25.427499 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" Nov 03 20:39:25.428157 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:25.427533 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" event={"ID":"9f2c7048-0565-4bd2-b05b-735757071fb6","Type":"ContainerDied","Data":"aa189ee45c1a7aa60d8380219508c000c3dd82c301f41e2f5b8b4a73880c7105"} Nov 03 20:39:25.428157 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:25.427575 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856" event={"ID":"9f2c7048-0565-4bd2-b05b-735757071fb6","Type":"ContainerDied","Data":"cb578c4d9633b9fb63d1621a269b7b333edc333d483e44c7eb8037dcd0e3ccb8"} Nov 03 20:39:25.428157 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:25.427593 5059 scope.go:122] "RemoveContainer" containerID="aa189ee45c1a7aa60d8380219508c000c3dd82c301f41e2f5b8b4a73880c7105" Nov 03 20:39:25.438099 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:25.438078 5059 scope.go:122] "RemoveContainer" containerID="aa189ee45c1a7aa60d8380219508c000c3dd82c301f41e2f5b8b4a73880c7105" Nov 03 20:39:25.438360 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:39:25.438339 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa189ee45c1a7aa60d8380219508c000c3dd82c301f41e2f5b8b4a73880c7105\": container with ID starting with aa189ee45c1a7aa60d8380219508c000c3dd82c301f41e2f5b8b4a73880c7105 not found: ID does not exist" containerID="aa189ee45c1a7aa60d8380219508c000c3dd82c301f41e2f5b8b4a73880c7105" Nov 03 20:39:25.438422 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:25.438372 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa189ee45c1a7aa60d8380219508c000c3dd82c301f41e2f5b8b4a73880c7105"} err="failed to get container status \"aa189ee45c1a7aa60d8380219508c000c3dd82c301f41e2f5b8b4a73880c7105\": rpc error: code = NotFound desc = could not find container \"aa189ee45c1a7aa60d8380219508c000c3dd82c301f41e2f5b8b4a73880c7105\": container with ID starting with aa189ee45c1a7aa60d8380219508c000c3dd82c301f41e2f5b8b4a73880c7105 not found: ID does not exist" Nov 03 20:39:25.451780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:25.451756 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856"] Nov 03 20:39:25.454993 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:25.454972 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcfde8/sub-test-model-bcfde8-kserve-854897f565-47856"] Nov 03 20:39:25.546125 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:25.546103 5059 scope.go:122] "RemoveContainer" containerID="3f775196318978b57c4c708d101e83f38ee80fd8c5d6c5eee6f5841b39c93535" Nov 03 20:39:25.555669 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:25.555641 5059 scope.go:122] "RemoveContainer" containerID="da8dcf56301a5bf859cd1267d1436e748641fe4be7bf5c9b6d88b7d504191f65" Nov 03 20:39:25.858246 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:25.858200 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42fe8135-7b7e-4292-ab0d-5ba57bbd3699" path="/var/lib/kubelet/pods/42fe8135-7b7e-4292-ab0d-5ba57bbd3699/volumes" Nov 03 20:39:25.858815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:25.858795 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f2c7048-0565-4bd2-b05b-735757071fb6" path="/var/lib/kubelet/pods/9f2c7048-0565-4bd2-b05b-735757071fb6/volumes" Nov 03 20:39:25.859360 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:25.859345 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b39ff65e-aa88-4534-a0f1-cd2af557c4ed" path="/var/lib/kubelet/pods/b39ff65e-aa88-4534-a0f1-cd2af557c4ed/volumes" Nov 03 20:39:26.850402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:26.850374 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:39:26.951290 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:26.951192 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-workload-certs\") pod \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " Nov 03 20:39:26.951290 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:26.951239 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-kube-api-access-c9wgn\" (UniqueName: \"kubernetes.io/projected/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-kube-api-access-c9wgn\") pod \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " Nov 03 20:39:26.951290 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:26.951262 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istiod-ca-cert\") pod \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " Nov 03 20:39:26.951593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:26.951292 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-workload-socket\") pod \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " Nov 03 20:39:26.951593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:26.951355 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-envoy\") pod \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " Nov 03 20:39:26.951593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:26.951393 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-credential-socket\") pod \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " Nov 03 20:39:26.951593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:26.951418 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-token\" (UniqueName: \"kubernetes.io/projected/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-token\") pod \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " Nov 03 20:39:26.951593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:26.951495 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-data\") pod \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " Nov 03 20:39:26.951593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:26.951549 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-podinfo\") pod \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\" (UID: \"e04b50ed-38f6-4c24-b4ca-046ca2f9d170\") " Nov 03 20:39:26.951961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:26.951600 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-workload-certs" pod "e04b50ed-38f6-4c24-b4ca-046ca2f9d170" (UID: "e04b50ed-38f6-4c24-b4ca-046ca2f9d170"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:26.951961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:26.951653 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-workload-socket" pod "e04b50ed-38f6-4c24-b4ca-046ca2f9d170" (UID: "e04b50ed-38f6-4c24-b4ca-046ca2f9d170"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:26.951961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:26.951856 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-data" pod "e04b50ed-38f6-4c24-b4ca-046ca2f9d170" (UID: "e04b50ed-38f6-4c24-b4ca-046ca2f9d170"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:26.951961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:26.951869 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-credential-socket" pod "e04b50ed-38f6-4c24-b4ca-046ca2f9d170" (UID: "e04b50ed-38f6-4c24-b4ca-046ca2f9d170"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:26.951961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:26.951863 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istiod-ca-cert" pod "e04b50ed-38f6-4c24-b4ca-046ca2f9d170" (UID: "e04b50ed-38f6-4c24-b4ca-046ca2f9d170"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:26.952269 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:26.952149 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:26.952269 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:26.952173 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:26.952269 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:26.952189 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:26.952269 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:26.952204 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:26.952269 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:26.952218 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:26.953850 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:26.953821 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-token" pod "e04b50ed-38f6-4c24-b4ca-046ca2f9d170" (UID: "e04b50ed-38f6-4c24-b4ca-046ca2f9d170"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:26.953850 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:26.953837 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-podinfo" pod "e04b50ed-38f6-4c24-b4ca-046ca2f9d170" (UID: "e04b50ed-38f6-4c24-b4ca-046ca2f9d170"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:39:26.953997 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:26.953931 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-envoy" pod "e04b50ed-38f6-4c24-b4ca-046ca2f9d170" (UID: "e04b50ed-38f6-4c24-b4ca-046ca2f9d170"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:26.953997 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:26.953931 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-kube-api-access-c9wgn" pod "e04b50ed-38f6-4c24-b4ca-046ca2f9d170" (UID: "e04b50ed-38f6-4c24-b4ca-046ca2f9d170"). InnerVolumeSpecName "kube-api-access-c9wgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:27.033619 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.033586 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls" podUID="8760fa0e-a78b-4db8-b71d-bf4086ddbfbd" containerName="payload-pre-processing" containerID="cri-o://cef80931e6aff9be1967d6d3105b1fc9be14a0fbc1100b81828051cdb49d5711" gracePeriod=25 Nov 03 20:39:27.054349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.053653 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9wgn\" (UniqueName: \"kubernetes.io/projected/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-kube-api-access-c9wgn\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:27.054434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.054363 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:27.054434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.054376 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:27.054434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.054386 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04b50ed-38f6-4c24-b4ca-046ca2f9d170-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:27.281731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.281702 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls" Nov 03 20:39:27.357576 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.357546 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8760fa0e-a78b-4db8-b71d-bf4086ddbfbd-kube-api-access-w8gkn\" (UniqueName: \"kubernetes.io/projected/8760fa0e-a78b-4db8-b71d-bf4086ddbfbd-kube-api-access-w8gkn\") pod \"8760fa0e-a78b-4db8-b71d-bf4086ddbfbd\" (UID: \"8760fa0e-a78b-4db8-b71d-bf4086ddbfbd\") " Nov 03 20:39:27.357737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.357600 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8760fa0e-a78b-4db8-b71d-bf4086ddbfbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8760fa0e-a78b-4db8-b71d-bf4086ddbfbd-plugins-config-volume\") pod \"8760fa0e-a78b-4db8-b71d-bf4086ddbfbd\" (UID: \"8760fa0e-a78b-4db8-b71d-bf4086ddbfbd\") " Nov 03 20:39:27.358053 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.358031 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8760fa0e-a78b-4db8-b71d-bf4086ddbfbd-plugins-config-volume" pod "8760fa0e-a78b-4db8-b71d-bf4086ddbfbd" (UID: "8760fa0e-a78b-4db8-b71d-bf4086ddbfbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:27.358172 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.358156 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8760fa0e-a78b-4db8-b71d-bf4086ddbfbd-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:27.359645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.359622 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8760fa0e-a78b-4db8-b71d-bf4086ddbfbd-kube-api-access-w8gkn" pod "8760fa0e-a78b-4db8-b71d-bf4086ddbfbd" (UID: "8760fa0e-a78b-4db8-b71d-bf4086ddbfbd"). InnerVolumeSpecName "kube-api-access-w8gkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:27.442007 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.441971 5059 generic.go:356] "Generic (PLEG): container finished" podID="e04b50ed-38f6-4c24-b4ca-046ca2f9d170" containerID="23edab384e4fee0335fdc43dcc0e0ae8943aa7e398e33d03b809e83016b240a5" exitCode=0 Nov 03 20:39:27.442160 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.442019 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" Nov 03 20:39:27.442160 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.442051 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" event={"ID":"e04b50ed-38f6-4c24-b4ca-046ca2f9d170","Type":"ContainerDied","Data":"23edab384e4fee0335fdc43dcc0e0ae8943aa7e398e33d03b809e83016b240a5"} Nov 03 20:39:27.442160 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.442102 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn" event={"ID":"e04b50ed-38f6-4c24-b4ca-046ca2f9d170","Type":"ContainerDied","Data":"35f501e7e614516b4b1d3a6e3904912ca4b78a1820a4dc30fd6522533bc077dd"} Nov 03 20:39:27.442160 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.442126 5059 scope.go:122] "RemoveContainer" containerID="23edab384e4fee0335fdc43dcc0e0ae8943aa7e398e33d03b809e83016b240a5" Nov 03 20:39:27.443747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.443722 5059 generic.go:356] "Generic (PLEG): container finished" podID="8760fa0e-a78b-4db8-b71d-bf4086ddbfbd" containerID="cef80931e6aff9be1967d6d3105b1fc9be14a0fbc1100b81828051cdb49d5711" exitCode=0 Nov 03 20:39:27.443872 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.443777 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls" event={"ID":"8760fa0e-a78b-4db8-b71d-bf4086ddbfbd","Type":"ContainerDied","Data":"cef80931e6aff9be1967d6d3105b1fc9be14a0fbc1100b81828051cdb49d5711"} Nov 03 20:39:27.443872 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.443807 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls" event={"ID":"8760fa0e-a78b-4db8-b71d-bf4086ddbfbd","Type":"ContainerDied","Data":"4959b0c7f42848c673b202788a758f9f274e1f3de52fca202971872398c3d9c7"} Nov 03 20:39:27.443872 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.443830 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls" Nov 03 20:39:27.453879 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.453860 5059 scope.go:122] "RemoveContainer" containerID="23edab384e4fee0335fdc43dcc0e0ae8943aa7e398e33d03b809e83016b240a5" Nov 03 20:39:27.454173 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:39:27.454145 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23edab384e4fee0335fdc43dcc0e0ae8943aa7e398e33d03b809e83016b240a5\": container with ID starting with 23edab384e4fee0335fdc43dcc0e0ae8943aa7e398e33d03b809e83016b240a5 not found: ID does not exist" containerID="23edab384e4fee0335fdc43dcc0e0ae8943aa7e398e33d03b809e83016b240a5" Nov 03 20:39:27.454262 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.454183 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23edab384e4fee0335fdc43dcc0e0ae8943aa7e398e33d03b809e83016b240a5"} err="failed to get container status \"23edab384e4fee0335fdc43dcc0e0ae8943aa7e398e33d03b809e83016b240a5\": rpc error: code = NotFound desc = could not find container \"23edab384e4fee0335fdc43dcc0e0ae8943aa7e398e33d03b809e83016b240a5\": container with ID starting with 23edab384e4fee0335fdc43dcc0e0ae8943aa7e398e33d03b809e83016b240a5 not found: ID does not exist" Nov 03 20:39:27.454262 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.454208 5059 scope.go:122] "RemoveContainer" containerID="cef80931e6aff9be1967d6d3105b1fc9be14a0fbc1100b81828051cdb49d5711" Nov 03 20:39:27.458773 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.458752 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8gkn\" (UniqueName: \"kubernetes.io/projected/8760fa0e-a78b-4db8-b71d-bf4086ddbfbd-kube-api-access-w8gkn\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:27.462980 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.462961 5059 scope.go:122] "RemoveContainer" containerID="cef80931e6aff9be1967d6d3105b1fc9be14a0fbc1100b81828051cdb49d5711" Nov 03 20:39:27.463227 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:39:27.463208 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cef80931e6aff9be1967d6d3105b1fc9be14a0fbc1100b81828051cdb49d5711\": container with ID starting with cef80931e6aff9be1967d6d3105b1fc9be14a0fbc1100b81828051cdb49d5711 not found: ID does not exist" containerID="cef80931e6aff9be1967d6d3105b1fc9be14a0fbc1100b81828051cdb49d5711" Nov 03 20:39:27.463271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.463234 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cef80931e6aff9be1967d6d3105b1fc9be14a0fbc1100b81828051cdb49d5711"} err="failed to get container status \"cef80931e6aff9be1967d6d3105b1fc9be14a0fbc1100b81828051cdb49d5711\": rpc error: code = NotFound desc = could not find container \"cef80931e6aff9be1967d6d3105b1fc9be14a0fbc1100b81828051cdb49d5711\": container with ID starting with cef80931e6aff9be1967d6d3105b1fc9be14a0fbc1100b81828051cdb49d5711 not found: ID does not exist" Nov 03 20:39:27.468076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.468054 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn"] Nov 03 20:39:27.472522 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.472501 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-k7cxn"] Nov 03 20:39:27.489731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.489696 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls"] Nov 03 20:39:27.494344 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.494321 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4c2a28-79968ddbb7-jh9ls"] Nov 03 20:39:27.863265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.859896 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8760fa0e-a78b-4db8-b71d-bf4086ddbfbd" path="/var/lib/kubelet/pods/8760fa0e-a78b-4db8-b71d-bf4086ddbfbd/volumes" Nov 03 20:39:27.863265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:27.860583 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e04b50ed-38f6-4c24-b4ca-046ca2f9d170" path="/var/lib/kubelet/pods/e04b50ed-38f6-4c24-b4ca-046ca2f9d170/volumes" Nov 03 20:39:30.056263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:30.056226 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth"] Nov 03 20:39:30.056795 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:30.056445 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" podUID="b1573256-23a1-4a26-9972-87bc9b27f2c1" containerName="istio-proxy" containerID="cri-o://f9261494f566e526c65e3e2a7bea4c6b25d6474679613298eca55296a266371f" gracePeriod=30 Nov 03 20:39:35.318260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.318216 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:39:35.360561 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.360481 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-data\") pod \"b1573256-23a1-4a26-9972-87bc9b27f2c1\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " Nov 03 20:39:35.360561 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.360519 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-credential-socket\") pod \"b1573256-23a1-4a26-9972-87bc9b27f2c1\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " Nov 03 20:39:35.360561 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.360546 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-workload-certs\") pod \"b1573256-23a1-4a26-9972-87bc9b27f2c1\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " Nov 03 20:39:35.360892 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.360570 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-token\" (UniqueName: \"kubernetes.io/projected/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-token\") pod \"b1573256-23a1-4a26-9972-87bc9b27f2c1\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " Nov 03 20:39:35.360892 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.360598 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1573256-23a1-4a26-9972-87bc9b27f2c1-kube-api-access-kfncw\" (UniqueName: \"kubernetes.io/projected/b1573256-23a1-4a26-9972-87bc9b27f2c1-kube-api-access-kfncw\") pod \"b1573256-23a1-4a26-9972-87bc9b27f2c1\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " Nov 03 20:39:35.360892 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.360793 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1573256-23a1-4a26-9972-87bc9b27f2c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1573256-23a1-4a26-9972-87bc9b27f2c1-istiod-ca-cert\") pod \"b1573256-23a1-4a26-9972-87bc9b27f2c1\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " Nov 03 20:39:35.360892 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.360820 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-podinfo\") pod \"b1573256-23a1-4a26-9972-87bc9b27f2c1\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " Nov 03 20:39:35.360892 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.360846 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-credential-socket" pod "b1573256-23a1-4a26-9972-87bc9b27f2c1" (UID: "b1573256-23a1-4a26-9972-87bc9b27f2c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:35.360892 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.360861 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-workload-certs" pod "b1573256-23a1-4a26-9972-87bc9b27f2c1" (UID: "b1573256-23a1-4a26-9972-87bc9b27f2c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:35.361222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.360908 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-data" pod "b1573256-23a1-4a26-9972-87bc9b27f2c1" (UID: "b1573256-23a1-4a26-9972-87bc9b27f2c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:35.361222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.360858 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-workload-socket\") pod \"b1573256-23a1-4a26-9972-87bc9b27f2c1\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " Nov 03 20:39:35.361222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.360986 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-envoy\") pod \"b1573256-23a1-4a26-9972-87bc9b27f2c1\" (UID: \"b1573256-23a1-4a26-9972-87bc9b27f2c1\") " Nov 03 20:39:35.361222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.361058 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-workload-socket" pod "b1573256-23a1-4a26-9972-87bc9b27f2c1" (UID: "b1573256-23a1-4a26-9972-87bc9b27f2c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:35.361222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.361199 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1573256-23a1-4a26-9972-87bc9b27f2c1-istiod-ca-cert" pod "b1573256-23a1-4a26-9972-87bc9b27f2c1" (UID: "b1573256-23a1-4a26-9972-87bc9b27f2c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:35.361482 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.361406 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1573256-23a1-4a26-9972-87bc9b27f2c1-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:35.361482 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.361426 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:35.361482 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.361438 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:35.361482 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.361453 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:35.361482 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.361465 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:35.363213 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.363188 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1573256-23a1-4a26-9972-87bc9b27f2c1-kube-api-access-kfncw" pod "b1573256-23a1-4a26-9972-87bc9b27f2c1" (UID: "b1573256-23a1-4a26-9972-87bc9b27f2c1"). InnerVolumeSpecName "kube-api-access-kfncw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:35.363730 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.363657 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-envoy" pod "b1573256-23a1-4a26-9972-87bc9b27f2c1" (UID: "b1573256-23a1-4a26-9972-87bc9b27f2c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:35.363730 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.363650 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-podinfo" pod "b1573256-23a1-4a26-9972-87bc9b27f2c1" (UID: "b1573256-23a1-4a26-9972-87bc9b27f2c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:39:35.363730 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.363714 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-token" pod "b1573256-23a1-4a26-9972-87bc9b27f2c1" (UID: "b1573256-23a1-4a26-9972-87bc9b27f2c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:35.462434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.462406 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:35.462434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.462434 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfncw\" (UniqueName: \"kubernetes.io/projected/b1573256-23a1-4a26-9972-87bc9b27f2c1-kube-api-access-kfncw\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:35.462567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.462446 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:35.462567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.462455 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1573256-23a1-4a26-9972-87bc9b27f2c1-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:35.506514 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.506480 5059 generic.go:356] "Generic (PLEG): container finished" podID="b1573256-23a1-4a26-9972-87bc9b27f2c1" containerID="f9261494f566e526c65e3e2a7bea4c6b25d6474679613298eca55296a266371f" exitCode=0 Nov 03 20:39:35.506725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.506528 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" Nov 03 20:39:35.506725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.506568 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" event={"ID":"b1573256-23a1-4a26-9972-87bc9b27f2c1","Type":"ContainerDied","Data":"f9261494f566e526c65e3e2a7bea4c6b25d6474679613298eca55296a266371f"} Nov 03 20:39:35.506725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.506628 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth" event={"ID":"b1573256-23a1-4a26-9972-87bc9b27f2c1","Type":"ContainerDied","Data":"e295e3f39ed26ad84473c631398dfe509800dfda3f929613c9ca1432d6bb0a75"} Nov 03 20:39:35.506725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.506650 5059 scope.go:122] "RemoveContainer" containerID="f9261494f566e526c65e3e2a7bea4c6b25d6474679613298eca55296a266371f" Nov 03 20:39:35.517756 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.517736 5059 scope.go:122] "RemoveContainer" containerID="f9261494f566e526c65e3e2a7bea4c6b25d6474679613298eca55296a266371f" Nov 03 20:39:35.518046 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:39:35.518027 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9261494f566e526c65e3e2a7bea4c6b25d6474679613298eca55296a266371f\": container with ID starting with f9261494f566e526c65e3e2a7bea4c6b25d6474679613298eca55296a266371f not found: ID does not exist" containerID="f9261494f566e526c65e3e2a7bea4c6b25d6474679613298eca55296a266371f" Nov 03 20:39:35.518087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.518057 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9261494f566e526c65e3e2a7bea4c6b25d6474679613298eca55296a266371f"} err="failed to get container status \"f9261494f566e526c65e3e2a7bea4c6b25d6474679613298eca55296a266371f\": rpc error: code = NotFound desc = could not find container \"f9261494f566e526c65e3e2a7bea4c6b25d6474679613298eca55296a266371f\": container with ID starting with f9261494f566e526c65e3e2a7bea4c6b25d6474679613298eca55296a266371f not found: ID does not exist" Nov 03 20:39:35.533591 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.533567 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth"] Nov 03 20:39:35.546002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.545977 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-fzkth"] Nov 03 20:39:35.857812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:35.857780 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1573256-23a1-4a26-9972-87bc9b27f2c1" path="/var/lib/kubelet/pods/b1573256-23a1-4a26-9972-87bc9b27f2c1/volumes" Nov 03 20:39:36.012257 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.012223 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw"] Nov 03 20:39:36.012459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.012421 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" podUID="e997605a-f2cc-4ffe-be15-d03219b3b25b" containerName="istio-proxy" containerID="cri-o://fd322771dcdddadbf22cfe8ba57e5548a7171a87db9d178ad5e8b09c84395e9b" gracePeriod=30 Nov 03 20:39:36.906535 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.906496 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5162af5-5t4dg"] Nov 03 20:39:36.907349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907186 5059 cpu_manager.go:414] "containerMap: removing container" podUID="b1573256-23a1-4a26-9972-87bc9b27f2c1" containerName="istio-proxy" Nov 03 20:39:36.907349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907211 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1573256-23a1-4a26-9972-87bc9b27f2c1" containerName="istio-proxy" Nov 03 20:39:36.907349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907236 5059 cpu_manager.go:414] "containerMap: removing container" podUID="9f2c7048-0565-4bd2-b05b-735757071fb6" containerName="main" Nov 03 20:39:36.907349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907245 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f2c7048-0565-4bd2-b05b-735757071fb6" containerName="main" Nov 03 20:39:36.907349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907259 5059 cpu_manager.go:414] "containerMap: removing container" podUID="bb66cb90-2ccf-4325-80aa-d8e43ac54a22" containerName="main" Nov 03 20:39:36.907349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907266 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb66cb90-2ccf-4325-80aa-d8e43ac54a22" containerName="main" Nov 03 20:39:36.907349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907275 5059 cpu_manager.go:414] "containerMap: removing container" podUID="af50a359-40ae-481c-a707-44a349210f33" containerName="main" Nov 03 20:39:36.907349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907282 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af50a359-40ae-481c-a707-44a349210f33" containerName="main" Nov 03 20:39:36.907349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907301 5059 cpu_manager.go:414] "containerMap: removing container" podUID="8760fa0e-a78b-4db8-b71d-bf4086ddbfbd" containerName="payload-pre-processing" Nov 03 20:39:36.907349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907309 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8760fa0e-a78b-4db8-b71d-bf4086ddbfbd" containerName="payload-pre-processing" Nov 03 20:39:36.907349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907322 5059 cpu_manager.go:414] "containerMap: removing container" podUID="e04b50ed-38f6-4c24-b4ca-046ca2f9d170" containerName="istio-proxy" Nov 03 20:39:36.907349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907329 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e04b50ed-38f6-4c24-b4ca-046ca2f9d170" containerName="istio-proxy" Nov 03 20:39:36.907349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907351 5059 cpu_manager.go:414] "containerMap: removing container" podUID="9bb4f711-0bcc-46f4-86b9-795c66613832" containerName="istio-proxy" Nov 03 20:39:36.907349 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907358 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb4f711-0bcc-46f4-86b9-795c66613832" containerName="istio-proxy" Nov 03 20:39:36.908154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907368 5059 cpu_manager.go:414] "containerMap: removing container" podUID="426d1445-9e48-4516-a920-fca04d918f22" containerName="istio-proxy" Nov 03 20:39:36.908154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907376 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426d1445-9e48-4516-a920-fca04d918f22" containerName="istio-proxy" Nov 03 20:39:36.908154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907387 5059 cpu_manager.go:414] "containerMap: removing container" podUID="765cb609-3515-4d3d-8158-da836d306b6c" containerName="main" Nov 03 20:39:36.908154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907394 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="765cb609-3515-4d3d-8158-da836d306b6c" containerName="main" Nov 03 20:39:36.908154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907411 5059 cpu_manager.go:414] "containerMap: removing container" podUID="0f4a0ec2-017b-4272-a8e6-a95122b3e845" containerName="revoke-keys" Nov 03 20:39:36.908154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907418 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f4a0ec2-017b-4272-a8e6-a95122b3e845" containerName="revoke-keys" Nov 03 20:39:36.908154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907541 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="765cb609-3515-4d3d-8158-da836d306b6c" containerName="main" Nov 03 20:39:36.908154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907559 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f4a0ec2-017b-4272-a8e6-a95122b3e845" containerName="revoke-keys" Nov 03 20:39:36.908154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907570 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1573256-23a1-4a26-9972-87bc9b27f2c1" containerName="istio-proxy" Nov 03 20:39:36.908154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907580 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f2c7048-0565-4bd2-b05b-735757071fb6" containerName="main" Nov 03 20:39:36.908154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907588 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb66cb90-2ccf-4325-80aa-d8e43ac54a22" containerName="main" Nov 03 20:39:36.908154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907597 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="af50a359-40ae-481c-a707-44a349210f33" containerName="main" Nov 03 20:39:36.908154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907607 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8760fa0e-a78b-4db8-b71d-bf4086ddbfbd" containerName="payload-pre-processing" Nov 03 20:39:36.908154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907618 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e04b50ed-38f6-4c24-b4ca-046ca2f9d170" containerName="istio-proxy" Nov 03 20:39:36.908154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907629 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb4f711-0bcc-46f4-86b9-795c66613832" containerName="istio-proxy" Nov 03 20:39:36.908154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.907637 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="426d1445-9e48-4516-a920-fca04d918f22" containerName="istio-proxy" Nov 03 20:39:36.913109 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.913048 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5162af5-5t4dg" Nov 03 20:39:36.926316 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.926286 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5162af5-5t4dg"] Nov 03 20:39:36.927380 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.927346 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:39:36.927676 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.927643 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:39:36.986081 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.986044 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x77k\" (UniqueName: \"kubernetes.io/projected/429244dc-47a6-43af-967f-91baa92f3bca-kube-api-access-7x77k\") pod \"maas-api-revoke-sub-d5162af5-5t4dg\" (UID: \"429244dc-47a6-43af-967f-91baa92f3bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5162af5-5t4dg" Nov 03 20:39:36.986265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.986096 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429244dc-47a6-43af-967f-91baa92f3bca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5162af5-5t4dg\" (UID: \"429244dc-47a6-43af-967f-91baa92f3bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5162af5-5t4dg" Nov 03 20:39:36.986346 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:36.986285 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429244dc-47a6-43af-967f-91baa92f3bca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5162af5-5t4dg\" (UID: \"429244dc-47a6-43af-967f-91baa92f3bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5162af5-5t4dg" Nov 03 20:39:37.036824 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:37.036790 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j8h6z"] Nov 03 20:39:37.039884 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:37.039860 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j8h6z"] Nov 03 20:39:37.087591 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:37.087557 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429244dc-47a6-43af-967f-91baa92f3bca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5162af5-5t4dg\" (UID: \"429244dc-47a6-43af-967f-91baa92f3bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5162af5-5t4dg" Nov 03 20:39:37.087770 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:37.087734 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429244dc-47a6-43af-967f-91baa92f3bca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5162af5-5t4dg\" (UID: \"429244dc-47a6-43af-967f-91baa92f3bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5162af5-5t4dg" Nov 03 20:39:37.087856 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:37.087829 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x77k\" (UniqueName: \"kubernetes.io/projected/429244dc-47a6-43af-967f-91baa92f3bca-kube-api-access-7x77k\") pod \"maas-api-revoke-sub-d5162af5-5t4dg\" (UID: \"429244dc-47a6-43af-967f-91baa92f3bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5162af5-5t4dg" Nov 03 20:39:37.088239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:37.088217 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429244dc-47a6-43af-967f-91baa92f3bca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5162af5-5t4dg\" (UID: \"429244dc-47a6-43af-967f-91baa92f3bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5162af5-5t4dg" Nov 03 20:39:37.090127 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:37.090104 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429244dc-47a6-43af-967f-91baa92f3bca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5162af5-5t4dg\" (UID: \"429244dc-47a6-43af-967f-91baa92f3bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5162af5-5t4dg" Nov 03 20:39:37.100238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:37.100214 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x77k\" (UniqueName: \"kubernetes.io/projected/429244dc-47a6-43af-967f-91baa92f3bca-kube-api-access-7x77k\") pod \"maas-api-revoke-sub-d5162af5-5t4dg\" (UID: \"429244dc-47a6-43af-967f-91baa92f3bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5162af5-5t4dg" Nov 03 20:39:37.266915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:37.266827 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5162af5-5t4dg" Nov 03 20:39:37.545154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:37.545126 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5162af5-5t4dg"] Nov 03 20:39:37.547429 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:39:37.547402 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429244dc_47a6_43af_967f_91baa92f3bca.slice/crio-538c2a0d80fa72699cadc3465b484a8a8be4f461ff126e046879e0876a6c3c2f WatchSource:0}: Error finding container 538c2a0d80fa72699cadc3465b484a8a8be4f461ff126e046879e0876a6c3c2f: Status 404 returned error can't find the container with id 538c2a0d80fa72699cadc3465b484a8a8be4f461ff126e046879e0876a6c3c2f Nov 03 20:39:37.856015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:37.855963 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddf9e671-9f34-424b-976f-2b9a25deddbf" path="/var/lib/kubelet/pods/ddf9e671-9f34-424b-976f-2b9a25deddbf/volumes" Nov 03 20:39:38.023980 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:38.023948 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-wvkx8"] Nov 03 20:39:38.026889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:38.026867 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-wvkx8"] Nov 03 20:39:38.527870 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:38.527832 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5162af5-5t4dg" event={"ID":"429244dc-47a6-43af-967f-91baa92f3bca","Type":"ContainerStarted","Data":"573561366a6d6b3b3f6054e0a9da378208d34e2a4ed82a68de8ff981c80e8f9e"} Nov 03 20:39:38.528034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:38.527915 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5162af5-5t4dg" event={"ID":"429244dc-47a6-43af-967f-91baa92f3bca","Type":"ContainerStarted","Data":"538c2a0d80fa72699cadc3465b484a8a8be4f461ff126e046879e0876a6c3c2f"} Nov 03 20:39:38.546821 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:38.546769 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5162af5-5t4dg" podStartSLOduration=2.546756785 podStartE2EDuration="2.546756785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:38.54554547 +0000 UTC m=+3772.803342728" watchObservedRunningTime="2026-11-03 20:39:38.546756785 +0000 UTC m=+3772.804554059" Nov 03 20:39:39.565510 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:39.565465 5059 generic.go:356] "Generic (PLEG): container finished" podID="429244dc-47a6-43af-967f-91baa92f3bca" containerID="573561366a6d6b3b3f6054e0a9da378208d34e2a4ed82a68de8ff981c80e8f9e" exitCode=0 Nov 03 20:39:39.568302 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:39.566276 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5162af5-5t4dg" event={"ID":"429244dc-47a6-43af-967f-91baa92f3bca","Type":"ContainerDied","Data":"573561366a6d6b3b3f6054e0a9da378208d34e2a4ed82a68de8ff981c80e8f9e"} Nov 03 20:39:39.857133 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:39.857096 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f544668b-01c9-4c84-bffe-d6f09f4566e5" path="/var/lib/kubelet/pods/f544668b-01c9-4c84-bffe-d6f09f4566e5/volumes" Nov 03 20:39:40.605130 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:39:40.605092 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:39:40.710180 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:40.710158 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5162af5-5t4dg" Nov 03 20:39:40.872240 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:40.872150 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/429244dc-47a6-43af-967f-91baa92f3bca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429244dc-47a6-43af-967f-91baa92f3bca-maas-api-xks-ca\") pod \"429244dc-47a6-43af-967f-91baa92f3bca\" (UID: \"429244dc-47a6-43af-967f-91baa92f3bca\") " Nov 03 20:39:40.872240 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:40.872236 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/429244dc-47a6-43af-967f-91baa92f3bca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429244dc-47a6-43af-967f-91baa92f3bca-maas-api-openshift-ca\") pod \"429244dc-47a6-43af-967f-91baa92f3bca\" (UID: \"429244dc-47a6-43af-967f-91baa92f3bca\") " Nov 03 20:39:40.872488 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:40.872363 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/429244dc-47a6-43af-967f-91baa92f3bca-kube-api-access-7x77k\" (UniqueName: \"kubernetes.io/projected/429244dc-47a6-43af-967f-91baa92f3bca-kube-api-access-7x77k\") pod \"429244dc-47a6-43af-967f-91baa92f3bca\" (UID: \"429244dc-47a6-43af-967f-91baa92f3bca\") " Nov 03 20:39:40.872936 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:40.872702 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/429244dc-47a6-43af-967f-91baa92f3bca-maas-api-openshift-ca" pod "429244dc-47a6-43af-967f-91baa92f3bca" (UID: "429244dc-47a6-43af-967f-91baa92f3bca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:40.872936 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:40.872856 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429244dc-47a6-43af-967f-91baa92f3bca-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:40.874424 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:40.874394 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/429244dc-47a6-43af-967f-91baa92f3bca-maas-api-xks-ca" pod "429244dc-47a6-43af-967f-91baa92f3bca" (UID: "429244dc-47a6-43af-967f-91baa92f3bca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:39:40.874836 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:40.874819 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/429244dc-47a6-43af-967f-91baa92f3bca-kube-api-access-7x77k" pod "429244dc-47a6-43af-967f-91baa92f3bca" (UID: "429244dc-47a6-43af-967f-91baa92f3bca"). InnerVolumeSpecName "kube-api-access-7x77k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:40.974018 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:40.973994 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x77k\" (UniqueName: \"kubernetes.io/projected/429244dc-47a6-43af-967f-91baa92f3bca-kube-api-access-7x77k\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:40.974018 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:40.974019 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429244dc-47a6-43af-967f-91baa92f3bca-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:41.247468 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.247443 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:39:41.378207 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.378172 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-envoy\") pod \"e997605a-f2cc-4ffe-be15-d03219b3b25b\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " Nov 03 20:39:41.378374 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.378233 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-workload-socket\") pod \"e997605a-f2cc-4ffe-be15-d03219b3b25b\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " Nov 03 20:39:41.378374 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.378303 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-podinfo\") pod \"e997605a-f2cc-4ffe-be15-d03219b3b25b\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " Nov 03 20:39:41.378374 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.378366 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e997605a-f2cc-4ffe-be15-d03219b3b25b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e997605a-f2cc-4ffe-be15-d03219b3b25b-istiod-ca-cert\") pod \"e997605a-f2cc-4ffe-be15-d03219b3b25b\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " Nov 03 20:39:41.378563 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.378398 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-credential-socket\") pod \"e997605a-f2cc-4ffe-be15-d03219b3b25b\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " Nov 03 20:39:41.378563 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.378481 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-workload-certs\") pod \"e997605a-f2cc-4ffe-be15-d03219b3b25b\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " Nov 03 20:39:41.378563 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.378530 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-token\" (UniqueName: \"kubernetes.io/projected/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-token\") pod \"e997605a-f2cc-4ffe-be15-d03219b3b25b\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " Nov 03 20:39:41.378724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.378571 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-data\") pod \"e997605a-f2cc-4ffe-be15-d03219b3b25b\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " Nov 03 20:39:41.378724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.378609 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e997605a-f2cc-4ffe-be15-d03219b3b25b-kube-api-access-6clpf\" (UniqueName: \"kubernetes.io/projected/e997605a-f2cc-4ffe-be15-d03219b3b25b-kube-api-access-6clpf\") pod \"e997605a-f2cc-4ffe-be15-d03219b3b25b\" (UID: \"e997605a-f2cc-4ffe-be15-d03219b3b25b\") " Nov 03 20:39:41.378864 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.378835 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-credential-socket" pod "e997605a-f2cc-4ffe-be15-d03219b3b25b" (UID: "e997605a-f2cc-4ffe-be15-d03219b3b25b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:41.379086 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.379054 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-workload-certs" pod "e997605a-f2cc-4ffe-be15-d03219b3b25b" (UID: "e997605a-f2cc-4ffe-be15-d03219b3b25b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:41.379201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.379175 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-data" pod "e997605a-f2cc-4ffe-be15-d03219b3b25b" (UID: "e997605a-f2cc-4ffe-be15-d03219b3b25b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:41.379540 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.379498 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e997605a-f2cc-4ffe-be15-d03219b3b25b-istiod-ca-cert" pod "e997605a-f2cc-4ffe-be15-d03219b3b25b" (UID: "e997605a-f2cc-4ffe-be15-d03219b3b25b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:41.379540 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.379514 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:41.379540 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.379542 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:41.379794 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.379556 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:41.379943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.379910 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-workload-socket" pod "e997605a-f2cc-4ffe-be15-d03219b3b25b" (UID: "e997605a-f2cc-4ffe-be15-d03219b3b25b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:41.381128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.381072 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-token" pod "e997605a-f2cc-4ffe-be15-d03219b3b25b" (UID: "e997605a-f2cc-4ffe-be15-d03219b3b25b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:41.381128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.381087 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-podinfo" pod "e997605a-f2cc-4ffe-be15-d03219b3b25b" (UID: "e997605a-f2cc-4ffe-be15-d03219b3b25b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:39:41.381258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.381152 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-envoy" pod "e997605a-f2cc-4ffe-be15-d03219b3b25b" (UID: "e997605a-f2cc-4ffe-be15-d03219b3b25b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:41.381403 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.381373 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e997605a-f2cc-4ffe-be15-d03219b3b25b-kube-api-access-6clpf" pod "e997605a-f2cc-4ffe-be15-d03219b3b25b" (UID: "e997605a-f2cc-4ffe-be15-d03219b3b25b"). InnerVolumeSpecName "kube-api-access-6clpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:41.481101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.481072 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:41.481101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.481100 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e997605a-f2cc-4ffe-be15-d03219b3b25b-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:41.481257 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.481114 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:41.481257 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.481128 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e997605a-f2cc-4ffe-be15-d03219b3b25b-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:41.481257 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.481141 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e997605a-f2cc-4ffe-be15-d03219b3b25b-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:41.481257 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.481154 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6clpf\" (UniqueName: \"kubernetes.io/projected/e997605a-f2cc-4ffe-be15-d03219b3b25b-kube-api-access-6clpf\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:41.580971 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.580930 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5162af5-5t4dg" Nov 03 20:39:41.581162 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.580970 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5162af5-5t4dg" event={"ID":"429244dc-47a6-43af-967f-91baa92f3bca","Type":"ContainerDied","Data":"538c2a0d80fa72699cadc3465b484a8a8be4f461ff126e046879e0876a6c3c2f"} Nov 03 20:39:41.581162 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.581039 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="538c2a0d80fa72699cadc3465b484a8a8be4f461ff126e046879e0876a6c3c2f" Nov 03 20:39:41.582893 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.582863 5059 generic.go:356] "Generic (PLEG): container finished" podID="e997605a-f2cc-4ffe-be15-d03219b3b25b" containerID="fd322771dcdddadbf22cfe8ba57e5548a7171a87db9d178ad5e8b09c84395e9b" exitCode=0 Nov 03 20:39:41.583049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.582949 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" event={"ID":"e997605a-f2cc-4ffe-be15-d03219b3b25b","Type":"ContainerDied","Data":"fd322771dcdddadbf22cfe8ba57e5548a7171a87db9d178ad5e8b09c84395e9b"} Nov 03 20:39:41.583049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.582985 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" event={"ID":"e997605a-f2cc-4ffe-be15-d03219b3b25b","Type":"ContainerDied","Data":"cb376944400b85358eb91951cc698aa63ba76455bb67c197f937eec674cf802c"} Nov 03 20:39:41.583049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.583014 5059 scope.go:122] "RemoveContainer" containerID="fd322771dcdddadbf22cfe8ba57e5548a7171a87db9d178ad5e8b09c84395e9b" Nov 03 20:39:41.583189 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.583159 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw" Nov 03 20:39:41.600133 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.600109 5059 scope.go:122] "RemoveContainer" containerID="fd322771dcdddadbf22cfe8ba57e5548a7171a87db9d178ad5e8b09c84395e9b" Nov 03 20:39:41.600449 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:39:41.600423 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd322771dcdddadbf22cfe8ba57e5548a7171a87db9d178ad5e8b09c84395e9b\": container with ID starting with fd322771dcdddadbf22cfe8ba57e5548a7171a87db9d178ad5e8b09c84395e9b not found: ID does not exist" containerID="fd322771dcdddadbf22cfe8ba57e5548a7171a87db9d178ad5e8b09c84395e9b" Nov 03 20:39:41.600557 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.600458 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd322771dcdddadbf22cfe8ba57e5548a7171a87db9d178ad5e8b09c84395e9b"} err="failed to get container status \"fd322771dcdddadbf22cfe8ba57e5548a7171a87db9d178ad5e8b09c84395e9b\": rpc error: code = NotFound desc = could not find container \"fd322771dcdddadbf22cfe8ba57e5548a7171a87db9d178ad5e8b09c84395e9b\": container with ID starting with fd322771dcdddadbf22cfe8ba57e5548a7171a87db9d178ad5e8b09c84395e9b not found: ID does not exist" Nov 03 20:39:41.616559 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.616533 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw"] Nov 03 20:39:41.621084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.621062 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cmxzw"] Nov 03 20:39:41.856355 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:41.856319 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e997605a-f2cc-4ffe-be15-d03219b3b25b" path="/var/lib/kubelet/pods/e997605a-f2cc-4ffe-be15-d03219b3b25b/volumes" Nov 03 20:39:44.856647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:44.856609 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5"] Nov 03 20:39:44.857108 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:44.856848 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" podUID="41778c8b-1334-4a0a-afbc-2d377d514e8d" containerName="istio-proxy" containerID="cri-o://6c9c6f5fc6b9469ca4ba0efeed4e557ba33c9e08b55f4b2d7e630a958c2f8963" gracePeriod=30 Nov 03 20:39:47.151081 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:47.151044 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252"] Nov 03 20:39:47.151500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:47.151249 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" podUID="9bff4885-a4dc-4b17-b596-e745e86fbccd" containerName="istio-proxy" containerID="cri-o://f98ae9867fc87a679d4159cab410d0f2e75a2ede6c5a0c454bba5459fd1c6a54" gracePeriod=30 Nov 03 20:39:47.486086 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:47.486008 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44e73305-xdqsv"] Nov 03 20:39:47.486509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:47.486492 5059 cpu_manager.go:414] "containerMap: removing container" podUID="e997605a-f2cc-4ffe-be15-d03219b3b25b" containerName="istio-proxy" Nov 03 20:39:47.486572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:47.486511 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e997605a-f2cc-4ffe-be15-d03219b3b25b" containerName="istio-proxy" Nov 03 20:39:47.486572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:47.486525 5059 cpu_manager.go:414] "containerMap: removing container" podUID="429244dc-47a6-43af-967f-91baa92f3bca" containerName="revoke-keys" Nov 03 20:39:47.486572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:47.486530 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="429244dc-47a6-43af-967f-91baa92f3bca" containerName="revoke-keys" Nov 03 20:39:47.486679 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:47.486616 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e997605a-f2cc-4ffe-be15-d03219b3b25b" containerName="istio-proxy" Nov 03 20:39:47.486679 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:47.486624 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="429244dc-47a6-43af-967f-91baa92f3bca" containerName="revoke-keys" Nov 03 20:39:47.490963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:47.490936 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e73305-xdqsv" Nov 03 20:39:47.496313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:47.496287 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:39:47.496484 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:47.496307 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:39:47.500426 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:47.500403 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44e73305-xdqsv"] Nov 03 20:39:47.542873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:47.542841 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4624bf15-7e01-4304-801c-a315f5ab38b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44e73305-xdqsv\" (UID: \"4624bf15-7e01-4304-801c-a315f5ab38b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e73305-xdqsv" Nov 03 20:39:47.543021 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:47.542900 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4624bf15-7e01-4304-801c-a315f5ab38b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44e73305-xdqsv\" (UID: \"4624bf15-7e01-4304-801c-a315f5ab38b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e73305-xdqsv" Nov 03 20:39:47.543087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:47.543028 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcbh8\" (UniqueName: \"kubernetes.io/projected/4624bf15-7e01-4304-801c-a315f5ab38b8-kube-api-access-lcbh8\") pod \"maas-api-revoke-sub-44e73305-xdqsv\" (UID: \"4624bf15-7e01-4304-801c-a315f5ab38b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e73305-xdqsv" Nov 03 20:39:47.644459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:47.644429 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcbh8\" (UniqueName: \"kubernetes.io/projected/4624bf15-7e01-4304-801c-a315f5ab38b8-kube-api-access-lcbh8\") pod \"maas-api-revoke-sub-44e73305-xdqsv\" (UID: \"4624bf15-7e01-4304-801c-a315f5ab38b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e73305-xdqsv" Nov 03 20:39:47.644572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:47.644514 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4624bf15-7e01-4304-801c-a315f5ab38b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44e73305-xdqsv\" (UID: \"4624bf15-7e01-4304-801c-a315f5ab38b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e73305-xdqsv" Nov 03 20:39:47.644572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:47.644561 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4624bf15-7e01-4304-801c-a315f5ab38b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44e73305-xdqsv\" (UID: \"4624bf15-7e01-4304-801c-a315f5ab38b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e73305-xdqsv" Nov 03 20:39:47.645176 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:47.645154 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4624bf15-7e01-4304-801c-a315f5ab38b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44e73305-xdqsv\" (UID: \"4624bf15-7e01-4304-801c-a315f5ab38b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e73305-xdqsv" Nov 03 20:39:47.646971 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:47.646946 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4624bf15-7e01-4304-801c-a315f5ab38b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44e73305-xdqsv\" (UID: \"4624bf15-7e01-4304-801c-a315f5ab38b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e73305-xdqsv" Nov 03 20:39:47.656770 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:47.656745 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcbh8\" (UniqueName: \"kubernetes.io/projected/4624bf15-7e01-4304-801c-a315f5ab38b8-kube-api-access-lcbh8\") pod \"maas-api-revoke-sub-44e73305-xdqsv\" (UID: \"4624bf15-7e01-4304-801c-a315f5ab38b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e73305-xdqsv" Nov 03 20:39:47.677559 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:47.677534 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m"] Nov 03 20:39:47.806755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:47.806656 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e73305-xdqsv" Nov 03 20:39:48.088692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:48.088651 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44e73305-xdqsv"] Nov 03 20:39:48.091011 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:39:48.090966 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4624bf15_7e01_4304_801c_a315f5ab38b8.slice/crio-3d5cba1e3f9e771c27c5e1ff794c01af57decbd9a16aaacccb90fb68cfc634be WatchSource:0}: Error finding container 3d5cba1e3f9e771c27c5e1ff794c01af57decbd9a16aaacccb90fb68cfc634be: Status 404 returned error can't find the container with id 3d5cba1e3f9e771c27c5e1ff794c01af57decbd9a16aaacccb90fb68cfc634be Nov 03 20:39:48.629506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:48.629418 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e73305-xdqsv" event={"ID":"4624bf15-7e01-4304-801c-a315f5ab38b8","Type":"ContainerStarted","Data":"759eaa50200c464b2b367e9ff03ae386e5484c3102d75acd506810534e98bfb0"} Nov 03 20:39:48.629506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:48.629479 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e73305-xdqsv" event={"ID":"4624bf15-7e01-4304-801c-a315f5ab38b8","Type":"ContainerStarted","Data":"3d5cba1e3f9e771c27c5e1ff794c01af57decbd9a16aaacccb90fb68cfc634be"} Nov 03 20:39:48.648174 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:48.648130 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e73305-xdqsv" podStartSLOduration=1.648115342 podStartE2EDuration="1.648115342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:48.64671132 +0000 UTC m=+3782.904508576" watchObservedRunningTime="2026-11-03 20:39:48.648115342 +0000 UTC m=+3782.905912612" Nov 03 20:39:49.637155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:49.637121 5059 generic.go:356] "Generic (PLEG): container finished" podID="4624bf15-7e01-4304-801c-a315f5ab38b8" containerID="759eaa50200c464b2b367e9ff03ae386e5484c3102d75acd506810534e98bfb0" exitCode=0 Nov 03 20:39:49.637537 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:49.637163 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e73305-xdqsv" event={"ID":"4624bf15-7e01-4304-801c-a315f5ab38b8","Type":"ContainerDied","Data":"759eaa50200c464b2b367e9ff03ae386e5484c3102d75acd506810534e98bfb0"} Nov 03 20:39:50.141952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.141930 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:39:50.171760 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.171577 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-data\") pod \"41778c8b-1334-4a0a-afbc-2d377d514e8d\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " Nov 03 20:39:50.172330 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.171939 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-workload-socket\") pod \"41778c8b-1334-4a0a-afbc-2d377d514e8d\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " Nov 03 20:39:50.172330 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.171963 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-data" pod "41778c8b-1334-4a0a-afbc-2d377d514e8d" (UID: "41778c8b-1334-4a0a-afbc-2d377d514e8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:50.172330 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.171996 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-token\" (UniqueName: \"kubernetes.io/projected/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-token\") pod \"41778c8b-1334-4a0a-afbc-2d377d514e8d\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " Nov 03 20:39:50.172330 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.172072 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-envoy\") pod \"41778c8b-1334-4a0a-afbc-2d377d514e8d\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " Nov 03 20:39:50.172330 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.172122 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41778c8b-1334-4a0a-afbc-2d377d514e8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41778c8b-1334-4a0a-afbc-2d377d514e8d-istiod-ca-cert\") pod \"41778c8b-1334-4a0a-afbc-2d377d514e8d\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " Nov 03 20:39:50.172330 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.172161 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-podinfo\") pod \"41778c8b-1334-4a0a-afbc-2d377d514e8d\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " Nov 03 20:39:50.172330 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.172216 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41778c8b-1334-4a0a-afbc-2d377d514e8d-kube-api-access-4wcqx\" (UniqueName: \"kubernetes.io/projected/41778c8b-1334-4a0a-afbc-2d377d514e8d-kube-api-access-4wcqx\") pod \"41778c8b-1334-4a0a-afbc-2d377d514e8d\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " Nov 03 20:39:50.172330 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.172242 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-workload-socket" pod "41778c8b-1334-4a0a-afbc-2d377d514e8d" (UID: "41778c8b-1334-4a0a-afbc-2d377d514e8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:50.172330 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.172253 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-workload-certs\") pod \"41778c8b-1334-4a0a-afbc-2d377d514e8d\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " Nov 03 20:39:50.172928 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.172350 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-credential-socket\") pod \"41778c8b-1334-4a0a-afbc-2d377d514e8d\" (UID: \"41778c8b-1334-4a0a-afbc-2d377d514e8d\") " Nov 03 20:39:50.172928 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.172815 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:50.172928 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.172840 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:50.173414 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.173336 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41778c8b-1334-4a0a-afbc-2d377d514e8d-istiod-ca-cert" pod "41778c8b-1334-4a0a-afbc-2d377d514e8d" (UID: "41778c8b-1334-4a0a-afbc-2d377d514e8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:50.173845 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.173800 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-workload-certs" pod "41778c8b-1334-4a0a-afbc-2d377d514e8d" (UID: "41778c8b-1334-4a0a-afbc-2d377d514e8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:50.176478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.176450 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-token" pod "41778c8b-1334-4a0a-afbc-2d377d514e8d" (UID: "41778c8b-1334-4a0a-afbc-2d377d514e8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:50.176686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.176636 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-envoy" pod "41778c8b-1334-4a0a-afbc-2d377d514e8d" (UID: "41778c8b-1334-4a0a-afbc-2d377d514e8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:50.176976 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.176936 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-podinfo" pod "41778c8b-1334-4a0a-afbc-2d377d514e8d" (UID: "41778c8b-1334-4a0a-afbc-2d377d514e8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:39:50.177089 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.177062 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41778c8b-1334-4a0a-afbc-2d377d514e8d-kube-api-access-4wcqx" pod "41778c8b-1334-4a0a-afbc-2d377d514e8d" (UID: "41778c8b-1334-4a0a-afbc-2d377d514e8d"). InnerVolumeSpecName "kube-api-access-4wcqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:50.192840 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.175813 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-credential-socket" pod "41778c8b-1334-4a0a-afbc-2d377d514e8d" (UID: "41778c8b-1334-4a0a-afbc-2d377d514e8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:50.274593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.274516 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wcqx\" (UniqueName: \"kubernetes.io/projected/41778c8b-1334-4a0a-afbc-2d377d514e8d-kube-api-access-4wcqx\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:50.274593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.274554 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:50.274593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.274570 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:50.274593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.274584 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:50.274892 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.274598 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:50.274892 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.274611 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41778c8b-1334-4a0a-afbc-2d377d514e8d-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:50.274892 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.274623 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41778c8b-1334-4a0a-afbc-2d377d514e8d-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:50.644398 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.644362 5059 generic.go:356] "Generic (PLEG): container finished" podID="41778c8b-1334-4a0a-afbc-2d377d514e8d" containerID="6c9c6f5fc6b9469ca4ba0efeed4e557ba33c9e08b55f4b2d7e630a958c2f8963" exitCode=0 Nov 03 20:39:50.644917 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.644413 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" Nov 03 20:39:50.644917 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.644455 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" event={"ID":"41778c8b-1334-4a0a-afbc-2d377d514e8d","Type":"ContainerDied","Data":"6c9c6f5fc6b9469ca4ba0efeed4e557ba33c9e08b55f4b2d7e630a958c2f8963"} Nov 03 20:39:50.644917 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.644528 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5" event={"ID":"41778c8b-1334-4a0a-afbc-2d377d514e8d","Type":"ContainerDied","Data":"14ceb520f6d4105401a7b46c0723e09fd8ad248fcbc4030ad7aa7c5f0ef82185"} Nov 03 20:39:50.644917 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.644562 5059 scope.go:122] "RemoveContainer" containerID="6c9c6f5fc6b9469ca4ba0efeed4e557ba33c9e08b55f4b2d7e630a958c2f8963" Nov 03 20:39:50.655932 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.655909 5059 scope.go:122] "RemoveContainer" containerID="6c9c6f5fc6b9469ca4ba0efeed4e557ba33c9e08b55f4b2d7e630a958c2f8963" Nov 03 20:39:50.656225 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:39:50.656208 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c9c6f5fc6b9469ca4ba0efeed4e557ba33c9e08b55f4b2d7e630a958c2f8963\": container with ID starting with 6c9c6f5fc6b9469ca4ba0efeed4e557ba33c9e08b55f4b2d7e630a958c2f8963 not found: ID does not exist" containerID="6c9c6f5fc6b9469ca4ba0efeed4e557ba33c9e08b55f4b2d7e630a958c2f8963" Nov 03 20:39:50.656279 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.656234 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9c6f5fc6b9469ca4ba0efeed4e557ba33c9e08b55f4b2d7e630a958c2f8963"} err="failed to get container status \"6c9c6f5fc6b9469ca4ba0efeed4e557ba33c9e08b55f4b2d7e630a958c2f8963\": rpc error: code = NotFound desc = could not find container \"6c9c6f5fc6b9469ca4ba0efeed4e557ba33c9e08b55f4b2d7e630a958c2f8963\": container with ID starting with 6c9c6f5fc6b9469ca4ba0efeed4e557ba33c9e08b55f4b2d7e630a958c2f8963 not found: ID does not exist" Nov 03 20:39:50.671569 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.671475 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5"] Nov 03 20:39:50.677005 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.676288 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-qprm5"] Nov 03 20:39:50.781409 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.781383 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e73305-xdqsv" Nov 03 20:39:50.880293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.880254 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4624bf15-7e01-4304-801c-a315f5ab38b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4624bf15-7e01-4304-801c-a315f5ab38b8-maas-api-xks-ca\") pod \"4624bf15-7e01-4304-801c-a315f5ab38b8\" (UID: \"4624bf15-7e01-4304-801c-a315f5ab38b8\") " Nov 03 20:39:50.880882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.880857 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4624bf15-7e01-4304-801c-a315f5ab38b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4624bf15-7e01-4304-801c-a315f5ab38b8-maas-api-openshift-ca\") pod \"4624bf15-7e01-4304-801c-a315f5ab38b8\" (UID: \"4624bf15-7e01-4304-801c-a315f5ab38b8\") " Nov 03 20:39:50.881024 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.880932 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4624bf15-7e01-4304-801c-a315f5ab38b8-kube-api-access-lcbh8\" (UniqueName: \"kubernetes.io/projected/4624bf15-7e01-4304-801c-a315f5ab38b8-kube-api-access-lcbh8\") pod \"4624bf15-7e01-4304-801c-a315f5ab38b8\" (UID: \"4624bf15-7e01-4304-801c-a315f5ab38b8\") " Nov 03 20:39:50.881884 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.881848 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4624bf15-7e01-4304-801c-a315f5ab38b8-maas-api-openshift-ca" pod "4624bf15-7e01-4304-801c-a315f5ab38b8" (UID: "4624bf15-7e01-4304-801c-a315f5ab38b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:50.883052 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.883029 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4624bf15-7e01-4304-801c-a315f5ab38b8-maas-api-xks-ca" pod "4624bf15-7e01-4304-801c-a315f5ab38b8" (UID: "4624bf15-7e01-4304-801c-a315f5ab38b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:39:50.883806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.883780 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4624bf15-7e01-4304-801c-a315f5ab38b8-kube-api-access-lcbh8" pod "4624bf15-7e01-4304-801c-a315f5ab38b8" (UID: "4624bf15-7e01-4304-801c-a315f5ab38b8"). InnerVolumeSpecName "kube-api-access-lcbh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:50.982249 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.982221 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4624bf15-7e01-4304-801c-a315f5ab38b8-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:50.982249 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.982247 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4624bf15-7e01-4304-801c-a315f5ab38b8-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:50.982419 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:50.982260 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcbh8\" (UniqueName: \"kubernetes.io/projected/4624bf15-7e01-4304-801c-a315f5ab38b8-kube-api-access-lcbh8\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:51.651032 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:51.651001 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e73305-xdqsv" Nov 03 20:39:51.651032 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:51.651021 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e73305-xdqsv" event={"ID":"4624bf15-7e01-4304-801c-a315f5ab38b8","Type":"ContainerDied","Data":"3d5cba1e3f9e771c27c5e1ff794c01af57decbd9a16aaacccb90fb68cfc634be"} Nov 03 20:39:51.651583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:51.651074 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d5cba1e3f9e771c27c5e1ff794c01af57decbd9a16aaacccb90fb68cfc634be" Nov 03 20:39:51.855745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:51.855705 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41778c8b-1334-4a0a-afbc-2d377d514e8d" path="/var/lib/kubelet/pods/41778c8b-1334-4a0a-afbc-2d377d514e8d/volumes" Nov 03 20:39:52.417367 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.417306 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:39:52.497372 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.497341 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-data\") pod \"9bff4885-a4dc-4b17-b596-e745e86fbccd\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " Nov 03 20:39:52.497508 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.497404 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-workload-socket\") pod \"9bff4885-a4dc-4b17-b596-e745e86fbccd\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " Nov 03 20:39:52.497508 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.497427 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-credential-socket\") pod \"9bff4885-a4dc-4b17-b596-e745e86fbccd\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " Nov 03 20:39:52.497508 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.497456 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bff4885-a4dc-4b17-b596-e745e86fbccd-kube-api-access-vjh84\" (UniqueName: \"kubernetes.io/projected/9bff4885-a4dc-4b17-b596-e745e86fbccd-kube-api-access-vjh84\") pod \"9bff4885-a4dc-4b17-b596-e745e86fbccd\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " Nov 03 20:39:52.497508 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.497491 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bff4885-a4dc-4b17-b596-e745e86fbccd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bff4885-a4dc-4b17-b596-e745e86fbccd-istiod-ca-cert\") pod \"9bff4885-a4dc-4b17-b596-e745e86fbccd\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " Nov 03 20:39:52.497786 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.497572 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-workload-certs\") pod \"9bff4885-a4dc-4b17-b596-e745e86fbccd\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " Nov 03 20:39:52.497786 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.497619 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-podinfo\") pod \"9bff4885-a4dc-4b17-b596-e745e86fbccd\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " Nov 03 20:39:52.497786 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.497648 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-token\" (UniqueName: \"kubernetes.io/projected/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-token\") pod \"9bff4885-a4dc-4b17-b596-e745e86fbccd\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " Nov 03 20:39:52.497786 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.497700 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-data" pod "9bff4885-a4dc-4b17-b596-e745e86fbccd" (UID: "9bff4885-a4dc-4b17-b596-e745e86fbccd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:52.497786 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.497736 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-envoy\") pod \"9bff4885-a4dc-4b17-b596-e745e86fbccd\" (UID: \"9bff4885-a4dc-4b17-b596-e745e86fbccd\") " Nov 03 20:39:52.498057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.497756 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-workload-socket" pod "9bff4885-a4dc-4b17-b596-e745e86fbccd" (UID: "9bff4885-a4dc-4b17-b596-e745e86fbccd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:52.498057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.497823 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-credential-socket" pod "9bff4885-a4dc-4b17-b596-e745e86fbccd" (UID: "9bff4885-a4dc-4b17-b596-e745e86fbccd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:52.498474 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.497934 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-workload-certs" pod "9bff4885-a4dc-4b17-b596-e745e86fbccd" (UID: "9bff4885-a4dc-4b17-b596-e745e86fbccd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:52.498619 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.498128 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bff4885-a4dc-4b17-b596-e745e86fbccd-istiod-ca-cert" pod "9bff4885-a4dc-4b17-b596-e745e86fbccd" (UID: "9bff4885-a4dc-4b17-b596-e745e86fbccd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:52.498619 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.498137 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:52.498619 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.498514 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:52.498619 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.498534 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:52.500107 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.500086 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-token" pod "9bff4885-a4dc-4b17-b596-e745e86fbccd" (UID: "9bff4885-a4dc-4b17-b596-e745e86fbccd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:52.500273 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.500252 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-envoy" pod "9bff4885-a4dc-4b17-b596-e745e86fbccd" (UID: "9bff4885-a4dc-4b17-b596-e745e86fbccd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:39:52.500416 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.500393 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-podinfo" pod "9bff4885-a4dc-4b17-b596-e745e86fbccd" (UID: "9bff4885-a4dc-4b17-b596-e745e86fbccd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:39:52.500559 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.500542 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bff4885-a4dc-4b17-b596-e745e86fbccd-kube-api-access-vjh84" pod "9bff4885-a4dc-4b17-b596-e745e86fbccd" (UID: "9bff4885-a4dc-4b17-b596-e745e86fbccd"). InnerVolumeSpecName "kube-api-access-vjh84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:52.599269 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.599242 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjh84\" (UniqueName: \"kubernetes.io/projected/9bff4885-a4dc-4b17-b596-e745e86fbccd-kube-api-access-vjh84\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:52.599269 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.599268 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bff4885-a4dc-4b17-b596-e745e86fbccd-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:52.599413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.599279 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:52.599413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.599289 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:52.599413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.599298 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:52.599413 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.599306 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bff4885-a4dc-4b17-b596-e745e86fbccd-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:52.659281 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.659255 5059 generic.go:356] "Generic (PLEG): container finished" podID="9bff4885-a4dc-4b17-b596-e745e86fbccd" containerID="f98ae9867fc87a679d4159cab410d0f2e75a2ede6c5a0c454bba5459fd1c6a54" exitCode=0 Nov 03 20:39:52.659642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.659295 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" Nov 03 20:39:52.659642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.659331 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" event={"ID":"9bff4885-a4dc-4b17-b596-e745e86fbccd","Type":"ContainerDied","Data":"f98ae9867fc87a679d4159cab410d0f2e75a2ede6c5a0c454bba5459fd1c6a54"} Nov 03 20:39:52.659642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.659387 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252" event={"ID":"9bff4885-a4dc-4b17-b596-e745e86fbccd","Type":"ContainerDied","Data":"b37f042b2b7ad71eb9819f0dbd537f0d279768df5494387b93bed653d485896b"} Nov 03 20:39:52.659642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.659412 5059 scope.go:122] "RemoveContainer" containerID="f98ae9867fc87a679d4159cab410d0f2e75a2ede6c5a0c454bba5459fd1c6a54" Nov 03 20:39:52.669738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.669716 5059 scope.go:122] "RemoveContainer" containerID="f98ae9867fc87a679d4159cab410d0f2e75a2ede6c5a0c454bba5459fd1c6a54" Nov 03 20:39:52.670011 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:39:52.669991 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f98ae9867fc87a679d4159cab410d0f2e75a2ede6c5a0c454bba5459fd1c6a54\": container with ID starting with f98ae9867fc87a679d4159cab410d0f2e75a2ede6c5a0c454bba5459fd1c6a54 not found: ID does not exist" containerID="f98ae9867fc87a679d4159cab410d0f2e75a2ede6c5a0c454bba5459fd1c6a54" Nov 03 20:39:52.670059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.670019 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98ae9867fc87a679d4159cab410d0f2e75a2ede6c5a0c454bba5459fd1c6a54"} err="failed to get container status \"f98ae9867fc87a679d4159cab410d0f2e75a2ede6c5a0c454bba5459fd1c6a54\": rpc error: code = NotFound desc = could not find container \"f98ae9867fc87a679d4159cab410d0f2e75a2ede6c5a0c454bba5459fd1c6a54\": container with ID starting with f98ae9867fc87a679d4159cab410d0f2e75a2ede6c5a0c454bba5459fd1c6a54 not found: ID does not exist" Nov 03 20:39:52.677923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.677896 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m" podUID="de4efb53-8b59-49f1-894f-ef366f00732a" containerName="payload-processing" containerID="cri-o://77b67d21274a3fb1e78412c9c2352788e224a8c423aaf976529b2a194a3814de" gracePeriod=25 Nov 03 20:39:52.685624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.685586 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252"] Nov 03 20:39:52.697076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.697049 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-5p252"] Nov 03 20:39:52.927575 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:52.927549 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m" Nov 03 20:39:53.003904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:53.003871 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de4efb53-8b59-49f1-894f-ef366f00732a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de4efb53-8b59-49f1-894f-ef366f00732a-plugins-config-volume\") pod \"de4efb53-8b59-49f1-894f-ef366f00732a\" (UID: \"de4efb53-8b59-49f1-894f-ef366f00732a\") " Nov 03 20:39:53.004039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:53.003942 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de4efb53-8b59-49f1-894f-ef366f00732a-kube-api-access-cpvhm\" (UniqueName: \"kubernetes.io/projected/de4efb53-8b59-49f1-894f-ef366f00732a-kube-api-access-cpvhm\") pod \"de4efb53-8b59-49f1-894f-ef366f00732a\" (UID: \"de4efb53-8b59-49f1-894f-ef366f00732a\") " Nov 03 20:39:53.004230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:53.004209 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de4efb53-8b59-49f1-894f-ef366f00732a-plugins-config-volume" pod "de4efb53-8b59-49f1-894f-ef366f00732a" (UID: "de4efb53-8b59-49f1-894f-ef366f00732a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:53.006026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:53.006006 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4efb53-8b59-49f1-894f-ef366f00732a-kube-api-access-cpvhm" pod "de4efb53-8b59-49f1-894f-ef366f00732a" (UID: "de4efb53-8b59-49f1-894f-ef366f00732a"). InnerVolumeSpecName "kube-api-access-cpvhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:53.105714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:53.105656 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de4efb53-8b59-49f1-894f-ef366f00732a-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:53.105714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:53.105710 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpvhm\" (UniqueName: \"kubernetes.io/projected/de4efb53-8b59-49f1-894f-ef366f00732a-kube-api-access-cpvhm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:53.668437 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:53.668394 5059 generic.go:356] "Generic (PLEG): container finished" podID="de4efb53-8b59-49f1-894f-ef366f00732a" containerID="77b67d21274a3fb1e78412c9c2352788e224a8c423aaf976529b2a194a3814de" exitCode=0 Nov 03 20:39:53.668437 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:53.668431 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m" Nov 03 20:39:53.669008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:53.668474 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m" event={"ID":"de4efb53-8b59-49f1-894f-ef366f00732a","Type":"ContainerDied","Data":"77b67d21274a3fb1e78412c9c2352788e224a8c423aaf976529b2a194a3814de"} Nov 03 20:39:53.669008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:53.668543 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m" event={"ID":"de4efb53-8b59-49f1-894f-ef366f00732a","Type":"ContainerDied","Data":"10af7362d56af3e4af6be86d228fac82334d40308666b2e0622589438ccc3e50"} Nov 03 20:39:53.669008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:53.668576 5059 scope.go:122] "RemoveContainer" containerID="77b67d21274a3fb1e78412c9c2352788e224a8c423aaf976529b2a194a3814de" Nov 03 20:39:53.678844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:53.678826 5059 scope.go:122] "RemoveContainer" containerID="77b67d21274a3fb1e78412c9c2352788e224a8c423aaf976529b2a194a3814de" Nov 03 20:39:53.679093 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:39:53.679072 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77b67d21274a3fb1e78412c9c2352788e224a8c423aaf976529b2a194a3814de\": container with ID starting with 77b67d21274a3fb1e78412c9c2352788e224a8c423aaf976529b2a194a3814de not found: ID does not exist" containerID="77b67d21274a3fb1e78412c9c2352788e224a8c423aaf976529b2a194a3814de" Nov 03 20:39:53.679158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:53.679101 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77b67d21274a3fb1e78412c9c2352788e224a8c423aaf976529b2a194a3814de"} err="failed to get container status \"77b67d21274a3fb1e78412c9c2352788e224a8c423aaf976529b2a194a3814de\": rpc error: code = NotFound desc = could not find container \"77b67d21274a3fb1e78412c9c2352788e224a8c423aaf976529b2a194a3814de\": container with ID starting with 77b67d21274a3fb1e78412c9c2352788e224a8c423aaf976529b2a194a3814de not found: ID does not exist" Nov 03 20:39:53.696005 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:53.695982 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m"] Nov 03 20:39:53.700412 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:53.700391 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-g457m"] Nov 03 20:39:53.856269 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:53.856237 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bff4885-a4dc-4b17-b596-e745e86fbccd" path="/var/lib/kubelet/pods/9bff4885-a4dc-4b17-b596-e745e86fbccd/volumes" Nov 03 20:39:53.856641 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:53.856626 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de4efb53-8b59-49f1-894f-ef366f00732a" path="/var/lib/kubelet/pods/de4efb53-8b59-49f1-894f-ef366f00732a/volumes" Nov 03 20:39:59.909354 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:59.909318 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz"] Nov 03 20:39:59.910003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:59.909547 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" podUID="cca72a7d-a69d-4e05-ac4d-da20e15a0243" containerName="istio-proxy" containerID="cri-o://f7ee1b1fba03037ed075aa90d678d3be16c71ebef4f5221a532f1d7a2e72e552" gracePeriod=30 Nov 03 20:39:59.912808 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:59.912781 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm"] Nov 03 20:39:59.913025 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:39:59.912992 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" podUID="896c119d-c988-4954-8606-72f0de8944cf" containerName="istio-proxy" containerID="cri-o://8eadfd624b310d22690f66e7e83531449a5166ef3ff1887788a0daf43a33c2bb" gracePeriod=30 Nov 03 20:40:00.744541 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.744507 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82e9ecf7-bk8kt"] Nov 03 20:40:00.745076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.745055 5059 cpu_manager.go:414] "containerMap: removing container" podUID="4624bf15-7e01-4304-801c-a315f5ab38b8" containerName="revoke-keys" Nov 03 20:40:00.745202 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.745078 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4624bf15-7e01-4304-801c-a315f5ab38b8" containerName="revoke-keys" Nov 03 20:40:00.745202 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.745099 5059 cpu_manager.go:414] "containerMap: removing container" podUID="41778c8b-1334-4a0a-afbc-2d377d514e8d" containerName="istio-proxy" Nov 03 20:40:00.745202 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.745106 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41778c8b-1334-4a0a-afbc-2d377d514e8d" containerName="istio-proxy" Nov 03 20:40:00.745202 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.745124 5059 cpu_manager.go:414] "containerMap: removing container" podUID="de4efb53-8b59-49f1-894f-ef366f00732a" containerName="payload-processing" Nov 03 20:40:00.745202 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.745134 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de4efb53-8b59-49f1-894f-ef366f00732a" containerName="payload-processing" Nov 03 20:40:00.745202 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.745162 5059 cpu_manager.go:414] "containerMap: removing container" podUID="9bff4885-a4dc-4b17-b596-e745e86fbccd" containerName="istio-proxy" Nov 03 20:40:00.745202 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.745170 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bff4885-a4dc-4b17-b596-e745e86fbccd" containerName="istio-proxy" Nov 03 20:40:00.745543 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.745266 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bff4885-a4dc-4b17-b596-e745e86fbccd" containerName="istio-proxy" Nov 03 20:40:00.745543 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.745286 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4624bf15-7e01-4304-801c-a315f5ab38b8" containerName="revoke-keys" Nov 03 20:40:00.745543 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.745297 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="41778c8b-1334-4a0a-afbc-2d377d514e8d" containerName="istio-proxy" Nov 03 20:40:00.745543 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.745307 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="de4efb53-8b59-49f1-894f-ef366f00732a" containerName="payload-processing" Nov 03 20:40:00.750034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.750010 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e9ecf7-bk8kt" Nov 03 20:40:00.757105 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.757076 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:40:00.757205 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.757136 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:40:00.763713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.763689 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82e9ecf7-bk8kt"] Nov 03 20:40:00.889360 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.889329 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b330e09-cf3a-4278-bc82-5386105badd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82e9ecf7-bk8kt\" (UID: \"3b330e09-cf3a-4278-bc82-5386105badd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e9ecf7-bk8kt" Nov 03 20:40:00.889498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.889382 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twckp\" (UniqueName: \"kubernetes.io/projected/3b330e09-cf3a-4278-bc82-5386105badd1-kube-api-access-twckp\") pod \"maas-api-revoke-sub-82e9ecf7-bk8kt\" (UID: \"3b330e09-cf3a-4278-bc82-5386105badd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e9ecf7-bk8kt" Nov 03 20:40:00.889498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.889451 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b330e09-cf3a-4278-bc82-5386105badd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82e9ecf7-bk8kt\" (UID: \"3b330e09-cf3a-4278-bc82-5386105badd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e9ecf7-bk8kt" Nov 03 20:40:00.990820 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.990790 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b330e09-cf3a-4278-bc82-5386105badd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82e9ecf7-bk8kt\" (UID: \"3b330e09-cf3a-4278-bc82-5386105badd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e9ecf7-bk8kt" Nov 03 20:40:00.991238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.990846 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twckp\" (UniqueName: \"kubernetes.io/projected/3b330e09-cf3a-4278-bc82-5386105badd1-kube-api-access-twckp\") pod \"maas-api-revoke-sub-82e9ecf7-bk8kt\" (UID: \"3b330e09-cf3a-4278-bc82-5386105badd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e9ecf7-bk8kt" Nov 03 20:40:00.991238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.990867 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b330e09-cf3a-4278-bc82-5386105badd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82e9ecf7-bk8kt\" (UID: \"3b330e09-cf3a-4278-bc82-5386105badd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e9ecf7-bk8kt" Nov 03 20:40:00.991535 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.991513 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b330e09-cf3a-4278-bc82-5386105badd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82e9ecf7-bk8kt\" (UID: \"3b330e09-cf3a-4278-bc82-5386105badd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e9ecf7-bk8kt" Nov 03 20:40:00.993215 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:00.993197 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b330e09-cf3a-4278-bc82-5386105badd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82e9ecf7-bk8kt\" (UID: \"3b330e09-cf3a-4278-bc82-5386105badd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e9ecf7-bk8kt" Nov 03 20:40:01.002929 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:01.002880 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twckp\" (UniqueName: \"kubernetes.io/projected/3b330e09-cf3a-4278-bc82-5386105badd1-kube-api-access-twckp\") pod \"maas-api-revoke-sub-82e9ecf7-bk8kt\" (UID: \"3b330e09-cf3a-4278-bc82-5386105badd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e9ecf7-bk8kt" Nov 03 20:40:01.063865 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:01.063842 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e9ecf7-bk8kt" Nov 03 20:40:01.343334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:01.343297 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82e9ecf7-bk8kt"] Nov 03 20:40:01.344129 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:40:01.344102 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b330e09_cf3a_4278_bc82_5386105badd1.slice/crio-4e2a7b58779a6ecf0e5bd6d0f33c61c5fc31fe17a7a3231c7587842cfe8a1ca1 WatchSource:0}: Error finding container 4e2a7b58779a6ecf0e5bd6d0f33c61c5fc31fe17a7a3231c7587842cfe8a1ca1: Status 404 returned error can't find the container with id 4e2a7b58779a6ecf0e5bd6d0f33c61c5fc31fe17a7a3231c7587842cfe8a1ca1 Nov 03 20:40:01.721539 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:01.721503 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e9ecf7-bk8kt" event={"ID":"3b330e09-cf3a-4278-bc82-5386105badd1","Type":"ContainerStarted","Data":"73901e37b4ea8f0d28c4915f0e106f668be2fea2c0e7ad02397fa7f5ca74a3a0"} Nov 03 20:40:01.721716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:01.721562 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e9ecf7-bk8kt" event={"ID":"3b330e09-cf3a-4278-bc82-5386105badd1","Type":"ContainerStarted","Data":"4e2a7b58779a6ecf0e5bd6d0f33c61c5fc31fe17a7a3231c7587842cfe8a1ca1"} Nov 03 20:40:01.742166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:01.742115 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e9ecf7-bk8kt" podStartSLOduration=1.7421019260000001 podStartE2EDuration="1.742101926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:01.738433639 +0000 UTC m=+3795.996230896" watchObservedRunningTime="2026-11-03 20:40:01.742101926 +0000 UTC m=+3795.999899227" Nov 03 20:40:02.729478 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:02.729440 5059 generic.go:356] "Generic (PLEG): container finished" podID="3b330e09-cf3a-4278-bc82-5386105badd1" containerID="73901e37b4ea8f0d28c4915f0e106f668be2fea2c0e7ad02397fa7f5ca74a3a0" exitCode=0 Nov 03 20:40:02.729892 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:02.729526 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e9ecf7-bk8kt" event={"ID":"3b330e09-cf3a-4278-bc82-5386105badd1","Type":"ContainerDied","Data":"73901e37b4ea8f0d28c4915f0e106f668be2fea2c0e7ad02397fa7f5ca74a3a0"} Nov 03 20:40:03.482504 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.482446 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s"] Nov 03 20:40:03.483455 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.483418 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" containerID="cri-o://b3821841d889962619728e3cfd3742dffe6108759aefe0be61ddae8c78d55106" gracePeriod=30 Nov 03 20:40:03.683777 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.683740 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd"] Nov 03 20:40:03.687490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.687472 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.709512 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.709479 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd"] Nov 03 20:40:03.709701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.709555 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm"] Nov 03 20:40:03.713828 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.713803 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.785134 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.785099 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm"] Nov 03 20:40:03.823400 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.823358 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrqtf\" (UniqueName: \"kubernetes.io/projected/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-kube-api-access-lrqtf\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.823400 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.823406 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.823593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.823427 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.823593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.823526 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.823593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.823552 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt8lx\" (UniqueName: \"kubernetes.io/projected/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-kube-api-access-mt8lx\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.823759 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.823600 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.823759 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.823625 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.823759 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.823641 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.823759 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.823696 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.823894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.823843 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.823928 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.823898 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.823959 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.823948 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.824011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.823996 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.824046 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.824020 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.824082 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.824044 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.824082 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.824070 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.824147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.824093 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.824147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.824124 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.925121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.925089 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.925238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.925139 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.925238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.925164 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.925238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.925198 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.925238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.925231 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.925463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.925280 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.925463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.925313 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.925463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.925355 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.925463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.925385 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.925463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.925413 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.925463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.925453 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.925789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.925488 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.925789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.925514 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.925789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.925526 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.925789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.925578 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.925789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.925582 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrqtf\" (UniqueName: \"kubernetes.io/projected/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-kube-api-access-lrqtf\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.925789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.925615 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.925789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.925643 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.926183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.925981 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.926183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.925992 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.926183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.926049 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.926183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.926079 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.926183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.926086 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt8lx\" (UniqueName: \"kubernetes.io/projected/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-kube-api-access-mt8lx\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.926450 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.926271 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.926450 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.926272 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.926559 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.926482 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.926624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.926602 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.927036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.927010 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.928060 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.928040 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.928351 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.928319 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.928786 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.928764 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.928942 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.928919 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.931239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.931220 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e9ecf7-bk8kt" Nov 03 20:40:03.937304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.937284 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.937595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.937565 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrqtf\" (UniqueName: \"kubernetes.io/projected/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-kube-api-access-lrqtf\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:03.938170 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.938148 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt8lx\" (UniqueName: \"kubernetes.io/projected/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-kube-api-access-mt8lx\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:03.938283 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:03.938194 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:04.002489 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.002464 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:04.026762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.026738 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b330e09-cf3a-4278-bc82-5386105badd1-kube-api-access-twckp\" (UniqueName: \"kubernetes.io/projected/3b330e09-cf3a-4278-bc82-5386105badd1-kube-api-access-twckp\") pod \"3b330e09-cf3a-4278-bc82-5386105badd1\" (UID: \"3b330e09-cf3a-4278-bc82-5386105badd1\") " Nov 03 20:40:04.026876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.026823 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b330e09-cf3a-4278-bc82-5386105badd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b330e09-cf3a-4278-bc82-5386105badd1-maas-api-xks-ca\") pod \"3b330e09-cf3a-4278-bc82-5386105badd1\" (UID: \"3b330e09-cf3a-4278-bc82-5386105badd1\") " Nov 03 20:40:04.026876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.026852 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b330e09-cf3a-4278-bc82-5386105badd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b330e09-cf3a-4278-bc82-5386105badd1-maas-api-openshift-ca\") pod \"3b330e09-cf3a-4278-bc82-5386105badd1\" (UID: \"3b330e09-cf3a-4278-bc82-5386105badd1\") " Nov 03 20:40:04.027266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.027240 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b330e09-cf3a-4278-bc82-5386105badd1-maas-api-openshift-ca" pod "3b330e09-cf3a-4278-bc82-5386105badd1" (UID: "3b330e09-cf3a-4278-bc82-5386105badd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:04.028819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.028793 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b330e09-cf3a-4278-bc82-5386105badd1-kube-api-access-twckp" pod "3b330e09-cf3a-4278-bc82-5386105badd1" (UID: "3b330e09-cf3a-4278-bc82-5386105badd1"). InnerVolumeSpecName "kube-api-access-twckp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:04.028909 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.028866 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b330e09-cf3a-4278-bc82-5386105badd1-maas-api-xks-ca" pod "3b330e09-cf3a-4278-bc82-5386105badd1" (UID: "3b330e09-cf3a-4278-bc82-5386105badd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:04.085221 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.085188 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:04.134220 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.128312 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b330e09-cf3a-4278-bc82-5386105badd1-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:04.134220 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.134053 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twckp\" (UniqueName: \"kubernetes.io/projected/3b330e09-cf3a-4278-bc82-5386105badd1-kube-api-access-twckp\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:04.134220 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.134078 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b330e09-cf3a-4278-bc82-5386105badd1-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:04.285049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.285016 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd"] Nov 03 20:40:04.285382 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:40:04.285362 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076156cc_fc7a_4a9e_bca0_7bee7bc7b065.slice/crio-14458aee810e410292116b9ba9f156232f2d2ffdacfbdc8b17ec4c26bdd61d2c WatchSource:0}: Error finding container 14458aee810e410292116b9ba9f156232f2d2ffdacfbdc8b17ec4c26bdd61d2c: Status 404 returned error can't find the container with id 14458aee810e410292116b9ba9f156232f2d2ffdacfbdc8b17ec4c26bdd61d2c Nov 03 20:40:04.287684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.287633 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:40:04.287776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.287719 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:40:04.287776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.287750 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:40:04.380772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.380741 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm"] Nov 03 20:40:04.381247 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:40:04.381221 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92cecfa5_c5dc_4c44_9a7e_c0a521a2ff80.slice/crio-c181f1616bb52cf2e444bc18ce32f514106fa23a552cad2c4bf8fa9160b5fed0 WatchSource:0}: Error finding container c181f1616bb52cf2e444bc18ce32f514106fa23a552cad2c4bf8fa9160b5fed0: Status 404 returned error can't find the container with id c181f1616bb52cf2e444bc18ce32f514106fa23a552cad2c4bf8fa9160b5fed0 Nov 03 20:40:04.384373 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.384336 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:40:04.384456 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.384427 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:40:04.384517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.384492 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:40:04.788648 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.788601 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" event={"ID":"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80","Type":"ContainerStarted","Data":"717313c3216c7a6ab47bc1c7f439ba1789a4c63aaf1014eec9612da6b76faf0e"} Nov 03 20:40:04.789158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.788691 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" event={"ID":"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80","Type":"ContainerStarted","Data":"c181f1616bb52cf2e444bc18ce32f514106fa23a552cad2c4bf8fa9160b5fed0"} Nov 03 20:40:04.790016 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.789985 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e9ecf7-bk8kt" Nov 03 20:40:04.790163 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.790015 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e9ecf7-bk8kt" event={"ID":"3b330e09-cf3a-4278-bc82-5386105badd1","Type":"ContainerDied","Data":"4e2a7b58779a6ecf0e5bd6d0f33c61c5fc31fe17a7a3231c7587842cfe8a1ca1"} Nov 03 20:40:04.790163 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.790062 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e2a7b58779a6ecf0e5bd6d0f33c61c5fc31fe17a7a3231c7587842cfe8a1ca1" Nov 03 20:40:04.791852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.791817 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" event={"ID":"076156cc-fc7a-4a9e-bca0-7bee7bc7b065","Type":"ContainerStarted","Data":"e6f40dbc1e4393730e4807046b3eb1684e1ea3495a4793427c27e44c8ba4ee95"} Nov 03 20:40:04.791986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.791864 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" event={"ID":"076156cc-fc7a-4a9e-bca0-7bee7bc7b065","Type":"ContainerStarted","Data":"14458aee810e410292116b9ba9f156232f2d2ffdacfbdc8b17ec4c26bdd61d2c"} Nov 03 20:40:04.812090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.812037 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" podStartSLOduration=1.812021007 podStartE2EDuration="1.812021007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:04.811389302 +0000 UTC m=+3799.069186561" watchObservedRunningTime="2026-11-03 20:40:04.812021007 +0000 UTC m=+3799.069818265" Nov 03 20:40:04.842840 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:04.842795 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" podStartSLOduration=1.842781921 podStartE2EDuration="1.842781921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:04.839562231 +0000 UTC m=+3799.097359491" watchObservedRunningTime="2026-11-03 20:40:04.842781921 +0000 UTC m=+3799.100579178" Nov 03 20:40:05.013706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.005412 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:05.013706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.009722 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" start-of-body= Nov 03 20:40:05.013706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.009798 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" podUID="076156cc-fc7a-4a9e-bca0-7bee7bc7b065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" Nov 03 20:40:05.054363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.054284 5059 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:40:05.054363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.054355 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:40:05.088709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.086564 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:05.088917 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.088887 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.173:15021/healthz/ready\": dial tcp 10.132.1.173:15021: connect: connection refused" start-of-body= Nov 03 20:40:05.089007 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.088945 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" podUID="92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.173:15021/healthz/ready\": dial tcp 10.132.1.173:15021: connect: connection refused" Nov 03 20:40:05.162143 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.162113 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:40:05.197412 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.197388 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:40:05.246519 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.246487 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istiod-ca-cert\") pod \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " Nov 03 20:40:05.246691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.246542 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-envoy\") pod \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " Nov 03 20:40:05.246691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.246578 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-podinfo\") pod \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " Nov 03 20:40:05.246691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.246605 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-token\" (UniqueName: \"kubernetes.io/projected/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-token\") pod \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " Nov 03 20:40:05.246691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.246635 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-credential-socket\") pod \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " Nov 03 20:40:05.246691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.246689 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-data\") pod \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " Nov 03 20:40:05.246972 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.246717 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cca72a7d-a69d-4e05-ac4d-da20e15a0243-kube-api-access-r4698\" (UniqueName: \"kubernetes.io/projected/cca72a7d-a69d-4e05-ac4d-da20e15a0243-kube-api-access-r4698\") pod \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " Nov 03 20:40:05.246972 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.246751 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-workload-socket\") pod \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " Nov 03 20:40:05.246972 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.246866 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istiod-ca-cert" pod "cca72a7d-a69d-4e05-ac4d-da20e15a0243" (UID: "cca72a7d-a69d-4e05-ac4d-da20e15a0243"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:05.246972 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.246925 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-workload-certs\") pod \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\" (UID: \"cca72a7d-a69d-4e05-ac4d-da20e15a0243\") " Nov 03 20:40:05.247299 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.247136 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-data" pod "cca72a7d-a69d-4e05-ac4d-da20e15a0243" (UID: "cca72a7d-a69d-4e05-ac4d-da20e15a0243"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:05.247375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.247344 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-workload-certs" pod "cca72a7d-a69d-4e05-ac4d-da20e15a0243" (UID: "cca72a7d-a69d-4e05-ac4d-da20e15a0243"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:05.247480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.247455 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:05.247552 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.247481 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:05.247552 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.247499 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:05.247690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.247552 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-workload-socket" pod "cca72a7d-a69d-4e05-ac4d-da20e15a0243" (UID: "cca72a7d-a69d-4e05-ac4d-da20e15a0243"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:05.247690 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.247550 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-credential-socket" pod "cca72a7d-a69d-4e05-ac4d-da20e15a0243" (UID: "cca72a7d-a69d-4e05-ac4d-da20e15a0243"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:05.249248 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.249222 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-envoy" pod "cca72a7d-a69d-4e05-ac4d-da20e15a0243" (UID: "cca72a7d-a69d-4e05-ac4d-da20e15a0243"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:05.249561 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.249539 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca72a7d-a69d-4e05-ac4d-da20e15a0243-kube-api-access-r4698" pod "cca72a7d-a69d-4e05-ac4d-da20e15a0243" (UID: "cca72a7d-a69d-4e05-ac4d-da20e15a0243"). InnerVolumeSpecName "kube-api-access-r4698". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:05.249684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.249637 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-podinfo" pod "cca72a7d-a69d-4e05-ac4d-da20e15a0243" (UID: "cca72a7d-a69d-4e05-ac4d-da20e15a0243"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:40:05.249876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.249852 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-token" pod "cca72a7d-a69d-4e05-ac4d-da20e15a0243" (UID: "cca72a7d-a69d-4e05-ac4d-da20e15a0243"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:05.348487 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.348459 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/896c119d-c988-4954-8606-72f0de8944cf-kube-api-access-nzn96\" (UniqueName: \"kubernetes.io/projected/896c119d-c988-4954-8606-72f0de8944cf-kube-api-access-nzn96\") pod \"896c119d-c988-4954-8606-72f0de8944cf\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " Nov 03 20:40:05.348624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.348516 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-workload-socket\") pod \"896c119d-c988-4954-8606-72f0de8944cf\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " Nov 03 20:40:05.348624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.348556 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/896c119d-c988-4954-8606-72f0de8944cf-istio-token\" (UniqueName: \"kubernetes.io/projected/896c119d-c988-4954-8606-72f0de8944cf-istio-token\") pod \"896c119d-c988-4954-8606-72f0de8944cf\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " Nov 03 20:40:05.348624 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.348616 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-workload-certs\") pod \"896c119d-c988-4954-8606-72f0de8944cf\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " Nov 03 20:40:05.348834 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.348710 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-istio-data\") pod \"896c119d-c988-4954-8606-72f0de8944cf\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " Nov 03 20:40:05.348834 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.348757 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/896c119d-c988-4954-8606-72f0de8944cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896c119d-c988-4954-8606-72f0de8944cf-istio-podinfo\") pod \"896c119d-c988-4954-8606-72f0de8944cf\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " Nov 03 20:40:05.348834 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.348789 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/896c119d-c988-4954-8606-72f0de8944cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896c119d-c988-4954-8606-72f0de8944cf-istiod-ca-cert\") pod \"896c119d-c988-4954-8606-72f0de8944cf\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " Nov 03 20:40:05.348976 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.348831 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-credential-socket\") pod \"896c119d-c988-4954-8606-72f0de8944cf\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " Nov 03 20:40:05.348976 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.348891 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-workload-socket" pod "896c119d-c988-4954-8606-72f0de8944cf" (UID: "896c119d-c988-4954-8606-72f0de8944cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:05.348976 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.348939 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-istio-envoy\") pod \"896c119d-c988-4954-8606-72f0de8944cf\" (UID: \"896c119d-c988-4954-8606-72f0de8944cf\") " Nov 03 20:40:05.349124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.348970 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-workload-certs" pod "896c119d-c988-4954-8606-72f0de8944cf" (UID: "896c119d-c988-4954-8606-72f0de8944cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:05.349124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.348986 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-istio-data" pod "896c119d-c988-4954-8606-72f0de8944cf" (UID: "896c119d-c988-4954-8606-72f0de8944cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:05.349214 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.349182 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-credential-socket" pod "896c119d-c988-4954-8606-72f0de8944cf" (UID: "896c119d-c988-4954-8606-72f0de8944cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:05.349263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.349226 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/896c119d-c988-4954-8606-72f0de8944cf-istiod-ca-cert" pod "896c119d-c988-4954-8606-72f0de8944cf" (UID: "896c119d-c988-4954-8606-72f0de8944cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:05.349380 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.349361 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:05.349438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.349385 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:05.349438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.349401 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:05.349438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.349416 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:05.349438 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.349430 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:05.349573 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.349442 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cca72a7d-a69d-4e05-ac4d-da20e15a0243-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:05.349573 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.349457 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:05.349573 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.349470 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4698\" (UniqueName: \"kubernetes.io/projected/cca72a7d-a69d-4e05-ac4d-da20e15a0243-kube-api-access-r4698\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:05.349573 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.349487 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896c119d-c988-4954-8606-72f0de8944cf-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:05.349573 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.349501 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca72a7d-a69d-4e05-ac4d-da20e15a0243-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:05.349573 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.349517 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:05.350686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.350647 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/896c119d-c988-4954-8606-72f0de8944cf-kube-api-access-nzn96" pod "896c119d-c988-4954-8606-72f0de8944cf" (UID: "896c119d-c988-4954-8606-72f0de8944cf"). InnerVolumeSpecName "kube-api-access-nzn96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:05.350851 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.350822 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/896c119d-c988-4954-8606-72f0de8944cf-istio-podinfo" pod "896c119d-c988-4954-8606-72f0de8944cf" (UID: "896c119d-c988-4954-8606-72f0de8944cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:40:05.351168 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.351141 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/896c119d-c988-4954-8606-72f0de8944cf-istio-token" pod "896c119d-c988-4954-8606-72f0de8944cf" (UID: "896c119d-c988-4954-8606-72f0de8944cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:05.351284 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.351266 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-istio-envoy" pod "896c119d-c988-4954-8606-72f0de8944cf" (UID: "896c119d-c988-4954-8606-72f0de8944cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:05.450937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.450895 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/896c119d-c988-4954-8606-72f0de8944cf-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:05.450937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.450932 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896c119d-c988-4954-8606-72f0de8944cf-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:05.450937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.450944 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896c119d-c988-4954-8606-72f0de8944cf-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:05.451164 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.450954 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzn96\" (UniqueName: \"kubernetes.io/projected/896c119d-c988-4954-8606-72f0de8944cf-kube-api-access-nzn96\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:05.812744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.812293 5059 generic.go:356] "Generic (PLEG): container finished" podID="cca72a7d-a69d-4e05-ac4d-da20e15a0243" containerID="f7ee1b1fba03037ed075aa90d678d3be16c71ebef4f5221a532f1d7a2e72e552" exitCode=0 Nov 03 20:40:05.812744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.812358 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" Nov 03 20:40:05.812744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.812387 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" event={"ID":"cca72a7d-a69d-4e05-ac4d-da20e15a0243","Type":"ContainerDied","Data":"f7ee1b1fba03037ed075aa90d678d3be16c71ebef4f5221a532f1d7a2e72e552"} Nov 03 20:40:05.812744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.812457 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz" event={"ID":"cca72a7d-a69d-4e05-ac4d-da20e15a0243","Type":"ContainerDied","Data":"f004ee6b7be2108660e25a6868948ec67f63eae85339a25ac11570bd56bd1c93"} Nov 03 20:40:05.812744 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.812491 5059 scope.go:122] "RemoveContainer" containerID="f7ee1b1fba03037ed075aa90d678d3be16c71ebef4f5221a532f1d7a2e72e552" Nov 03 20:40:05.814893 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.814865 5059 generic.go:356] "Generic (PLEG): container finished" podID="896c119d-c988-4954-8606-72f0de8944cf" containerID="8eadfd624b310d22690f66e7e83531449a5166ef3ff1887788a0daf43a33c2bb" exitCode=0 Nov 03 20:40:05.815911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.814897 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" event={"ID":"896c119d-c988-4954-8606-72f0de8944cf","Type":"ContainerDied","Data":"8eadfd624b310d22690f66e7e83531449a5166ef3ff1887788a0daf43a33c2bb"} Nov 03 20:40:05.815911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.814915 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" Nov 03 20:40:05.815911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.815149 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm" event={"ID":"896c119d-c988-4954-8606-72f0de8944cf","Type":"ContainerDied","Data":"4081a1dd2fe8268ac5c31286bc25269fc5b237bcdc5b2ba48213f729201a653d"} Nov 03 20:40:05.825355 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.825337 5059 scope.go:122] "RemoveContainer" containerID="f7ee1b1fba03037ed075aa90d678d3be16c71ebef4f5221a532f1d7a2e72e552" Nov 03 20:40:05.831021 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:40:05.825924 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7ee1b1fba03037ed075aa90d678d3be16c71ebef4f5221a532f1d7a2e72e552\": container with ID starting with f7ee1b1fba03037ed075aa90d678d3be16c71ebef4f5221a532f1d7a2e72e552 not found: ID does not exist" containerID="f7ee1b1fba03037ed075aa90d678d3be16c71ebef4f5221a532f1d7a2e72e552" Nov 03 20:40:05.831021 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.825963 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7ee1b1fba03037ed075aa90d678d3be16c71ebef4f5221a532f1d7a2e72e552"} err="failed to get container status \"f7ee1b1fba03037ed075aa90d678d3be16c71ebef4f5221a532f1d7a2e72e552\": rpc error: code = NotFound desc = could not find container \"f7ee1b1fba03037ed075aa90d678d3be16c71ebef4f5221a532f1d7a2e72e552\": container with ID starting with f7ee1b1fba03037ed075aa90d678d3be16c71ebef4f5221a532f1d7a2e72e552 not found: ID does not exist" Nov 03 20:40:05.831021 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.825984 5059 scope.go:122] "RemoveContainer" containerID="8eadfd624b310d22690f66e7e83531449a5166ef3ff1887788a0daf43a33c2bb" Nov 03 20:40:05.837375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.837357 5059 scope.go:122] "RemoveContainer" containerID="8eadfd624b310d22690f66e7e83531449a5166ef3ff1887788a0daf43a33c2bb" Nov 03 20:40:05.838530 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:40:05.837632 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eadfd624b310d22690f66e7e83531449a5166ef3ff1887788a0daf43a33c2bb\": container with ID starting with 8eadfd624b310d22690f66e7e83531449a5166ef3ff1887788a0daf43a33c2bb not found: ID does not exist" containerID="8eadfd624b310d22690f66e7e83531449a5166ef3ff1887788a0daf43a33c2bb" Nov 03 20:40:05.838530 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.837684 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eadfd624b310d22690f66e7e83531449a5166ef3ff1887788a0daf43a33c2bb"} err="failed to get container status \"8eadfd624b310d22690f66e7e83531449a5166ef3ff1887788a0daf43a33c2bb\": rpc error: code = NotFound desc = could not find container \"8eadfd624b310d22690f66e7e83531449a5166ef3ff1887788a0daf43a33c2bb\": container with ID starting with 8eadfd624b310d22690f66e7e83531449a5166ef3ff1887788a0daf43a33c2bb not found: ID does not exist" Nov 03 20:40:05.845940 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.845916 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm"] Nov 03 20:40:05.913628 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.912311 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-nznsm"] Nov 03 20:40:05.933609 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.933578 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz"] Nov 03 20:40:05.940009 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:05.939981 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-j2rzz"] Nov 03 20:40:06.003118 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:06.003076 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" start-of-body= Nov 03 20:40:06.003967 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:06.003569 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" podUID="076156cc-fc7a-4a9e-bca0-7bee7bc7b065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" Nov 03 20:40:06.092686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:06.092127 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.173:15021/healthz/ready\": dial tcp 10.132.1.173:15021: connect: connection refused" start-of-body= Nov 03 20:40:06.092686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:06.092195 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" podUID="92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.173:15021/healthz/ready\": dial tcp 10.132.1.173:15021: connect: connection refused" Nov 03 20:40:07.007830 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:07.007791 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" start-of-body= Nov 03 20:40:07.008392 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:07.008356 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" podUID="076156cc-fc7a-4a9e-bca0-7bee7bc7b065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" Nov 03 20:40:07.086685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:07.086086 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.173:15021/healthz/ready\": dial tcp 10.132.1.173:15021: connect: connection refused" start-of-body= Nov 03 20:40:07.086685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:07.086150 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" podUID="92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.173:15021/healthz/ready\": dial tcp 10.132.1.173:15021: connect: connection refused" Nov 03 20:40:07.857326 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:07.857292 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="896c119d-c988-4954-8606-72f0de8944cf" path="/var/lib/kubelet/pods/896c119d-c988-4954-8606-72f0de8944cf/volumes" Nov 03 20:40:07.865684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:07.857922 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cca72a7d-a69d-4e05-ac4d-da20e15a0243" path="/var/lib/kubelet/pods/cca72a7d-a69d-4e05-ac4d-da20e15a0243/volumes" Nov 03 20:40:08.007233 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.007197 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" start-of-body= Nov 03 20:40:08.007497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.007471 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" podUID="076156cc-fc7a-4a9e-bca0-7bee7bc7b065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" Nov 03 20:40:08.089141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.089098 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.173:15021/healthz/ready\": dial tcp 10.132.1.173:15021: connect: connection refused" start-of-body= Nov 03 20:40:08.089764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.089730 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" podUID="92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.173:15021/healthz/ready\": dial tcp 10.132.1.173:15021: connect: connection refused" Nov 03 20:40:08.838126 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.838099 5059 generic.go:356] "Generic (PLEG): container finished" podID="9069949b-78ef-42cc-99b0-1feffc84b679" containerID="b3821841d889962619728e3cfd3742dffe6108759aefe0be61ddae8c78d55106" exitCode=0 Nov 03 20:40:08.838227 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.838171 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" event={"ID":"9069949b-78ef-42cc-99b0-1feffc84b679","Type":"ContainerDied","Data":"b3821841d889962619728e3cfd3742dffe6108759aefe0be61ddae8c78d55106"} Nov 03 20:40:08.838278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.838230 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" event={"ID":"9069949b-78ef-42cc-99b0-1feffc84b679","Type":"ContainerDied","Data":"eccf70e6270bc299851bddc769809e6c8081a6a29606df42984f367920b3621e"} Nov 03 20:40:08.838278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.838253 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eccf70e6270bc299851bddc769809e6c8081a6a29606df42984f367920b3621e" Nov 03 20:40:08.839698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.839680 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:40:08.932777 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.932736 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-credential-socket\") pod \"9069949b-78ef-42cc-99b0-1feffc84b679\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " Nov 03 20:40:08.932938 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.932793 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-workload-certs\") pod \"9069949b-78ef-42cc-99b0-1feffc84b679\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " Nov 03 20:40:08.932938 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.932822 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-workload-socket\") pod \"9069949b-78ef-42cc-99b0-1feffc84b679\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " Nov 03 20:40:08.932938 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.932857 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-istio-envoy\") pod \"9069949b-78ef-42cc-99b0-1feffc84b679\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " Nov 03 20:40:08.932938 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.932889 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9069949b-78ef-42cc-99b0-1feffc84b679-istio-token\" (UniqueName: \"kubernetes.io/projected/9069949b-78ef-42cc-99b0-1feffc84b679-istio-token\") pod \"9069949b-78ef-42cc-99b0-1feffc84b679\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " Nov 03 20:40:08.933070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.933006 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-istio-data\") pod \"9069949b-78ef-42cc-99b0-1feffc84b679\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " Nov 03 20:40:08.933070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.933033 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9069949b-78ef-42cc-99b0-1feffc84b679-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9069949b-78ef-42cc-99b0-1feffc84b679-istio-podinfo\") pod \"9069949b-78ef-42cc-99b0-1feffc84b679\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " Nov 03 20:40:08.933133 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.933076 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9069949b-78ef-42cc-99b0-1feffc84b679-kube-api-access-mhlgt\" (UniqueName: \"kubernetes.io/projected/9069949b-78ef-42cc-99b0-1feffc84b679-kube-api-access-mhlgt\") pod \"9069949b-78ef-42cc-99b0-1feffc84b679\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " Nov 03 20:40:08.933133 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.933081 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-credential-socket" pod "9069949b-78ef-42cc-99b0-1feffc84b679" (UID: "9069949b-78ef-42cc-99b0-1feffc84b679"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:08.933192 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.933138 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9069949b-78ef-42cc-99b0-1feffc84b679-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9069949b-78ef-42cc-99b0-1feffc84b679-istiod-ca-cert\") pod \"9069949b-78ef-42cc-99b0-1feffc84b679\" (UID: \"9069949b-78ef-42cc-99b0-1feffc84b679\") " Nov 03 20:40:08.933495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.933472 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:08.934070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.934043 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-workload-certs" pod "9069949b-78ef-42cc-99b0-1feffc84b679" (UID: "9069949b-78ef-42cc-99b0-1feffc84b679"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:08.934184 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.934087 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-workload-socket" pod "9069949b-78ef-42cc-99b0-1feffc84b679" (UID: "9069949b-78ef-42cc-99b0-1feffc84b679"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:08.934294 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.934271 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-istio-data" pod "9069949b-78ef-42cc-99b0-1feffc84b679" (UID: "9069949b-78ef-42cc-99b0-1feffc84b679"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:08.934347 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.934279 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9069949b-78ef-42cc-99b0-1feffc84b679-istiod-ca-cert" pod "9069949b-78ef-42cc-99b0-1feffc84b679" (UID: "9069949b-78ef-42cc-99b0-1feffc84b679"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:08.935724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.935702 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9069949b-78ef-42cc-99b0-1feffc84b679-istio-token" pod "9069949b-78ef-42cc-99b0-1feffc84b679" (UID: "9069949b-78ef-42cc-99b0-1feffc84b679"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:08.936036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.936013 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9069949b-78ef-42cc-99b0-1feffc84b679-kube-api-access-mhlgt" pod "9069949b-78ef-42cc-99b0-1feffc84b679" (UID: "9069949b-78ef-42cc-99b0-1feffc84b679"). InnerVolumeSpecName "kube-api-access-mhlgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:08.936101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.936083 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9069949b-78ef-42cc-99b0-1feffc84b679-istio-podinfo" pod "9069949b-78ef-42cc-99b0-1feffc84b679" (UID: "9069949b-78ef-42cc-99b0-1feffc84b679"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:40:08.936361 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:08.936330 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-istio-envoy" pod "9069949b-78ef-42cc-99b0-1feffc84b679" (UID: "9069949b-78ef-42cc-99b0-1feffc84b679"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:09.008626 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:09.008126 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" start-of-body= Nov 03 20:40:09.008626 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:09.008205 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" podUID="076156cc-fc7a-4a9e-bca0-7bee7bc7b065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" Nov 03 20:40:09.035170 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:09.035136 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:09.035170 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:09.035170 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9069949b-78ef-42cc-99b0-1feffc84b679-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:09.035322 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:09.035182 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhlgt\" (UniqueName: \"kubernetes.io/projected/9069949b-78ef-42cc-99b0-1feffc84b679-kube-api-access-mhlgt\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:09.035322 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:09.035192 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9069949b-78ef-42cc-99b0-1feffc84b679-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:09.035322 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:09.035201 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:09.035322 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:09.035210 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:09.035322 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:09.035218 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9069949b-78ef-42cc-99b0-1feffc84b679-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:09.035322 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:09.035226 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9069949b-78ef-42cc-99b0-1feffc84b679-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:09.093746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:09.086057 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.173:15021/healthz/ready\": dial tcp 10.132.1.173:15021: connect: connection refused" start-of-body= Nov 03 20:40:09.093746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:09.086121 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" podUID="92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.173:15021/healthz/ready\": dial tcp 10.132.1.173:15021: connect: connection refused" Nov 03 20:40:09.846836 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:09.846801 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s" Nov 03 20:40:09.940898 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:09.940854 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s"] Nov 03 20:40:09.943537 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:09.943509 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dbx4s"] Nov 03 20:40:10.007174 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:10.007135 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" start-of-body= Nov 03 20:40:10.007819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:10.007414 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" podUID="076156cc-fc7a-4a9e-bca0-7bee7bc7b065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" Nov 03 20:40:10.086093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:10.086056 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.173:15021/healthz/ready\": dial tcp 10.132.1.173:15021: connect: connection refused" start-of-body= Nov 03 20:40:10.086375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:10.086346 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" podUID="92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.173:15021/healthz/ready\": dial tcp 10.132.1.173:15021: connect: connection refused" Nov 03 20:40:10.634422 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:40:10.634387 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:40:11.002974 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:11.002889 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" start-of-body= Nov 03 20:40:11.002974 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:11.002957 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" podUID="076156cc-fc7a-4a9e-bca0-7bee7bc7b065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.172:15021/healthz/ready\": dial tcp 10.132.1.172:15021: connect: connection refused" Nov 03 20:40:11.086187 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:11.086140 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.173:15021/healthz/ready\": dial tcp 10.132.1.173:15021: connect: connection refused" start-of-body= Nov 03 20:40:11.086340 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:11.086203 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" podUID="92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.173:15021/healthz/ready\": dial tcp 10.132.1.173:15021: connect: connection refused" Nov 03 20:40:11.862560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:11.862519 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" path="/var/lib/kubelet/pods/9069949b-78ef-42cc-99b0-1feffc84b679/volumes" Nov 03 20:40:12.061112 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:12.061076 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:12.061424 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:12.061393 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:12.062218 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:12.062193 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:12.090210 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:12.090183 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:12.090475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:12.090453 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:12.091248 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:12.091231 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:12.510834 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:12.510797 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn"] Nov 03 20:40:12.524193 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:12.524163 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c"] Nov 03 20:40:13.017604 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.017567 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de0676fc-bk5xz"] Nov 03 20:40:13.018014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.017983 5059 cpu_manager.go:414] "containerMap: removing container" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" Nov 03 20:40:13.018014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.017995 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" Nov 03 20:40:13.018102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.018026 5059 cpu_manager.go:414] "containerMap: removing container" podUID="3b330e09-cf3a-4278-bc82-5386105badd1" containerName="revoke-keys" Nov 03 20:40:13.018102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.018033 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b330e09-cf3a-4278-bc82-5386105badd1" containerName="revoke-keys" Nov 03 20:40:13.018102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.018042 5059 cpu_manager.go:414] "containerMap: removing container" podUID="896c119d-c988-4954-8606-72f0de8944cf" containerName="istio-proxy" Nov 03 20:40:13.018102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.018048 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="896c119d-c988-4954-8606-72f0de8944cf" containerName="istio-proxy" Nov 03 20:40:13.018102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.018057 5059 cpu_manager.go:414] "containerMap: removing container" podUID="cca72a7d-a69d-4e05-ac4d-da20e15a0243" containerName="istio-proxy" Nov 03 20:40:13.018102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.018062 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca72a7d-a69d-4e05-ac4d-da20e15a0243" containerName="istio-proxy" Nov 03 20:40:13.018294 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.018113 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b330e09-cf3a-4278-bc82-5386105badd1" containerName="revoke-keys" Nov 03 20:40:13.018294 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.018124 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="896c119d-c988-4954-8606-72f0de8944cf" containerName="istio-proxy" Nov 03 20:40:13.018294 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.018130 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="cca72a7d-a69d-4e05-ac4d-da20e15a0243" containerName="istio-proxy" Nov 03 20:40:13.018294 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.018136 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9069949b-78ef-42cc-99b0-1feffc84b679" containerName="istio-proxy" Nov 03 20:40:13.022510 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.022491 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0676fc-bk5xz" Nov 03 20:40:13.028629 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.028598 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:40:13.029022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.028997 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28cac4fe-lrfxw"] Nov 03 20:40:13.029862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.029841 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:40:13.033131 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.033108 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de0676fc-bk5xz"] Nov 03 20:40:13.036206 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.036184 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28cac4fe-lrfxw"] Nov 03 20:40:13.098315 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.098282 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pfcg\" (UniqueName: \"kubernetes.io/projected/75ed2a98-0045-44d3-ac8b-c9c32ea1b658-kube-api-access-5pfcg\") pod \"maas-api-revoke-sub-de0676fc-bk5xz\" (UID: \"75ed2a98-0045-44d3-ac8b-c9c32ea1b658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0676fc-bk5xz" Nov 03 20:40:13.098462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.098442 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ed2a98-0045-44d3-ac8b-c9c32ea1b658-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de0676fc-bk5xz\" (UID: \"75ed2a98-0045-44d3-ac8b-c9c32ea1b658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0676fc-bk5xz" Nov 03 20:40:13.098525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.098473 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ed2a98-0045-44d3-ac8b-c9c32ea1b658-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de0676fc-bk5xz\" (UID: \"75ed2a98-0045-44d3-ac8b-c9c32ea1b658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0676fc-bk5xz" Nov 03 20:40:13.199355 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.199323 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ed2a98-0045-44d3-ac8b-c9c32ea1b658-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de0676fc-bk5xz\" (UID: \"75ed2a98-0045-44d3-ac8b-c9c32ea1b658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0676fc-bk5xz" Nov 03 20:40:13.199472 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.199358 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ed2a98-0045-44d3-ac8b-c9c32ea1b658-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de0676fc-bk5xz\" (UID: \"75ed2a98-0045-44d3-ac8b-c9c32ea1b658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0676fc-bk5xz" Nov 03 20:40:13.199472 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.199399 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pfcg\" (UniqueName: \"kubernetes.io/projected/75ed2a98-0045-44d3-ac8b-c9c32ea1b658-kube-api-access-5pfcg\") pod \"maas-api-revoke-sub-de0676fc-bk5xz\" (UID: \"75ed2a98-0045-44d3-ac8b-c9c32ea1b658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0676fc-bk5xz" Nov 03 20:40:13.200071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.200049 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ed2a98-0045-44d3-ac8b-c9c32ea1b658-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de0676fc-bk5xz\" (UID: \"75ed2a98-0045-44d3-ac8b-c9c32ea1b658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0676fc-bk5xz" Nov 03 20:40:13.201782 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.201762 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ed2a98-0045-44d3-ac8b-c9c32ea1b658-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de0676fc-bk5xz\" (UID: \"75ed2a98-0045-44d3-ac8b-c9c32ea1b658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0676fc-bk5xz" Nov 03 20:40:13.211580 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.211558 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pfcg\" (UniqueName: \"kubernetes.io/projected/75ed2a98-0045-44d3-ac8b-c9c32ea1b658-kube-api-access-5pfcg\") pod \"maas-api-revoke-sub-de0676fc-bk5xz\" (UID: \"75ed2a98-0045-44d3-ac8b-c9c32ea1b658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0676fc-bk5xz" Nov 03 20:40:13.339062 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.339036 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0676fc-bk5xz" Nov 03 20:40:13.627922 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.627889 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de0676fc-bk5xz"] Nov 03 20:40:13.628124 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:40:13.628101 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ed2a98_0045_44d3_ac8b_c9c32ea1b658.slice/crio-44a9f8293974795f9f7773cded9175f89ab777d537eb0379941ab0eac511f734 WatchSource:0}: Error finding container 44a9f8293974795f9f7773cded9175f89ab777d537eb0379941ab0eac511f734: Status 404 returned error can't find the container with id 44a9f8293974795f9f7773cded9175f89ab777d537eb0379941ab0eac511f734 Nov 03 20:40:13.855989 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.855957 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acf514f7-7a99-44ca-b62c-058eb4a39d26" path="/var/lib/kubelet/pods/acf514f7-7a99-44ca-b62c-058eb4a39d26/volumes" Nov 03 20:40:13.877417 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.877380 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0676fc-bk5xz" event={"ID":"75ed2a98-0045-44d3-ac8b-c9c32ea1b658","Type":"ContainerStarted","Data":"339b9db0f75cc3254388d71fefa4fc52189afbbc0916496dc7cc5536d5c39ccd"} Nov 03 20:40:13.877509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.877442 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0676fc-bk5xz" event={"ID":"75ed2a98-0045-44d3-ac8b-c9c32ea1b658","Type":"ContainerStarted","Data":"44a9f8293974795f9f7773cded9175f89ab777d537eb0379941ab0eac511f734"} Nov 03 20:40:13.896343 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:13.896301 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0676fc-bk5xz" podStartSLOduration=1.896286682 podStartE2EDuration="1.896286682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:13.894571358 +0000 UTC m=+3808.152368613" watchObservedRunningTime="2026-11-03 20:40:13.896286682 +0000 UTC m=+3808.154083938" Nov 03 20:40:14.887676 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:14.887631 5059 generic.go:356] "Generic (PLEG): container finished" podID="75ed2a98-0045-44d3-ac8b-c9c32ea1b658" containerID="339b9db0f75cc3254388d71fefa4fc52189afbbc0916496dc7cc5536d5c39ccd" exitCode=0 Nov 03 20:40:14.888034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:14.887716 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0676fc-bk5xz" event={"ID":"75ed2a98-0045-44d3-ac8b-c9c32ea1b658","Type":"ContainerDied","Data":"339b9db0f75cc3254388d71fefa4fc52189afbbc0916496dc7cc5536d5c39ccd"} Nov 03 20:40:16.033642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:16.033617 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0676fc-bk5xz" Nov 03 20:40:16.128649 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:16.128620 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75ed2a98-0045-44d3-ac8b-c9c32ea1b658-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ed2a98-0045-44d3-ac8b-c9c32ea1b658-maas-api-openshift-ca\") pod \"75ed2a98-0045-44d3-ac8b-c9c32ea1b658\" (UID: \"75ed2a98-0045-44d3-ac8b-c9c32ea1b658\") " Nov 03 20:40:16.128827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:16.128695 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ed2a98-0045-44d3-ac8b-c9c32ea1b658-kube-api-access-5pfcg\" (UniqueName: \"kubernetes.io/projected/75ed2a98-0045-44d3-ac8b-c9c32ea1b658-kube-api-access-5pfcg\") pod \"75ed2a98-0045-44d3-ac8b-c9c32ea1b658\" (UID: \"75ed2a98-0045-44d3-ac8b-c9c32ea1b658\") " Nov 03 20:40:16.128827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:16.128735 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75ed2a98-0045-44d3-ac8b-c9c32ea1b658-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ed2a98-0045-44d3-ac8b-c9c32ea1b658-maas-api-xks-ca\") pod \"75ed2a98-0045-44d3-ac8b-c9c32ea1b658\" (UID: \"75ed2a98-0045-44d3-ac8b-c9c32ea1b658\") " Nov 03 20:40:16.129035 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:16.129008 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ed2a98-0045-44d3-ac8b-c9c32ea1b658-maas-api-openshift-ca" pod "75ed2a98-0045-44d3-ac8b-c9c32ea1b658" (UID: "75ed2a98-0045-44d3-ac8b-c9c32ea1b658"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:16.130840 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:16.130812 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75ed2a98-0045-44d3-ac8b-c9c32ea1b658-maas-api-xks-ca" pod "75ed2a98-0045-44d3-ac8b-c9c32ea1b658" (UID: "75ed2a98-0045-44d3-ac8b-c9c32ea1b658"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:16.130937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:16.130843 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ed2a98-0045-44d3-ac8b-c9c32ea1b658-kube-api-access-5pfcg" pod "75ed2a98-0045-44d3-ac8b-c9c32ea1b658" (UID: "75ed2a98-0045-44d3-ac8b-c9c32ea1b658"). InnerVolumeSpecName "kube-api-access-5pfcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:16.230106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:16.230028 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ed2a98-0045-44d3-ac8b-c9c32ea1b658-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:16.230106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:16.230062 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pfcg\" (UniqueName: \"kubernetes.io/projected/75ed2a98-0045-44d3-ac8b-c9c32ea1b658-kube-api-access-5pfcg\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:16.230106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:16.230073 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ed2a98-0045-44d3-ac8b-c9c32ea1b658-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:16.901571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:16.901535 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0676fc-bk5xz" event={"ID":"75ed2a98-0045-44d3-ac8b-c9c32ea1b658","Type":"ContainerDied","Data":"44a9f8293974795f9f7773cded9175f89ab777d537eb0379941ab0eac511f734"} Nov 03 20:40:16.901571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:16.901557 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de0676fc-bk5xz" Nov 03 20:40:16.901795 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:16.901590 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44a9f8293974795f9f7773cded9175f89ab777d537eb0379941ab0eac511f734" Nov 03 20:40:17.512053 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.512010 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn" podUID="c9ead1ac-0ace-4761-9fde-b7f97a9dfa31" containerName="payload-processing" containerID="cri-o://9c318b2e852863d226308b67f4a9c795472db2505655e98f81679d2b9182e1e4" gracePeriod=25 Nov 03 20:40:17.524623 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.524594 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c" podUID="ece17ed8-afcd-41ab-85e0-adb35875ecd6" containerName="payload-pre-processing" containerID="cri-o://2f439e80b9652e884e5ddcb17983cc536b4f3f4964293cac6b93378706a542b3" gracePeriod=25 Nov 03 20:40:17.771279 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.771227 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn" Nov 03 20:40:17.807106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.807079 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c" Nov 03 20:40:17.850155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.850119 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9ead1ac-0ace-4761-9fde-b7f97a9dfa31-kube-api-access-wcrnp\" (UniqueName: \"kubernetes.io/projected/c9ead1ac-0ace-4761-9fde-b7f97a9dfa31-kube-api-access-wcrnp\") pod \"c9ead1ac-0ace-4761-9fde-b7f97a9dfa31\" (UID: \"c9ead1ac-0ace-4761-9fde-b7f97a9dfa31\") " Nov 03 20:40:17.850292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.850168 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9ead1ac-0ace-4761-9fde-b7f97a9dfa31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9ead1ac-0ace-4761-9fde-b7f97a9dfa31-plugins-config-volume\") pod \"c9ead1ac-0ace-4761-9fde-b7f97a9dfa31\" (UID: \"c9ead1ac-0ace-4761-9fde-b7f97a9dfa31\") " Nov 03 20:40:17.850292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.850235 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ece17ed8-afcd-41ab-85e0-adb35875ecd6-kube-api-access-dtgkq\" (UniqueName: \"kubernetes.io/projected/ece17ed8-afcd-41ab-85e0-adb35875ecd6-kube-api-access-dtgkq\") pod \"ece17ed8-afcd-41ab-85e0-adb35875ecd6\" (UID: \"ece17ed8-afcd-41ab-85e0-adb35875ecd6\") " Nov 03 20:40:17.850425 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.850307 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ece17ed8-afcd-41ab-85e0-adb35875ecd6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece17ed8-afcd-41ab-85e0-adb35875ecd6-plugins-config-volume\") pod \"ece17ed8-afcd-41ab-85e0-adb35875ecd6\" (UID: \"ece17ed8-afcd-41ab-85e0-adb35875ecd6\") " Nov 03 20:40:17.850763 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.850732 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ead1ac-0ace-4761-9fde-b7f97a9dfa31-plugins-config-volume" pod "c9ead1ac-0ace-4761-9fde-b7f97a9dfa31" (UID: "c9ead1ac-0ace-4761-9fde-b7f97a9dfa31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:17.850876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.850786 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece17ed8-afcd-41ab-85e0-adb35875ecd6-plugins-config-volume" pod "ece17ed8-afcd-41ab-85e0-adb35875ecd6" (UID: "ece17ed8-afcd-41ab-85e0-adb35875ecd6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:17.852373 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.852350 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece17ed8-afcd-41ab-85e0-adb35875ecd6-kube-api-access-dtgkq" pod "ece17ed8-afcd-41ab-85e0-adb35875ecd6" (UID: "ece17ed8-afcd-41ab-85e0-adb35875ecd6"). InnerVolumeSpecName "kube-api-access-dtgkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:17.852911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.852885 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ead1ac-0ace-4761-9fde-b7f97a9dfa31-kube-api-access-wcrnp" pod "c9ead1ac-0ace-4761-9fde-b7f97a9dfa31" (UID: "c9ead1ac-0ace-4761-9fde-b7f97a9dfa31"). InnerVolumeSpecName "kube-api-access-wcrnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:17.910440 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.910400 5059 generic.go:356] "Generic (PLEG): container finished" podID="ece17ed8-afcd-41ab-85e0-adb35875ecd6" containerID="2f439e80b9652e884e5ddcb17983cc536b4f3f4964293cac6b93378706a542b3" exitCode=0 Nov 03 20:40:17.910646 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.910617 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c" event={"ID":"ece17ed8-afcd-41ab-85e0-adb35875ecd6","Type":"ContainerDied","Data":"2f439e80b9652e884e5ddcb17983cc536b4f3f4964293cac6b93378706a542b3"} Nov 03 20:40:17.910735 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.910652 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c" Nov 03 20:40:17.910783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.910737 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c" event={"ID":"ece17ed8-afcd-41ab-85e0-adb35875ecd6","Type":"ContainerDied","Data":"1fb871bbe1a064bd6c076f10538e03887172e9b7c42b6bc26be22afe9e84cc0b"} Nov 03 20:40:17.910783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.910777 5059 scope.go:122] "RemoveContainer" containerID="2f439e80b9652e884e5ddcb17983cc536b4f3f4964293cac6b93378706a542b3" Nov 03 20:40:17.912630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.912595 5059 generic.go:356] "Generic (PLEG): container finished" podID="c9ead1ac-0ace-4761-9fde-b7f97a9dfa31" containerID="9c318b2e852863d226308b67f4a9c795472db2505655e98f81679d2b9182e1e4" exitCode=0 Nov 03 20:40:17.912809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.912788 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn" event={"ID":"c9ead1ac-0ace-4761-9fde-b7f97a9dfa31","Type":"ContainerDied","Data":"9c318b2e852863d226308b67f4a9c795472db2505655e98f81679d2b9182e1e4"} Nov 03 20:40:17.912887 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.912845 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn" event={"ID":"c9ead1ac-0ace-4761-9fde-b7f97a9dfa31","Type":"ContainerDied","Data":"2944eaaae47115096c908bfd9d08b4061d4dafe6b809a6b937a0b15f266b2edf"} Nov 03 20:40:17.912971 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.912955 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn" Nov 03 20:40:17.928839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.928817 5059 scope.go:122] "RemoveContainer" containerID="2f439e80b9652e884e5ddcb17983cc536b4f3f4964293cac6b93378706a542b3" Nov 03 20:40:17.929197 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:40:17.929169 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f439e80b9652e884e5ddcb17983cc536b4f3f4964293cac6b93378706a542b3\": container with ID starting with 2f439e80b9652e884e5ddcb17983cc536b4f3f4964293cac6b93378706a542b3 not found: ID does not exist" containerID="2f439e80b9652e884e5ddcb17983cc536b4f3f4964293cac6b93378706a542b3" Nov 03 20:40:17.929290 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.929201 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f439e80b9652e884e5ddcb17983cc536b4f3f4964293cac6b93378706a542b3"} err="failed to get container status \"2f439e80b9652e884e5ddcb17983cc536b4f3f4964293cac6b93378706a542b3\": rpc error: code = NotFound desc = could not find container \"2f439e80b9652e884e5ddcb17983cc536b4f3f4964293cac6b93378706a542b3\": container with ID starting with 2f439e80b9652e884e5ddcb17983cc536b4f3f4964293cac6b93378706a542b3 not found: ID does not exist" Nov 03 20:40:17.929290 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.929220 5059 scope.go:122] "RemoveContainer" containerID="9c318b2e852863d226308b67f4a9c795472db2505655e98f81679d2b9182e1e4" Nov 03 20:40:17.936492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.936467 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn"] Nov 03 20:40:17.938883 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.938861 5059 scope.go:122] "RemoveContainer" containerID="9c318b2e852863d226308b67f4a9c795472db2505655e98f81679d2b9182e1e4" Nov 03 20:40:17.939170 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:40:17.939152 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c318b2e852863d226308b67f4a9c795472db2505655e98f81679d2b9182e1e4\": container with ID starting with 9c318b2e852863d226308b67f4a9c795472db2505655e98f81679d2b9182e1e4 not found: ID does not exist" containerID="9c318b2e852863d226308b67f4a9c795472db2505655e98f81679d2b9182e1e4" Nov 03 20:40:17.939243 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.939175 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c318b2e852863d226308b67f4a9c795472db2505655e98f81679d2b9182e1e4"} err="failed to get container status \"9c318b2e852863d226308b67f4a9c795472db2505655e98f81679d2b9182e1e4\": rpc error: code = NotFound desc = could not find container \"9c318b2e852863d226308b67f4a9c795472db2505655e98f81679d2b9182e1e4\": container with ID starting with 9c318b2e852863d226308b67f4a9c795472db2505655e98f81679d2b9182e1e4 not found: ID does not exist" Nov 03 20:40:17.941642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.941619 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-32e482-5647cf6c5b-mx7tn"] Nov 03 20:40:17.951329 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.951304 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtgkq\" (UniqueName: \"kubernetes.io/projected/ece17ed8-afcd-41ab-85e0-adb35875ecd6-kube-api-access-dtgkq\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:17.951329 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.951330 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece17ed8-afcd-41ab-85e0-adb35875ecd6-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:17.951448 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.951341 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcrnp\" (UniqueName: \"kubernetes.io/projected/c9ead1ac-0ace-4761-9fde-b7f97a9dfa31-kube-api-access-wcrnp\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:17.951448 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.951349 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9ead1ac-0ace-4761-9fde-b7f97a9dfa31-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:17.953804 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.953785 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c"] Nov 03 20:40:17.962348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:17.962328 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-rrj8c"] Nov 03 20:40:18.703943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.703910 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79"] Nov 03 20:40:18.704322 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.704305 5059 cpu_manager.go:414] "containerMap: removing container" podUID="75ed2a98-0045-44d3-ac8b-c9c32ea1b658" containerName="revoke-keys" Nov 03 20:40:18.704382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.704326 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ed2a98-0045-44d3-ac8b-c9c32ea1b658" containerName="revoke-keys" Nov 03 20:40:18.704382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.704366 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c9ead1ac-0ace-4761-9fde-b7f97a9dfa31" containerName="payload-processing" Nov 03 20:40:18.704382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.704373 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9ead1ac-0ace-4761-9fde-b7f97a9dfa31" containerName="payload-processing" Nov 03 20:40:18.704480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.704385 5059 cpu_manager.go:414] "containerMap: removing container" podUID="ece17ed8-afcd-41ab-85e0-adb35875ecd6" containerName="payload-pre-processing" Nov 03 20:40:18.704480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.704390 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ece17ed8-afcd-41ab-85e0-adb35875ecd6" containerName="payload-pre-processing" Nov 03 20:40:18.704480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.704463 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9ead1ac-0ace-4761-9fde-b7f97a9dfa31" containerName="payload-processing" Nov 03 20:40:18.704480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.704474 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ece17ed8-afcd-41ab-85e0-adb35875ecd6" containerName="payload-pre-processing" Nov 03 20:40:18.704596 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.704482 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ed2a98-0045-44d3-ac8b-c9c32ea1b658" containerName="revoke-keys" Nov 03 20:40:18.708983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.708966 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.720067 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.720045 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79"] Nov 03 20:40:18.757414 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.757386 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cl85\" (UniqueName: \"kubernetes.io/projected/35287c50-78d5-4061-8ead-d2325d6800c8-kube-api-access-2cl85\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.757537 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.757450 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.757537 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.757513 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35287c50-78d5-4061-8ead-d2325d6800c8-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.757608 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.757572 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.757643 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.757625 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.757723 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.757709 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.757767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.757753 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35287c50-78d5-4061-8ead-d2325d6800c8-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.757799 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.757776 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.757799 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.757792 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35287c50-78d5-4061-8ead-d2325d6800c8-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.858630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.858600 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cl85\" (UniqueName: \"kubernetes.io/projected/35287c50-78d5-4061-8ead-d2325d6800c8-kube-api-access-2cl85\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.858796 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.858709 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.858796 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.858737 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35287c50-78d5-4061-8ead-d2325d6800c8-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.858913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.858793 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.858913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.858820 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.858998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.858963 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.859524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.859054 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35287c50-78d5-4061-8ead-d2325d6800c8-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.859524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.859101 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.859524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.859129 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35287c50-78d5-4061-8ead-d2325d6800c8-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.859524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.859191 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.859524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.859248 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.859524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.859372 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.859524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.859488 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.860013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.859710 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35287c50-78d5-4061-8ead-d2325d6800c8-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.860984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.860956 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.861515 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.861498 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35287c50-78d5-4061-8ead-d2325d6800c8-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.870980 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.870948 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35287c50-78d5-4061-8ead-d2325d6800c8-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:18.871170 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:18.871152 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cl85\" (UniqueName: \"kubernetes.io/projected/35287c50-78d5-4061-8ead-d2325d6800c8-kube-api-access-2cl85\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:19.023682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:19.023629 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:19.346253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:19.346222 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79"] Nov 03 20:40:19.347213 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:40:19.347176 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35287c50_78d5_4061_8ead_d2325d6800c8.slice/crio-bd54a2cc127b7cc1307cdbc8cf3d32d098941a34a44956e7f61888e30f352aa0 WatchSource:0}: Error finding container bd54a2cc127b7cc1307cdbc8cf3d32d098941a34a44956e7f61888e30f352aa0: Status 404 returned error can't find the container with id bd54a2cc127b7cc1307cdbc8cf3d32d098941a34a44956e7f61888e30f352aa0 Nov 03 20:40:19.349470 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:19.349436 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:40:19.349519 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:19.349504 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:40:19.349550 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:19.349537 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:40:19.855894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:19.855858 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9ead1ac-0ace-4761-9fde-b7f97a9dfa31" path="/var/lib/kubelet/pods/c9ead1ac-0ace-4761-9fde-b7f97a9dfa31/volumes" Nov 03 20:40:19.856289 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:19.856278 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ece17ed8-afcd-41ab-85e0-adb35875ecd6" path="/var/lib/kubelet/pods/ece17ed8-afcd-41ab-85e0-adb35875ecd6/volumes" Nov 03 20:40:19.928610 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:19.928579 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" event={"ID":"35287c50-78d5-4061-8ead-d2325d6800c8","Type":"ContainerStarted","Data":"e98bbba4fdaf72a992af1b602abe2af37f79924046bf9fdd52e8d228e42f5f34"} Nov 03 20:40:19.928768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:19.928625 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" event={"ID":"35287c50-78d5-4061-8ead-d2325d6800c8","Type":"ContainerStarted","Data":"bd54a2cc127b7cc1307cdbc8cf3d32d098941a34a44956e7f61888e30f352aa0"} Nov 03 20:40:19.953364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:19.953317 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" podStartSLOduration=1.953304298 podStartE2EDuration="1.953304298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:19.951182075 +0000 UTC m=+3814.208979333" watchObservedRunningTime="2026-11-03 20:40:19.953304298 +0000 UTC m=+3814.211101595" Nov 03 20:40:20.024042 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:20.023998 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:20.025546 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:20.025518 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" start-of-body= Nov 03 20:40:20.025654 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:20.025570 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" podUID="35287c50-78d5-4061-8ead-d2325d6800c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" Nov 03 20:40:20.501718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:20.501619 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz"] Nov 03 20:40:20.502263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:20.502229 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" podUID="abce531c-32e5-4fee-b6fa-7b489da29704" containerName="istio-proxy" containerID="cri-o://b45a6bff1af76d642a374e7fb94b73d65da2201d87f7ddea070a688d25f2fb96" gracePeriod=30 Nov 03 20:40:21.024442 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:21.024405 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" start-of-body= Nov 03 20:40:21.024841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:21.024469 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" podUID="35287c50-78d5-4061-8ead-d2325d6800c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" Nov 03 20:40:21.536705 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:21.536657 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70e9c51f-2g5tq"] Nov 03 20:40:21.540245 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:21.540228 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e9c51f-2g5tq" Nov 03 20:40:21.546127 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:21.546105 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:40:21.553831 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:21.553805 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70e9c51f-2g5tq"] Nov 03 20:40:21.598149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:21.598118 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7dae9ac-15d8-4751-87db-2ca6ed359f2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70e9c51f-2g5tq\" (UID: \"c7dae9ac-15d8-4751-87db-2ca6ed359f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e9c51f-2g5tq" Nov 03 20:40:21.598276 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:21.598174 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97ql9\" (UniqueName: \"kubernetes.io/projected/c7dae9ac-15d8-4751-87db-2ca6ed359f2e-kube-api-access-97ql9\") pod \"maas-api-revoke-sub-70e9c51f-2g5tq\" (UID: \"c7dae9ac-15d8-4751-87db-2ca6ed359f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e9c51f-2g5tq" Nov 03 20:40:21.598334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:21.598312 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7dae9ac-15d8-4751-87db-2ca6ed359f2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70e9c51f-2g5tq\" (UID: \"c7dae9ac-15d8-4751-87db-2ca6ed359f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e9c51f-2g5tq" Nov 03 20:40:21.645823 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:21.645792 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:40:21.699779 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:21.699748 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7dae9ac-15d8-4751-87db-2ca6ed359f2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70e9c51f-2g5tq\" (UID: \"c7dae9ac-15d8-4751-87db-2ca6ed359f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e9c51f-2g5tq" Nov 03 20:40:21.699922 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:21.699828 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97ql9\" (UniqueName: \"kubernetes.io/projected/c7dae9ac-15d8-4751-87db-2ca6ed359f2e-kube-api-access-97ql9\") pod \"maas-api-revoke-sub-70e9c51f-2g5tq\" (UID: \"c7dae9ac-15d8-4751-87db-2ca6ed359f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e9c51f-2g5tq" Nov 03 20:40:21.700000 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:21.699921 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7dae9ac-15d8-4751-87db-2ca6ed359f2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70e9c51f-2g5tq\" (UID: \"c7dae9ac-15d8-4751-87db-2ca6ed359f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e9c51f-2g5tq" Nov 03 20:40:21.700572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:21.700545 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7dae9ac-15d8-4751-87db-2ca6ed359f2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70e9c51f-2g5tq\" (UID: \"c7dae9ac-15d8-4751-87db-2ca6ed359f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e9c51f-2g5tq" Nov 03 20:40:21.702322 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:21.702281 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7dae9ac-15d8-4751-87db-2ca6ed359f2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70e9c51f-2g5tq\" (UID: \"c7dae9ac-15d8-4751-87db-2ca6ed359f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e9c51f-2g5tq" Nov 03 20:40:21.715561 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:21.715539 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97ql9\" (UniqueName: \"kubernetes.io/projected/c7dae9ac-15d8-4751-87db-2ca6ed359f2e-kube-api-access-97ql9\") pod \"maas-api-revoke-sub-70e9c51f-2g5tq\" (UID: \"c7dae9ac-15d8-4751-87db-2ca6ed359f2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e9c51f-2g5tq" Nov 03 20:40:21.884688 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:21.880864 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e9c51f-2g5tq" Nov 03 20:40:22.024461 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:22.024422 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" start-of-body= Nov 03 20:40:22.024844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:22.024473 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" podUID="35287c50-78d5-4061-8ead-d2325d6800c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" Nov 03 20:40:22.198938 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:40:22.198887 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7dae9ac_15d8_4751_87db_2ca6ed359f2e.slice/crio-6ff1f489bfb931d1f9bad41bb0624d69455c35bb1803cb243d02430843bb74b6 WatchSource:0}: Error finding container 6ff1f489bfb931d1f9bad41bb0624d69455c35bb1803cb243d02430843bb74b6: Status 404 returned error can't find the container with id 6ff1f489bfb931d1f9bad41bb0624d69455c35bb1803cb243d02430843bb74b6 Nov 03 20:40:22.199415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:22.199389 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70e9c51f-2g5tq"] Nov 03 20:40:22.955948 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:22.955910 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e9c51f-2g5tq" event={"ID":"c7dae9ac-15d8-4751-87db-2ca6ed359f2e","Type":"ContainerStarted","Data":"59cacebca1eec8a19ee9119d3b5833976b021ea541538d9597b0a7409ce592e0"} Nov 03 20:40:22.956241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:22.956215 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e9c51f-2g5tq" event={"ID":"c7dae9ac-15d8-4751-87db-2ca6ed359f2e","Type":"ContainerStarted","Data":"6ff1f489bfb931d1f9bad41bb0624d69455c35bb1803cb243d02430843bb74b6"} Nov 03 20:40:22.990531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:22.990480 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e9c51f-2g5tq" podStartSLOduration=1.990465511 podStartE2EDuration="1.990465511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:22.989391534 +0000 UTC m=+3817.247188791" watchObservedRunningTime="2026-11-03 20:40:22.990465511 +0000 UTC m=+3817.248262767" Nov 03 20:40:23.025024 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:23.024985 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" start-of-body= Nov 03 20:40:23.025459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:23.025053 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" podUID="35287c50-78d5-4061-8ead-d2325d6800c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" Nov 03 20:40:23.484821 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:23.484729 5059 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:40:23.484821 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:23.484803 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" podUID="abce531c-32e5-4fee-b6fa-7b489da29704" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:40:24.000350 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:24.000310 5059 generic.go:356] "Generic (PLEG): container finished" podID="c7dae9ac-15d8-4751-87db-2ca6ed359f2e" containerID="59cacebca1eec8a19ee9119d3b5833976b021ea541538d9597b0a7409ce592e0" exitCode=0 Nov 03 20:40:24.000507 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:24.000478 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e9c51f-2g5tq" event={"ID":"c7dae9ac-15d8-4751-87db-2ca6ed359f2e","Type":"ContainerDied","Data":"59cacebca1eec8a19ee9119d3b5833976b021ea541538d9597b0a7409ce592e0"} Nov 03 20:40:24.024712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:24.024673 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" start-of-body= Nov 03 20:40:24.025591 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:24.024807 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" podUID="35287c50-78d5-4061-8ead-d2325d6800c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" Nov 03 20:40:24.031138 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:24.031114 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8095d1bf-dtz7r"] Nov 03 20:40:24.034899 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:24.034874 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8095d1bf-dtz7r"] Nov 03 20:40:25.024282 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.024250 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" start-of-body= Nov 03 20:40:25.024443 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.024333 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" podUID="35287c50-78d5-4061-8ead-d2325d6800c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.175:15021/healthz/ready\": dial tcp 10.132.1.175:15021: connect: connection refused" Nov 03 20:40:25.154809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.154784 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e9c51f-2g5tq" Nov 03 20:40:25.236741 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.236704 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7dae9ac-15d8-4751-87db-2ca6ed359f2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7dae9ac-15d8-4751-87db-2ca6ed359f2e-maas-api-openshift-ca\") pod \"c7dae9ac-15d8-4751-87db-2ca6ed359f2e\" (UID: \"c7dae9ac-15d8-4751-87db-2ca6ed359f2e\") " Nov 03 20:40:25.236922 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.236813 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7dae9ac-15d8-4751-87db-2ca6ed359f2e-kube-api-access-97ql9\" (UniqueName: \"kubernetes.io/projected/c7dae9ac-15d8-4751-87db-2ca6ed359f2e-kube-api-access-97ql9\") pod \"c7dae9ac-15d8-4751-87db-2ca6ed359f2e\" (UID: \"c7dae9ac-15d8-4751-87db-2ca6ed359f2e\") " Nov 03 20:40:25.236922 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.236901 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7dae9ac-15d8-4751-87db-2ca6ed359f2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7dae9ac-15d8-4751-87db-2ca6ed359f2e-maas-api-xks-ca\") pod \"c7dae9ac-15d8-4751-87db-2ca6ed359f2e\" (UID: \"c7dae9ac-15d8-4751-87db-2ca6ed359f2e\") " Nov 03 20:40:25.237127 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.237077 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7dae9ac-15d8-4751-87db-2ca6ed359f2e-maas-api-openshift-ca" pod "c7dae9ac-15d8-4751-87db-2ca6ed359f2e" (UID: "c7dae9ac-15d8-4751-87db-2ca6ed359f2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:25.237287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.237267 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7dae9ac-15d8-4751-87db-2ca6ed359f2e-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:25.238938 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.238913 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7dae9ac-15d8-4751-87db-2ca6ed359f2e-maas-api-xks-ca" pod "c7dae9ac-15d8-4751-87db-2ca6ed359f2e" (UID: "c7dae9ac-15d8-4751-87db-2ca6ed359f2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:25.239063 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.238942 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7dae9ac-15d8-4751-87db-2ca6ed359f2e-kube-api-access-97ql9" pod "c7dae9ac-15d8-4751-87db-2ca6ed359f2e" (UID: "c7dae9ac-15d8-4751-87db-2ca6ed359f2e"). InnerVolumeSpecName "kube-api-access-97ql9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:25.338149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.338119 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97ql9\" (UniqueName: \"kubernetes.io/projected/c7dae9ac-15d8-4751-87db-2ca6ed359f2e-kube-api-access-97ql9\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:25.338149 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.338149 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7dae9ac-15d8-4751-87db-2ca6ed359f2e-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:25.612314 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.612249 5059 scope.go:122] "RemoveContainer" containerID="bf14b10e6eed091103f0a91d03c839e8e63da4240c83391a08fcc3b818d4bdc8" Nov 03 20:40:25.626570 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.626544 5059 scope.go:122] "RemoveContainer" containerID="b3821841d889962619728e3cfd3742dffe6108759aefe0be61ddae8c78d55106" Nov 03 20:40:25.635576 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.635552 5059 scope.go:122] "RemoveContainer" containerID="9dfc60c513c642a762a15433fbf7ab06f633fb6d150365a70a84fd6ecc7aeee0" Nov 03 20:40:25.646313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.646286 5059 scope.go:122] "RemoveContainer" containerID="af2790ddecc29d9fe9f64c8e69ec5c674fb2d6254584e35b58eeafbca6d951e5" Nov 03 20:40:25.656775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.656753 5059 scope.go:122] "RemoveContainer" containerID="7dbfa1ff25e2a926d9d9feaa2ace0d152c5e6229ded2d8b9803f7a1302037649" Nov 03 20:40:25.667160 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.667137 5059 scope.go:122] "RemoveContainer" containerID="34b644e595dd4c7fa011e16e93e0d748f3b8f70ba73413e7d75dd921cc60c8b5" Nov 03 20:40:25.675699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.675676 5059 scope.go:122] "RemoveContainer" containerID="26f0370e8a062fc5d6b350cb6562de91ff7496aa5deead5b631ec0ec052ef2dc" Nov 03 20:40:25.683860 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.683836 5059 scope.go:122] "RemoveContainer" containerID="8452ce584ec710dade06364894274fbcd2c189f440545b82d81e7ad1fc47dfb7" Nov 03 20:40:25.694622 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.694603 5059 scope.go:122] "RemoveContainer" containerID="26d622763433e981c524cf0ddaec2006cc9ffcb67eb091acf6792a109cd90315" Nov 03 20:40:25.704767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.704748 5059 scope.go:122] "RemoveContainer" containerID="608a9becc0865d2fa635a042e78e2d293a90887169f9f7a9a78ac4715c431fc8" Nov 03 20:40:25.772873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.772851 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:40:25.843644 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.843610 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abce531c-32e5-4fee-b6fa-7b489da29704-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abce531c-32e5-4fee-b6fa-7b489da29704-istio-podinfo\") pod \"abce531c-32e5-4fee-b6fa-7b489da29704\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " Nov 03 20:40:25.843779 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.843723 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abce531c-32e5-4fee-b6fa-7b489da29704-istio-token\" (UniqueName: \"kubernetes.io/projected/abce531c-32e5-4fee-b6fa-7b489da29704-istio-token\") pod \"abce531c-32e5-4fee-b6fa-7b489da29704\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " Nov 03 20:40:25.843779 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.843761 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-istio-data\") pod \"abce531c-32e5-4fee-b6fa-7b489da29704\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " Nov 03 20:40:25.843974 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.843947 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-workload-certs\") pod \"abce531c-32e5-4fee-b6fa-7b489da29704\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " Nov 03 20:40:25.844071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.844000 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abce531c-32e5-4fee-b6fa-7b489da29704-kube-api-access-bxmp4\" (UniqueName: \"kubernetes.io/projected/abce531c-32e5-4fee-b6fa-7b489da29704-kube-api-access-bxmp4\") pod \"abce531c-32e5-4fee-b6fa-7b489da29704\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " Nov 03 20:40:25.844071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.844031 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-istio-data" pod "abce531c-32e5-4fee-b6fa-7b489da29704" (UID: "abce531c-32e5-4fee-b6fa-7b489da29704"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:25.844185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.844077 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-workload-socket\") pod \"abce531c-32e5-4fee-b6fa-7b489da29704\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " Nov 03 20:40:25.844185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.844108 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-credential-socket\") pod \"abce531c-32e5-4fee-b6fa-7b489da29704\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " Nov 03 20:40:25.844185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.844148 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-istio-envoy\") pod \"abce531c-32e5-4fee-b6fa-7b489da29704\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " Nov 03 20:40:25.844185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.844176 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-workload-certs" pod "abce531c-32e5-4fee-b6fa-7b489da29704" (UID: "abce531c-32e5-4fee-b6fa-7b489da29704"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:25.844399 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.844204 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abce531c-32e5-4fee-b6fa-7b489da29704-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abce531c-32e5-4fee-b6fa-7b489da29704-istiod-ca-cert\") pod \"abce531c-32e5-4fee-b6fa-7b489da29704\" (UID: \"abce531c-32e5-4fee-b6fa-7b489da29704\") " Nov 03 20:40:25.844399 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.844384 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-credential-socket" pod "abce531c-32e5-4fee-b6fa-7b489da29704" (UID: "abce531c-32e5-4fee-b6fa-7b489da29704"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:25.844593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.844565 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-workload-socket" pod "abce531c-32e5-4fee-b6fa-7b489da29704" (UID: "abce531c-32e5-4fee-b6fa-7b489da29704"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:25.844725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.844637 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:25.844725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.844685 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:25.844725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.844702 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:25.844969 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.844906 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abce531c-32e5-4fee-b6fa-7b489da29704-istiod-ca-cert" pod "abce531c-32e5-4fee-b6fa-7b489da29704" (UID: "abce531c-32e5-4fee-b6fa-7b489da29704"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:25.846179 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.846147 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abce531c-32e5-4fee-b6fa-7b489da29704-istio-podinfo" pod "abce531c-32e5-4fee-b6fa-7b489da29704" (UID: "abce531c-32e5-4fee-b6fa-7b489da29704"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:40:25.846642 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.846621 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abce531c-32e5-4fee-b6fa-7b489da29704-istio-token" pod "abce531c-32e5-4fee-b6fa-7b489da29704" (UID: "abce531c-32e5-4fee-b6fa-7b489da29704"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:25.846852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.846835 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abce531c-32e5-4fee-b6fa-7b489da29704-kube-api-access-bxmp4" pod "abce531c-32e5-4fee-b6fa-7b489da29704" (UID: "abce531c-32e5-4fee-b6fa-7b489da29704"). InnerVolumeSpecName "kube-api-access-bxmp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:25.847087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.847057 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-istio-envoy" pod "abce531c-32e5-4fee-b6fa-7b489da29704" (UID: "abce531c-32e5-4fee-b6fa-7b489da29704"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:25.858544 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.858512 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a004055f-9ef2-4be0-b531-6a65b0caa7c2" path="/var/lib/kubelet/pods/a004055f-9ef2-4be0-b531-6a65b0caa7c2/volumes" Nov 03 20:40:25.945807 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.945778 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:25.945913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.945812 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abce531c-32e5-4fee-b6fa-7b489da29704-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:25.945913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.945827 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abce531c-32e5-4fee-b6fa-7b489da29704-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:25.945913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.945839 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abce531c-32e5-4fee-b6fa-7b489da29704-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:25.945913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.945853 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxmp4\" (UniqueName: \"kubernetes.io/projected/abce531c-32e5-4fee-b6fa-7b489da29704-kube-api-access-bxmp4\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:25.945913 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:25.945869 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abce531c-32e5-4fee-b6fa-7b489da29704-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:26.013374 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:26.013340 5059 generic.go:356] "Generic (PLEG): container finished" podID="abce531c-32e5-4fee-b6fa-7b489da29704" containerID="b45a6bff1af76d642a374e7fb94b73d65da2201d87f7ddea070a688d25f2fb96" exitCode=0 Nov 03 20:40:26.013526 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:26.013386 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" Nov 03 20:40:26.013526 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:26.013422 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" event={"ID":"abce531c-32e5-4fee-b6fa-7b489da29704","Type":"ContainerDied","Data":"b45a6bff1af76d642a374e7fb94b73d65da2201d87f7ddea070a688d25f2fb96"} Nov 03 20:40:26.013526 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:26.013475 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz" event={"ID":"abce531c-32e5-4fee-b6fa-7b489da29704","Type":"ContainerDied","Data":"1bfeee2fc9ea6eb7dc1380fb81842c27e5c42d85ed11286b3d86d298a93ef9fb"} Nov 03 20:40:26.013526 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:26.013517 5059 scope.go:122] "RemoveContainer" containerID="b45a6bff1af76d642a374e7fb94b73d65da2201d87f7ddea070a688d25f2fb96" Nov 03 20:40:26.015086 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:26.015067 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e9c51f-2g5tq" Nov 03 20:40:26.015177 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:26.015072 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e9c51f-2g5tq" event={"ID":"c7dae9ac-15d8-4751-87db-2ca6ed359f2e","Type":"ContainerDied","Data":"6ff1f489bfb931d1f9bad41bb0624d69455c35bb1803cb243d02430843bb74b6"} Nov 03 20:40:26.015227 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:26.015179 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ff1f489bfb931d1f9bad41bb0624d69455c35bb1803cb243d02430843bb74b6" Nov 03 20:40:26.023685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:26.023647 5059 scope.go:122] "RemoveContainer" containerID="b45a6bff1af76d642a374e7fb94b73d65da2201d87f7ddea070a688d25f2fb96" Nov 03 20:40:26.024026 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:40:26.023986 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b45a6bff1af76d642a374e7fb94b73d65da2201d87f7ddea070a688d25f2fb96\": container with ID starting with b45a6bff1af76d642a374e7fb94b73d65da2201d87f7ddea070a688d25f2fb96 not found: ID does not exist" containerID="b45a6bff1af76d642a374e7fb94b73d65da2201d87f7ddea070a688d25f2fb96" Nov 03 20:40:26.024126 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:26.024025 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45a6bff1af76d642a374e7fb94b73d65da2201d87f7ddea070a688d25f2fb96"} err="failed to get container status \"b45a6bff1af76d642a374e7fb94b73d65da2201d87f7ddea070a688d25f2fb96\": rpc error: code = NotFound desc = could not find container \"b45a6bff1af76d642a374e7fb94b73d65da2201d87f7ddea070a688d25f2fb96\": container with ID starting with b45a6bff1af76d642a374e7fb94b73d65da2201d87f7ddea070a688d25f2fb96 not found: ID does not exist" Nov 03 20:40:26.028323 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:26.028303 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:26.028588 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:26.028570 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:26.029444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:26.029423 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:26.044097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:26.044075 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz"] Nov 03 20:40:26.050353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:26.050331 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-l7ghz"] Nov 03 20:40:27.880373 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:27.880330 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abce531c-32e5-4fee-b6fa-7b489da29704" path="/var/lib/kubelet/pods/abce531c-32e5-4fee-b6fa-7b489da29704/volumes" Nov 03 20:40:29.543725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:29.543685 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2"] Nov 03 20:40:29.544141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:29.543870 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" podUID="09d624de-215a-40be-9b38-0ba5a08743ef" containerName="istio-proxy" containerID="cri-o://2b4e52afd7a499fdc801448c104ffdb86585ae2b2371d359f5cc52f6255a59a8" gracePeriod=30 Nov 03 20:40:31.258860 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:31.258820 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-78xj2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:40:31.259366 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:31.258880 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" podUID="09d624de-215a-40be-9b38-0ba5a08743ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:40:32.163945 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.163903 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s"] Nov 03 20:40:32.164368 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.164347 5059 cpu_manager.go:414] "containerMap: removing container" podUID="abce531c-32e5-4fee-b6fa-7b489da29704" containerName="istio-proxy" Nov 03 20:40:32.164461 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.164371 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abce531c-32e5-4fee-b6fa-7b489da29704" containerName="istio-proxy" Nov 03 20:40:32.164461 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.164400 5059 cpu_manager.go:414] "containerMap: removing container" podUID="c7dae9ac-15d8-4751-87db-2ca6ed359f2e" containerName="revoke-keys" Nov 03 20:40:32.164461 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.164408 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7dae9ac-15d8-4751-87db-2ca6ed359f2e" containerName="revoke-keys" Nov 03 20:40:32.164621 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.164513 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7dae9ac-15d8-4751-87db-2ca6ed359f2e" containerName="revoke-keys" Nov 03 20:40:32.164621 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.164532 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="abce531c-32e5-4fee-b6fa-7b489da29704" containerName="istio-proxy" Nov 03 20:40:32.169819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.169796 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" Nov 03 20:40:32.191844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.187542 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s"] Nov 03 20:40:32.223494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.223463 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ce656e-8f66-4adc-9e57-d7d3b565e937-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s\" (UID: \"33ce656e-8f66-4adc-9e57-d7d3b565e937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" Nov 03 20:40:32.223635 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.223514 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8djjs\" (UniqueName: \"kubernetes.io/projected/33ce656e-8f66-4adc-9e57-d7d3b565e937-kube-api-access-8djjs\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s\" (UID: \"33ce656e-8f66-4adc-9e57-d7d3b565e937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" Nov 03 20:40:32.223725 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.223702 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33ce656e-8f66-4adc-9e57-d7d3b565e937-metrics-certs\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s\" (UID: \"33ce656e-8f66-4adc-9e57-d7d3b565e937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" Nov 03 20:40:32.223773 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.223764 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33ce656e-8f66-4adc-9e57-d7d3b565e937-maas-api-tls\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s\" (UID: \"33ce656e-8f66-4adc-9e57-d7d3b565e937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" Nov 03 20:40:32.237811 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.237779 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-2b5703-serving-cert\"" Nov 03 20:40:32.237919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.237857 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-942q2\"" Nov 03 20:40:32.237919 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.237857 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:40:32.325029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.324990 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ce656e-8f66-4adc-9e57-d7d3b565e937-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s\" (UID: \"33ce656e-8f66-4adc-9e57-d7d3b565e937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" Nov 03 20:40:32.325431 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.325071 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8djjs\" (UniqueName: \"kubernetes.io/projected/33ce656e-8f66-4adc-9e57-d7d3b565e937-kube-api-access-8djjs\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s\" (UID: \"33ce656e-8f66-4adc-9e57-d7d3b565e937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" Nov 03 20:40:32.325431 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.325148 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33ce656e-8f66-4adc-9e57-d7d3b565e937-metrics-certs\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s\" (UID: \"33ce656e-8f66-4adc-9e57-d7d3b565e937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" Nov 03 20:40:32.325431 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.325201 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33ce656e-8f66-4adc-9e57-d7d3b565e937-maas-api-tls\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s\" (UID: \"33ce656e-8f66-4adc-9e57-d7d3b565e937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" Nov 03 20:40:32.325431 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:40:32.325320 5059 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-serving-cert: secret "maas-api-e2e-worker-w2-2b5703-serving-cert" not found Nov 03 20:40:32.325431 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:40:32.325380 5059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33ce656e-8f66-4adc-9e57-d7d3b565e937-maas-api-tls podName:33ce656e-8f66-4adc-9e57-d7d3b565e937 nodeName:}" failed. No retries permitted until 2026-11-03 20:40:32.825363707 +0000 UTC m=+3827.083160942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33ce656e-8f66-4adc-9e57-d7d3b565e937-maas-api-tls") pod "maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" (UID: "33ce656e-8f66-4adc-9e57-d7d3b565e937") : secret "maas-api-e2e-worker-w2-2b5703-serving-cert" not found Nov 03 20:40:32.326208 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.326177 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ce656e-8f66-4adc-9e57-d7d3b565e937-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s\" (UID: \"33ce656e-8f66-4adc-9e57-d7d3b565e937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" Nov 03 20:40:32.329085 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.329058 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33ce656e-8f66-4adc-9e57-d7d3b565e937-metrics-certs\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s\" (UID: \"33ce656e-8f66-4adc-9e57-d7d3b565e937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" Nov 03 20:40:32.339100 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.339070 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8djjs\" (UniqueName: \"kubernetes.io/projected/33ce656e-8f66-4adc-9e57-d7d3b565e937-kube-api-access-8djjs\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s\" (UID: \"33ce656e-8f66-4adc-9e57-d7d3b565e937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" Nov 03 20:40:32.831075 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.831025 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33ce656e-8f66-4adc-9e57-d7d3b565e937-maas-api-tls\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s\" (UID: \"33ce656e-8f66-4adc-9e57-d7d3b565e937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" Nov 03 20:40:32.834045 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:32.834013 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33ce656e-8f66-4adc-9e57-d7d3b565e937-maas-api-tls\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s\" (UID: \"33ce656e-8f66-4adc-9e57-d7d3b565e937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" Nov 03 20:40:33.099528 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:33.099443 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" Nov 03 20:40:33.389916 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:33.389888 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s"] Nov 03 20:40:33.391233 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:40:33.391203 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ce656e_8f66_4adc_9e57_d7d3b565e937.slice/crio-6d97f8c8265e643b31584b5888a98169dbee4c9cb851a64c6d1470f519413d81 WatchSource:0}: Error finding container 6d97f8c8265e643b31584b5888a98169dbee4c9cb851a64c6d1470f519413d81: Status 404 returned error can't find the container with id 6d97f8c8265e643b31584b5888a98169dbee4c9cb851a64c6d1470f519413d81 Nov 03 20:40:33.392513 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:33.392493 5059 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:40:34.060406 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.060365 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" event={"ID":"33ce656e-8f66-4adc-9e57-d7d3b565e937","Type":"ContainerStarted","Data":"912a8cea40988edb7390e043dc3c81ea0713bc051eb2f4f042803ca4e810b145"} Nov 03 20:40:34.060559 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.060437 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" event={"ID":"33ce656e-8f66-4adc-9e57-d7d3b565e937","Type":"ContainerStarted","Data":"6d97f8c8265e643b31584b5888a98169dbee4c9cb851a64c6d1470f519413d81"} Nov 03 20:40:34.060559 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.060471 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" Nov 03 20:40:34.083139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.083091 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" podStartSLOduration=1.581017659 podStartE2EDuration="2.083078111s" totalImagesPullingTime="502.060452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:34.080559451 +0000 UTC m=+3828.338356708" watchObservedRunningTime="2026-11-03 20:40:34.083078111 +0000 UTC m=+3828.340875367" Nov 03 20:40:34.795529 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.795498 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:40:34.850262 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.850232 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-workload-certs\") pod \"09d624de-215a-40be-9b38-0ba5a08743ef\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " Nov 03 20:40:34.850430 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.850290 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d624de-215a-40be-9b38-0ba5a08743ef-istio-token\" (UniqueName: \"kubernetes.io/projected/09d624de-215a-40be-9b38-0ba5a08743ef-istio-token\") pod \"09d624de-215a-40be-9b38-0ba5a08743ef\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " Nov 03 20:40:34.850430 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.850342 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09d624de-215a-40be-9b38-0ba5a08743ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09d624de-215a-40be-9b38-0ba5a08743ef-istiod-ca-cert\") pod \"09d624de-215a-40be-9b38-0ba5a08743ef\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " Nov 03 20:40:34.850430 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.850374 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09d624de-215a-40be-9b38-0ba5a08743ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09d624de-215a-40be-9b38-0ba5a08743ef-istio-podinfo\") pod \"09d624de-215a-40be-9b38-0ba5a08743ef\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " Nov 03 20:40:34.850618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.850446 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d624de-215a-40be-9b38-0ba5a08743ef-kube-api-access-c2lp6\" (UniqueName: \"kubernetes.io/projected/09d624de-215a-40be-9b38-0ba5a08743ef-kube-api-access-c2lp6\") pod \"09d624de-215a-40be-9b38-0ba5a08743ef\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " Nov 03 20:40:34.850618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.850485 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-workload-socket\") pod \"09d624de-215a-40be-9b38-0ba5a08743ef\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " Nov 03 20:40:34.850618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.850505 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-workload-certs" pod "09d624de-215a-40be-9b38-0ba5a08743ef" (UID: "09d624de-215a-40be-9b38-0ba5a08743ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:34.850618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.850512 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-istio-envoy\") pod \"09d624de-215a-40be-9b38-0ba5a08743ef\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " Nov 03 20:40:34.850914 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.850871 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-workload-socket" pod "09d624de-215a-40be-9b38-0ba5a08743ef" (UID: "09d624de-215a-40be-9b38-0ba5a08743ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:34.850982 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.850776 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d624de-215a-40be-9b38-0ba5a08743ef-istiod-ca-cert" pod "09d624de-215a-40be-9b38-0ba5a08743ef" (UID: "09d624de-215a-40be-9b38-0ba5a08743ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:34.851161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.851131 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-istio-data" pod "09d624de-215a-40be-9b38-0ba5a08743ef" (UID: "09d624de-215a-40be-9b38-0ba5a08743ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:34.851247 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.850575 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-istio-data\") pod \"09d624de-215a-40be-9b38-0ba5a08743ef\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " Nov 03 20:40:34.851306 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.851252 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-credential-socket\") pod \"09d624de-215a-40be-9b38-0ba5a08743ef\" (UID: \"09d624de-215a-40be-9b38-0ba5a08743ef\") " Nov 03 20:40:34.851548 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.851520 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-credential-socket" pod "09d624de-215a-40be-9b38-0ba5a08743ef" (UID: "09d624de-215a-40be-9b38-0ba5a08743ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:34.851700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.851655 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:34.851784 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.851707 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:34.851784 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.851725 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:34.851784 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.851740 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:34.851784 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.851754 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09d624de-215a-40be-9b38-0ba5a08743ef-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:34.853086 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.853061 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09d624de-215a-40be-9b38-0ba5a08743ef-istio-podinfo" pod "09d624de-215a-40be-9b38-0ba5a08743ef" (UID: "09d624de-215a-40be-9b38-0ba5a08743ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:40:34.853197 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.853081 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d624de-215a-40be-9b38-0ba5a08743ef-istio-token" pod "09d624de-215a-40be-9b38-0ba5a08743ef" (UID: "09d624de-215a-40be-9b38-0ba5a08743ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:34.853337 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.853313 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-istio-envoy" pod "09d624de-215a-40be-9b38-0ba5a08743ef" (UID: "09d624de-215a-40be-9b38-0ba5a08743ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:34.853410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.853315 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d624de-215a-40be-9b38-0ba5a08743ef-kube-api-access-c2lp6" pod "09d624de-215a-40be-9b38-0ba5a08743ef" (UID: "09d624de-215a-40be-9b38-0ba5a08743ef"). InnerVolumeSpecName "kube-api-access-c2lp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:34.952449 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.952404 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2lp6\" (UniqueName: \"kubernetes.io/projected/09d624de-215a-40be-9b38-0ba5a08743ef-kube-api-access-c2lp6\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:34.952449 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.952444 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09d624de-215a-40be-9b38-0ba5a08743ef-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:34.952699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.952460 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09d624de-215a-40be-9b38-0ba5a08743ef-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:34.952699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:34.952475 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09d624de-215a-40be-9b38-0ba5a08743ef-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:35.066890 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:35.066859 5059 generic.go:356] "Generic (PLEG): container finished" podID="09d624de-215a-40be-9b38-0ba5a08743ef" containerID="2b4e52afd7a499fdc801448c104ffdb86585ae2b2371d359f5cc52f6255a59a8" exitCode=0 Nov 03 20:40:35.067019 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:35.066907 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" Nov 03 20:40:35.067019 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:35.066944 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" event={"ID":"09d624de-215a-40be-9b38-0ba5a08743ef","Type":"ContainerDied","Data":"2b4e52afd7a499fdc801448c104ffdb86585ae2b2371d359f5cc52f6255a59a8"} Nov 03 20:40:35.067019 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:35.066997 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2" event={"ID":"09d624de-215a-40be-9b38-0ba5a08743ef","Type":"ContainerDied","Data":"e333907c379f464c17e6a1b2bdafaba5c2ec787715543737b32901885a0ca269"} Nov 03 20:40:35.067125 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:35.067020 5059 scope.go:122] "RemoveContainer" containerID="2b4e52afd7a499fdc801448c104ffdb86585ae2b2371d359f5cc52f6255a59a8" Nov 03 20:40:35.077219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:35.077196 5059 scope.go:122] "RemoveContainer" containerID="2b4e52afd7a499fdc801448c104ffdb86585ae2b2371d359f5cc52f6255a59a8" Nov 03 20:40:35.077466 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:40:35.077446 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b4e52afd7a499fdc801448c104ffdb86585ae2b2371d359f5cc52f6255a59a8\": container with ID starting with 2b4e52afd7a499fdc801448c104ffdb86585ae2b2371d359f5cc52f6255a59a8 not found: ID does not exist" containerID="2b4e52afd7a499fdc801448c104ffdb86585ae2b2371d359f5cc52f6255a59a8" Nov 03 20:40:35.077522 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:35.077476 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b4e52afd7a499fdc801448c104ffdb86585ae2b2371d359f5cc52f6255a59a8"} err="failed to get container status \"2b4e52afd7a499fdc801448c104ffdb86585ae2b2371d359f5cc52f6255a59a8\": rpc error: code = NotFound desc = could not find container \"2b4e52afd7a499fdc801448c104ffdb86585ae2b2371d359f5cc52f6255a59a8\": container with ID starting with 2b4e52afd7a499fdc801448c104ffdb86585ae2b2371d359f5cc52f6255a59a8 not found: ID does not exist" Nov 03 20:40:35.091303 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:35.091281 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2"] Nov 03 20:40:35.095837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:35.095819 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-78xj2"] Nov 03 20:40:35.856553 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:35.856512 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09d624de-215a-40be-9b38-0ba5a08743ef" path="/var/lib/kubelet/pods/09d624de-215a-40be-9b38-0ba5a08743ef/volumes" Nov 03 20:40:39.045991 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.045955 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch"] Nov 03 20:40:39.046402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.046386 5059 cpu_manager.go:414] "containerMap: removing container" podUID="09d624de-215a-40be-9b38-0ba5a08743ef" containerName="istio-proxy" Nov 03 20:40:39.046447 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.046403 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d624de-215a-40be-9b38-0ba5a08743ef" containerName="istio-proxy" Nov 03 20:40:39.046485 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.046478 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d624de-215a-40be-9b38-0ba5a08743ef" containerName="istio-proxy" Nov 03 20:40:39.051095 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.051072 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.057631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.057608 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-32e482-openshift-default-dockercfg-5pv2v\"" Nov 03 20:40:39.065078 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.065049 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd"] Nov 03 20:40:39.070339 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.070317 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw"] Nov 03 20:40:39.070481 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.070457 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.074180 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.074160 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch"] Nov 03 20:40:39.074308 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.074265 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.081410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.081385 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw"] Nov 03 20:40:39.086141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.086120 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd"] Nov 03 20:40:39.102173 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.102141 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b46nt\" (UniqueName: \"kubernetes.io/projected/044fe062-415e-43bf-a759-fc5ed2f81610-kube-api-access-b46nt\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.102408 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.102209 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.102408 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.102310 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.102408 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.102349 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.102408 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.102373 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.102408 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.102393 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.102715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.102415 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/044fe062-415e-43bf-a759-fc5ed2f81610-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.102715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.102442 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/044fe062-415e-43bf-a759-fc5ed2f81610-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.102715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.102569 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/044fe062-415e-43bf-a759-fc5ed2f81610-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.203320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.203285 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/044fe062-415e-43bf-a759-fc5ed2f81610-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.203489 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.203327 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.203489 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.203449 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.203567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.203531 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b46nt\" (UniqueName: \"kubernetes.io/projected/044fe062-415e-43bf-a759-fc5ed2f81610-kube-api-access-b46nt\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.203611 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.203566 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.203611 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.203585 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.203611 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.203608 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.203782 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.203626 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.203782 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.203713 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.203864 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.203791 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2beae172-d2d5-4e41-97fb-5772fca8e05d-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.203864 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.203817 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x74r7\" (UniqueName: \"kubernetes.io/projected/4717868b-fcbe-4f25-ae92-ab98166199d8-kube-api-access-x74r7\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.203864 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.203842 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.204017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.203924 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.204017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.203961 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.204017 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.203994 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.204184 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.204058 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.204184 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.204084 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.204184 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.204132 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.204320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.204231 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.204320 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.204276 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/044fe062-415e-43bf-a759-fc5ed2f81610-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.204410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.204308 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.204410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.204313 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.204410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.204359 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.204410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.204387 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/044fe062-415e-43bf-a759-fc5ed2f81610-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.204592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.204435 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4717868b-fcbe-4f25-ae92-ab98166199d8-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.204592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.204458 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.204592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.204482 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.204592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.204534 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hsq4\" (UniqueName: \"kubernetes.io/projected/2beae172-d2d5-4e41-97fb-5772fca8e05d-kube-api-access-8hsq4\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.204592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.204570 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.204900 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.204603 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.204900 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.204676 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.205208 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.205185 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/044fe062-415e-43bf-a759-fc5ed2f81610-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.206153 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.206133 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/044fe062-415e-43bf-a759-fc5ed2f81610-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.206255 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.206232 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.215843 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.215814 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/044fe062-415e-43bf-a759-fc5ed2f81610-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.215843 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.215827 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b46nt\" (UniqueName: \"kubernetes.io/projected/044fe062-415e-43bf-a759-fc5ed2f81610-kube-api-access-b46nt\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.305449 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.305356 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2beae172-d2d5-4e41-97fb-5772fca8e05d-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.305449 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.305397 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x74r7\" (UniqueName: \"kubernetes.io/projected/4717868b-fcbe-4f25-ae92-ab98166199d8-kube-api-access-x74r7\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.305449 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.305426 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.305449 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.305448 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.305883 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.305470 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.305883 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.305488 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.305883 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.305517 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.305883 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.305559 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4717868b-fcbe-4f25-ae92-ab98166199d8-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.305883 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.305599 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.305883 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.305627 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hsq4\" (UniqueName: \"kubernetes.io/projected/2beae172-d2d5-4e41-97fb-5772fca8e05d-kube-api-access-8hsq4\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.305883 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.305683 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.306259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.305919 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.306259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.305984 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.306259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.305989 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.306259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.306022 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.306259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.306022 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.306259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.306063 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.306259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.306110 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.306259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.306161 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.306259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.306192 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.306259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.306256 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.306805 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.306285 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.306805 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.306292 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.306805 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.306424 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.306805 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.306459 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.306805 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.306493 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.306805 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.306621 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4717868b-fcbe-4f25-ae92-ab98166199d8-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.307129 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.306972 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2beae172-d2d5-4e41-97fb-5772fca8e05d-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.308432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.308408 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.308595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.308577 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.308873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.308853 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.309205 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.309184 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.319364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.319287 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.319364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.319336 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.319364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.319339 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x74r7\" (UniqueName: \"kubernetes.io/projected/4717868b-fcbe-4f25-ae92-ab98166199d8-kube-api-access-x74r7\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.319722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.319705 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hsq4\" (UniqueName: \"kubernetes.io/projected/2beae172-d2d5-4e41-97fb-5772fca8e05d-kube-api-access-8hsq4\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.346704 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.346681 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14917091-d5vtz"] Nov 03 20:40:39.351797 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.351781 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14917091-d5vtz" Nov 03 20:40:39.356174 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.356147 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:40:39.356313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.356239 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\"" Nov 03 20:40:39.359827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.359804 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14917091-d5vtz"] Nov 03 20:40:39.367400 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.367376 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:39.393409 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.393387 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:39.397296 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.397272 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:39.408336 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.408169 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/998d8105-98ae-451b-8d6f-639a2c61bf12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14917091-d5vtz\" (UID: \"998d8105-98ae-451b-8d6f-639a2c61bf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14917091-d5vtz" Nov 03 20:40:39.408336 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.408266 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zqsq\" (UniqueName: \"kubernetes.io/projected/998d8105-98ae-451b-8d6f-639a2c61bf12-kube-api-access-2zqsq\") pod \"maas-api-revoke-sub-14917091-d5vtz\" (UID: \"998d8105-98ae-451b-8d6f-639a2c61bf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14917091-d5vtz" Nov 03 20:40:39.408336 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.408312 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/998d8105-98ae-451b-8d6f-639a2c61bf12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14917091-d5vtz\" (UID: \"998d8105-98ae-451b-8d6f-639a2c61bf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14917091-d5vtz" Nov 03 20:40:39.492604 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.492575 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q"] Nov 03 20:40:39.507538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.506280 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb"] Nov 03 20:40:39.509446 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.509399 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/998d8105-98ae-451b-8d6f-639a2c61bf12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14917091-d5vtz\" (UID: \"998d8105-98ae-451b-8d6f-639a2c61bf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14917091-d5vtz" Nov 03 20:40:39.509555 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.509520 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zqsq\" (UniqueName: \"kubernetes.io/projected/998d8105-98ae-451b-8d6f-639a2c61bf12-kube-api-access-2zqsq\") pod \"maas-api-revoke-sub-14917091-d5vtz\" (UID: \"998d8105-98ae-451b-8d6f-639a2c61bf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14917091-d5vtz" Nov 03 20:40:39.509630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.509554 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/998d8105-98ae-451b-8d6f-639a2c61bf12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14917091-d5vtz\" (UID: \"998d8105-98ae-451b-8d6f-639a2c61bf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14917091-d5vtz" Nov 03 20:40:39.510267 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.510248 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/998d8105-98ae-451b-8d6f-639a2c61bf12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14917091-d5vtz\" (UID: \"998d8105-98ae-451b-8d6f-639a2c61bf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14917091-d5vtz" Nov 03 20:40:39.512925 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.512899 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/998d8105-98ae-451b-8d6f-639a2c61bf12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14917091-d5vtz\" (UID: \"998d8105-98ae-451b-8d6f-639a2c61bf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14917091-d5vtz" Nov 03 20:40:39.526244 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.526218 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zqsq\" (UniqueName: \"kubernetes.io/projected/998d8105-98ae-451b-8d6f-639a2c61bf12-kube-api-access-2zqsq\") pod \"maas-api-revoke-sub-14917091-d5vtz\" (UID: \"998d8105-98ae-451b-8d6f-639a2c61bf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14917091-d5vtz" Nov 03 20:40:39.650601 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.650568 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch"] Nov 03 20:40:39.651762 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:40:39.651738 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044fe062_415e_43bf_a759_fc5ed2f81610.slice/crio-5bad706e57526771579a0aa03a11635f7682040328d08676619819b68de1780e WatchSource:0}: Error finding container 5bad706e57526771579a0aa03a11635f7682040328d08676619819b68de1780e: Status 404 returned error can't find the container with id 5bad706e57526771579a0aa03a11635f7682040328d08676619819b68de1780e Nov 03 20:40:39.654010 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.653978 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:40:39.654094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.654050 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:40:39.654094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.654081 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:40:39.669046 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.669023 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14917091-d5vtz" Nov 03 20:40:39.698818 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:40:39.698788 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2beae172_d2d5_4e41_97fb_5772fca8e05d.slice/crio-78d2b0c7d23c2d99d6872c8a523accdfa80a5b673d8ff92d30e01aae9fbd81e5 WatchSource:0}: Error finding container 78d2b0c7d23c2d99d6872c8a523accdfa80a5b673d8ff92d30e01aae9fbd81e5: Status 404 returned error can't find the container with id 78d2b0c7d23c2d99d6872c8a523accdfa80a5b673d8ff92d30e01aae9fbd81e5 Nov 03 20:40:39.698987 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.698952 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw"] Nov 03 20:40:39.701330 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.701296 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:40:39.701437 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.701400 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:40:39.701503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.701450 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:40:39.702000 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:40:39.701977 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4717868b_fcbe_4f25_ae92_ab98166199d8.slice/crio-fb1f07b9081559d0429cd6f964a03139d1374b68822e62a24b15da0b514d5637 WatchSource:0}: Error finding container fb1f07b9081559d0429cd6f964a03139d1374b68822e62a24b15da0b514d5637: Status 404 returned error can't find the container with id fb1f07b9081559d0429cd6f964a03139d1374b68822e62a24b15da0b514d5637 Nov 03 20:40:39.702923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.702900 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd"] Nov 03 20:40:39.704279 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.704246 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:40:39.704341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.704323 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:40:39.704393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.704374 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:40:39.973711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:39.973652 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14917091-d5vtz"] Nov 03 20:40:39.974607 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:40:39.974580 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998d8105_98ae_451b_8d6f_639a2c61bf12.slice/crio-ffa01adc8c1a86a49bd1e68d51bfc244e9cd24a7857038dec1f2f635fbd1015a WatchSource:0}: Error finding container ffa01adc8c1a86a49bd1e68d51bfc244e9cd24a7857038dec1f2f635fbd1015a: Status 404 returned error can't find the container with id ffa01adc8c1a86a49bd1e68d51bfc244e9cd24a7857038dec1f2f635fbd1015a Nov 03 20:40:40.074856 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:40.074826 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" Nov 03 20:40:40.107249 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:40.107207 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" event={"ID":"044fe062-415e-43bf-a759-fc5ed2f81610","Type":"ContainerStarted","Data":"fd0e55c0414776a8d531c0501b3753f49d70fa87c6521726b180fe554f5d3553"} Nov 03 20:40:40.107592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:40.107280 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" event={"ID":"044fe062-415e-43bf-a759-fc5ed2f81610","Type":"ContainerStarted","Data":"5bad706e57526771579a0aa03a11635f7682040328d08676619819b68de1780e"} Nov 03 20:40:40.110067 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:40.110023 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" event={"ID":"4717868b-fcbe-4f25-ae92-ab98166199d8","Type":"ContainerStarted","Data":"cf488f087c3e78d0028cc0d2e3cdb4283b6b8e552e2e983e51bf0cde78a34dcd"} Nov 03 20:40:40.110241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:40.110104 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" event={"ID":"4717868b-fcbe-4f25-ae92-ab98166199d8","Type":"ContainerStarted","Data":"fb1f07b9081559d0429cd6f964a03139d1374b68822e62a24b15da0b514d5637"} Nov 03 20:40:40.112251 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:40.112211 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14917091-d5vtz" event={"ID":"998d8105-98ae-451b-8d6f-639a2c61bf12","Type":"ContainerStarted","Data":"ffa01adc8c1a86a49bd1e68d51bfc244e9cd24a7857038dec1f2f635fbd1015a"} Nov 03 20:40:40.118560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:40.118522 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" event={"ID":"2beae172-d2d5-4e41-97fb-5772fca8e05d","Type":"ContainerStarted","Data":"b6d218d235549ee069a2989a61510bf7592be0af650590d12ee551d028bef486"} Nov 03 20:40:40.118720 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:40.118580 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" event={"ID":"2beae172-d2d5-4e41-97fb-5772fca8e05d","Type":"ContainerStarted","Data":"78d2b0c7d23c2d99d6872c8a523accdfa80a5b673d8ff92d30e01aae9fbd81e5"} Nov 03 20:40:40.175060 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:40.166094 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" podStartSLOduration=1.166070025 podStartE2EDuration="1.166070025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:40.141533593 +0000 UTC m=+3834.399330851" watchObservedRunningTime="2026-11-03 20:40:40.166070025 +0000 UTC m=+3834.423867282" Nov 03 20:40:40.226934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:40.226836 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" podStartSLOduration=1.226816992 podStartE2EDuration="1.226816992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:40.208014215 +0000 UTC m=+3834.465811477" watchObservedRunningTime="2026-11-03 20:40:40.226816992 +0000 UTC m=+3834.484614250" Nov 03 20:40:40.236352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:40.236299 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" podStartSLOduration=1.236281947 podStartE2EDuration="1.236281947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:40.233805981 +0000 UTC m=+3834.491603242" watchObservedRunningTime="2026-11-03 20:40:40.236281947 +0000 UTC m=+3834.494079203" Nov 03 20:40:40.368992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:40.368953 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:40.389858 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:40.389811 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused" start-of-body= Nov 03 20:40:40.390435 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:40.390400 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" podUID="044fe062-415e-43bf-a759-fc5ed2f81610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused" Nov 03 20:40:40.395013 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:40.394987 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:40.402410 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:40.402381 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:40.402729 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:40.402707 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused" start-of-body= Nov 03 20:40:40.402886 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:40.402863 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" podUID="4717868b-fcbe-4f25-ae92-ab98166199d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused" Nov 03 20:40:40.403066 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:40.403049 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.180:15021/healthz/ready\": dial tcp 10.132.1.180:15021: connect: connection refused" start-of-body= Nov 03 20:40:40.403175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:40.403159 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" podUID="2beae172-d2d5-4e41-97fb-5772fca8e05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.180:15021/healthz/ready\": dial tcp 10.132.1.180:15021: connect: connection refused" Nov 03 20:40:40.711687 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:40:40.690419 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:40:41.147293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:41.146927 5059 generic.go:356] "Generic (PLEG): container finished" podID="998d8105-98ae-451b-8d6f-639a2c61bf12" containerID="d721492c32ba0aa19654d57c84d125b4b5495409ea04136336c07ae71e367369" exitCode=0 Nov 03 20:40:41.149963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:41.148428 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14917091-d5vtz" event={"ID":"998d8105-98ae-451b-8d6f-639a2c61bf12","Type":"ContainerDied","Data":"d721492c32ba0aa19654d57c84d125b4b5495409ea04136336c07ae71e367369"} Nov 03 20:40:41.374123 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:41.374076 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused" start-of-body= Nov 03 20:40:41.377686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:41.374401 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" podUID="044fe062-415e-43bf-a759-fc5ed2f81610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused" Nov 03 20:40:41.421685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:41.414360 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused" start-of-body= Nov 03 20:40:41.421976 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:41.421939 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" podUID="4717868b-fcbe-4f25-ae92-ab98166199d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused" Nov 03 20:40:41.438260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:41.425147 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.180:15021/healthz/ready\": dial tcp 10.132.1.180:15021: connect: connection refused" start-of-body= Nov 03 20:40:41.438260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:41.437807 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" podUID="2beae172-d2d5-4e41-97fb-5772fca8e05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.180:15021/healthz/ready\": dial tcp 10.132.1.180:15021: connect: connection refused" Nov 03 20:40:42.426686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:42.382914 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused" start-of-body= Nov 03 20:40:42.426686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:42.382981 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" podUID="044fe062-415e-43bf-a759-fc5ed2f81610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused" Nov 03 20:40:42.442687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:42.434702 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.180:15021/healthz/ready\": dial tcp 10.132.1.180:15021: connect: connection refused" start-of-body= Nov 03 20:40:42.442687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:42.434788 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" podUID="2beae172-d2d5-4e41-97fb-5772fca8e05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.180:15021/healthz/ready\": dial tcp 10.132.1.180:15021: connect: connection refused" Nov 03 20:40:42.442687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:42.436970 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused" start-of-body= Nov 03 20:40:42.442687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:42.437017 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" podUID="4717868b-fcbe-4f25-ae92-ab98166199d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused" Nov 03 20:40:43.136819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:43.134387 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14917091-d5vtz" Nov 03 20:40:43.153473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:43.153439 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998d8105-98ae-451b-8d6f-639a2c61bf12-kube-api-access-2zqsq\" (UniqueName: \"kubernetes.io/projected/998d8105-98ae-451b-8d6f-639a2c61bf12-kube-api-access-2zqsq\") pod \"998d8105-98ae-451b-8d6f-639a2c61bf12\" (UID: \"998d8105-98ae-451b-8d6f-639a2c61bf12\") " Nov 03 20:40:43.153809 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:43.153781 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/998d8105-98ae-451b-8d6f-639a2c61bf12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/998d8105-98ae-451b-8d6f-639a2c61bf12-maas-api-xks-ca\") pod \"998d8105-98ae-451b-8d6f-639a2c61bf12\" (UID: \"998d8105-98ae-451b-8d6f-639a2c61bf12\") " Nov 03 20:40:43.155224 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:43.155201 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/998d8105-98ae-451b-8d6f-639a2c61bf12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/998d8105-98ae-451b-8d6f-639a2c61bf12-maas-api-openshift-ca\") pod \"998d8105-98ae-451b-8d6f-639a2c61bf12\" (UID: \"998d8105-98ae-451b-8d6f-639a2c61bf12\") " Nov 03 20:40:43.155680 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:43.155584 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998d8105-98ae-451b-8d6f-639a2c61bf12-maas-api-openshift-ca" pod "998d8105-98ae-451b-8d6f-639a2c61bf12" (UID: "998d8105-98ae-451b-8d6f-639a2c61bf12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:43.156036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:43.156003 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/998d8105-98ae-451b-8d6f-639a2c61bf12-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:43.157389 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:43.157352 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998d8105-98ae-451b-8d6f-639a2c61bf12-kube-api-access-2zqsq" pod "998d8105-98ae-451b-8d6f-639a2c61bf12" (UID: "998d8105-98ae-451b-8d6f-639a2c61bf12"). InnerVolumeSpecName "kube-api-access-2zqsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:43.172831 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:43.172784 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/998d8105-98ae-451b-8d6f-639a2c61bf12-maas-api-xks-ca" pod "998d8105-98ae-451b-8d6f-639a2c61bf12" (UID: "998d8105-98ae-451b-8d6f-639a2c61bf12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:43.257074 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:43.256996 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zqsq\" (UniqueName: \"kubernetes.io/projected/998d8105-98ae-451b-8d6f-639a2c61bf12-kube-api-access-2zqsq\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:43.257074 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:43.257024 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/998d8105-98ae-451b-8d6f-639a2c61bf12-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:43.317697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:43.313573 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14917091-d5vtz" event={"ID":"998d8105-98ae-451b-8d6f-639a2c61bf12","Type":"ContainerDied","Data":"ffa01adc8c1a86a49bd1e68d51bfc244e9cd24a7857038dec1f2f635fbd1015a"} Nov 03 20:40:43.317697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:43.313640 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffa01adc8c1a86a49bd1e68d51bfc244e9cd24a7857038dec1f2f635fbd1015a" Nov 03 20:40:43.317697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:43.313735 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14917091-d5vtz" Nov 03 20:40:43.375484 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:43.375035 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused" start-of-body= Nov 03 20:40:43.375484 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:43.375102 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" podUID="044fe062-415e-43bf-a759-fc5ed2f81610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused" Nov 03 20:40:43.407739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:43.407333 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused" start-of-body= Nov 03 20:40:43.407739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:43.407392 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" podUID="4717868b-fcbe-4f25-ae92-ab98166199d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused" Nov 03 20:40:43.407999 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:43.407967 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.180:15021/healthz/ready\": dial tcp 10.132.1.180:15021: connect: connection refused" start-of-body= Nov 03 20:40:43.408066 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:43.408010 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" podUID="2beae172-d2d5-4e41-97fb-5772fca8e05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.180:15021/healthz/ready\": dial tcp 10.132.1.180:15021: connect: connection refused" Nov 03 20:40:44.370088 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:44.370045 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused" start-of-body= Nov 03 20:40:44.371174 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:44.371137 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" podUID="044fe062-415e-43bf-a759-fc5ed2f81610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused" Nov 03 20:40:44.402686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:44.402263 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.180:15021/healthz/ready\": dial tcp 10.132.1.180:15021: connect: connection refused" start-of-body= Nov 03 20:40:44.402686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:44.402336 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" podUID="2beae172-d2d5-4e41-97fb-5772fca8e05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.180:15021/healthz/ready\": dial tcp 10.132.1.180:15021: connect: connection refused" Nov 03 20:40:44.403329 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:44.403300 5059 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused" start-of-body= Nov 03 20:40:44.403540 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:44.403517 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" podUID="4717868b-fcbe-4f25-ae92-ab98166199d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.179:15021/healthz/ready\": dial tcp 10.132.1.179:15021: connect: connection refused" Nov 03 20:40:44.494924 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:44.494722 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q" podUID="8e9aefe5-8450-4eb9-8d8b-df89be81ebb3" containerName="payload-processing" containerID="cri-o://0c8c232fc6576eb6c8950c19f0d5c165e8a2cdcac6e16524643b9275e091b3df" gracePeriod=25 Nov 03 20:40:44.525163 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:44.518159 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb" podUID="26576b40-fd9c-4298-9129-5b341d2c0b3d" containerName="payload-pre-processing" containerID="cri-o://99a0d9d2e14726b927ef1738be56c6ac25f24a2999b1b04a05311141ddb6c181" gracePeriod=25 Nov 03 20:40:45.193687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.184294 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch"] Nov 03 20:40:45.193687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.184679 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" podUID="044fe062-415e-43bf-a759-fc5ed2f81610" containerName="istio-proxy" containerID="cri-o://fd0e55c0414776a8d531c0501b3753f49d70fa87c6521726b180fe554f5d3553" gracePeriod=30 Nov 03 20:40:45.265991 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.257369 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd"] Nov 03 20:40:45.287873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.282098 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" podUID="4717868b-fcbe-4f25-ae92-ab98166199d8" containerName="istio-proxy" containerID="cri-o://cf488f087c3e78d0028cc0d2e3cdb4283b6b8e552e2e983e51bf0cde78a34dcd" gracePeriod=30 Nov 03 20:40:45.301283 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.293250 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw"] Nov 03 20:40:45.301283 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.293517 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" podUID="2beae172-d2d5-4e41-97fb-5772fca8e05d" containerName="istio-proxy" containerID="cri-o://b6d218d235549ee069a2989a61510bf7592be0af650590d12ee551d028bef486" gracePeriod=30 Nov 03 20:40:45.355707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.355646 5059 generic.go:356] "Generic (PLEG): container finished" podID="26576b40-fd9c-4298-9129-5b341d2c0b3d" containerID="99a0d9d2e14726b927ef1738be56c6ac25f24a2999b1b04a05311141ddb6c181" exitCode=0 Nov 03 20:40:45.355871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.355823 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb" event={"ID":"26576b40-fd9c-4298-9129-5b341d2c0b3d","Type":"ContainerDied","Data":"99a0d9d2e14726b927ef1738be56c6ac25f24a2999b1b04a05311141ddb6c181"} Nov 03 20:40:45.355950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.355874 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb" event={"ID":"26576b40-fd9c-4298-9129-5b341d2c0b3d","Type":"ContainerDied","Data":"843a76e69bdcc8b626cf0ba45546fd6598ae7eb974cbddb9144d57af0fad2b12"} Nov 03 20:40:45.355950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.355898 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="843a76e69bdcc8b626cf0ba45546fd6598ae7eb974cbddb9144d57af0fad2b12" Nov 03 20:40:45.356076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.356057 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb" Nov 03 20:40:45.357833 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.357808 5059 generic.go:356] "Generic (PLEG): container finished" podID="8e9aefe5-8450-4eb9-8d8b-df89be81ebb3" containerID="0c8c232fc6576eb6c8950c19f0d5c165e8a2cdcac6e16524643b9275e091b3df" exitCode=0 Nov 03 20:40:45.377853 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.373971 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q" event={"ID":"8e9aefe5-8450-4eb9-8d8b-df89be81ebb3","Type":"ContainerDied","Data":"0c8c232fc6576eb6c8950c19f0d5c165e8a2cdcac6e16524643b9275e091b3df"} Nov 03 20:40:45.438804 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.438763 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26576b40-fd9c-4298-9129-5b341d2c0b3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26576b40-fd9c-4298-9129-5b341d2c0b3d-plugins-config-volume\") pod \"26576b40-fd9c-4298-9129-5b341d2c0b3d\" (UID: \"26576b40-fd9c-4298-9129-5b341d2c0b3d\") " Nov 03 20:40:45.444999 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.444783 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26576b40-fd9c-4298-9129-5b341d2c0b3d-plugins-config-volume" pod "26576b40-fd9c-4298-9129-5b341d2c0b3d" (UID: "26576b40-fd9c-4298-9129-5b341d2c0b3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:45.445530 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.445510 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26576b40-fd9c-4298-9129-5b341d2c0b3d-kube-api-access-htk2g\" (UniqueName: \"kubernetes.io/projected/26576b40-fd9c-4298-9129-5b341d2c0b3d-kube-api-access-htk2g\") pod \"26576b40-fd9c-4298-9129-5b341d2c0b3d\" (UID: \"26576b40-fd9c-4298-9129-5b341d2c0b3d\") " Nov 03 20:40:45.446337 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.446315 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26576b40-fd9c-4298-9129-5b341d2c0b3d-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:45.449693 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.449232 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26576b40-fd9c-4298-9129-5b341d2c0b3d-kube-api-access-htk2g" pod "26576b40-fd9c-4298-9129-5b341d2c0b3d" (UID: "26576b40-fd9c-4298-9129-5b341d2c0b3d"). InnerVolumeSpecName "kube-api-access-htk2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:45.544840 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.544810 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q" Nov 03 20:40:45.547497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.547468 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htk2g\" (UniqueName: \"kubernetes.io/projected/26576b40-fd9c-4298-9129-5b341d2c0b3d-kube-api-access-htk2g\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:45.648473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.648428 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e9aefe5-8450-4eb9-8d8b-df89be81ebb3-kube-api-access-dbg8q\" (UniqueName: \"kubernetes.io/projected/8e9aefe5-8450-4eb9-8d8b-df89be81ebb3-kube-api-access-dbg8q\") pod \"8e9aefe5-8450-4eb9-8d8b-df89be81ebb3\" (UID: \"8e9aefe5-8450-4eb9-8d8b-df89be81ebb3\") " Nov 03 20:40:45.648708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.648538 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e9aefe5-8450-4eb9-8d8b-df89be81ebb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9aefe5-8450-4eb9-8d8b-df89be81ebb3-plugins-config-volume\") pod \"8e9aefe5-8450-4eb9-8d8b-df89be81ebb3\" (UID: \"8e9aefe5-8450-4eb9-8d8b-df89be81ebb3\") " Nov 03 20:40:45.649316 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.649286 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e9aefe5-8450-4eb9-8d8b-df89be81ebb3-plugins-config-volume" pod "8e9aefe5-8450-4eb9-8d8b-df89be81ebb3" (UID: "8e9aefe5-8450-4eb9-8d8b-df89be81ebb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:45.652268 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.652241 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e9aefe5-8450-4eb9-8d8b-df89be81ebb3-kube-api-access-dbg8q" pod "8e9aefe5-8450-4eb9-8d8b-df89be81ebb3" (UID: "8e9aefe5-8450-4eb9-8d8b-df89be81ebb3"). InnerVolumeSpecName "kube-api-access-dbg8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:45.749572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.749479 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9aefe5-8450-4eb9-8d8b-df89be81ebb3-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:45.749572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:45.749517 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbg8q\" (UniqueName: \"kubernetes.io/projected/8e9aefe5-8450-4eb9-8d8b-df89be81ebb3-kube-api-access-dbg8q\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:46.388969 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:46.388932 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb" Nov 03 20:40:46.389765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:46.389576 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q" Nov 03 20:40:46.389907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:46.389779 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q" event={"ID":"8e9aefe5-8450-4eb9-8d8b-df89be81ebb3","Type":"ContainerDied","Data":"6c0c462e53b0f3e601530d1f63a4aa75eb98bbedf81934f22d8ce6d5eb1793b7"} Nov 03 20:40:46.389907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:46.389855 5059 scope.go:122] "RemoveContainer" containerID="0c8c232fc6576eb6c8950c19f0d5c165e8a2cdcac6e16524643b9275e091b3df" Nov 03 20:40:46.427277 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:46.427240 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb"] Nov 03 20:40:46.435569 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:46.435540 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9f46e5-7b7655d547-smrhb"] Nov 03 20:40:46.449817 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:46.449789 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q"] Nov 03 20:40:46.456701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:46.456656 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9f46e5-74d4f56bf8-b6x8q"] Nov 03 20:40:47.855515 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:47.855483 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26576b40-fd9c-4298-9129-5b341d2c0b3d" path="/var/lib/kubelet/pods/26576b40-fd9c-4298-9129-5b341d2c0b3d/volumes" Nov 03 20:40:47.855957 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:47.855941 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e9aefe5-8450-4eb9-8d8b-df89be81ebb3" path="/var/lib/kubelet/pods/8e9aefe5-8450-4eb9-8d8b-df89be81ebb3/volumes" Nov 03 20:40:51.474741 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:40:51.474707 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044fe062_415e_43bf_a759_fc5ed2f81610.slice/crio-fd0e55c0414776a8d531c0501b3753f49d70fa87c6521726b180fe554f5d3553.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044fe062_415e_43bf_a759_fc5ed2f81610.slice/crio-conmon-fd0e55c0414776a8d531c0501b3753f49d70fa87c6521726b180fe554f5d3553.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:40:51.560771 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:40:51.560738 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4717868b_fcbe_4f25_ae92_ab98166199d8.slice/crio-conmon-cf488f087c3e78d0028cc0d2e3cdb4283b6b8e552e2e983e51bf0cde78a34dcd.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2beae172_d2d5_4e41_97fb_5772fca8e05d.slice/crio-78d2b0c7d23c2d99d6872c8a523accdfa80a5b673d8ff92d30e01aae9fbd81e5\": RecentStats: unable to find data in memory cache]" Nov 03 20:40:51.682412 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.682368 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:51.712278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.712217 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:51.733895 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.733871 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:51.812932 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.812902 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2beae172-d2d5-4e41-97fb-5772fca8e05d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2beae172-d2d5-4e41-97fb-5772fca8e05d-istiod-ca-cert\") pod \"2beae172-d2d5-4e41-97fb-5772fca8e05d\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " Nov 03 20:40:51.813097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.812938 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-data\") pod \"2beae172-d2d5-4e41-97fb-5772fca8e05d\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " Nov 03 20:40:51.813097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.812956 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-workload-certs\") pod \"044fe062-415e-43bf-a759-fc5ed2f81610\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " Nov 03 20:40:51.813097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.812983 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-envoy\") pod \"2beae172-d2d5-4e41-97fb-5772fca8e05d\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " Nov 03 20:40:51.813097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813001 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4717868b-fcbe-4f25-ae92-ab98166199d8-kube-api-access-x74r7\" (UniqueName: \"kubernetes.io/projected/4717868b-fcbe-4f25-ae92-ab98166199d8-kube-api-access-x74r7\") pod \"4717868b-fcbe-4f25-ae92-ab98166199d8\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " Nov 03 20:40:51.813097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813039 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-podinfo\") pod \"2beae172-d2d5-4e41-97fb-5772fca8e05d\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " Nov 03 20:40:51.813097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813067 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-credential-socket\") pod \"4717868b-fcbe-4f25-ae92-ab98166199d8\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " Nov 03 20:40:51.813097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813097 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-token\" (UniqueName: \"kubernetes.io/projected/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-token\") pod \"4717868b-fcbe-4f25-ae92-ab98166199d8\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " Nov 03 20:40:51.813465 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813137 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-workload-certs\") pod \"4717868b-fcbe-4f25-ae92-ab98166199d8\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " Nov 03 20:40:51.813465 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813180 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4717868b-fcbe-4f25-ae92-ab98166199d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4717868b-fcbe-4f25-ae92-ab98166199d8-istiod-ca-cert\") pod \"4717868b-fcbe-4f25-ae92-ab98166199d8\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " Nov 03 20:40:51.813465 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813206 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/044fe062-415e-43bf-a759-fc5ed2f81610-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/044fe062-415e-43bf-a759-fc5ed2f81610-istio-podinfo\") pod \"044fe062-415e-43bf-a759-fc5ed2f81610\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " Nov 03 20:40:51.813465 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813234 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-envoy\") pod \"4717868b-fcbe-4f25-ae92-ab98166199d8\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " Nov 03 20:40:51.813465 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813264 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-workload-socket\") pod \"4717868b-fcbe-4f25-ae92-ab98166199d8\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " Nov 03 20:40:51.813465 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813236 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-data" pod "2beae172-d2d5-4e41-97fb-5772fca8e05d" (UID: "2beae172-d2d5-4e41-97fb-5772fca8e05d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:51.813465 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813266 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-workload-certs" pod "044fe062-415e-43bf-a759-fc5ed2f81610" (UID: "044fe062-415e-43bf-a759-fc5ed2f81610"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:51.813465 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813307 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-istio-envoy\") pod \"044fe062-415e-43bf-a759-fc5ed2f81610\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " Nov 03 20:40:51.813465 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813335 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/044fe062-415e-43bf-a759-fc5ed2f81610-kube-api-access-b46nt\" (UniqueName: \"kubernetes.io/projected/044fe062-415e-43bf-a759-fc5ed2f81610-kube-api-access-b46nt\") pod \"044fe062-415e-43bf-a759-fc5ed2f81610\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " Nov 03 20:40:51.813465 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813368 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/044fe062-415e-43bf-a759-fc5ed2f81610-istio-token\" (UniqueName: \"kubernetes.io/projected/044fe062-415e-43bf-a759-fc5ed2f81610-istio-token\") pod \"044fe062-415e-43bf-a759-fc5ed2f81610\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " Nov 03 20:40:51.813465 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813398 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-credential-socket\") pod \"044fe062-415e-43bf-a759-fc5ed2f81610\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " Nov 03 20:40:51.813465 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813438 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-podinfo\") pod \"4717868b-fcbe-4f25-ae92-ab98166199d8\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " Nov 03 20:40:51.813465 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813440 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-credential-socket" pod "4717868b-fcbe-4f25-ae92-ab98166199d8" (UID: "4717868b-fcbe-4f25-ae92-ab98166199d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:51.813465 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813460 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2beae172-d2d5-4e41-97fb-5772fca8e05d-istiod-ca-cert" pod "2beae172-d2d5-4e41-97fb-5772fca8e05d" (UID: "2beae172-d2d5-4e41-97fb-5772fca8e05d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:51.814182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813466 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-data\") pod \"4717868b-fcbe-4f25-ae92-ab98166199d8\" (UID: \"4717868b-fcbe-4f25-ae92-ab98166199d8\") " Nov 03 20:40:51.814182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813556 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-workload-certs\") pod \"2beae172-d2d5-4e41-97fb-5772fca8e05d\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " Nov 03 20:40:51.814182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813596 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4717868b-fcbe-4f25-ae92-ab98166199d8-istiod-ca-cert" pod "4717868b-fcbe-4f25-ae92-ab98166199d8" (UID: "4717868b-fcbe-4f25-ae92-ab98166199d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:51.814182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813600 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-token\" (UniqueName: \"kubernetes.io/projected/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-token\") pod \"2beae172-d2d5-4e41-97fb-5772fca8e05d\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " Nov 03 20:40:51.814182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813739 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/044fe062-415e-43bf-a759-fc5ed2f81610-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/044fe062-415e-43bf-a759-fc5ed2f81610-istiod-ca-cert\") pod \"044fe062-415e-43bf-a759-fc5ed2f81610\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " Nov 03 20:40:51.814182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813803 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-workload-socket\") pod \"2beae172-d2d5-4e41-97fb-5772fca8e05d\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " Nov 03 20:40:51.814182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813834 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2beae172-d2d5-4e41-97fb-5772fca8e05d-kube-api-access-8hsq4\" (UniqueName: \"kubernetes.io/projected/2beae172-d2d5-4e41-97fb-5772fca8e05d-kube-api-access-8hsq4\") pod \"2beae172-d2d5-4e41-97fb-5772fca8e05d\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " Nov 03 20:40:51.814182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813868 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-istio-data\") pod \"044fe062-415e-43bf-a759-fc5ed2f81610\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " Nov 03 20:40:51.814182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813903 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-workload-socket\") pod \"044fe062-415e-43bf-a759-fc5ed2f81610\" (UID: \"044fe062-415e-43bf-a759-fc5ed2f81610\") " Nov 03 20:40:51.814182 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813948 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-credential-socket\") pod \"2beae172-d2d5-4e41-97fb-5772fca8e05d\" (UID: \"2beae172-d2d5-4e41-97fb-5772fca8e05d\") " Nov 03 20:40:51.816525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.814964 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2beae172-d2d5-4e41-97fb-5772fca8e05d-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.816525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.814990 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.816525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.815020 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.816525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.815034 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.816525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.815047 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4717868b-fcbe-4f25-ae92-ab98166199d8-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.816525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813686 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-data" pod "4717868b-fcbe-4f25-ae92-ab98166199d8" (UID: "4717868b-fcbe-4f25-ae92-ab98166199d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:51.816525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.813868 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-workload-socket" pod "4717868b-fcbe-4f25-ae92-ab98166199d8" (UID: "4717868b-fcbe-4f25-ae92-ab98166199d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:51.816525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.815309 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-credential-socket" pod "2beae172-d2d5-4e41-97fb-5772fca8e05d" (UID: "2beae172-d2d5-4e41-97fb-5772fca8e05d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:51.816525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.815530 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-workload-certs" pod "4717868b-fcbe-4f25-ae92-ab98166199d8" (UID: "4717868b-fcbe-4f25-ae92-ab98166199d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:51.816525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.816006 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044fe062-415e-43bf-a759-fc5ed2f81610-istiod-ca-cert" pod "044fe062-415e-43bf-a759-fc5ed2f81610" (UID: "044fe062-415e-43bf-a759-fc5ed2f81610"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:51.816525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.816092 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4717868b-fcbe-4f25-ae92-ab98166199d8-kube-api-access-x74r7" pod "4717868b-fcbe-4f25-ae92-ab98166199d8" (UID: "4717868b-fcbe-4f25-ae92-ab98166199d8"). InnerVolumeSpecName "kube-api-access-x74r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:51.816525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.816434 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-workload-socket" pod "2beae172-d2d5-4e41-97fb-5772fca8e05d" (UID: "2beae172-d2d5-4e41-97fb-5772fca8e05d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:51.817933 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.817908 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-istio-envoy" pod "044fe062-415e-43bf-a759-fc5ed2f81610" (UID: "044fe062-415e-43bf-a759-fc5ed2f81610"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:51.819900 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.819780 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-podinfo" pod "2beae172-d2d5-4e41-97fb-5772fca8e05d" (UID: "2beae172-d2d5-4e41-97fb-5772fca8e05d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:40:51.820059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.820035 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-istio-data" pod "044fe062-415e-43bf-a759-fc5ed2f81610" (UID: "044fe062-415e-43bf-a759-fc5ed2f81610"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:51.820222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.820205 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-workload-socket" pod "044fe062-415e-43bf-a759-fc5ed2f81610" (UID: "044fe062-415e-43bf-a759-fc5ed2f81610"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:51.820376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.820359 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-credential-socket" pod "044fe062-415e-43bf-a759-fc5ed2f81610" (UID: "044fe062-415e-43bf-a759-fc5ed2f81610"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:51.821860 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.821036 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-token" pod "2beae172-d2d5-4e41-97fb-5772fca8e05d" (UID: "2beae172-d2d5-4e41-97fb-5772fca8e05d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:51.821860 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.821262 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-workload-certs" pod "2beae172-d2d5-4e41-97fb-5772fca8e05d" (UID: "2beae172-d2d5-4e41-97fb-5772fca8e05d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:51.821860 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.821436 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-token" pod "4717868b-fcbe-4f25-ae92-ab98166199d8" (UID: "4717868b-fcbe-4f25-ae92-ab98166199d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:51.821860 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.821746 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044fe062-415e-43bf-a759-fc5ed2f81610-kube-api-access-b46nt" pod "044fe062-415e-43bf-a759-fc5ed2f81610" (UID: "044fe062-415e-43bf-a759-fc5ed2f81610"). InnerVolumeSpecName "kube-api-access-b46nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:51.822100 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.822057 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2beae172-d2d5-4e41-97fb-5772fca8e05d-kube-api-access-8hsq4" pod "2beae172-d2d5-4e41-97fb-5772fca8e05d" (UID: "2beae172-d2d5-4e41-97fb-5772fca8e05d"). InnerVolumeSpecName "kube-api-access-8hsq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:51.822815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.822496 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-envoy" pod "2beae172-d2d5-4e41-97fb-5772fca8e05d" (UID: "2beae172-d2d5-4e41-97fb-5772fca8e05d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:51.822815 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.822528 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-envoy" pod "4717868b-fcbe-4f25-ae92-ab98166199d8" (UID: "4717868b-fcbe-4f25-ae92-ab98166199d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:51.823075 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.823052 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/044fe062-415e-43bf-a759-fc5ed2f81610-istio-podinfo" pod "044fe062-415e-43bf-a759-fc5ed2f81610" (UID: "044fe062-415e-43bf-a759-fc5ed2f81610"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:40:51.823762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.823698 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044fe062-415e-43bf-a759-fc5ed2f81610-istio-token" pod "044fe062-415e-43bf-a759-fc5ed2f81610" (UID: "044fe062-415e-43bf-a759-fc5ed2f81610"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:51.828036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.828013 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-podinfo" pod "4717868b-fcbe-4f25-ae92-ab98166199d8" (UID: "4717868b-fcbe-4f25-ae92-ab98166199d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:40:51.832714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.832686 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s"] Nov 03 20:40:51.832994 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.832955 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" podUID="33ce656e-8f66-4adc-9e57-d7d3b565e937" containerName="maas-api" containerID="cri-o://912a8cea40988edb7390e043dc3c81ea0713bc051eb2f4f042803ca4e810b145" gracePeriod=30 Nov 03 20:40:51.916517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.916495 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.916600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.916520 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.916600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.916530 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/044fe062-415e-43bf-a759-fc5ed2f81610-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.916600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.916539 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.916600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.916548 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hsq4\" (UniqueName: \"kubernetes.io/projected/2beae172-d2d5-4e41-97fb-5772fca8e05d-kube-api-access-8hsq4\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.916600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.916557 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.916600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.916567 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.916600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.916575 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.916600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.916584 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.916600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.916592 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x74r7\" (UniqueName: \"kubernetes.io/projected/4717868b-fcbe-4f25-ae92-ab98166199d8-kube-api-access-x74r7\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.916600 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.916600 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2beae172-d2d5-4e41-97fb-5772fca8e05d-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.916934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.916610 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.916934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.916618 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.916934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.916627 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/044fe062-415e-43bf-a759-fc5ed2f81610-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.916934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.916635 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.916934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.916643 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.916934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.916651 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.916934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.916678 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b46nt\" (UniqueName: \"kubernetes.io/projected/044fe062-415e-43bf-a759-fc5ed2f81610-kube-api-access-b46nt\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.916934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.916691 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/044fe062-415e-43bf-a759-fc5ed2f81610-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.916934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.916703 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/044fe062-415e-43bf-a759-fc5ed2f81610-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.916934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.916711 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.916934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:51.916719 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4717868b-fcbe-4f25-ae92-ab98166199d8-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:52.076999 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.076972 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" Nov 03 20:40:52.117529 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.117493 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33ce656e-8f66-4adc-9e57-d7d3b565e937-metrics-certs\" (UniqueName: \"kubernetes.io/secret/33ce656e-8f66-4adc-9e57-d7d3b565e937-metrics-certs\") pod \"33ce656e-8f66-4adc-9e57-d7d3b565e937\" (UID: \"33ce656e-8f66-4adc-9e57-d7d3b565e937\") " Nov 03 20:40:52.117650 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.117555 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33ce656e-8f66-4adc-9e57-d7d3b565e937-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ce656e-8f66-4adc-9e57-d7d3b565e937-openshift-service-ca\") pod \"33ce656e-8f66-4adc-9e57-d7d3b565e937\" (UID: \"33ce656e-8f66-4adc-9e57-d7d3b565e937\") " Nov 03 20:40:52.117650 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.117580 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ce656e-8f66-4adc-9e57-d7d3b565e937-kube-api-access-8djjs\" (UniqueName: \"kubernetes.io/projected/33ce656e-8f66-4adc-9e57-d7d3b565e937-kube-api-access-8djjs\") pod \"33ce656e-8f66-4adc-9e57-d7d3b565e937\" (UID: \"33ce656e-8f66-4adc-9e57-d7d3b565e937\") " Nov 03 20:40:52.117650 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.117627 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33ce656e-8f66-4adc-9e57-d7d3b565e937-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33ce656e-8f66-4adc-9e57-d7d3b565e937-maas-api-tls\") pod \"33ce656e-8f66-4adc-9e57-d7d3b565e937\" (UID: \"33ce656e-8f66-4adc-9e57-d7d3b565e937\") " Nov 03 20:40:52.117982 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.117948 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ce656e-8f66-4adc-9e57-d7d3b565e937-openshift-service-ca" pod "33ce656e-8f66-4adc-9e57-d7d3b565e937" (UID: "33ce656e-8f66-4adc-9e57-d7d3b565e937"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:52.118109 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.118047 5059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ce656e-8f66-4adc-9e57-d7d3b565e937-openshift-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:52.119819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.119792 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ce656e-8f66-4adc-9e57-d7d3b565e937-kube-api-access-8djjs" pod "33ce656e-8f66-4adc-9e57-d7d3b565e937" (UID: "33ce656e-8f66-4adc-9e57-d7d3b565e937"). InnerVolumeSpecName "kube-api-access-8djjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:52.120174 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.120156 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ce656e-8f66-4adc-9e57-d7d3b565e937-metrics-certs" pod "33ce656e-8f66-4adc-9e57-d7d3b565e937" (UID: "33ce656e-8f66-4adc-9e57-d7d3b565e937"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:52.120232 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.120179 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ce656e-8f66-4adc-9e57-d7d3b565e937-maas-api-tls" pod "33ce656e-8f66-4adc-9e57-d7d3b565e937" (UID: "33ce656e-8f66-4adc-9e57-d7d3b565e937"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:52.229742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.229355 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33ce656e-8f66-4adc-9e57-d7d3b565e937-maas-api-tls\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:52.229742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.229400 5059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33ce656e-8f66-4adc-9e57-d7d3b565e937-metrics-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:52.229742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.229419 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8djjs\" (UniqueName: \"kubernetes.io/projected/33ce656e-8f66-4adc-9e57-d7d3b565e937-kube-api-access-8djjs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:52.396068 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.396028 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm"] Nov 03 20:40:52.396441 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.396282 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" podUID="92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80" containerName="istio-proxy" containerID="cri-o://717313c3216c7a6ab47bc1c7f439ba1789a4c63aaf1014eec9612da6b76faf0e" gracePeriod=30 Nov 03 20:40:52.399414 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.399383 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j"] Nov 03 20:40:52.399589 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.399568 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" podUID="bbd92cdc-e2bd-4f72-8c63-1221069ca3fd" containerName="istio-proxy" containerID="cri-o://0e34ccd56022844f4709995fa385f307c970109d38e0aa73908019a3605706a5" gracePeriod=30 Nov 03 20:40:52.403117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.403085 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79"] Nov 03 20:40:52.403348 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.403326 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" podUID="35287c50-78d5-4061-8ead-d2325d6800c8" containerName="istio-proxy" containerID="cri-o://e98bbba4fdaf72a992af1b602abe2af37f79924046bf9fdd52e8d228e42f5f34" gracePeriod=30 Nov 03 20:40:52.406003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.405966 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd"] Nov 03 20:40:52.406171 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.406150 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" podUID="076156cc-fc7a-4a9e-bca0-7bee7bc7b065" containerName="istio-proxy" containerID="cri-o://e6f40dbc1e4393730e4807046b3eb1684e1ea3495a4793427c27e44c8ba4ee95" gracePeriod=30 Nov 03 20:40:52.422952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.422929 5059 generic.go:356] "Generic (PLEG): container finished" podID="2beae172-d2d5-4e41-97fb-5772fca8e05d" containerID="b6d218d235549ee069a2989a61510bf7592be0af650590d12ee551d028bef486" exitCode=0 Nov 03 20:40:52.423058 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.422974 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" Nov 03 20:40:52.423058 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.423016 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" event={"ID":"2beae172-d2d5-4e41-97fb-5772fca8e05d","Type":"ContainerDied","Data":"b6d218d235549ee069a2989a61510bf7592be0af650590d12ee551d028bef486"} Nov 03 20:40:52.423154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.423080 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw" event={"ID":"2beae172-d2d5-4e41-97fb-5772fca8e05d","Type":"ContainerDied","Data":"78d2b0c7d23c2d99d6872c8a523accdfa80a5b673d8ff92d30e01aae9fbd81e5"} Nov 03 20:40:52.423154 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.423114 5059 scope.go:122] "RemoveContainer" containerID="b6d218d235549ee069a2989a61510bf7592be0af650590d12ee551d028bef486" Nov 03 20:40:52.425049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.425023 5059 generic.go:356] "Generic (PLEG): container finished" podID="33ce656e-8f66-4adc-9e57-d7d3b565e937" containerID="912a8cea40988edb7390e043dc3c81ea0713bc051eb2f4f042803ca4e810b145" exitCode=0 Nov 03 20:40:52.425153 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.425064 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" Nov 03 20:40:52.425209 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.425171 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" event={"ID":"33ce656e-8f66-4adc-9e57-d7d3b565e937","Type":"ContainerDied","Data":"912a8cea40988edb7390e043dc3c81ea0713bc051eb2f4f042803ca4e810b145"} Nov 03 20:40:52.425250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.425206 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s" event={"ID":"33ce656e-8f66-4adc-9e57-d7d3b565e937","Type":"ContainerDied","Data":"6d97f8c8265e643b31584b5888a98169dbee4c9cb851a64c6d1470f519413d81"} Nov 03 20:40:52.427024 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.427000 5059 generic.go:356] "Generic (PLEG): container finished" podID="044fe062-415e-43bf-a759-fc5ed2f81610" containerID="fd0e55c0414776a8d531c0501b3753f49d70fa87c6521726b180fe554f5d3553" exitCode=0 Nov 03 20:40:52.427130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.427046 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" Nov 03 20:40:52.427130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.427070 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" event={"ID":"044fe062-415e-43bf-a759-fc5ed2f81610","Type":"ContainerDied","Data":"fd0e55c0414776a8d531c0501b3753f49d70fa87c6521726b180fe554f5d3553"} Nov 03 20:40:52.427130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.427108 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch" event={"ID":"044fe062-415e-43bf-a759-fc5ed2f81610","Type":"ContainerDied","Data":"5bad706e57526771579a0aa03a11635f7682040328d08676619819b68de1780e"} Nov 03 20:40:52.434752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.434625 5059 generic.go:356] "Generic (PLEG): container finished" podID="4717868b-fcbe-4f25-ae92-ab98166199d8" containerID="cf488f087c3e78d0028cc0d2e3cdb4283b6b8e552e2e983e51bf0cde78a34dcd" exitCode=0 Nov 03 20:40:52.434752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.434717 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" event={"ID":"4717868b-fcbe-4f25-ae92-ab98166199d8","Type":"ContainerDied","Data":"cf488f087c3e78d0028cc0d2e3cdb4283b6b8e552e2e983e51bf0cde78a34dcd"} Nov 03 20:40:52.434752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.434730 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" Nov 03 20:40:52.437612 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.434796 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd" event={"ID":"4717868b-fcbe-4f25-ae92-ab98166199d8","Type":"ContainerDied","Data":"fb1f07b9081559d0429cd6f964a03139d1374b68822e62a24b15da0b514d5637"} Nov 03 20:40:52.445294 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.445271 5059 scope.go:122] "RemoveContainer" containerID="b6d218d235549ee069a2989a61510bf7592be0af650590d12ee551d028bef486" Nov 03 20:40:52.447436 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:40:52.447407 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6d218d235549ee069a2989a61510bf7592be0af650590d12ee551d028bef486\": container with ID starting with b6d218d235549ee069a2989a61510bf7592be0af650590d12ee551d028bef486 not found: ID does not exist" containerID="b6d218d235549ee069a2989a61510bf7592be0af650590d12ee551d028bef486" Nov 03 20:40:52.447524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.447446 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6d218d235549ee069a2989a61510bf7592be0af650590d12ee551d028bef486"} err="failed to get container status \"b6d218d235549ee069a2989a61510bf7592be0af650590d12ee551d028bef486\": rpc error: code = NotFound desc = could not find container \"b6d218d235549ee069a2989a61510bf7592be0af650590d12ee551d028bef486\": container with ID starting with b6d218d235549ee069a2989a61510bf7592be0af650590d12ee551d028bef486 not found: ID does not exist" Nov 03 20:40:52.447524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.447469 5059 scope.go:122] "RemoveContainer" containerID="912a8cea40988edb7390e043dc3c81ea0713bc051eb2f4f042803ca4e810b145" Nov 03 20:40:52.456915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.456697 5059 scope.go:122] "RemoveContainer" containerID="912a8cea40988edb7390e043dc3c81ea0713bc051eb2f4f042803ca4e810b145" Nov 03 20:40:52.456982 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:40:52.456957 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"912a8cea40988edb7390e043dc3c81ea0713bc051eb2f4f042803ca4e810b145\": container with ID starting with 912a8cea40988edb7390e043dc3c81ea0713bc051eb2f4f042803ca4e810b145 not found: ID does not exist" containerID="912a8cea40988edb7390e043dc3c81ea0713bc051eb2f4f042803ca4e810b145" Nov 03 20:40:52.457037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.456996 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"912a8cea40988edb7390e043dc3c81ea0713bc051eb2f4f042803ca4e810b145"} err="failed to get container status \"912a8cea40988edb7390e043dc3c81ea0713bc051eb2f4f042803ca4e810b145\": rpc error: code = NotFound desc = could not find container \"912a8cea40988edb7390e043dc3c81ea0713bc051eb2f4f042803ca4e810b145\": container with ID starting with 912a8cea40988edb7390e043dc3c81ea0713bc051eb2f4f042803ca4e810b145 not found: ID does not exist" Nov 03 20:40:52.457037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.457015 5059 scope.go:122] "RemoveContainer" containerID="fd0e55c0414776a8d531c0501b3753f49d70fa87c6521726b180fe554f5d3553" Nov 03 20:40:52.466014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.465995 5059 scope.go:122] "RemoveContainer" containerID="fd0e55c0414776a8d531c0501b3753f49d70fa87c6521726b180fe554f5d3553" Nov 03 20:40:52.466263 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:40:52.466241 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd0e55c0414776a8d531c0501b3753f49d70fa87c6521726b180fe554f5d3553\": container with ID starting with fd0e55c0414776a8d531c0501b3753f49d70fa87c6521726b180fe554f5d3553 not found: ID does not exist" containerID="fd0e55c0414776a8d531c0501b3753f49d70fa87c6521726b180fe554f5d3553" Nov 03 20:40:52.466339 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.466269 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0e55c0414776a8d531c0501b3753f49d70fa87c6521726b180fe554f5d3553"} err="failed to get container status \"fd0e55c0414776a8d531c0501b3753f49d70fa87c6521726b180fe554f5d3553\": rpc error: code = NotFound desc = could not find container \"fd0e55c0414776a8d531c0501b3753f49d70fa87c6521726b180fe554f5d3553\": container with ID starting with fd0e55c0414776a8d531c0501b3753f49d70fa87c6521726b180fe554f5d3553 not found: ID does not exist" Nov 03 20:40:52.466339 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.466288 5059 scope.go:122] "RemoveContainer" containerID="cf488f087c3e78d0028cc0d2e3cdb4283b6b8e552e2e983e51bf0cde78a34dcd" Nov 03 20:40:52.474493 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.474472 5059 scope.go:122] "RemoveContainer" containerID="cf488f087c3e78d0028cc0d2e3cdb4283b6b8e552e2e983e51bf0cde78a34dcd" Nov 03 20:40:52.474761 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:40:52.474743 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf488f087c3e78d0028cc0d2e3cdb4283b6b8e552e2e983e51bf0cde78a34dcd\": container with ID starting with cf488f087c3e78d0028cc0d2e3cdb4283b6b8e552e2e983e51bf0cde78a34dcd not found: ID does not exist" containerID="cf488f087c3e78d0028cc0d2e3cdb4283b6b8e552e2e983e51bf0cde78a34dcd" Nov 03 20:40:52.475018 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.474768 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf488f087c3e78d0028cc0d2e3cdb4283b6b8e552e2e983e51bf0cde78a34dcd"} err="failed to get container status \"cf488f087c3e78d0028cc0d2e3cdb4283b6b8e552e2e983e51bf0cde78a34dcd\": rpc error: code = NotFound desc = could not find container \"cf488f087c3e78d0028cc0d2e3cdb4283b6b8e552e2e983e51bf0cde78a34dcd\": container with ID starting with cf488f087c3e78d0028cc0d2e3cdb4283b6b8e552e2e983e51bf0cde78a34dcd not found: ID does not exist" Nov 03 20:40:52.537806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.532069 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw"] Nov 03 20:40:52.537806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.536849 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-x8dkw"] Nov 03 20:40:52.543938 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.543903 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s"] Nov 03 20:40:52.547164 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.547135 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-4br5s"] Nov 03 20:40:52.561286 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.561224 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd"] Nov 03 20:40:52.565406 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.565379 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-46gtd"] Nov 03 20:40:52.576267 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.576238 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch"] Nov 03 20:40:52.580133 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:52.580111 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-wbpch"] Nov 03 20:40:53.855672 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:53.855631 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="044fe062-415e-43bf-a759-fc5ed2f81610" path="/var/lib/kubelet/pods/044fe062-415e-43bf-a759-fc5ed2f81610/volumes" Nov 03 20:40:53.856104 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:53.856087 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2beae172-d2d5-4e41-97fb-5772fca8e05d" path="/var/lib/kubelet/pods/2beae172-d2d5-4e41-97fb-5772fca8e05d/volumes" Nov 03 20:40:53.856481 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:53.856469 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33ce656e-8f66-4adc-9e57-d7d3b565e937" path="/var/lib/kubelet/pods/33ce656e-8f66-4adc-9e57-d7d3b565e937/volumes" Nov 03 20:40:53.857173 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:53.857156 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4717868b-fcbe-4f25-ae92-ab98166199d8" path="/var/lib/kubelet/pods/4717868b-fcbe-4f25-ae92-ab98166199d8/volumes" Nov 03 20:40:56.031324 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:56.031287 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:40:56.031753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:56.031358 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" podUID="35287c50-78d5-4061-8ead-d2325d6800c8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:40:57.064086 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.064046 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:40:57.064540 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.064138 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" podUID="076156cc-fc7a-4a9e-bca0-7bee7bc7b065" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:40:57.092277 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.092250 5059 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:40:57.092374 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.092316 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" podUID="92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:40:57.670255 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.670172 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:57.723057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.723028 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:40:57.726068 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.726047 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:57.729820 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.729797 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:57.784222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784194 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-credential-socket\") pod \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " Nov 03 20:40:57.784341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784232 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-token\" (UniqueName: \"kubernetes.io/projected/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-token\") pod \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " Nov 03 20:40:57.784341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784255 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-workload-certs\") pod \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " Nov 03 20:40:57.784341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784273 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-podinfo\") pod \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " Nov 03 20:40:57.784341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784291 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istiod-ca-cert\") pod \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " Nov 03 20:40:57.784341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784326 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istiod-ca-cert\") pod \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " Nov 03 20:40:57.784609 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784361 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-envoy\") pod \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " Nov 03 20:40:57.784609 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784410 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-kube-api-access-4r4ct\" (UniqueName: \"kubernetes.io/projected/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-kube-api-access-4r4ct\") pod \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " Nov 03 20:40:57.784609 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784449 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istiod-ca-cert\") pod \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " Nov 03 20:40:57.784609 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784495 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-envoy\") pod \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " Nov 03 20:40:57.784609 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784512 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-credential-socket" pod "076156cc-fc7a-4a9e-bca0-7bee7bc7b065" (UID: "076156cc-fc7a-4a9e-bca0-7bee7bc7b065"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:57.784609 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784533 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-workload-socket\") pod \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " Nov 03 20:40:57.784609 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784594 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-workload-certs\") pod \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " Nov 03 20:40:57.785002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784650 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-token\" (UniqueName: \"kubernetes.io/projected/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-token\") pod \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " Nov 03 20:40:57.785002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784694 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-credential-socket\") pod \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " Nov 03 20:40:57.785002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784733 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-token\" (UniqueName: \"kubernetes.io/projected/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-token\") pod \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " Nov 03 20:40:57.785002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784792 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-data\") pod \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " Nov 03 20:40:57.785002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784825 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-data\") pod \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " Nov 03 20:40:57.785002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784856 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-podinfo\") pod \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " Nov 03 20:40:57.785002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784885 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-podinfo\") pod \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " Nov 03 20:40:57.785002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784924 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-workload-socket\") pod \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " Nov 03 20:40:57.785002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784958 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-workload-socket\") pod \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " Nov 03 20:40:57.785002 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784990 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-data\") pod \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " Nov 03 20:40:57.785509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.785054 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-credential-socket\") pod \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " Nov 03 20:40:57.785509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.785080 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-kube-api-access-mt8lx\" (UniqueName: \"kubernetes.io/projected/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-kube-api-access-mt8lx\") pod \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\" (UID: \"076156cc-fc7a-4a9e-bca0-7bee7bc7b065\") " Nov 03 20:40:57.785509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.785110 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-envoy\") pod \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " Nov 03 20:40:57.785509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.785145 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-workload-certs\") pod \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\" (UID: \"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd\") " Nov 03 20:40:57.785509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.785186 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-kube-api-access-lrqtf\" (UniqueName: \"kubernetes.io/projected/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-kube-api-access-lrqtf\") pod \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\" (UID: \"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80\") " Nov 03 20:40:57.785782 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784691 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-workload-certs" pod "92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80" (UID: "92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:57.785782 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.785598 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.785782 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.785619 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.785928 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.785901 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-credential-socket" pod "bbd92cdc-e2bd-4f72-8c63-1221069ca3fd" (UID: "bbd92cdc-e2bd-4f72-8c63-1221069ca3fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:57.786054 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.786028 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-workload-socket" pod "92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80" (UID: "92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:57.786156 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.786131 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-data" pod "076156cc-fc7a-4a9e-bca0-7bee7bc7b065" (UID: "076156cc-fc7a-4a9e-bca0-7bee7bc7b065"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:57.786395 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.786372 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-data" pod "bbd92cdc-e2bd-4f72-8c63-1221069ca3fd" (UID: "bbd92cdc-e2bd-4f72-8c63-1221069ca3fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:57.786477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784850 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istiod-ca-cert" pod "bbd92cdc-e2bd-4f72-8c63-1221069ca3fd" (UID: "bbd92cdc-e2bd-4f72-8c63-1221069ca3fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:57.786477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.784890 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istiod-ca-cert" pod "92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80" (UID: "92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:57.786477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.785151 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-credential-socket" pod "92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80" (UID: "92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:57.786477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.785182 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-workload-socket" pod "bbd92cdc-e2bd-4f72-8c63-1221069ca3fd" (UID: "bbd92cdc-e2bd-4f72-8c63-1221069ca3fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:57.786477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.785302 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istiod-ca-cert" pod "076156cc-fc7a-4a9e-bca0-7bee7bc7b065" (UID: "076156cc-fc7a-4a9e-bca0-7bee7bc7b065"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:57.786477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.785323 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-workload-certs" pod "076156cc-fc7a-4a9e-bca0-7bee7bc7b065" (UID: "076156cc-fc7a-4a9e-bca0-7bee7bc7b065"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:57.786477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.785505 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-workload-socket" pod "076156cc-fc7a-4a9e-bca0-7bee7bc7b065" (UID: "076156cc-fc7a-4a9e-bca0-7bee7bc7b065"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:57.786904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.786475 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-workload-certs" pod "bbd92cdc-e2bd-4f72-8c63-1221069ca3fd" (UID: "bbd92cdc-e2bd-4f72-8c63-1221069ca3fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:57.786904 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.786546 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-data" pod "92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80" (UID: "92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:57.788536 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.788508 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-token" pod "92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80" (UID: "92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:57.788783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.788733 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-envoy" pod "92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80" (UID: "92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:57.788996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.788972 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-token" pod "bbd92cdc-e2bd-4f72-8c63-1221069ca3fd" (UID: "bbd92cdc-e2bd-4f72-8c63-1221069ca3fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:57.789071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.788990 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-token" pod "076156cc-fc7a-4a9e-bca0-7bee7bc7b065" (UID: "076156cc-fc7a-4a9e-bca0-7bee7bc7b065"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:57.789071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.789030 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-envoy" pod "076156cc-fc7a-4a9e-bca0-7bee7bc7b065" (UID: "076156cc-fc7a-4a9e-bca0-7bee7bc7b065"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:57.789071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.789064 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-kube-api-access-mt8lx" pod "076156cc-fc7a-4a9e-bca0-7bee7bc7b065" (UID: "076156cc-fc7a-4a9e-bca0-7bee7bc7b065"). InnerVolumeSpecName "kube-api-access-mt8lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:57.789292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.789273 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-podinfo" pod "bbd92cdc-e2bd-4f72-8c63-1221069ca3fd" (UID: "bbd92cdc-e2bd-4f72-8c63-1221069ca3fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:40:57.789513 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.789491 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-kube-api-access-4r4ct" pod "bbd92cdc-e2bd-4f72-8c63-1221069ca3fd" (UID: "bbd92cdc-e2bd-4f72-8c63-1221069ca3fd"). InnerVolumeSpecName "kube-api-access-4r4ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:57.790052 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.790033 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-kube-api-access-lrqtf" pod "92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80" (UID: "92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80"). InnerVolumeSpecName "kube-api-access-lrqtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:57.790645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.790624 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-envoy" pod "bbd92cdc-e2bd-4f72-8c63-1221069ca3fd" (UID: "bbd92cdc-e2bd-4f72-8c63-1221069ca3fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:57.790759 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.790622 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-podinfo" pod "92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80" (UID: "92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:40:57.790873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.790851 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-podinfo" pod "076156cc-fc7a-4a9e-bca0-7bee7bc7b065" (UID: "076156cc-fc7a-4a9e-bca0-7bee7bc7b065"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:40:57.886867 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.886843 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35287c50-78d5-4061-8ead-d2325d6800c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35287c50-78d5-4061-8ead-d2325d6800c8-istiod-ca-cert\") pod \"35287c50-78d5-4061-8ead-d2325d6800c8\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " Nov 03 20:40:57.886953 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.886880 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-workload-socket\") pod \"35287c50-78d5-4061-8ead-d2325d6800c8\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " Nov 03 20:40:57.886953 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.886934 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35287c50-78d5-4061-8ead-d2325d6800c8-kube-api-access-2cl85\" (UniqueName: \"kubernetes.io/projected/35287c50-78d5-4061-8ead-d2325d6800c8-kube-api-access-2cl85\") pod \"35287c50-78d5-4061-8ead-d2325d6800c8\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " Nov 03 20:40:57.887048 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887018 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35287c50-78d5-4061-8ead-d2325d6800c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35287c50-78d5-4061-8ead-d2325d6800c8-istio-podinfo\") pod \"35287c50-78d5-4061-8ead-d2325d6800c8\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " Nov 03 20:40:57.887105 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887053 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-workload-certs\") pod \"35287c50-78d5-4061-8ead-d2325d6800c8\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " Nov 03 20:40:57.887105 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887088 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-credential-socket\") pod \"35287c50-78d5-4061-8ead-d2325d6800c8\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " Nov 03 20:40:57.887183 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887143 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35287c50-78d5-4061-8ead-d2325d6800c8-istio-token\" (UniqueName: \"kubernetes.io/projected/35287c50-78d5-4061-8ead-d2325d6800c8-istio-token\") pod \"35287c50-78d5-4061-8ead-d2325d6800c8\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " Nov 03 20:40:57.887239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887196 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-istio-data\") pod \"35287c50-78d5-4061-8ead-d2325d6800c8\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " Nov 03 20:40:57.887291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887224 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-workload-socket" pod "35287c50-78d5-4061-8ead-d2325d6800c8" (UID: "35287c50-78d5-4061-8ead-d2325d6800c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:57.887291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887242 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35287c50-78d5-4061-8ead-d2325d6800c8-istiod-ca-cert" pod "35287c50-78d5-4061-8ead-d2325d6800c8" (UID: "35287c50-78d5-4061-8ead-d2325d6800c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:57.887291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887232 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-istio-envoy\") pod \"35287c50-78d5-4061-8ead-d2325d6800c8\" (UID: \"35287c50-78d5-4061-8ead-d2325d6800c8\") " Nov 03 20:40:57.887439 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887408 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-credential-socket" pod "35287c50-78d5-4061-8ead-d2325d6800c8" (UID: "35287c50-78d5-4061-8ead-d2325d6800c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:57.887439 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887423 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-workload-certs" pod "35287c50-78d5-4061-8ead-d2325d6800c8" (UID: "35287c50-78d5-4061-8ead-d2325d6800c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:57.887623 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887604 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-istio-data" pod "35287c50-78d5-4061-8ead-d2325d6800c8" (UID: "35287c50-78d5-4061-8ead-d2325d6800c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:57.887750 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887735 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.887812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887752 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt8lx\" (UniqueName: \"kubernetes.io/projected/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-kube-api-access-mt8lx\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.887812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887762 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.887812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887774 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.887812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887789 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrqtf\" (UniqueName: \"kubernetes.io/projected/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-kube-api-access-lrqtf\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.887812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887803 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.888070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887817 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.888070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887829 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.888070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887838 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35287c50-78d5-4061-8ead-d2325d6800c8-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.888070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887847 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.888070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887859 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.888070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887875 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.888070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887887 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.888070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887897 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r4ct\" (UniqueName: \"kubernetes.io/projected/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-kube-api-access-4r4ct\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.888070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887905 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.888070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887913 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.888070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887922 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.888070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887931 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.888070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887939 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.888070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887948 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.888070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887958 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.888070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887967 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.888070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887975 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.888070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887983 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.888070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887992 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.888070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.887999 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.888070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.888008 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.888070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.888016 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.888070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.888024 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.888070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.888032 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076156cc-fc7a-4a9e-bca0-7bee7bc7b065-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.889401 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.889307 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-istio-envoy" pod "35287c50-78d5-4061-8ead-d2325d6800c8" (UID: "35287c50-78d5-4061-8ead-d2325d6800c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:57.889638 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.889615 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35287c50-78d5-4061-8ead-d2325d6800c8-istio-podinfo" pod "35287c50-78d5-4061-8ead-d2325d6800c8" (UID: "35287c50-78d5-4061-8ead-d2325d6800c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:40:57.889768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.889637 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35287c50-78d5-4061-8ead-d2325d6800c8-kube-api-access-2cl85" pod "35287c50-78d5-4061-8ead-d2325d6800c8" (UID: "35287c50-78d5-4061-8ead-d2325d6800c8"). InnerVolumeSpecName "kube-api-access-2cl85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:57.889768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.889648 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35287c50-78d5-4061-8ead-d2325d6800c8-istio-token" pod "35287c50-78d5-4061-8ead-d2325d6800c8" (UID: "35287c50-78d5-4061-8ead-d2325d6800c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:57.989236 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.989180 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35287c50-78d5-4061-8ead-d2325d6800c8-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.989236 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.989202 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35287c50-78d5-4061-8ead-d2325d6800c8-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.989236 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.989213 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cl85\" (UniqueName: \"kubernetes.io/projected/35287c50-78d5-4061-8ead-d2325d6800c8-kube-api-access-2cl85\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.989236 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:57.989224 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35287c50-78d5-4061-8ead-d2325d6800c8-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:58.472375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.472335 5059 generic.go:356] "Generic (PLEG): container finished" podID="92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80" containerID="717313c3216c7a6ab47bc1c7f439ba1789a4c63aaf1014eec9612da6b76faf0e" exitCode=0 Nov 03 20:40:58.472876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.472391 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" Nov 03 20:40:58.472876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.472428 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" event={"ID":"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80","Type":"ContainerDied","Data":"717313c3216c7a6ab47bc1c7f439ba1789a4c63aaf1014eec9612da6b76faf0e"} Nov 03 20:40:58.472876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.472481 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm" event={"ID":"92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80","Type":"ContainerDied","Data":"c181f1616bb52cf2e444bc18ce32f514106fa23a552cad2c4bf8fa9160b5fed0"} Nov 03 20:40:58.472876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.472502 5059 scope.go:122] "RemoveContainer" containerID="717313c3216c7a6ab47bc1c7f439ba1789a4c63aaf1014eec9612da6b76faf0e" Nov 03 20:40:58.473894 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.473873 5059 generic.go:356] "Generic (PLEG): container finished" podID="35287c50-78d5-4061-8ead-d2325d6800c8" containerID="e98bbba4fdaf72a992af1b602abe2af37f79924046bf9fdd52e8d228e42f5f34" exitCode=0 Nov 03 20:40:58.473997 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.473918 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" Nov 03 20:40:58.473997 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.473958 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" event={"ID":"35287c50-78d5-4061-8ead-d2325d6800c8","Type":"ContainerDied","Data":"e98bbba4fdaf72a992af1b602abe2af37f79924046bf9fdd52e8d228e42f5f34"} Nov 03 20:40:58.474080 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.474003 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79" event={"ID":"35287c50-78d5-4061-8ead-d2325d6800c8","Type":"ContainerDied","Data":"bd54a2cc127b7cc1307cdbc8cf3d32d098941a34a44956e7f61888e30f352aa0"} Nov 03 20:40:58.476229 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.476194 5059 generic.go:356] "Generic (PLEG): container finished" podID="bbd92cdc-e2bd-4f72-8c63-1221069ca3fd" containerID="0e34ccd56022844f4709995fa385f307c970109d38e0aa73908019a3605706a5" exitCode=0 Nov 03 20:40:58.476329 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.476224 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" event={"ID":"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd","Type":"ContainerDied","Data":"0e34ccd56022844f4709995fa385f307c970109d38e0aa73908019a3605706a5"} Nov 03 20:40:58.476329 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.476268 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" event={"ID":"bbd92cdc-e2bd-4f72-8c63-1221069ca3fd","Type":"ContainerDied","Data":"2d0da3958f8a206492b090761f8d10fc84d3560053575f8560f7d743955337a2"} Nov 03 20:40:58.476446 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.476362 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j" Nov 03 20:40:58.477855 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.477831 5059 generic.go:356] "Generic (PLEG): container finished" podID="076156cc-fc7a-4a9e-bca0-7bee7bc7b065" containerID="e6f40dbc1e4393730e4807046b3eb1684e1ea3495a4793427c27e44c8ba4ee95" exitCode=0 Nov 03 20:40:58.478012 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.477875 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" Nov 03 20:40:58.478012 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.477914 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" event={"ID":"076156cc-fc7a-4a9e-bca0-7bee7bc7b065","Type":"ContainerDied","Data":"e6f40dbc1e4393730e4807046b3eb1684e1ea3495a4793427c27e44c8ba4ee95"} Nov 03 20:40:58.478012 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.477961 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd" event={"ID":"076156cc-fc7a-4a9e-bca0-7bee7bc7b065","Type":"ContainerDied","Data":"14458aee810e410292116b9ba9f156232f2d2ffdacfbdc8b17ec4c26bdd61d2c"} Nov 03 20:40:58.488574 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.488556 5059 scope.go:122] "RemoveContainer" containerID="717313c3216c7a6ab47bc1c7f439ba1789a4c63aaf1014eec9612da6b76faf0e" Nov 03 20:40:58.488915 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:40:58.488894 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"717313c3216c7a6ab47bc1c7f439ba1789a4c63aaf1014eec9612da6b76faf0e\": container with ID starting with 717313c3216c7a6ab47bc1c7f439ba1789a4c63aaf1014eec9612da6b76faf0e not found: ID does not exist" containerID="717313c3216c7a6ab47bc1c7f439ba1789a4c63aaf1014eec9612da6b76faf0e" Nov 03 20:40:58.489000 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.488926 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"717313c3216c7a6ab47bc1c7f439ba1789a4c63aaf1014eec9612da6b76faf0e"} err="failed to get container status \"717313c3216c7a6ab47bc1c7f439ba1789a4c63aaf1014eec9612da6b76faf0e\": rpc error: code = NotFound desc = could not find container \"717313c3216c7a6ab47bc1c7f439ba1789a4c63aaf1014eec9612da6b76faf0e\": container with ID starting with 717313c3216c7a6ab47bc1c7f439ba1789a4c63aaf1014eec9612da6b76faf0e not found: ID does not exist" Nov 03 20:40:58.489000 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.488943 5059 scope.go:122] "RemoveContainer" containerID="e98bbba4fdaf72a992af1b602abe2af37f79924046bf9fdd52e8d228e42f5f34" Nov 03 20:40:58.497818 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.497796 5059 scope.go:122] "RemoveContainer" containerID="e98bbba4fdaf72a992af1b602abe2af37f79924046bf9fdd52e8d228e42f5f34" Nov 03 20:40:58.498053 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:40:58.498035 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e98bbba4fdaf72a992af1b602abe2af37f79924046bf9fdd52e8d228e42f5f34\": container with ID starting with e98bbba4fdaf72a992af1b602abe2af37f79924046bf9fdd52e8d228e42f5f34 not found: ID does not exist" containerID="e98bbba4fdaf72a992af1b602abe2af37f79924046bf9fdd52e8d228e42f5f34" Nov 03 20:40:58.498098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.498069 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e98bbba4fdaf72a992af1b602abe2af37f79924046bf9fdd52e8d228e42f5f34"} err="failed to get container status \"e98bbba4fdaf72a992af1b602abe2af37f79924046bf9fdd52e8d228e42f5f34\": rpc error: code = NotFound desc = could not find container \"e98bbba4fdaf72a992af1b602abe2af37f79924046bf9fdd52e8d228e42f5f34\": container with ID starting with e98bbba4fdaf72a992af1b602abe2af37f79924046bf9fdd52e8d228e42f5f34 not found: ID does not exist" Nov 03 20:40:58.498098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.498085 5059 scope.go:122] "RemoveContainer" containerID="0e34ccd56022844f4709995fa385f307c970109d38e0aa73908019a3605706a5" Nov 03 20:40:58.504458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.504438 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm"] Nov 03 20:40:58.508727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.508704 5059 scope.go:122] "RemoveContainer" containerID="0e34ccd56022844f4709995fa385f307c970109d38e0aa73908019a3605706a5" Nov 03 20:40:58.509018 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:40:58.508994 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e34ccd56022844f4709995fa385f307c970109d38e0aa73908019a3605706a5\": container with ID starting with 0e34ccd56022844f4709995fa385f307c970109d38e0aa73908019a3605706a5 not found: ID does not exist" containerID="0e34ccd56022844f4709995fa385f307c970109d38e0aa73908019a3605706a5" Nov 03 20:40:58.509099 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.509026 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e34ccd56022844f4709995fa385f307c970109d38e0aa73908019a3605706a5"} err="failed to get container status \"0e34ccd56022844f4709995fa385f307c970109d38e0aa73908019a3605706a5\": rpc error: code = NotFound desc = could not find container \"0e34ccd56022844f4709995fa385f307c970109d38e0aa73908019a3605706a5\": container with ID starting with 0e34ccd56022844f4709995fa385f307c970109d38e0aa73908019a3605706a5 not found: ID does not exist" Nov 03 20:40:58.509099 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.509041 5059 scope.go:122] "RemoveContainer" containerID="e6f40dbc1e4393730e4807046b3eb1684e1ea3495a4793427c27e44c8ba4ee95" Nov 03 20:40:58.511210 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.511190 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-hztmm"] Nov 03 20:40:58.517430 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.517293 5059 scope.go:122] "RemoveContainer" containerID="e6f40dbc1e4393730e4807046b3eb1684e1ea3495a4793427c27e44c8ba4ee95" Nov 03 20:40:58.517591 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:40:58.517568 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f40dbc1e4393730e4807046b3eb1684e1ea3495a4793427c27e44c8ba4ee95\": container with ID starting with e6f40dbc1e4393730e4807046b3eb1684e1ea3495a4793427c27e44c8ba4ee95 not found: ID does not exist" containerID="e6f40dbc1e4393730e4807046b3eb1684e1ea3495a4793427c27e44c8ba4ee95" Nov 03 20:40:58.517641 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.517597 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f40dbc1e4393730e4807046b3eb1684e1ea3495a4793427c27e44c8ba4ee95"} err="failed to get container status \"e6f40dbc1e4393730e4807046b3eb1684e1ea3495a4793427c27e44c8ba4ee95\": rpc error: code = NotFound desc = could not find container \"e6f40dbc1e4393730e4807046b3eb1684e1ea3495a4793427c27e44c8ba4ee95\": container with ID starting with e6f40dbc1e4393730e4807046b3eb1684e1ea3495a4793427c27e44c8ba4ee95 not found: ID does not exist" Nov 03 20:40:58.525329 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.525300 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79"] Nov 03 20:40:58.530195 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.530168 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-crz79"] Nov 03 20:40:58.542101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.542081 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j"] Nov 03 20:40:58.548557 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.548538 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-67f2j"] Nov 03 20:40:58.562525 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.562504 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd"] Nov 03 20:40:58.566585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:58.566563 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-rhcqd"] Nov 03 20:40:59.855629 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:59.855591 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="076156cc-fc7a-4a9e-bca0-7bee7bc7b065" path="/var/lib/kubelet/pods/076156cc-fc7a-4a9e-bca0-7bee7bc7b065/volumes" Nov 03 20:40:59.856224 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:59.856206 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35287c50-78d5-4061-8ead-d2325d6800c8" path="/var/lib/kubelet/pods/35287c50-78d5-4061-8ead-d2325d6800c8/volumes" Nov 03 20:40:59.856787 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:59.856771 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80" path="/var/lib/kubelet/pods/92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80/volumes" Nov 03 20:40:59.857783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:40:59.857766 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbd92cdc-e2bd-4f72-8c63-1221069ca3fd" path="/var/lib/kubelet/pods/bbd92cdc-e2bd-4f72-8c63-1221069ca3fd/volumes" Nov 03 20:41:03.276517 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.276484 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v"] Nov 03 20:41:03.277034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.276865 5059 cpu_manager.go:414] "containerMap: removing container" podUID="26576b40-fd9c-4298-9129-5b341d2c0b3d" containerName="payload-pre-processing" Nov 03 20:41:03.277034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.276881 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26576b40-fd9c-4298-9129-5b341d2c0b3d" containerName="payload-pre-processing" Nov 03 20:41:03.277034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.276892 5059 cpu_manager.go:414] "containerMap: removing container" podUID="8e9aefe5-8450-4eb9-8d8b-df89be81ebb3" containerName="payload-processing" Nov 03 20:41:03.277034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.276898 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e9aefe5-8450-4eb9-8d8b-df89be81ebb3" containerName="payload-processing" Nov 03 20:41:03.277034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.276907 5059 cpu_manager.go:414] "containerMap: removing container" podUID="076156cc-fc7a-4a9e-bca0-7bee7bc7b065" containerName="istio-proxy" Nov 03 20:41:03.277034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.276912 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="076156cc-fc7a-4a9e-bca0-7bee7bc7b065" containerName="istio-proxy" Nov 03 20:41:03.277034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.276926 5059 cpu_manager.go:414] "containerMap: removing container" podUID="998d8105-98ae-451b-8d6f-639a2c61bf12" containerName="revoke-keys" Nov 03 20:41:03.277034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.276935 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998d8105-98ae-451b-8d6f-639a2c61bf12" containerName="revoke-keys" Nov 03 20:41:03.277034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.276947 5059 cpu_manager.go:414] "containerMap: removing container" podUID="33ce656e-8f66-4adc-9e57-d7d3b565e937" containerName="maas-api" Nov 03 20:41:03.277034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.276952 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ce656e-8f66-4adc-9e57-d7d3b565e937" containerName="maas-api" Nov 03 20:41:03.277034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.276959 5059 cpu_manager.go:414] "containerMap: removing container" podUID="044fe062-415e-43bf-a759-fc5ed2f81610" containerName="istio-proxy" Nov 03 20:41:03.277034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.276963 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="044fe062-415e-43bf-a759-fc5ed2f81610" containerName="istio-proxy" Nov 03 20:41:03.277034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.276974 5059 cpu_manager.go:414] "containerMap: removing container" podUID="bbd92cdc-e2bd-4f72-8c63-1221069ca3fd" containerName="istio-proxy" Nov 03 20:41:03.277034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.276979 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd92cdc-e2bd-4f72-8c63-1221069ca3fd" containerName="istio-proxy" Nov 03 20:41:03.277034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.276987 5059 cpu_manager.go:414] "containerMap: removing container" podUID="2beae172-d2d5-4e41-97fb-5772fca8e05d" containerName="istio-proxy" Nov 03 20:41:03.277034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.276991 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2beae172-d2d5-4e41-97fb-5772fca8e05d" containerName="istio-proxy" Nov 03 20:41:03.277034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.276999 5059 cpu_manager.go:414] "containerMap: removing container" podUID="4717868b-fcbe-4f25-ae92-ab98166199d8" containerName="istio-proxy" Nov 03 20:41:03.277034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.277003 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4717868b-fcbe-4f25-ae92-ab98166199d8" containerName="istio-proxy" Nov 03 20:41:03.277034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.277010 5059 cpu_manager.go:414] "containerMap: removing container" podUID="35287c50-78d5-4061-8ead-d2325d6800c8" containerName="istio-proxy" Nov 03 20:41:03.277034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.277015 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35287c50-78d5-4061-8ead-d2325d6800c8" containerName="istio-proxy" Nov 03 20:41:03.277034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.277025 5059 cpu_manager.go:414] "containerMap: removing container" podUID="92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80" containerName="istio-proxy" Nov 03 20:41:03.277034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.277030 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80" containerName="istio-proxy" Nov 03 20:41:03.277884 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.277080 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="076156cc-fc7a-4a9e-bca0-7bee7bc7b065" containerName="istio-proxy" Nov 03 20:41:03.277884 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.277090 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="998d8105-98ae-451b-8d6f-639a2c61bf12" containerName="revoke-keys" Nov 03 20:41:03.277884 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.277097 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="33ce656e-8f66-4adc-9e57-d7d3b565e937" containerName="maas-api" Nov 03 20:41:03.277884 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.277103 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="044fe062-415e-43bf-a759-fc5ed2f81610" containerName="istio-proxy" Nov 03 20:41:03.277884 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.277109 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd92cdc-e2bd-4f72-8c63-1221069ca3fd" containerName="istio-proxy" Nov 03 20:41:03.277884 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.277115 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2beae172-d2d5-4e41-97fb-5772fca8e05d" containerName="istio-proxy" Nov 03 20:41:03.277884 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.277121 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4717868b-fcbe-4f25-ae92-ab98166199d8" containerName="istio-proxy" Nov 03 20:41:03.277884 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.277127 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="35287c50-78d5-4061-8ead-d2325d6800c8" containerName="istio-proxy" Nov 03 20:41:03.277884 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.277133 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="92cecfa5-c5dc-4c44-9a7e-c0a521a2ff80" containerName="istio-proxy" Nov 03 20:41:03.277884 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.277138 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="26576b40-fd9c-4298-9129-5b341d2c0b3d" containerName="payload-pre-processing" Nov 03 20:41:03.277884 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.277143 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e9aefe5-8450-4eb9-8d8b-df89be81ebb3" containerName="payload-processing" Nov 03 20:41:03.282198 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.282170 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.291881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.291857 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h"] Nov 03 20:41:03.295698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.295676 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v"] Nov 03 20:41:03.295821 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.295765 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.308451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.308432 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h"] Nov 03 20:41:03.434633 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.434606 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.434801 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.434638 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69597629-8533-4612-8f61-7e3cc63ecea6-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.434801 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.434686 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.434801 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.434764 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69597629-8533-4612-8f61-7e3cc63ecea6-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.434933 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.434803 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.434933 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.434824 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.434933 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.434893 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.435080 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.434938 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1411a3da-bb2f-465f-a391-7c3063fbe83d-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.435080 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.434964 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.435080 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.434986 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.435080 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.435002 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.435271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.435124 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgqsx\" (UniqueName: \"kubernetes.io/projected/69597629-8533-4612-8f61-7e3cc63ecea6-kube-api-access-pgqsx\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.435271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.435182 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.435271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.435210 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.435271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.435244 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69597629-8533-4612-8f61-7e3cc63ecea6-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.435271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.435263 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.435426 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.435285 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sblt7\" (UniqueName: \"kubernetes.io/projected/1411a3da-bb2f-465f-a391-7c3063fbe83d-kube-api-access-sblt7\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.435426 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.435342 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.536145 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.536070 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sblt7\" (UniqueName: \"kubernetes.io/projected/1411a3da-bb2f-465f-a391-7c3063fbe83d-kube-api-access-sblt7\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.536301 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.536213 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.536301 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.536271 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.536415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.536387 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69597629-8533-4612-8f61-7e3cc63ecea6-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.536475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.536431 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.536475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.536455 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69597629-8533-4612-8f61-7e3cc63ecea6-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.536589 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.536481 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.536589 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.536502 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.536733 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.536651 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.536733 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.536720 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1411a3da-bb2f-465f-a391-7c3063fbe83d-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.536851 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.536735 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.536851 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.536764 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.536851 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.536797 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.536851 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.536828 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.537072 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.536873 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgqsx\" (UniqueName: \"kubernetes.io/projected/69597629-8533-4612-8f61-7e3cc63ecea6-kube-api-access-pgqsx\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.537072 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.536924 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.537072 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.536950 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.537072 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.536958 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.537072 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.536976 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69597629-8533-4612-8f61-7e3cc63ecea6-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.537072 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.537011 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.537072 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.537036 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.537455 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.537177 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.537515 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.537473 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.537576 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.537491 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69597629-8533-4612-8f61-7e3cc63ecea6-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.537576 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.537526 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.537576 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.537504 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.537818 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.537788 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.538277 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.538249 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1411a3da-bb2f-465f-a391-7c3063fbe83d-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.539221 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.539199 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.539334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.539270 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.539395 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.539343 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.539838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.539821 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69597629-8533-4612-8f61-7e3cc63ecea6-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.548996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.548965 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.548996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.548985 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69597629-8533-4612-8f61-7e3cc63ecea6-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.549266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.549250 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgqsx\" (UniqueName: \"kubernetes.io/projected/69597629-8533-4612-8f61-7e3cc63ecea6-kube-api-access-pgqsx\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.549388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.549367 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sblt7\" (UniqueName: \"kubernetes.io/projected/1411a3da-bb2f-465f-a391-7c3063fbe83d-kube-api-access-sblt7\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.596914 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.596882 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:03.611717 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.611693 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:03.886317 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.886289 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v"] Nov 03 20:41:03.888253 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:41:03.888216 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69597629_8533_4612_8f61_7e3cc63ecea6.slice/crio-0bbcaf5ddd2f4cb917d98308662cd9f135fd664e6092641cf86ea6e41b115bc4 WatchSource:0}: Error finding container 0bbcaf5ddd2f4cb917d98308662cd9f135fd664e6092641cf86ea6e41b115bc4: Status 404 returned error can't find the container with id 0bbcaf5ddd2f4cb917d98308662cd9f135fd664e6092641cf86ea6e41b115bc4 Nov 03 20:41:03.890372 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.890304 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:41:03.890491 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.890388 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:41:03.890491 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.890436 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:41:03.907776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.907751 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h"] Nov 03 20:41:03.908251 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:41:03.908230 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1411a3da_bb2f_465f_a391_7c3063fbe83d.slice/crio-8adeb197cb7f3a3aa2114c0d3e414b0e2982795d38d14cb5133324e80c9c7914 WatchSource:0}: Error finding container 8adeb197cb7f3a3aa2114c0d3e414b0e2982795d38d14cb5133324e80c9c7914: Status 404 returned error can't find the container with id 8adeb197cb7f3a3aa2114c0d3e414b0e2982795d38d14cb5133324e80c9c7914 Nov 03 20:41:03.910265 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.910235 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:41:03.910337 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.910291 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:41:03.910337 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:03.910319 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:41:04.523003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:04.522004 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" event={"ID":"69597629-8533-4612-8f61-7e3cc63ecea6","Type":"ContainerStarted","Data":"18ec72802d4c036166466a28084769c1609327a8f03d2e21981f0d1acc78e272"} Nov 03 20:41:04.523003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:04.522077 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" event={"ID":"69597629-8533-4612-8f61-7e3cc63ecea6","Type":"ContainerStarted","Data":"0bbcaf5ddd2f4cb917d98308662cd9f135fd664e6092641cf86ea6e41b115bc4"} Nov 03 20:41:04.527474 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:04.525882 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" event={"ID":"1411a3da-bb2f-465f-a391-7c3063fbe83d","Type":"ContainerStarted","Data":"b9ce56eda5802519fa78d6929644c7783a43e7adefc0b3c42a6a007603216778"} Nov 03 20:41:04.527474 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:04.525944 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" event={"ID":"1411a3da-bb2f-465f-a391-7c3063fbe83d","Type":"ContainerStarted","Data":"8adeb197cb7f3a3aa2114c0d3e414b0e2982795d38d14cb5133324e80c9c7914"} Nov 03 20:41:04.549062 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:04.548993 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" podStartSLOduration=1.548971575 podStartE2EDuration="1.548971575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:04.546169797 +0000 UTC m=+3858.803967086" watchObservedRunningTime="2026-11-03 20:41:04.548971575 +0000 UTC m=+3858.806768837" Nov 03 20:41:04.582686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:04.581784 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" podStartSLOduration=1.58176295 podStartE2EDuration="1.58176295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:04.579131067 +0000 UTC m=+3858.836928352" watchObservedRunningTime="2026-11-03 20:41:04.58176295 +0000 UTC m=+3858.839560209" Nov 03 20:41:04.602696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:04.601830 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:04.604683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:04.604366 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused" start-of-body= Nov 03 20:41:04.604683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:04.604423 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" podUID="69597629-8533-4612-8f61-7e3cc63ecea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused" Nov 03 20:41:04.613739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:04.613046 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:04.614138 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:04.614106 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused" start-of-body= Nov 03 20:41:04.614253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:04.614160 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" podUID="1411a3da-bb2f-465f-a391-7c3063fbe83d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused" Nov 03 20:41:05.599685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:05.599060 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused" start-of-body= Nov 03 20:41:05.599685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:05.599127 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" podUID="69597629-8533-4612-8f61-7e3cc63ecea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused" Nov 03 20:41:05.613707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:05.612945 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused" start-of-body= Nov 03 20:41:05.613707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:05.613004 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" podUID="1411a3da-bb2f-465f-a391-7c3063fbe83d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused" Nov 03 20:41:06.601686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:06.601220 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused" start-of-body= Nov 03 20:41:06.601686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:06.601302 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" podUID="69597629-8533-4612-8f61-7e3cc63ecea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused" Nov 03 20:41:06.613016 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:06.612981 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused" start-of-body= Nov 03 20:41:06.613248 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:06.613225 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" podUID="1411a3da-bb2f-465f-a391-7c3063fbe83d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused" Nov 03 20:41:07.601250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:07.600765 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused" start-of-body= Nov 03 20:41:07.601250 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:07.600834 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" podUID="69597629-8533-4612-8f61-7e3cc63ecea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused" Nov 03 20:41:07.612160 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:07.612075 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused" start-of-body= Nov 03 20:41:07.612160 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:07.612121 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" podUID="1411a3da-bb2f-465f-a391-7c3063fbe83d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused" Nov 03 20:41:08.602685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:08.602064 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused" start-of-body= Nov 03 20:41:08.602685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:08.602131 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" podUID="69597629-8533-4612-8f61-7e3cc63ecea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused" Nov 03 20:41:08.612339 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:08.612308 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused" start-of-body= Nov 03 20:41:08.612779 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:08.612755 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" podUID="1411a3da-bb2f-465f-a391-7c3063fbe83d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused" Nov 03 20:41:09.600685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:09.600054 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused" start-of-body= Nov 03 20:41:09.600685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:09.600137 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" podUID="69597629-8533-4612-8f61-7e3cc63ecea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused" Nov 03 20:41:09.612607 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:09.612567 5059 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused" start-of-body= Nov 03 20:41:09.613025 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:09.612628 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" podUID="1411a3da-bb2f-465f-a391-7c3063fbe83d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused" Nov 03 20:41:10.602611 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:10.602578 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:10.603063 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:41:10.603026 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:41:10.603190 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:10.603038 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:10.603769 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:10.603749 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:10.616897 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:10.616877 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:10.617185 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:10.617117 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:10.617726 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:10.617708 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:11.272464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:11.272420 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h"] Nov 03 20:41:11.277521 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:11.277475 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz"] Nov 03 20:41:11.277722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:11.277699 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" podUID="4b43d3ba-a346-45d6-8077-8a52b42de081" containerName="istio-proxy" containerID="cri-o://1b5c2d5036db0940f7c2add6187517d59bd2c5c0560de10c0ec0d49cbf6bb032" gracePeriod=30 Nov 03 20:41:11.280429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:11.280400 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v"] Nov 03 20:41:11.576115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:11.576083 5059 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" secret="" err="secret \"e2e-worker-w3-9f46e5-openshift-default-dockercfg-4djl8\" not found" Nov 03 20:41:11.576290 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:11.576195 5059 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" secret="" err="secret \"e2e-worker-w3-9f46e5-openshift-default-dockercfg-4djl8\" not found" Nov 03 20:41:12.497272 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:12.497236 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-v5xhc_d50891bc-16c2-497d-8e00-b79ef492ab6e/manager/0.log" Nov 03 20:41:12.580890 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:12.580827 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" podUID="69597629-8533-4612-8f61-7e3cc63ecea6" containerName="istio-proxy" containerID="cri-o://18ec72802d4c036166466a28084769c1609327a8f03d2e21981f0d1acc78e272" gracePeriod=30 Nov 03 20:41:12.581069 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:12.580877 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" podUID="1411a3da-bb2f-465f-a391-7c3063fbe83d" containerName="istio-proxy" containerID="cri-o://b9ce56eda5802519fa78d6929644c7783a43e7adefc0b3c42a6a007603216778" gracePeriod=30 Nov 03 20:41:13.659755 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:13.659721 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dcrh4_f44d81a0-55a2-4021-b34f-3400deb30c82/manager/0.log" Nov 03 20:41:16.499559 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.499524 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-fzr95_191c7922-07dd-4c9c-89d2-d804f638854d/manager/0.log" Nov 03 20:41:16.526747 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.526720 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:41:16.598571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.598536 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-workload-socket\") pod \"4b43d3ba-a346-45d6-8077-8a52b42de081\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " Nov 03 20:41:16.599455 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.599422 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-data\") pod \"4b43d3ba-a346-45d6-8077-8a52b42de081\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " Nov 03 20:41:16.599589 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.599505 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-envoy\") pod \"4b43d3ba-a346-45d6-8077-8a52b42de081\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " Nov 03 20:41:16.599589 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.599542 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-credential-socket\") pod \"4b43d3ba-a346-45d6-8077-8a52b42de081\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " Nov 03 20:41:16.599589 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.599576 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-workload-certs\") pod \"4b43d3ba-a346-45d6-8077-8a52b42de081\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " Nov 03 20:41:16.599765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.599613 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-podinfo\") pod \"4b43d3ba-a346-45d6-8077-8a52b42de081\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " Nov 03 20:41:16.599765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.599686 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b43d3ba-a346-45d6-8077-8a52b42de081-kube-api-access-5b5t4\" (UniqueName: \"kubernetes.io/projected/4b43d3ba-a346-45d6-8077-8a52b42de081-kube-api-access-5b5t4\") pod \"4b43d3ba-a346-45d6-8077-8a52b42de081\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " Nov 03 20:41:16.599765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.599717 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-token\" (UniqueName: \"kubernetes.io/projected/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-token\") pod \"4b43d3ba-a346-45d6-8077-8a52b42de081\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " Nov 03 20:41:16.599864 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.599825 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b43d3ba-a346-45d6-8077-8a52b42de081-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b43d3ba-a346-45d6-8077-8a52b42de081-istiod-ca-cert\") pod \"4b43d3ba-a346-45d6-8077-8a52b42de081\" (UID: \"4b43d3ba-a346-45d6-8077-8a52b42de081\") " Nov 03 20:41:16.600596 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.600568 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-workload-socket" pod "4b43d3ba-a346-45d6-8077-8a52b42de081" (UID: "4b43d3ba-a346-45d6-8077-8a52b42de081"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:41:16.600932 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.600901 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-workload-certs" pod "4b43d3ba-a346-45d6-8077-8a52b42de081" (UID: "4b43d3ba-a346-45d6-8077-8a52b42de081"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:41:16.601120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.601099 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-data" pod "4b43d3ba-a346-45d6-8077-8a52b42de081" (UID: "4b43d3ba-a346-45d6-8077-8a52b42de081"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:41:16.601266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.601245 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b43d3ba-a346-45d6-8077-8a52b42de081-istiod-ca-cert" pod "4b43d3ba-a346-45d6-8077-8a52b42de081" (UID: "4b43d3ba-a346-45d6-8077-8a52b42de081"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:16.601620 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.601599 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-credential-socket" pod "4b43d3ba-a346-45d6-8077-8a52b42de081" (UID: "4b43d3ba-a346-45d6-8077-8a52b42de081"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:41:16.604819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.604727 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-podinfo" pod "4b43d3ba-a346-45d6-8077-8a52b42de081" (UID: "4b43d3ba-a346-45d6-8077-8a52b42de081"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:41:16.604819 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.604781 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-token" pod "4b43d3ba-a346-45d6-8077-8a52b42de081" (UID: "4b43d3ba-a346-45d6-8077-8a52b42de081"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:16.605778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.605652 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b43d3ba-a346-45d6-8077-8a52b42de081-kube-api-access-5b5t4" pod "4b43d3ba-a346-45d6-8077-8a52b42de081" (UID: "4b43d3ba-a346-45d6-8077-8a52b42de081"). InnerVolumeSpecName "kube-api-access-5b5t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:16.606931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.606906 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-envoy" pod "4b43d3ba-a346-45d6-8077-8a52b42de081" (UID: "4b43d3ba-a346-45d6-8077-8a52b42de081"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:41:16.607328 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.607303 5059 generic.go:356] "Generic (PLEG): container finished" podID="4b43d3ba-a346-45d6-8077-8a52b42de081" containerID="1b5c2d5036db0940f7c2add6187517d59bd2c5c0560de10c0ec0d49cbf6bb032" exitCode=0 Nov 03 20:41:16.607405 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.607380 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" Nov 03 20:41:16.607501 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.607477 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" event={"ID":"4b43d3ba-a346-45d6-8077-8a52b42de081","Type":"ContainerDied","Data":"1b5c2d5036db0940f7c2add6187517d59bd2c5c0560de10c0ec0d49cbf6bb032"} Nov 03 20:41:16.607588 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.607538 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz" event={"ID":"4b43d3ba-a346-45d6-8077-8a52b42de081","Type":"ContainerDied","Data":"076de4cff2c0552482a581fd5c826904e8d033b83f6fa9bd083ed79471c950bc"} Nov 03 20:41:16.607588 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.607575 5059 scope.go:122] "RemoveContainer" containerID="1b5c2d5036db0940f7c2add6187517d59bd2c5c0560de10c0ec0d49cbf6bb032" Nov 03 20:41:16.626873 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.626850 5059 scope.go:122] "RemoveContainer" containerID="1b5c2d5036db0940f7c2add6187517d59bd2c5c0560de10c0ec0d49cbf6bb032" Nov 03 20:41:16.627186 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:41:16.627163 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b5c2d5036db0940f7c2add6187517d59bd2c5c0560de10c0ec0d49cbf6bb032\": container with ID starting with 1b5c2d5036db0940f7c2add6187517d59bd2c5c0560de10c0ec0d49cbf6bb032 not found: ID does not exist" containerID="1b5c2d5036db0940f7c2add6187517d59bd2c5c0560de10c0ec0d49cbf6bb032" Nov 03 20:41:16.627280 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.627192 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b5c2d5036db0940f7c2add6187517d59bd2c5c0560de10c0ec0d49cbf6bb032"} err="failed to get container status \"1b5c2d5036db0940f7c2add6187517d59bd2c5c0560de10c0ec0d49cbf6bb032\": rpc error: code = NotFound desc = could not find container \"1b5c2d5036db0940f7c2add6187517d59bd2c5c0560de10c0ec0d49cbf6bb032\": container with ID starting with 1b5c2d5036db0940f7c2add6187517d59bd2c5c0560de10c0ec0d49cbf6bb032 not found: ID does not exist" Nov 03 20:41:16.639522 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.639499 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz"] Nov 03 20:41:16.644499 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.644473 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lzkcz"] Nov 03 20:41:16.701371 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.701348 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:16.701371 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.701370 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:16.701497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.701380 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:16.701497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.701389 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:16.701497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.701400 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b5t4\" (UniqueName: \"kubernetes.io/projected/4b43d3ba-a346-45d6-8077-8a52b42de081-kube-api-access-5b5t4\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:16.701497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.701409 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:16.701497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.701416 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b43d3ba-a346-45d6-8077-8a52b42de081-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:16.701497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.701426 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:16.701497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:16.701434 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b43d3ba-a346-45d6-8077-8a52b42de081-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:17.831701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.831655 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:17.856781 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.856753 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b43d3ba-a346-45d6-8077-8a52b42de081" path="/var/lib/kubelet/pods/4b43d3ba-a346-45d6-8077-8a52b42de081/volumes" Nov 03 20:41:17.857981 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.857955 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:17.912673 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.912630 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69597629-8533-4612-8f61-7e3cc63ecea6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69597629-8533-4612-8f61-7e3cc63ecea6-istio-podinfo\") pod \"69597629-8533-4612-8f61-7e3cc63ecea6\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " Nov 03 20:41:17.912844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.912685 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69597629-8533-4612-8f61-7e3cc63ecea6-istio-token\" (UniqueName: \"kubernetes.io/projected/69597629-8533-4612-8f61-7e3cc63ecea6-istio-token\") pod \"69597629-8533-4612-8f61-7e3cc63ecea6\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " Nov 03 20:41:17.912844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.912715 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-data\") pod \"1411a3da-bb2f-465f-a391-7c3063fbe83d\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " Nov 03 20:41:17.912844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.912757 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1411a3da-bb2f-465f-a391-7c3063fbe83d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1411a3da-bb2f-465f-a391-7c3063fbe83d-istiod-ca-cert\") pod \"1411a3da-bb2f-465f-a391-7c3063fbe83d\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " Nov 03 20:41:17.912844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.912774 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1411a3da-bb2f-465f-a391-7c3063fbe83d-kube-api-access-sblt7\" (UniqueName: \"kubernetes.io/projected/1411a3da-bb2f-465f-a391-7c3063fbe83d-kube-api-access-sblt7\") pod \"1411a3da-bb2f-465f-a391-7c3063fbe83d\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " Nov 03 20:41:17.912844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.912804 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-envoy\") pod \"1411a3da-bb2f-465f-a391-7c3063fbe83d\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " Nov 03 20:41:17.912844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.912830 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-istio-data\") pod \"69597629-8533-4612-8f61-7e3cc63ecea6\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " Nov 03 20:41:17.913169 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.912880 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-token\" (UniqueName: \"kubernetes.io/projected/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-token\") pod \"1411a3da-bb2f-465f-a391-7c3063fbe83d\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " Nov 03 20:41:17.913169 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.912908 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-workload-socket\") pod \"69597629-8533-4612-8f61-7e3cc63ecea6\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " Nov 03 20:41:17.913169 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.912938 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-istio-envoy\") pod \"69597629-8533-4612-8f61-7e3cc63ecea6\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " Nov 03 20:41:17.913169 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.912976 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-credential-socket\") pod \"69597629-8533-4612-8f61-7e3cc63ecea6\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " Nov 03 20:41:17.913169 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.913016 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-workload-certs\") pod \"69597629-8533-4612-8f61-7e3cc63ecea6\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " Nov 03 20:41:17.913169 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.913043 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-data" pod "1411a3da-bb2f-465f-a391-7c3063fbe83d" (UID: "1411a3da-bb2f-465f-a391-7c3063fbe83d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:41:17.913169 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.913055 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69597629-8533-4612-8f61-7e3cc63ecea6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69597629-8533-4612-8f61-7e3cc63ecea6-istiod-ca-cert\") pod \"69597629-8533-4612-8f61-7e3cc63ecea6\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " Nov 03 20:41:17.913169 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.913122 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-podinfo\") pod \"1411a3da-bb2f-465f-a391-7c3063fbe83d\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " Nov 03 20:41:17.913169 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.913159 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1411a3da-bb2f-465f-a391-7c3063fbe83d-istiod-ca-cert" pod "1411a3da-bb2f-465f-a391-7c3063fbe83d" (UID: "1411a3da-bb2f-465f-a391-7c3063fbe83d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:17.913611 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.913182 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-credential-socket\") pod \"1411a3da-bb2f-465f-a391-7c3063fbe83d\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " Nov 03 20:41:17.913611 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.913255 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-workload-certs\") pod \"1411a3da-bb2f-465f-a391-7c3063fbe83d\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " Nov 03 20:41:17.913611 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.913293 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-workload-socket\") pod \"1411a3da-bb2f-465f-a391-7c3063fbe83d\" (UID: \"1411a3da-bb2f-465f-a391-7c3063fbe83d\") " Nov 03 20:41:17.913611 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.913327 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69597629-8533-4612-8f61-7e3cc63ecea6-kube-api-access-pgqsx\" (UniqueName: \"kubernetes.io/projected/69597629-8533-4612-8f61-7e3cc63ecea6-kube-api-access-pgqsx\") pod \"69597629-8533-4612-8f61-7e3cc63ecea6\" (UID: \"69597629-8533-4612-8f61-7e3cc63ecea6\") " Nov 03 20:41:17.913611 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.913355 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69597629-8533-4612-8f61-7e3cc63ecea6-istiod-ca-cert" pod "69597629-8533-4612-8f61-7e3cc63ecea6" (UID: "69597629-8533-4612-8f61-7e3cc63ecea6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:17.913611 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.913573 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-workload-certs" pod "1411a3da-bb2f-465f-a391-7c3063fbe83d" (UID: "1411a3da-bb2f-465f-a391-7c3063fbe83d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:41:17.913961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.913699 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:17.913961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.913720 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1411a3da-bb2f-465f-a391-7c3063fbe83d-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:17.913961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.913736 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69597629-8533-4612-8f61-7e3cc63ecea6-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:17.913961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.913751 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:17.913961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.913917 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-workload-socket" pod "1411a3da-bb2f-465f-a391-7c3063fbe83d" (UID: "1411a3da-bb2f-465f-a391-7c3063fbe83d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:41:17.914233 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.913981 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-credential-socket" pod "1411a3da-bb2f-465f-a391-7c3063fbe83d" (UID: "1411a3da-bb2f-465f-a391-7c3063fbe83d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:41:17.914233 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.914137 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-workload-socket" pod "69597629-8533-4612-8f61-7e3cc63ecea6" (UID: "69597629-8533-4612-8f61-7e3cc63ecea6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:41:17.914233 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.914197 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-credential-socket" pod "69597629-8533-4612-8f61-7e3cc63ecea6" (UID: "69597629-8533-4612-8f61-7e3cc63ecea6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:41:17.915420 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.915393 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69597629-8533-4612-8f61-7e3cc63ecea6-istio-podinfo" pod "69597629-8533-4612-8f61-7e3cc63ecea6" (UID: "69597629-8533-4612-8f61-7e3cc63ecea6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:41:17.915704 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.915675 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-workload-certs" pod "69597629-8533-4612-8f61-7e3cc63ecea6" (UID: "69597629-8533-4612-8f61-7e3cc63ecea6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:41:17.915841 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.915711 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-istio-data" pod "69597629-8533-4612-8f61-7e3cc63ecea6" (UID: "69597629-8533-4612-8f61-7e3cc63ecea6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:41:17.915957 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.915743 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-podinfo" pod "1411a3da-bb2f-465f-a391-7c3063fbe83d" (UID: "1411a3da-bb2f-465f-a391-7c3063fbe83d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:41:17.916093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.916034 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69597629-8533-4612-8f61-7e3cc63ecea6-kube-api-access-pgqsx" pod "69597629-8533-4612-8f61-7e3cc63ecea6" (UID: "69597629-8533-4612-8f61-7e3cc63ecea6"). InnerVolumeSpecName "kube-api-access-pgqsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:17.916536 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.916508 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69597629-8533-4612-8f61-7e3cc63ecea6-istio-token" pod "69597629-8533-4612-8f61-7e3cc63ecea6" (UID: "69597629-8533-4612-8f61-7e3cc63ecea6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:17.916630 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.916593 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-envoy" pod "1411a3da-bb2f-465f-a391-7c3063fbe83d" (UID: "1411a3da-bb2f-465f-a391-7c3063fbe83d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:41:17.916749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.916630 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1411a3da-bb2f-465f-a391-7c3063fbe83d-kube-api-access-sblt7" pod "1411a3da-bb2f-465f-a391-7c3063fbe83d" (UID: "1411a3da-bb2f-465f-a391-7c3063fbe83d"). InnerVolumeSpecName "kube-api-access-sblt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:17.917568 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.917547 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-token" pod "1411a3da-bb2f-465f-a391-7c3063fbe83d" (UID: "1411a3da-bb2f-465f-a391-7c3063fbe83d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:17.917735 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:17.917719 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-istio-envoy" pod "69597629-8533-4612-8f61-7e3cc63ecea6" (UID: "69597629-8533-4612-8f61-7e3cc63ecea6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:41:18.014714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.014678 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:18.014714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.014711 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:18.014868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.014722 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:18.014868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.014732 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:18.014868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.014749 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:18.014868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.014761 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:18.014868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.014770 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:18.014868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.014779 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:18.014868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.014787 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgqsx\" (UniqueName: \"kubernetes.io/projected/69597629-8533-4612-8f61-7e3cc63ecea6-kube-api-access-pgqsx\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:18.014868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.014796 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69597629-8533-4612-8f61-7e3cc63ecea6-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:18.014868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.014805 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69597629-8533-4612-8f61-7e3cc63ecea6-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:18.014868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.014819 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sblt7\" (UniqueName: \"kubernetes.io/projected/1411a3da-bb2f-465f-a391-7c3063fbe83d-kube-api-access-sblt7\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:18.014868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.014829 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1411a3da-bb2f-465f-a391-7c3063fbe83d-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:18.014868 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.014838 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69597629-8533-4612-8f61-7e3cc63ecea6-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:18.621796 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.621760 5059 generic.go:356] "Generic (PLEG): container finished" podID="69597629-8533-4612-8f61-7e3cc63ecea6" containerID="18ec72802d4c036166466a28084769c1609327a8f03d2e21981f0d1acc78e272" exitCode=0 Nov 03 20:41:18.621948 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.621828 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" Nov 03 20:41:18.621948 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.621842 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" event={"ID":"69597629-8533-4612-8f61-7e3cc63ecea6","Type":"ContainerDied","Data":"18ec72802d4c036166466a28084769c1609327a8f03d2e21981f0d1acc78e272"} Nov 03 20:41:18.621948 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.621933 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v" event={"ID":"69597629-8533-4612-8f61-7e3cc63ecea6","Type":"ContainerDied","Data":"0bbcaf5ddd2f4cb917d98308662cd9f135fd664e6092641cf86ea6e41b115bc4"} Nov 03 20:41:18.622107 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.621958 5059 scope.go:122] "RemoveContainer" containerID="18ec72802d4c036166466a28084769c1609327a8f03d2e21981f0d1acc78e272" Nov 03 20:41:18.623233 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.623210 5059 generic.go:356] "Generic (PLEG): container finished" podID="1411a3da-bb2f-465f-a391-7c3063fbe83d" containerID="b9ce56eda5802519fa78d6929644c7783a43e7adefc0b3c42a6a007603216778" exitCode=0 Nov 03 20:41:18.623308 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.623273 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" Nov 03 20:41:18.623364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.623302 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" event={"ID":"1411a3da-bb2f-465f-a391-7c3063fbe83d","Type":"ContainerDied","Data":"b9ce56eda5802519fa78d6929644c7783a43e7adefc0b3c42a6a007603216778"} Nov 03 20:41:18.623364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.623349 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h" event={"ID":"1411a3da-bb2f-465f-a391-7c3063fbe83d","Type":"ContainerDied","Data":"8adeb197cb7f3a3aa2114c0d3e414b0e2982795d38d14cb5133324e80c9c7914"} Nov 03 20:41:18.631826 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.631737 5059 scope.go:122] "RemoveContainer" containerID="18ec72802d4c036166466a28084769c1609327a8f03d2e21981f0d1acc78e272" Nov 03 20:41:18.632197 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:41:18.632169 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18ec72802d4c036166466a28084769c1609327a8f03d2e21981f0d1acc78e272\": container with ID starting with 18ec72802d4c036166466a28084769c1609327a8f03d2e21981f0d1acc78e272 not found: ID does not exist" containerID="18ec72802d4c036166466a28084769c1609327a8f03d2e21981f0d1acc78e272" Nov 03 20:41:18.632271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.632206 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ec72802d4c036166466a28084769c1609327a8f03d2e21981f0d1acc78e272"} err="failed to get container status \"18ec72802d4c036166466a28084769c1609327a8f03d2e21981f0d1acc78e272\": rpc error: code = NotFound desc = could not find container \"18ec72802d4c036166466a28084769c1609327a8f03d2e21981f0d1acc78e272\": container with ID starting with 18ec72802d4c036166466a28084769c1609327a8f03d2e21981f0d1acc78e272 not found: ID does not exist" Nov 03 20:41:18.632271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.632228 5059 scope.go:122] "RemoveContainer" containerID="b9ce56eda5802519fa78d6929644c7783a43e7adefc0b3c42a6a007603216778" Nov 03 20:41:18.640914 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.640895 5059 scope.go:122] "RemoveContainer" containerID="b9ce56eda5802519fa78d6929644c7783a43e7adefc0b3c42a6a007603216778" Nov 03 20:41:18.641176 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:41:18.641157 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9ce56eda5802519fa78d6929644c7783a43e7adefc0b3c42a6a007603216778\": container with ID starting with b9ce56eda5802519fa78d6929644c7783a43e7adefc0b3c42a6a007603216778 not found: ID does not exist" containerID="b9ce56eda5802519fa78d6929644c7783a43e7adefc0b3c42a6a007603216778" Nov 03 20:41:18.641242 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.641182 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9ce56eda5802519fa78d6929644c7783a43e7adefc0b3c42a6a007603216778"} err="failed to get container status \"b9ce56eda5802519fa78d6929644c7783a43e7adefc0b3c42a6a007603216778\": rpc error: code = NotFound desc = could not find container \"b9ce56eda5802519fa78d6929644c7783a43e7adefc0b3c42a6a007603216778\": container with ID starting with b9ce56eda5802519fa78d6929644c7783a43e7adefc0b3c42a6a007603216778 not found: ID does not exist" Nov 03 20:41:18.648786 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.648716 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v"] Nov 03 20:41:18.651732 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.651700 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qsd9v"] Nov 03 20:41:18.663998 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.663978 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h"] Nov 03 20:41:18.669249 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:18.669225 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-qkg6h"] Nov 03 20:41:19.854802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:19.854766 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1411a3da-bb2f-465f-a391-7c3063fbe83d" path="/var/lib/kubelet/pods/1411a3da-bb2f-465f-a391-7c3063fbe83d/volumes" Nov 03 20:41:19.855249 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:19.855233 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69597629-8533-4612-8f61-7e3cc63ecea6" path="/var/lib/kubelet/pods/69597629-8533-4612-8f61-7e3cc63ecea6/volumes" Nov 03 20:41:25.838040 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:25.838008 5059 scope.go:122] "RemoveContainer" containerID="a61b0ec38265a7b62ed139983e4c7f404bbb92940a0beb66d08a48f6a22de06b" Nov 03 20:41:25.846850 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:25.846831 5059 scope.go:122] "RemoveContainer" containerID="99a0d9d2e14726b927ef1738be56c6ac25f24a2999b1b04a05311141ddb6c181" Nov 03 20:41:25.856623 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:25.856606 5059 scope.go:122] "RemoveContainer" containerID="4b50d662eaf88e93735d86eb301d9af01f88938380cdae493dd558f07e9ddfaf" Nov 03 20:41:25.864379 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:25.864349 5059 scope.go:122] "RemoveContainer" containerID="6f43ee19f2585664e349b56b9bab1c96bb6a783ac14fe8a9f0e4970b2f827788" Nov 03 20:41:25.871704 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:25.871686 5059 scope.go:122] "RemoveContainer" containerID="de718ea41d179627b84ca65669c1d61a191409b00fbc8e497064dc92a4410c03" Nov 03 20:41:27.024538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:27.024495 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b792ade-c6v6j"] Nov 03 20:41:27.027371 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:27.027349 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b792ade-c6v6j"] Nov 03 20:41:27.854363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:27.854322 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e531663b-ff8f-426c-b4c4-362049ae9294" path="/var/lib/kubelet/pods/e531663b-ff8f-426c-b4c4-362049ae9294/volumes" Nov 03 20:41:38.024273 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:38.024232 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddfa3748-d2b62"] Nov 03 20:41:38.029594 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:38.029565 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddfa3748-d2b62"] Nov 03 20:41:39.854604 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:39.854562 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9e87c74-5443-4f7e-b782-d46308c58b9c" path="/var/lib/kubelet/pods/c9e87c74-5443-4f7e-b782-d46308c58b9c/volumes" Nov 03 20:41:40.603340 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:41:40.603299 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:41:41.603460 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.603402 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm"] Nov 03 20:41:41.603982 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.603961 5059 cpu_manager.go:414] "containerMap: removing container" podUID="69597629-8533-4612-8f61-7e3cc63ecea6" containerName="istio-proxy" Nov 03 20:41:41.604054 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.603983 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69597629-8533-4612-8f61-7e3cc63ecea6" containerName="istio-proxy" Nov 03 20:41:41.604054 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.604003 5059 cpu_manager.go:414] "containerMap: removing container" podUID="1411a3da-bb2f-465f-a391-7c3063fbe83d" containerName="istio-proxy" Nov 03 20:41:41.604054 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.604009 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1411a3da-bb2f-465f-a391-7c3063fbe83d" containerName="istio-proxy" Nov 03 20:41:41.604054 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.604023 5059 cpu_manager.go:414] "containerMap: removing container" podUID="4b43d3ba-a346-45d6-8077-8a52b42de081" containerName="istio-proxy" Nov 03 20:41:41.604054 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.604028 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b43d3ba-a346-45d6-8077-8a52b42de081" containerName="istio-proxy" Nov 03 20:41:41.604319 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.604086 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="69597629-8533-4612-8f61-7e3cc63ecea6" containerName="istio-proxy" Nov 03 20:41:41.604319 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.604096 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1411a3da-bb2f-465f-a391-7c3063fbe83d" containerName="istio-proxy" Nov 03 20:41:41.604319 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.604103 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b43d3ba-a346-45d6-8077-8a52b42de081" containerName="istio-proxy" Nov 03 20:41:41.609248 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.609221 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.617676 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.617635 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d790d69a-openshift-default-dockercfg-xlqnf\"" Nov 03 20:41:41.622402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.622376 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm"] Nov 03 20:41:41.703132 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.703085 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-workload-socket\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.703318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.703151 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-podinfo\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.703318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.703191 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-token\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.703318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.703216 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-data\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.703318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.703267 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-workload-certs\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.703519 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.703343 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e72bc583-3867-48ed-a5b6-56642cd222c9-istiod-ca-cert\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.703519 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.703372 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-envoy\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.703519 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.703417 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-credential-socket\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.703519 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.703474 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm296\" (UniqueName: \"kubernetes.io/projected/e72bc583-3867-48ed-a5b6-56642cd222c9-kube-api-access-pm296\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.804045 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.804007 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-workload-socket\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.804238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.804058 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-podinfo\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.804238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.804085 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-token\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.804238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.804100 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-data\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.804238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.804120 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-workload-certs\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.804238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.804153 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e72bc583-3867-48ed-a5b6-56642cd222c9-istiod-ca-cert\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.804238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.804171 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-envoy\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.804238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.804210 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-credential-socket\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.804238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.804235 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm296\" (UniqueName: \"kubernetes.io/projected/e72bc583-3867-48ed-a5b6-56642cd222c9-kube-api-access-pm296\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.804711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.804544 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-workload-certs\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.804711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.804575 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-data\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.804711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.804695 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-workload-socket\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.804880 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.804784 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-credential-socket\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.804986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.804968 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e72bc583-3867-48ed-a5b6-56642cd222c9-istiod-ca-cert\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.806549 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.806528 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-envoy\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.806908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.806885 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-podinfo\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.818464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.818443 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-token\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.818570 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.818511 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm296\" (UniqueName: \"kubernetes.io/projected/e72bc583-3867-48ed-a5b6-56642cd222c9-kube-api-access-pm296\") pod \"e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:41.925356 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:41.925272 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:42.207627 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:42.207592 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm"] Nov 03 20:41:42.209485 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:41:42.209457 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode72bc583_3867_48ed_a5b6_56642cd222c9.slice/crio-a7d8923ca228d82cb59cac73f18af1082f593e910d63741ba0ac80b538c2708d WatchSource:0}: Error finding container a7d8923ca228d82cb59cac73f18af1082f593e910d63741ba0ac80b538c2708d: Status 404 returned error can't find the container with id a7d8923ca228d82cb59cac73f18af1082f593e910d63741ba0ac80b538c2708d Nov 03 20:41:42.211595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:42.211561 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:41:42.211694 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:42.211623 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:41:42.211742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:42.211655 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:41:42.740863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:42.740824 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" event={"ID":"e72bc583-3867-48ed-a5b6-56642cd222c9","Type":"ContainerStarted","Data":"1ac1d8d071bd35b5e3b867e05bbcd1508144eb7713b3f646b40e7e290b89f552"} Nov 03 20:41:42.741270 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:42.740897 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" event={"ID":"e72bc583-3867-48ed-a5b6-56642cd222c9","Type":"ContainerStarted","Data":"a7d8923ca228d82cb59cac73f18af1082f593e910d63741ba0ac80b538c2708d"} Nov 03 20:41:42.766193 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:42.766136 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" podStartSLOduration=1.766124355 podStartE2EDuration="1.766124355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:42.764797543 +0000 UTC m=+3897.022594800" watchObservedRunningTime="2026-11-03 20:41:42.766124355 +0000 UTC m=+3897.023921612" Nov 03 20:41:42.926114 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:42.926077 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:43.930708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:43.930655 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:44.749816 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:44.749786 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:44.750560 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:44.750540 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:45.368273 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.368229 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws"] Nov 03 20:41:45.373344 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.373234 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws" Nov 03 20:41:45.383449 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.383418 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:41:45.383605 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.383556 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-942q2\"" Nov 03 20:41:45.383901 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.383877 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d790d69a-serving-cert\"" Nov 03 20:41:45.388306 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.388278 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws"] Nov 03 20:41:45.434834 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.434802 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcvzc\" (UniqueName: \"kubernetes.io/projected/7083b660-34b1-479c-9647-e7a4271e16a3-kube-api-access-mcvzc\") pod \"maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws\" (UID: \"7083b660-34b1-479c-9647-e7a4271e16a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws" Nov 03 20:41:45.434968 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.434848 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7083b660-34b1-479c-9647-e7a4271e16a3-metrics-certs\") pod \"maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws\" (UID: \"7083b660-34b1-479c-9647-e7a4271e16a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws" Nov 03 20:41:45.435023 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.434997 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7083b660-34b1-479c-9647-e7a4271e16a3-maas-api-tls\") pod \"maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws\" (UID: \"7083b660-34b1-479c-9647-e7a4271e16a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws" Nov 03 20:41:45.435121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.435099 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7083b660-34b1-479c-9647-e7a4271e16a3-openshift-service-ca\") pod \"maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws\" (UID: \"7083b660-34b1-479c-9647-e7a4271e16a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws" Nov 03 20:41:45.459093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.459064 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h"] Nov 03 20:41:45.462842 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.462822 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h" Nov 03 20:41:45.463521 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.463500 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8"] Nov 03 20:41:45.466784 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.466766 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8" Nov 03 20:41:45.471762 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.471741 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d790d69a\"" Nov 03 20:41:45.475030 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.475005 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h"] Nov 03 20:41:45.478069 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.478046 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8"] Nov 03 20:41:45.536113 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.536079 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpfsg\" (UniqueName: \"kubernetes.io/projected/86b27128-166c-4d7b-abf0-5ebb0a846f5c-kube-api-access-dpfsg\") pod \"payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h\" (UID: \"86b27128-166c-4d7b-abf0-5ebb0a846f5c\") " pod="openshift-ingress/payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h" Nov 03 20:41:45.536239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.536117 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7083b660-34b1-479c-9647-e7a4271e16a3-maas-api-tls\") pod \"maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws\" (UID: \"7083b660-34b1-479c-9647-e7a4271e16a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws" Nov 03 20:41:45.536239 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.536136 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b27128-166c-4d7b-abf0-5ebb0a846f5c-plugins-config-volume\") pod \"payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h\" (UID: \"86b27128-166c-4d7b-abf0-5ebb0a846f5c\") " pod="openshift-ingress/payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h" Nov 03 20:41:45.536309 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.536236 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtxgz\" (UniqueName: \"kubernetes.io/projected/3d7eb2d5-2413-4e4b-83e2-87475614c70d-kube-api-access-qtxgz\") pod \"payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8\" (UID: \"3d7eb2d5-2413-4e4b-83e2-87475614c70d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8" Nov 03 20:41:45.536309 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.536302 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7083b660-34b1-479c-9647-e7a4271e16a3-openshift-service-ca\") pod \"maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws\" (UID: \"7083b660-34b1-479c-9647-e7a4271e16a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws" Nov 03 20:41:45.536373 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.536325 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcvzc\" (UniqueName: \"kubernetes.io/projected/7083b660-34b1-479c-9647-e7a4271e16a3-kube-api-access-mcvzc\") pod \"maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws\" (UID: \"7083b660-34b1-479c-9647-e7a4271e16a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws" Nov 03 20:41:45.536373 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.536350 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d7eb2d5-2413-4e4b-83e2-87475614c70d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8\" (UID: \"3d7eb2d5-2413-4e4b-83e2-87475614c70d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8" Nov 03 20:41:45.536455 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.536433 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7083b660-34b1-479c-9647-e7a4271e16a3-metrics-certs\") pod \"maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws\" (UID: \"7083b660-34b1-479c-9647-e7a4271e16a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws" Nov 03 20:41:45.536917 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.536894 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7083b660-34b1-479c-9647-e7a4271e16a3-openshift-service-ca\") pod \"maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws\" (UID: \"7083b660-34b1-479c-9647-e7a4271e16a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws" Nov 03 20:41:45.538608 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.538585 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7083b660-34b1-479c-9647-e7a4271e16a3-maas-api-tls\") pod \"maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws\" (UID: \"7083b660-34b1-479c-9647-e7a4271e16a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws" Nov 03 20:41:45.538784 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.538768 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7083b660-34b1-479c-9647-e7a4271e16a3-metrics-certs\") pod \"maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws\" (UID: \"7083b660-34b1-479c-9647-e7a4271e16a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws" Nov 03 20:41:45.548876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.548854 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcvzc\" (UniqueName: \"kubernetes.io/projected/7083b660-34b1-479c-9647-e7a4271e16a3-kube-api-access-mcvzc\") pod \"maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws\" (UID: \"7083b660-34b1-479c-9647-e7a4271e16a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws" Nov 03 20:41:45.637882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.637786 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpfsg\" (UniqueName: \"kubernetes.io/projected/86b27128-166c-4d7b-abf0-5ebb0a846f5c-kube-api-access-dpfsg\") pod \"payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h\" (UID: \"86b27128-166c-4d7b-abf0-5ebb0a846f5c\") " pod="openshift-ingress/payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h" Nov 03 20:41:45.637882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.637844 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b27128-166c-4d7b-abf0-5ebb0a846f5c-plugins-config-volume\") pod \"payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h\" (UID: \"86b27128-166c-4d7b-abf0-5ebb0a846f5c\") " pod="openshift-ingress/payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h" Nov 03 20:41:45.638058 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.637884 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtxgz\" (UniqueName: \"kubernetes.io/projected/3d7eb2d5-2413-4e4b-83e2-87475614c70d-kube-api-access-qtxgz\") pod \"payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8\" (UID: \"3d7eb2d5-2413-4e4b-83e2-87475614c70d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8" Nov 03 20:41:45.638058 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.637964 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d7eb2d5-2413-4e4b-83e2-87475614c70d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8\" (UID: \"3d7eb2d5-2413-4e4b-83e2-87475614c70d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8" Nov 03 20:41:45.638638 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.638617 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d7eb2d5-2413-4e4b-83e2-87475614c70d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8\" (UID: \"3d7eb2d5-2413-4e4b-83e2-87475614c70d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8" Nov 03 20:41:45.638701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.638617 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b27128-166c-4d7b-abf0-5ebb0a846f5c-plugins-config-volume\") pod \"payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h\" (UID: \"86b27128-166c-4d7b-abf0-5ebb0a846f5c\") " pod="openshift-ingress/payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h" Nov 03 20:41:45.650164 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.650138 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtxgz\" (UniqueName: \"kubernetes.io/projected/3d7eb2d5-2413-4e4b-83e2-87475614c70d-kube-api-access-qtxgz\") pod \"payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8\" (UID: \"3d7eb2d5-2413-4e4b-83e2-87475614c70d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8" Nov 03 20:41:45.650292 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.650208 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpfsg\" (UniqueName: \"kubernetes.io/projected/86b27128-166c-4d7b-abf0-5ebb0a846f5c-kube-api-access-dpfsg\") pod \"payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h\" (UID: \"86b27128-166c-4d7b-abf0-5ebb0a846f5c\") " pod="openshift-ingress/payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h" Nov 03 20:41:45.700537 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.700510 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-942q2\"" Nov 03 20:41:45.708635 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.708614 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws" Nov 03 20:41:45.778047 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.777770 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h" Nov 03 20:41:45.787045 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.786803 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8" Nov 03 20:41:45.989497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:45.989471 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws"] Nov 03 20:41:45.991097 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:41:45.991075 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7083b660_34b1_479c_9647_e7a4271e16a3.slice/crio-85e94f3981a3c3d483823db82a67709900f5af0a2545f9350d5733f03951ef49 WatchSource:0}: Error finding container 85e94f3981a3c3d483823db82a67709900f5af0a2545f9350d5733f03951ef49: Status 404 returned error can't find the container with id 85e94f3981a3c3d483823db82a67709900f5af0a2545f9350d5733f03951ef49 Nov 03 20:41:46.067241 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:46.067200 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8"] Nov 03 20:41:46.069301 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:41:46.069273 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7eb2d5_2413_4e4b_83e2_87475614c70d.slice/crio-7f2f1c2909fc341066372af136a0471a085bd22bfc54a11209e390bb07742c58 WatchSource:0}: Error finding container 7f2f1c2909fc341066372af136a0471a085bd22bfc54a11209e390bb07742c58: Status 404 returned error can't find the container with id 7f2f1c2909fc341066372af136a0471a085bd22bfc54a11209e390bb07742c58 Nov 03 20:41:46.070768 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:46.070741 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h"] Nov 03 20:41:46.070856 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:41:46.070787 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b27128_166c_4d7b_abf0_5ebb0a846f5c.slice/crio-e083c6e661db49e2b1e8e72c207c5742b1dfd39e8b6e424cc49b2d0308364088 WatchSource:0}: Error finding container e083c6e661db49e2b1e8e72c207c5742b1dfd39e8b6e424cc49b2d0308364088: Status 404 returned error can't find the container with id e083c6e661db49e2b1e8e72c207c5742b1dfd39e8b6e424cc49b2d0308364088 Nov 03 20:41:46.760709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:46.760604 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8" event={"ID":"3d7eb2d5-2413-4e4b-83e2-87475614c70d","Type":"ContainerStarted","Data":"2cce3298486b983a933756c94e1f62630df237b33c753ef15caf2d81b43c8493"} Nov 03 20:41:46.760709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:46.760677 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8" event={"ID":"3d7eb2d5-2413-4e4b-83e2-87475614c70d","Type":"ContainerStarted","Data":"7f2f1c2909fc341066372af136a0471a085bd22bfc54a11209e390bb07742c58"} Nov 03 20:41:46.760709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:46.760704 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8" Nov 03 20:41:46.762309 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:46.762280 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws" event={"ID":"7083b660-34b1-479c-9647-e7a4271e16a3","Type":"ContainerStarted","Data":"59d49c36ecf2f487a82e14d56fa538bf7d0da0a501d095773a07a0c1fcbd6894"} Nov 03 20:41:46.762446 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:46.762324 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws" event={"ID":"7083b660-34b1-479c-9647-e7a4271e16a3","Type":"ContainerStarted","Data":"85e94f3981a3c3d483823db82a67709900f5af0a2545f9350d5733f03951ef49"} Nov 03 20:41:46.762446 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:46.762354 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws" Nov 03 20:41:46.763924 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:46.763892 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h" event={"ID":"86b27128-166c-4d7b-abf0-5ebb0a846f5c","Type":"ContainerStarted","Data":"28fdc2a6c6596d143c2b046d4811513dfa3b1d09e6c50a60d4280442ebeb60d9"} Nov 03 20:41:46.764022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:46.763949 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h" event={"ID":"86b27128-166c-4d7b-abf0-5ebb0a846f5c","Type":"ContainerStarted","Data":"e083c6e661db49e2b1e8e72c207c5742b1dfd39e8b6e424cc49b2d0308364088"} Nov 03 20:41:46.764069 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:46.764048 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h" Nov 03 20:41:46.780018 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:46.779980 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8" podStartSLOduration=1.7799678239999999 podStartE2EDuration="1.779967824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:46.777572879 +0000 UTC m=+3901.035370129" watchObservedRunningTime="2026-11-03 20:41:46.779967824 +0000 UTC m=+3901.037765080" Nov 03 20:41:46.799215 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:46.799168 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws" podStartSLOduration=1.297288976 podStartE2EDuration="1.79915593s" totalImagesPullingTime="501.866954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:46.796542845 +0000 UTC m=+3901.054340102" watchObservedRunningTime="2026-11-03 20:41:46.79915593 +0000 UTC m=+3901.056953187" Nov 03 20:41:46.815722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:46.815685 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h" podStartSLOduration=1.815674368 podStartE2EDuration="1.815674368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:46.813367951 +0000 UTC m=+3901.071165209" watchObservedRunningTime="2026-11-03 20:41:46.815674368 +0000 UTC m=+3901.073471618" Nov 03 20:41:52.026048 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:52.025959 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce2e52c1-nh59n"] Nov 03 20:41:52.028461 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:52.028435 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce2e52c1-nh59n"] Nov 03 20:41:52.785483 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:52.785452 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws" Nov 03 20:41:53.855651 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:53.855614 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb070666-da11-410c-a670-46da774feef2" path="/var/lib/kubelet/pods/eb070666-da11-410c-a670-46da774feef2/volumes" Nov 03 20:41:56.823732 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:56.823697 5059 generic.go:356] "Generic (PLEG): container finished" podID="e72bc583-3867-48ed-a5b6-56642cd222c9" containerID="1ac1d8d071bd35b5e3b867e05bbcd1508144eb7713b3f646b40e7e290b89f552" exitCode=0 Nov 03 20:41:56.824141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:56.823770 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" event={"ID":"e72bc583-3867-48ed-a5b6-56642cd222c9","Type":"ContainerDied","Data":"1ac1d8d071bd35b5e3b867e05bbcd1508144eb7713b3f646b40e7e290b89f552"} Nov 03 20:41:56.824141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:56.824135 5059 scope.go:122] "RemoveContainer" containerID="1ac1d8d071bd35b5e3b867e05bbcd1508144eb7713b3f646b40e7e290b89f552" Nov 03 20:41:56.825224 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:56.825194 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:41:56.825280 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:56.825265 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:41:56.825315 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:56.825296 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:41:57.785076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:57.785039 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8" Nov 03 20:41:57.785818 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:57.785794 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h" Nov 03 20:41:57.831608 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:57.831571 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" event={"ID":"e72bc583-3867-48ed-a5b6-56642cd222c9","Type":"ContainerStarted","Data":"baf4059ca56187e9149810641c20609196d44cd18b49f20543bad26e1ebb6c01"} Nov 03 20:41:57.926188 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:57.926157 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:57.927363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:57.927337 5059 patch_prober.go:29] interesting pod/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.184:15021/healthz/ready\": dial tcp 10.132.1.184:15021: connect: connection refused" start-of-body= Nov 03 20:41:57.927427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:57.927393 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" podUID="e72bc583-3867-48ed-a5b6-56642cd222c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.184:15021/healthz/ready\": dial tcp 10.132.1.184:15021: connect: connection refused" Nov 03 20:41:58.926602 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:58.926570 5059 patch_prober.go:29] interesting pod/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.184:15021/healthz/ready\": dial tcp 10.132.1.184:15021: connect: connection refused" start-of-body= Nov 03 20:41:58.927020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:58.926623 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" podUID="e72bc583-3867-48ed-a5b6-56642cd222c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.184:15021/healthz/ready\": dial tcp 10.132.1.184:15021: connect: connection refused" Nov 03 20:41:59.750125 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:59.750092 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:41:59.926081 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:59.926049 5059 patch_prober.go:29] interesting pod/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.184:15021/healthz/ready\": dial tcp 10.132.1.184:15021: connect: connection refused" start-of-body= Nov 03 20:41:59.926260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:41:59.926098 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" podUID="e72bc583-3867-48ed-a5b6-56642cd222c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.184:15021/healthz/ready\": dial tcp 10.132.1.184:15021: connect: connection refused" Nov 03 20:42:00.925921 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:00.925887 5059 patch_prober.go:29] interesting pod/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.184:15021/healthz/ready\": dial tcp 10.132.1.184:15021: connect: connection refused" start-of-body= Nov 03 20:42:00.926324 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:00.925943 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" podUID="e72bc583-3867-48ed-a5b6-56642cd222c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.184:15021/healthz/ready\": dial tcp 10.132.1.184:15021: connect: connection refused" Nov 03 20:42:01.926233 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:01.926199 5059 patch_prober.go:29] interesting pod/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.184:15021/healthz/ready\": dial tcp 10.132.1.184:15021: connect: connection refused" start-of-body= Nov 03 20:42:01.926635 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:01.926252 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" podUID="e72bc583-3867-48ed-a5b6-56642cd222c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.184:15021/healthz/ready\": dial tcp 10.132.1.184:15021: connect: connection refused" Nov 03 20:42:02.929966 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:02.929934 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:42:02.930887 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:02.930865 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:42:03.023049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:03.023012 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2390e8d3-6b6b5"] Nov 03 20:42:03.025416 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:03.025386 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2390e8d3-6b6b5"] Nov 03 20:42:03.855118 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:03.855080 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c43a0d3c-3581-4b3b-8e77-9758d17fd710" path="/var/lib/kubelet/pods/c43a0d3c-3581-4b3b-8e77-9758d17fd710/volumes" Nov 03 20:42:05.167536 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:05.167504 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h"] Nov 03 20:42:05.184650 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:05.184613 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8"] Nov 03 20:42:05.903784 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:05.903746 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws"] Nov 03 20:42:05.903964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:05.903931 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws" podUID="7083b660-34b1-479c-9647-e7a4271e16a3" containerName="maas-api" containerID="cri-o://59d49c36ecf2f487a82e14d56fa538bf7d0da0a501d095773a07a0c1fcbd6894" gracePeriod=30 Nov 03 20:42:06.151109 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:06.151081 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws" Nov 03 20:42:06.250714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:06.250587 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7083b660-34b1-479c-9647-e7a4271e16a3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7083b660-34b1-479c-9647-e7a4271e16a3-maas-api-tls\") pod \"7083b660-34b1-479c-9647-e7a4271e16a3\" (UID: \"7083b660-34b1-479c-9647-e7a4271e16a3\") " Nov 03 20:42:06.251180 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:06.250795 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7083b660-34b1-479c-9647-e7a4271e16a3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7083b660-34b1-479c-9647-e7a4271e16a3-metrics-certs\") pod \"7083b660-34b1-479c-9647-e7a4271e16a3\" (UID: \"7083b660-34b1-479c-9647-e7a4271e16a3\") " Nov 03 20:42:06.251180 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:06.250897 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7083b660-34b1-479c-9647-e7a4271e16a3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7083b660-34b1-479c-9647-e7a4271e16a3-openshift-service-ca\") pod \"7083b660-34b1-479c-9647-e7a4271e16a3\" (UID: \"7083b660-34b1-479c-9647-e7a4271e16a3\") " Nov 03 20:42:06.251314 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:06.251212 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7083b660-34b1-479c-9647-e7a4271e16a3-openshift-service-ca" pod "7083b660-34b1-479c-9647-e7a4271e16a3" (UID: "7083b660-34b1-479c-9647-e7a4271e16a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:42:06.252771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:06.252748 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7083b660-34b1-479c-9647-e7a4271e16a3-maas-api-tls" pod "7083b660-34b1-479c-9647-e7a4271e16a3" (UID: "7083b660-34b1-479c-9647-e7a4271e16a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:42:06.253029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:06.253015 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7083b660-34b1-479c-9647-e7a4271e16a3-metrics-certs" pod "7083b660-34b1-479c-9647-e7a4271e16a3" (UID: "7083b660-34b1-479c-9647-e7a4271e16a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:42:06.352048 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:06.352020 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7083b660-34b1-479c-9647-e7a4271e16a3-kube-api-access-mcvzc\" (UniqueName: \"kubernetes.io/projected/7083b660-34b1-479c-9647-e7a4271e16a3-kube-api-access-mcvzc\") pod \"7083b660-34b1-479c-9647-e7a4271e16a3\" (UID: \"7083b660-34b1-479c-9647-e7a4271e16a3\") " Nov 03 20:42:06.352220 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:06.352207 5059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7083b660-34b1-479c-9647-e7a4271e16a3-metrics-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:06.352264 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:06.352222 5059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7083b660-34b1-479c-9647-e7a4271e16a3-openshift-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:06.352264 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:06.352232 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7083b660-34b1-479c-9647-e7a4271e16a3-maas-api-tls\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:06.353981 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:06.353959 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7083b660-34b1-479c-9647-e7a4271e16a3-kube-api-access-mcvzc" pod "7083b660-34b1-479c-9647-e7a4271e16a3" (UID: "7083b660-34b1-479c-9647-e7a4271e16a3"). InnerVolumeSpecName "kube-api-access-mcvzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:42:06.453626 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:06.453596 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcvzc\" (UniqueName: \"kubernetes.io/projected/7083b660-34b1-479c-9647-e7a4271e16a3-kube-api-access-mcvzc\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:06.876655 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:06.876610 5059 generic.go:356] "Generic (PLEG): container finished" podID="7083b660-34b1-479c-9647-e7a4271e16a3" containerID="59d49c36ecf2f487a82e14d56fa538bf7d0da0a501d095773a07a0c1fcbd6894" exitCode=0 Nov 03 20:42:06.876863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:06.876686 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws" Nov 03 20:42:06.876863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:06.876696 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws" event={"ID":"7083b660-34b1-479c-9647-e7a4271e16a3","Type":"ContainerDied","Data":"59d49c36ecf2f487a82e14d56fa538bf7d0da0a501d095773a07a0c1fcbd6894"} Nov 03 20:42:06.876863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:06.876748 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws" event={"ID":"7083b660-34b1-479c-9647-e7a4271e16a3","Type":"ContainerDied","Data":"85e94f3981a3c3d483823db82a67709900f5af0a2545f9350d5733f03951ef49"} Nov 03 20:42:06.876863 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:06.876775 5059 scope.go:122] "RemoveContainer" containerID="59d49c36ecf2f487a82e14d56fa538bf7d0da0a501d095773a07a0c1fcbd6894" Nov 03 20:42:06.886400 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:06.886383 5059 scope.go:122] "RemoveContainer" containerID="59d49c36ecf2f487a82e14d56fa538bf7d0da0a501d095773a07a0c1fcbd6894" Nov 03 20:42:06.886695 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:42:06.886649 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59d49c36ecf2f487a82e14d56fa538bf7d0da0a501d095773a07a0c1fcbd6894\": container with ID starting with 59d49c36ecf2f487a82e14d56fa538bf7d0da0a501d095773a07a0c1fcbd6894 not found: ID does not exist" containerID="59d49c36ecf2f487a82e14d56fa538bf7d0da0a501d095773a07a0c1fcbd6894" Nov 03 20:42:06.886756 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:06.886705 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59d49c36ecf2f487a82e14d56fa538bf7d0da0a501d095773a07a0c1fcbd6894"} err="failed to get container status \"59d49c36ecf2f487a82e14d56fa538bf7d0da0a501d095773a07a0c1fcbd6894\": rpc error: code = NotFound desc = could not find container \"59d49c36ecf2f487a82e14d56fa538bf7d0da0a501d095773a07a0c1fcbd6894\": container with ID starting with 59d49c36ecf2f487a82e14d56fa538bf7d0da0a501d095773a07a0c1fcbd6894 not found: ID does not exist" Nov 03 20:42:06.901084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:06.901058 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws"] Nov 03 20:42:06.905805 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:06.905783 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d790d69a-7fd7d94f5d-blpws"] Nov 03 20:42:07.855940 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:07.855897 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7083b660-34b1-479c-9647-e7a4271e16a3" path="/var/lib/kubelet/pods/7083b660-34b1-479c-9647-e7a4271e16a3/volumes" Nov 03 20:42:10.168276 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.168223 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h" podUID="86b27128-166c-4d7b-abf0-5ebb0a846f5c" containerName="payload-processing" containerID="cri-o://28fdc2a6c6596d143c2b046d4811513dfa3b1d09e6c50a60d4280442ebeb60d9" gracePeriod=25 Nov 03 20:42:10.185903 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.185874 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8" podUID="3d7eb2d5-2413-4e4b-83e2-87475614c70d" containerName="payload-pre-processing" containerID="cri-o://2cce3298486b983a933756c94e1f62630df237b33c753ef15caf2d81b43c8493" gracePeriod=25 Nov 03 20:42:10.449201 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.449176 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h" Nov 03 20:42:10.452184 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.452166 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8" Nov 03 20:42:10.483780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.483757 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86b27128-166c-4d7b-abf0-5ebb0a846f5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b27128-166c-4d7b-abf0-5ebb0a846f5c-plugins-config-volume\") pod \"86b27128-166c-4d7b-abf0-5ebb0a846f5c\" (UID: \"86b27128-166c-4d7b-abf0-5ebb0a846f5c\") " Nov 03 20:42:10.483909 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.483796 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d7eb2d5-2413-4e4b-83e2-87475614c70d-kube-api-access-qtxgz\" (UniqueName: \"kubernetes.io/projected/3d7eb2d5-2413-4e4b-83e2-87475614c70d-kube-api-access-qtxgz\") pod \"3d7eb2d5-2413-4e4b-83e2-87475614c70d\" (UID: \"3d7eb2d5-2413-4e4b-83e2-87475614c70d\") " Nov 03 20:42:10.483909 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.483819 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d7eb2d5-2413-4e4b-83e2-87475614c70d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d7eb2d5-2413-4e4b-83e2-87475614c70d-plugins-config-volume\") pod \"3d7eb2d5-2413-4e4b-83e2-87475614c70d\" (UID: \"3d7eb2d5-2413-4e4b-83e2-87475614c70d\") " Nov 03 20:42:10.483909 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.483885 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b27128-166c-4d7b-abf0-5ebb0a846f5c-kube-api-access-dpfsg\" (UniqueName: \"kubernetes.io/projected/86b27128-166c-4d7b-abf0-5ebb0a846f5c-kube-api-access-dpfsg\") pod \"86b27128-166c-4d7b-abf0-5ebb0a846f5c\" (UID: \"86b27128-166c-4d7b-abf0-5ebb0a846f5c\") " Nov 03 20:42:10.484138 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.484113 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86b27128-166c-4d7b-abf0-5ebb0a846f5c-plugins-config-volume" pod "86b27128-166c-4d7b-abf0-5ebb0a846f5c" (UID: "86b27128-166c-4d7b-abf0-5ebb0a846f5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:42:10.484219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.484199 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d7eb2d5-2413-4e4b-83e2-87475614c70d-plugins-config-volume" pod "3d7eb2d5-2413-4e4b-83e2-87475614c70d" (UID: "3d7eb2d5-2413-4e4b-83e2-87475614c70d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:42:10.488793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.488763 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b27128-166c-4d7b-abf0-5ebb0a846f5c-kube-api-access-dpfsg" pod "86b27128-166c-4d7b-abf0-5ebb0a846f5c" (UID: "86b27128-166c-4d7b-abf0-5ebb0a846f5c"). InnerVolumeSpecName "kube-api-access-dpfsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:42:10.489840 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.489815 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7eb2d5-2413-4e4b-83e2-87475614c70d-kube-api-access-qtxgz" pod "3d7eb2d5-2413-4e4b-83e2-87475614c70d" (UID: "3d7eb2d5-2413-4e4b-83e2-87475614c70d"). InnerVolumeSpecName "kube-api-access-qtxgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:42:10.585498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.585471 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtxgz\" (UniqueName: \"kubernetes.io/projected/3d7eb2d5-2413-4e4b-83e2-87475614c70d-kube-api-access-qtxgz\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:10.585498 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.585499 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d7eb2d5-2413-4e4b-83e2-87475614c70d-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:10.585647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.585514 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpfsg\" (UniqueName: \"kubernetes.io/projected/86b27128-166c-4d7b-abf0-5ebb0a846f5c-kube-api-access-dpfsg\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:10.585647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.585526 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b27128-166c-4d7b-abf0-5ebb0a846f5c-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:10.603435 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:42:10.603407 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:42:10.899405 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.899374 5059 generic.go:356] "Generic (PLEG): container finished" podID="3d7eb2d5-2413-4e4b-83e2-87475614c70d" containerID="2cce3298486b983a933756c94e1f62630df237b33c753ef15caf2d81b43c8493" exitCode=0 Nov 03 20:42:10.899575 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.899417 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8" Nov 03 20:42:10.899575 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.899455 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8" event={"ID":"3d7eb2d5-2413-4e4b-83e2-87475614c70d","Type":"ContainerDied","Data":"2cce3298486b983a933756c94e1f62630df237b33c753ef15caf2d81b43c8493"} Nov 03 20:42:10.899575 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.899509 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8" event={"ID":"3d7eb2d5-2413-4e4b-83e2-87475614c70d","Type":"ContainerDied","Data":"7f2f1c2909fc341066372af136a0471a085bd22bfc54a11209e390bb07742c58"} Nov 03 20:42:10.899575 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.899541 5059 scope.go:122] "RemoveContainer" containerID="2cce3298486b983a933756c94e1f62630df237b33c753ef15caf2d81b43c8493" Nov 03 20:42:10.901102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.901072 5059 generic.go:356] "Generic (PLEG): container finished" podID="86b27128-166c-4d7b-abf0-5ebb0a846f5c" containerID="28fdc2a6c6596d143c2b046d4811513dfa3b1d09e6c50a60d4280442ebeb60d9" exitCode=0 Nov 03 20:42:10.901223 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.901127 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h" Nov 03 20:42:10.901223 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.901172 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h" event={"ID":"86b27128-166c-4d7b-abf0-5ebb0a846f5c","Type":"ContainerDied","Data":"28fdc2a6c6596d143c2b046d4811513dfa3b1d09e6c50a60d4280442ebeb60d9"} Nov 03 20:42:10.901223 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.901208 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h" event={"ID":"86b27128-166c-4d7b-abf0-5ebb0a846f5c","Type":"ContainerDied","Data":"e083c6e661db49e2b1e8e72c207c5742b1dfd39e8b6e424cc49b2d0308364088"} Nov 03 20:42:10.916364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.913985 5059 scope.go:122] "RemoveContainer" containerID="2cce3298486b983a933756c94e1f62630df237b33c753ef15caf2d81b43c8493" Nov 03 20:42:10.916364 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:42:10.914314 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cce3298486b983a933756c94e1f62630df237b33c753ef15caf2d81b43c8493\": container with ID starting with 2cce3298486b983a933756c94e1f62630df237b33c753ef15caf2d81b43c8493 not found: ID does not exist" containerID="2cce3298486b983a933756c94e1f62630df237b33c753ef15caf2d81b43c8493" Nov 03 20:42:10.916364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.914351 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cce3298486b983a933756c94e1f62630df237b33c753ef15caf2d81b43c8493"} err="failed to get container status \"2cce3298486b983a933756c94e1f62630df237b33c753ef15caf2d81b43c8493\": rpc error: code = NotFound desc = could not find container \"2cce3298486b983a933756c94e1f62630df237b33c753ef15caf2d81b43c8493\": container with ID starting with 2cce3298486b983a933756c94e1f62630df237b33c753ef15caf2d81b43c8493 not found: ID does not exist" Nov 03 20:42:10.916364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.914377 5059 scope.go:122] "RemoveContainer" containerID="28fdc2a6c6596d143c2b046d4811513dfa3b1d09e6c50a60d4280442ebeb60d9" Nov 03 20:42:10.923366 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.923347 5059 scope.go:122] "RemoveContainer" containerID="28fdc2a6c6596d143c2b046d4811513dfa3b1d09e6c50a60d4280442ebeb60d9" Nov 03 20:42:10.923612 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:42:10.923595 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28fdc2a6c6596d143c2b046d4811513dfa3b1d09e6c50a60d4280442ebeb60d9\": container with ID starting with 28fdc2a6c6596d143c2b046d4811513dfa3b1d09e6c50a60d4280442ebeb60d9 not found: ID does not exist" containerID="28fdc2a6c6596d143c2b046d4811513dfa3b1d09e6c50a60d4280442ebeb60d9" Nov 03 20:42:10.923651 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.923619 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28fdc2a6c6596d143c2b046d4811513dfa3b1d09e6c50a60d4280442ebeb60d9"} err="failed to get container status \"28fdc2a6c6596d143c2b046d4811513dfa3b1d09e6c50a60d4280442ebeb60d9\": rpc error: code = NotFound desc = could not find container \"28fdc2a6c6596d143c2b046d4811513dfa3b1d09e6c50a60d4280442ebeb60d9\": container with ID starting with 28fdc2a6c6596d143c2b046d4811513dfa3b1d09e6c50a60d4280442ebeb60d9 not found: ID does not exist" Nov 03 20:42:10.932049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.932015 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h"] Nov 03 20:42:10.936417 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.936394 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d790d69a-5c7d995866-ppm5h"] Nov 03 20:42:10.946510 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.946488 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8"] Nov 03 20:42:10.950686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:10.950633 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d790d69a-64c65cc55f-59vp8"] Nov 03 20:42:11.451310 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:11.451226 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm"] Nov 03 20:42:11.451783 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:11.451486 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" podUID="e72bc583-3867-48ed-a5b6-56642cd222c9" containerName="istio-proxy" containerID="cri-o://baf4059ca56187e9149810641c20609196d44cd18b49f20543bad26e1ebb6c01" gracePeriod=30 Nov 03 20:42:11.855952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:11.855920 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d7eb2d5-2413-4e4b-83e2-87475614c70d" path="/var/lib/kubelet/pods/3d7eb2d5-2413-4e4b-83e2-87475614c70d/volumes" Nov 03 20:42:11.856359 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:11.856344 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86b27128-166c-4d7b-abf0-5ebb0a846f5c" path="/var/lib/kubelet/pods/86b27128-166c-4d7b-abf0-5ebb0a846f5c/volumes" Nov 03 20:42:14.314365 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.314318 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55c765799f-vxzm4"] Nov 03 20:42:14.314923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.314903 5059 cpu_manager.go:414] "containerMap: removing container" podUID="7083b660-34b1-479c-9647-e7a4271e16a3" containerName="maas-api" Nov 03 20:42:14.314968 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.314927 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7083b660-34b1-479c-9647-e7a4271e16a3" containerName="maas-api" Nov 03 20:42:14.314968 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.314951 5059 cpu_manager.go:414] "containerMap: removing container" podUID="86b27128-166c-4d7b-abf0-5ebb0a846f5c" containerName="payload-processing" Nov 03 20:42:14.314968 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.314960 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86b27128-166c-4d7b-abf0-5ebb0a846f5c" containerName="payload-processing" Nov 03 20:42:14.315064 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.315005 5059 cpu_manager.go:414] "containerMap: removing container" podUID="3d7eb2d5-2413-4e4b-83e2-87475614c70d" containerName="payload-pre-processing" Nov 03 20:42:14.315064 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.315014 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d7eb2d5-2413-4e4b-83e2-87475614c70d" containerName="payload-pre-processing" Nov 03 20:42:14.315126 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.315090 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d7eb2d5-2413-4e4b-83e2-87475614c70d" containerName="payload-pre-processing" Nov 03 20:42:14.315126 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.315100 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7083b660-34b1-479c-9647-e7a4271e16a3" containerName="maas-api" Nov 03 20:42:14.315126 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.315110 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="86b27128-166c-4d7b-abf0-5ebb0a846f5c" containerName="payload-processing" Nov 03 20:42:14.320256 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.320228 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55c765799f-vxzm4" Nov 03 20:42:14.327108 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.327080 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 20:42:14.327944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.327918 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55c765799f-vxzm4"] Nov 03 20:42:14.328303 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.328280 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xc6tf\"" Nov 03 20:42:14.328303 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.328290 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 20:42:14.328465 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.328352 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 20:42:14.418999 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.418965 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a6e3c07-e75c-4bee-9699-5a421ecc0005-metrics-certs\") pod \"maas-controller-55c765799f-vxzm4\" (UID: \"0a6e3c07-e75c-4bee-9699-5a421ecc0005\") " pod="opendatahub/maas-controller-55c765799f-vxzm4" Nov 03 20:42:14.418999 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.419005 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwt4c\" (UniqueName: \"kubernetes.io/projected/0a6e3c07-e75c-4bee-9699-5a421ecc0005-kube-api-access-hwt4c\") pod \"maas-controller-55c765799f-vxzm4\" (UID: \"0a6e3c07-e75c-4bee-9699-5a421ecc0005\") " pod="opendatahub/maas-controller-55c765799f-vxzm4" Nov 03 20:42:14.419253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.419197 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a6e3c07-e75c-4bee-9699-5a421ecc0005-webhook-cert\") pod \"maas-controller-55c765799f-vxzm4\" (UID: \"0a6e3c07-e75c-4bee-9699-5a421ecc0005\") " pod="opendatahub/maas-controller-55c765799f-vxzm4" Nov 03 20:42:14.520075 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.520046 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a6e3c07-e75c-4bee-9699-5a421ecc0005-webhook-cert\") pod \"maas-controller-55c765799f-vxzm4\" (UID: \"0a6e3c07-e75c-4bee-9699-5a421ecc0005\") " pod="opendatahub/maas-controller-55c765799f-vxzm4" Nov 03 20:42:14.520259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.520092 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a6e3c07-e75c-4bee-9699-5a421ecc0005-metrics-certs\") pod \"maas-controller-55c765799f-vxzm4\" (UID: \"0a6e3c07-e75c-4bee-9699-5a421ecc0005\") " pod="opendatahub/maas-controller-55c765799f-vxzm4" Nov 03 20:42:14.520259 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.520126 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwt4c\" (UniqueName: \"kubernetes.io/projected/0a6e3c07-e75c-4bee-9699-5a421ecc0005-kube-api-access-hwt4c\") pod \"maas-controller-55c765799f-vxzm4\" (UID: \"0a6e3c07-e75c-4bee-9699-5a421ecc0005\") " pod="opendatahub/maas-controller-55c765799f-vxzm4" Nov 03 20:42:14.522885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.522848 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a6e3c07-e75c-4bee-9699-5a421ecc0005-metrics-certs\") pod \"maas-controller-55c765799f-vxzm4\" (UID: \"0a6e3c07-e75c-4bee-9699-5a421ecc0005\") " pod="opendatahub/maas-controller-55c765799f-vxzm4" Nov 03 20:42:14.523011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.522977 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a6e3c07-e75c-4bee-9699-5a421ecc0005-webhook-cert\") pod \"maas-controller-55c765799f-vxzm4\" (UID: \"0a6e3c07-e75c-4bee-9699-5a421ecc0005\") " pod="opendatahub/maas-controller-55c765799f-vxzm4" Nov 03 20:42:14.532740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.532716 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwt4c\" (UniqueName: \"kubernetes.io/projected/0a6e3c07-e75c-4bee-9699-5a421ecc0005-kube-api-access-hwt4c\") pod \"maas-controller-55c765799f-vxzm4\" (UID: \"0a6e3c07-e75c-4bee-9699-5a421ecc0005\") " pod="opendatahub/maas-controller-55c765799f-vxzm4" Nov 03 20:42:14.638099 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.638069 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55c765799f-vxzm4" Nov 03 20:42:14.954955 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:14.954921 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55c765799f-vxzm4"] Nov 03 20:42:15.936440 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:15.936387 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55c765799f-vxzm4" event={"ID":"0a6e3c07-e75c-4bee-9699-5a421ecc0005","Type":"ContainerStarted","Data":"f7af9c96c8137ebb1fa3f569a77a95892549c9249f5ae11ad624a13c6c8792f9"} Nov 03 20:42:16.742592 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.742532 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:42:16.845723 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.845682 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-workload-socket\") pod \"e72bc583-3867-48ed-a5b6-56642cd222c9\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " Nov 03 20:42:16.845907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.845759 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-credential-socket\") pod \"e72bc583-3867-48ed-a5b6-56642cd222c9\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " Nov 03 20:42:16.845907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.845796 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e72bc583-3867-48ed-a5b6-56642cd222c9-kube-api-access-pm296\" (UniqueName: \"kubernetes.io/projected/e72bc583-3867-48ed-a5b6-56642cd222c9-kube-api-access-pm296\") pod \"e72bc583-3867-48ed-a5b6-56642cd222c9\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " Nov 03 20:42:16.845907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.845880 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-data\") pod \"e72bc583-3867-48ed-a5b6-56642cd222c9\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " Nov 03 20:42:16.846083 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.845911 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-podinfo\") pod \"e72bc583-3867-48ed-a5b6-56642cd222c9\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " Nov 03 20:42:16.846083 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.845949 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e72bc583-3867-48ed-a5b6-56642cd222c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e72bc583-3867-48ed-a5b6-56642cd222c9-istiod-ca-cert\") pod \"e72bc583-3867-48ed-a5b6-56642cd222c9\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " Nov 03 20:42:16.846083 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.845992 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-token\" (UniqueName: \"kubernetes.io/projected/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-token\") pod \"e72bc583-3867-48ed-a5b6-56642cd222c9\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " Nov 03 20:42:16.846083 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.846032 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-envoy\") pod \"e72bc583-3867-48ed-a5b6-56642cd222c9\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " Nov 03 20:42:16.846083 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.846060 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-workload-certs\") pod \"e72bc583-3867-48ed-a5b6-56642cd222c9\" (UID: \"e72bc583-3867-48ed-a5b6-56642cd222c9\") " Nov 03 20:42:16.846738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.846529 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-data" pod "e72bc583-3867-48ed-a5b6-56642cd222c9" (UID: "e72bc583-3867-48ed-a5b6-56642cd222c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:42:16.846738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.846703 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-credential-socket" pod "e72bc583-3867-48ed-a5b6-56642cd222c9" (UID: "e72bc583-3867-48ed-a5b6-56642cd222c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:42:16.846738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.846714 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-workload-socket" pod "e72bc583-3867-48ed-a5b6-56642cd222c9" (UID: "e72bc583-3867-48ed-a5b6-56642cd222c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:42:16.847404 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.847378 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e72bc583-3867-48ed-a5b6-56642cd222c9-istiod-ca-cert" pod "e72bc583-3867-48ed-a5b6-56642cd222c9" (UID: "e72bc583-3867-48ed-a5b6-56642cd222c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:42:16.847495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.847445 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-workload-certs" pod "e72bc583-3867-48ed-a5b6-56642cd222c9" (UID: "e72bc583-3867-48ed-a5b6-56642cd222c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:42:16.863704 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.862924 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e72bc583-3867-48ed-a5b6-56642cd222c9-kube-api-access-pm296" pod "e72bc583-3867-48ed-a5b6-56642cd222c9" (UID: "e72bc583-3867-48ed-a5b6-56642cd222c9"). InnerVolumeSpecName "kube-api-access-pm296". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:42:16.864040 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.863877 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-envoy" pod "e72bc583-3867-48ed-a5b6-56642cd222c9" (UID: "e72bc583-3867-48ed-a5b6-56642cd222c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:42:16.864040 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.864017 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-token" pod "e72bc583-3867-48ed-a5b6-56642cd222c9" (UID: "e72bc583-3867-48ed-a5b6-56642cd222c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:42:16.864589 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.864552 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-podinfo" pod "e72bc583-3867-48ed-a5b6-56642cd222c9" (UID: "e72bc583-3867-48ed-a5b6-56642cd222c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:42:16.944287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.944249 5059 generic.go:356] "Generic (PLEG): container finished" podID="e72bc583-3867-48ed-a5b6-56642cd222c9" containerID="baf4059ca56187e9149810641c20609196d44cd18b49f20543bad26e1ebb6c01" exitCode=0 Nov 03 20:42:16.944745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.944307 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" Nov 03 20:42:16.944745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.944327 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" event={"ID":"e72bc583-3867-48ed-a5b6-56642cd222c9","Type":"ContainerDied","Data":"baf4059ca56187e9149810641c20609196d44cd18b49f20543bad26e1ebb6c01"} Nov 03 20:42:16.944745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.944395 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm" event={"ID":"e72bc583-3867-48ed-a5b6-56642cd222c9","Type":"ContainerDied","Data":"a7d8923ca228d82cb59cac73f18af1082f593e910d63741ba0ac80b538c2708d"} Nov 03 20:42:16.944745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.944429 5059 scope.go:122] "RemoveContainer" containerID="baf4059ca56187e9149810641c20609196d44cd18b49f20543bad26e1ebb6c01" Nov 03 20:42:16.947073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.947046 5059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-workload-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:16.947195 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.947078 5059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-credential-socket\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:16.947195 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.947095 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm296\" (UniqueName: \"kubernetes.io/projected/e72bc583-3867-48ed-a5b6-56642cd222c9-kube-api-access-pm296\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:16.947195 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.947113 5059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-data\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:16.947195 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.947128 5059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-podinfo\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:16.947195 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.947142 5059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e72bc583-3867-48ed-a5b6-56642cd222c9-istiod-ca-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:16.947195 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.947156 5059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-token\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:16.947195 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.947170 5059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-istio-envoy\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:16.947195 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.947185 5059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e72bc583-3867-48ed-a5b6-56642cd222c9-workload-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:16.955233 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.955217 5059 scope.go:122] "RemoveContainer" containerID="1ac1d8d071bd35b5e3b867e05bbcd1508144eb7713b3f646b40e7e290b89f552" Nov 03 20:42:16.964439 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.964419 5059 scope.go:122] "RemoveContainer" containerID="baf4059ca56187e9149810641c20609196d44cd18b49f20543bad26e1ebb6c01" Nov 03 20:42:16.964759 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:42:16.964738 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baf4059ca56187e9149810641c20609196d44cd18b49f20543bad26e1ebb6c01\": container with ID starting with baf4059ca56187e9149810641c20609196d44cd18b49f20543bad26e1ebb6c01 not found: ID does not exist" containerID="baf4059ca56187e9149810641c20609196d44cd18b49f20543bad26e1ebb6c01" Nov 03 20:42:16.964838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.964768 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baf4059ca56187e9149810641c20609196d44cd18b49f20543bad26e1ebb6c01"} err="failed to get container status \"baf4059ca56187e9149810641c20609196d44cd18b49f20543bad26e1ebb6c01\": rpc error: code = NotFound desc = could not find container \"baf4059ca56187e9149810641c20609196d44cd18b49f20543bad26e1ebb6c01\": container with ID starting with baf4059ca56187e9149810641c20609196d44cd18b49f20543bad26e1ebb6c01 not found: ID does not exist" Nov 03 20:42:16.964838 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.964784 5059 scope.go:122] "RemoveContainer" containerID="1ac1d8d071bd35b5e3b867e05bbcd1508144eb7713b3f646b40e7e290b89f552" Nov 03 20:42:16.965061 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:42:16.965033 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ac1d8d071bd35b5e3b867e05bbcd1508144eb7713b3f646b40e7e290b89f552\": container with ID starting with 1ac1d8d071bd35b5e3b867e05bbcd1508144eb7713b3f646b40e7e290b89f552 not found: ID does not exist" containerID="1ac1d8d071bd35b5e3b867e05bbcd1508144eb7713b3f646b40e7e290b89f552" Nov 03 20:42:16.965118 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.965073 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ac1d8d071bd35b5e3b867e05bbcd1508144eb7713b3f646b40e7e290b89f552"} err="failed to get container status \"1ac1d8d071bd35b5e3b867e05bbcd1508144eb7713b3f646b40e7e290b89f552\": rpc error: code = NotFound desc = could not find container \"1ac1d8d071bd35b5e3b867e05bbcd1508144eb7713b3f646b40e7e290b89f552\": container with ID starting with 1ac1d8d071bd35b5e3b867e05bbcd1508144eb7713b3f646b40e7e290b89f552 not found: ID does not exist" Nov 03 20:42:16.970574 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.970551 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm"] Nov 03 20:42:16.975637 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:16.975606 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d790d69a-openshift-default-7f5f97fc65-cdqbm"] Nov 03 20:42:17.855100 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:17.855064 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e72bc583-3867-48ed-a5b6-56642cd222c9" path="/var/lib/kubelet/pods/e72bc583-3867-48ed-a5b6-56642cd222c9/volumes" Nov 03 20:42:17.951400 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:17.951365 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55c765799f-vxzm4" event={"ID":"0a6e3c07-e75c-4bee-9699-5a421ecc0005","Type":"ContainerStarted","Data":"20740c5fbcd9c0c0a9545d0fa3015464f1dcd65e643687bd7193d6b503467e83"} Nov 03 20:42:17.951776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:17.951537 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55c765799f-vxzm4" Nov 03 20:42:17.979780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:17.979728 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-55c765799f-vxzm4" podStartSLOduration=1.7359667380000001 podStartE2EDuration="3.979712809s" totalImagesPullingTime="2.243746071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:17.972741695 +0000 UTC m=+3932.230538954" watchObservedRunningTime="2026-11-03 20:42:17.979712809 +0000 UTC m=+3932.237510067" Nov 03 20:42:25.954770 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:25.954735 5059 scope.go:122] "RemoveContainer" containerID="2d95d4d5964dcdf538400f2dd8cdba6b546459884cd613156a703f0e2e134293" Nov 03 20:42:28.958781 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:28.958748 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55c765799f-vxzm4" Nov 03 20:42:35.925971 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:35.925936 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czzww"] Nov 03 20:42:35.926346 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:35.926287 5059 cpu_manager.go:414] "containerMap: removing container" podUID="e72bc583-3867-48ed-a5b6-56642cd222c9" containerName="istio-proxy" Nov 03 20:42:35.926346 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:35.926300 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e72bc583-3867-48ed-a5b6-56642cd222c9" containerName="istio-proxy" Nov 03 20:42:35.926434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:35.926379 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e72bc583-3867-48ed-a5b6-56642cd222c9" containerName="istio-proxy" Nov 03 20:42:35.931011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:35.930990 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czzww" Nov 03 20:42:35.938067 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:35.938043 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:42:35.938176 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:35.938063 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\"" Nov 03 20:42:35.945722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:35.945694 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czzww"] Nov 03 20:42:36.000753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:36.000716 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e3978e-1396-4270-8a8d-b2075ceadada-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-czzww\" (UID: \"54e3978e-1396-4270-8a8d-b2075ceadada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czzww" Nov 03 20:42:36.000942 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:36.000916 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e3978e-1396-4270-8a8d-b2075ceadada-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-czzww\" (UID: \"54e3978e-1396-4270-8a8d-b2075ceadada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czzww" Nov 03 20:42:36.001010 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:36.000958 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjw9w\" (UniqueName: \"kubernetes.io/projected/54e3978e-1396-4270-8a8d-b2075ceadada-kube-api-access-bjw9w\") pod \"maas-api-revoke-sub-85237e45-czzww\" (UID: \"54e3978e-1396-4270-8a8d-b2075ceadada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czzww" Nov 03 20:42:36.102415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:36.102381 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e3978e-1396-4270-8a8d-b2075ceadada-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-czzww\" (UID: \"54e3978e-1396-4270-8a8d-b2075ceadada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czzww" Nov 03 20:42:36.102415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:36.102417 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjw9w\" (UniqueName: \"kubernetes.io/projected/54e3978e-1396-4270-8a8d-b2075ceadada-kube-api-access-bjw9w\") pod \"maas-api-revoke-sub-85237e45-czzww\" (UID: \"54e3978e-1396-4270-8a8d-b2075ceadada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czzww" Nov 03 20:42:36.102606 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:36.102469 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e3978e-1396-4270-8a8d-b2075ceadada-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-czzww\" (UID: \"54e3978e-1396-4270-8a8d-b2075ceadada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czzww" Nov 03 20:42:36.103164 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:36.103132 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e3978e-1396-4270-8a8d-b2075ceadada-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-czzww\" (UID: \"54e3978e-1396-4270-8a8d-b2075ceadada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czzww" Nov 03 20:42:36.104835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:36.104813 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e3978e-1396-4270-8a8d-b2075ceadada-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-czzww\" (UID: \"54e3978e-1396-4270-8a8d-b2075ceadada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czzww" Nov 03 20:42:36.114934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:36.114913 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjw9w\" (UniqueName: \"kubernetes.io/projected/54e3978e-1396-4270-8a8d-b2075ceadada-kube-api-access-bjw9w\") pod \"maas-api-revoke-sub-85237e45-czzww\" (UID: \"54e3978e-1396-4270-8a8d-b2075ceadada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czzww" Nov 03 20:42:36.247461 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:36.247376 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czzww" Nov 03 20:42:36.523540 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:36.523510 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czzww"] Nov 03 20:42:36.525247 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:42:36.525218 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e3978e_1396_4270_8a8d_b2075ceadada.slice/crio-a0285ebdb191c1255df0cbcc9c257f7f7a84b6838559f2b4a802713b7b7fb8bc WatchSource:0}: Error finding container a0285ebdb191c1255df0cbcc9c257f7f7a84b6838559f2b4a802713b7b7fb8bc: Status 404 returned error can't find the container with id a0285ebdb191c1255df0cbcc9c257f7f7a84b6838559f2b4a802713b7b7fb8bc Nov 03 20:42:37.038416 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:37.038375 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czzww" event={"ID":"54e3978e-1396-4270-8a8d-b2075ceadada","Type":"ContainerStarted","Data":"0251afacbeae9bf1baa37258c48a970396374ddbbf7fcb5e8c880e8fbdfbd599"} Nov 03 20:42:37.038814 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:37.038442 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czzww" event={"ID":"54e3978e-1396-4270-8a8d-b2075ceadada","Type":"ContainerStarted","Data":"a0285ebdb191c1255df0cbcc9c257f7f7a84b6838559f2b4a802713b7b7fb8bc"} Nov 03 20:42:37.059090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:37.059032 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czzww" podStartSLOduration=2.059014777 podStartE2EDuration="2.059014777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:37.056033034 +0000 UTC m=+3951.313830293" watchObservedRunningTime="2026-11-03 20:42:37.059014777 +0000 UTC m=+3951.316812035" Nov 03 20:42:38.045781 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:38.045743 5059 generic.go:356] "Generic (PLEG): container finished" podID="54e3978e-1396-4270-8a8d-b2075ceadada" containerID="0251afacbeae9bf1baa37258c48a970396374ddbbf7fcb5e8c880e8fbdfbd599" exitCode=0 Nov 03 20:42:38.046291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:38.045837 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czzww" event={"ID":"54e3978e-1396-4270-8a8d-b2075ceadada","Type":"ContainerDied","Data":"0251afacbeae9bf1baa37258c48a970396374ddbbf7fcb5e8c880e8fbdfbd599"} Nov 03 20:42:39.243633 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:39.243610 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czzww" Nov 03 20:42:39.333387 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:39.333361 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54e3978e-1396-4270-8a8d-b2075ceadada-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e3978e-1396-4270-8a8d-b2075ceadada-maas-api-openshift-ca\") pod \"54e3978e-1396-4270-8a8d-b2075ceadada\" (UID: \"54e3978e-1396-4270-8a8d-b2075ceadada\") " Nov 03 20:42:39.333547 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:39.333419 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54e3978e-1396-4270-8a8d-b2075ceadada-kube-api-access-bjw9w\" (UniqueName: \"kubernetes.io/projected/54e3978e-1396-4270-8a8d-b2075ceadada-kube-api-access-bjw9w\") pod \"54e3978e-1396-4270-8a8d-b2075ceadada\" (UID: \"54e3978e-1396-4270-8a8d-b2075ceadada\") " Nov 03 20:42:39.333547 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:39.333482 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54e3978e-1396-4270-8a8d-b2075ceadada-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e3978e-1396-4270-8a8d-b2075ceadada-maas-api-xks-ca\") pod \"54e3978e-1396-4270-8a8d-b2075ceadada\" (UID: \"54e3978e-1396-4270-8a8d-b2075ceadada\") " Nov 03 20:42:39.333837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:39.333806 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54e3978e-1396-4270-8a8d-b2075ceadada-maas-api-openshift-ca" pod "54e3978e-1396-4270-8a8d-b2075ceadada" (UID: "54e3978e-1396-4270-8a8d-b2075ceadada"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:42:39.335652 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:39.335627 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54e3978e-1396-4270-8a8d-b2075ceadada-maas-api-xks-ca" pod "54e3978e-1396-4270-8a8d-b2075ceadada" (UID: "54e3978e-1396-4270-8a8d-b2075ceadada"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:42:39.335797 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:39.335678 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e3978e-1396-4270-8a8d-b2075ceadada-kube-api-access-bjw9w" pod "54e3978e-1396-4270-8a8d-b2075ceadada" (UID: "54e3978e-1396-4270-8a8d-b2075ceadada"). InnerVolumeSpecName "kube-api-access-bjw9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:42:39.434258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:39.434227 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjw9w\" (UniqueName: \"kubernetes.io/projected/54e3978e-1396-4270-8a8d-b2075ceadada-kube-api-access-bjw9w\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:39.434258 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:39.434257 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e3978e-1396-4270-8a8d-b2075ceadada-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:39.434434 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:39.434267 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e3978e-1396-4270-8a8d-b2075ceadada-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:40.057117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:40.057085 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czzww" Nov 03 20:42:40.057260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:40.057089 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czzww" event={"ID":"54e3978e-1396-4270-8a8d-b2075ceadada","Type":"ContainerDied","Data":"a0285ebdb191c1255df0cbcc9c257f7f7a84b6838559f2b4a802713b7b7fb8bc"} Nov 03 20:42:40.057260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:40.057213 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0285ebdb191c1255df0cbcc9c257f7f7a84b6838559f2b4a802713b7b7fb8bc" Nov 03 20:42:40.604094 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:42:40.604058 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:42:44.759379 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.759336 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz"] Nov 03 20:42:44.759977 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.759863 5059 cpu_manager.go:414] "containerMap: removing container" podUID="54e3978e-1396-4270-8a8d-b2075ceadada" containerName="revoke-keys" Nov 03 20:42:44.759977 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.759886 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e3978e-1396-4270-8a8d-b2075ceadada" containerName="revoke-keys" Nov 03 20:42:44.759977 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.759940 5059 cpu_manager.go:414] "containerMap: removing container" podUID="e72bc583-3867-48ed-a5b6-56642cd222c9" containerName="istio-proxy" Nov 03 20:42:44.759977 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.759948 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e72bc583-3867-48ed-a5b6-56642cd222c9" containerName="istio-proxy" Nov 03 20:42:44.760206 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.760022 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e72bc583-3867-48ed-a5b6-56642cd222c9" containerName="istio-proxy" Nov 03 20:42:44.760206 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.760039 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e3978e-1396-4270-8a8d-b2075ceadada" containerName="revoke-keys" Nov 03 20:42:44.763472 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.763449 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.784352 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.784324 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc"] Nov 03 20:42:44.788190 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.788169 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz"] Nov 03 20:42:44.788296 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.788272 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.795929 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.795906 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59"] Nov 03 20:42:44.800881 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.800858 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.803563 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.803543 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc"] Nov 03 20:42:44.813738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.813655 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59"] Nov 03 20:42:44.878120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878091 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6462b9bb-1018-4029-922d-eb71527fc301-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.878249 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878124 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14046b2-dd71-42a3-b239-0160088faa0a-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.878249 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878143 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6462b9bb-1018-4029-922d-eb71527fc301-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.878327 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878264 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab375b16-1371-437f-91b4-d5f8580e804d-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.878327 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878299 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d14046b2-dd71-42a3-b239-0160088faa0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.878327 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878317 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab375b16-1371-437f-91b4-d5f8580e804d-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.878427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878367 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab375b16-1371-437f-91b4-d5f8580e804d-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.878427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878411 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6462b9bb-1018-4029-922d-eb71527fc301-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.878493 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878440 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d14046b2-dd71-42a3-b239-0160088faa0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.878493 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878461 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbkzl\" (UniqueName: \"kubernetes.io/projected/6462b9bb-1018-4029-922d-eb71527fc301-kube-api-access-vbkzl\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.878558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878539 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d14046b2-dd71-42a3-b239-0160088faa0a-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.878606 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878563 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14046b2-dd71-42a3-b239-0160088faa0a-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.878606 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878584 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d14046b2-dd71-42a3-b239-0160088faa0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.878700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878636 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6462b9bb-1018-4029-922d-eb71527fc301-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.878700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878680 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab375b16-1371-437f-91b4-d5f8580e804d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.878772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878700 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6462b9bb-1018-4029-922d-eb71527fc301-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.878772 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878729 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab375b16-1371-437f-91b4-d5f8580e804d-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.878835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878785 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d14046b2-dd71-42a3-b239-0160088faa0a-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.878835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878802 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pksrm\" (UniqueName: \"kubernetes.io/projected/ab375b16-1371-437f-91b4-d5f8580e804d-kube-api-access-pksrm\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.878835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878819 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6462b9bb-1018-4029-922d-eb71527fc301-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.878931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878834 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab375b16-1371-437f-91b4-d5f8580e804d-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.878931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878853 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6462b9bb-1018-4029-922d-eb71527fc301-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.878931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878878 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn7m4\" (UniqueName: \"kubernetes.io/projected/d14046b2-dd71-42a3-b239-0160088faa0a-kube-api-access-vn7m4\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.879026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878952 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab375b16-1371-437f-91b4-d5f8580e804d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.879026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878972 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6462b9bb-1018-4029-922d-eb71527fc301-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.879026 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.878997 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab375b16-1371-437f-91b4-d5f8580e804d-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.879119 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.879035 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d14046b2-dd71-42a3-b239-0160088faa0a-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.979574 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.979548 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6462b9bb-1018-4029-922d-eb71527fc301-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.979722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.979578 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab375b16-1371-437f-91b4-d5f8580e804d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.979722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.979596 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6462b9bb-1018-4029-922d-eb71527fc301-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.979722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.979622 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab375b16-1371-437f-91b4-d5f8580e804d-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.979722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.979653 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d14046b2-dd71-42a3-b239-0160088faa0a-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.979722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.979701 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pksrm\" (UniqueName: \"kubernetes.io/projected/ab375b16-1371-437f-91b4-d5f8580e804d-kube-api-access-pksrm\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.979992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.979727 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6462b9bb-1018-4029-922d-eb71527fc301-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.979992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.979753 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab375b16-1371-437f-91b4-d5f8580e804d-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.979992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.979781 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6462b9bb-1018-4029-922d-eb71527fc301-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.979992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.979804 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn7m4\" (UniqueName: \"kubernetes.io/projected/d14046b2-dd71-42a3-b239-0160088faa0a-kube-api-access-vn7m4\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.979992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.979841 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab375b16-1371-437f-91b4-d5f8580e804d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.979992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.979874 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6462b9bb-1018-4029-922d-eb71527fc301-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.979992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.979912 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab375b16-1371-437f-91b4-d5f8580e804d-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.979992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.979939 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d14046b2-dd71-42a3-b239-0160088faa0a-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.979992 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.979980 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6462b9bb-1018-4029-922d-eb71527fc301-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.980472 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.980006 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14046b2-dd71-42a3-b239-0160088faa0a-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.980472 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.980031 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6462b9bb-1018-4029-922d-eb71527fc301-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.980472 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.980085 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6462b9bb-1018-4029-922d-eb71527fc301-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.980472 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.980099 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab375b16-1371-437f-91b4-d5f8580e804d-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.980472 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.980085 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab375b16-1371-437f-91b4-d5f8580e804d-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.980472 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.980129 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d14046b2-dd71-42a3-b239-0160088faa0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.980472 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.980157 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab375b16-1371-437f-91b4-d5f8580e804d-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.980472 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.980187 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab375b16-1371-437f-91b4-d5f8580e804d-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.980472 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.980227 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6462b9bb-1018-4029-922d-eb71527fc301-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.980472 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.980443 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6462b9bb-1018-4029-922d-eb71527fc301-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.980472 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.980125 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d14046b2-dd71-42a3-b239-0160088faa0a-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.981084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.980502 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab375b16-1371-437f-91b4-d5f8580e804d-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.981084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.980512 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6462b9bb-1018-4029-922d-eb71527fc301-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.981084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.980543 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d14046b2-dd71-42a3-b239-0160088faa0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.981084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.980573 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbkzl\" (UniqueName: \"kubernetes.io/projected/6462b9bb-1018-4029-922d-eb71527fc301-kube-api-access-vbkzl\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.981084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.980621 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d14046b2-dd71-42a3-b239-0160088faa0a-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.981084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.980657 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14046b2-dd71-42a3-b239-0160088faa0a-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.981084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.980733 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6462b9bb-1018-4029-922d-eb71527fc301-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.981084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.980747 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d14046b2-dd71-42a3-b239-0160088faa0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.981084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.980834 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab375b16-1371-437f-91b4-d5f8580e804d-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.981084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.980841 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6462b9bb-1018-4029-922d-eb71527fc301-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.981084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.980998 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab375b16-1371-437f-91b4-d5f8580e804d-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.981084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.980452 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab375b16-1371-437f-91b4-d5f8580e804d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.981691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.981379 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d14046b2-dd71-42a3-b239-0160088faa0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.981691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.981398 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14046b2-dd71-42a3-b239-0160088faa0a-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.981816 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.981770 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d14046b2-dd71-42a3-b239-0160088faa0a-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.981952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.981930 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d14046b2-dd71-42a3-b239-0160088faa0a-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.983278 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.983247 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab375b16-1371-437f-91b4-d5f8580e804d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.983387 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.983289 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab375b16-1371-437f-91b4-d5f8580e804d-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.983473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.983451 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6462b9bb-1018-4029-922d-eb71527fc301-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.983524 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.983508 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d14046b2-dd71-42a3-b239-0160088faa0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.983598 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.983583 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d14046b2-dd71-42a3-b239-0160088faa0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.983876 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.983860 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6462b9bb-1018-4029-922d-eb71527fc301-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.993479 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.993431 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6462b9bb-1018-4029-922d-eb71527fc301-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.993798 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.993753 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab375b16-1371-437f-91b4-d5f8580e804d-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:44.993914 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.993857 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d14046b2-dd71-42a3-b239-0160088faa0a-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.993981 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.993940 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbkzl\" (UniqueName: \"kubernetes.io/projected/6462b9bb-1018-4029-922d-eb71527fc301-kube-api-access-vbkzl\") pod \"maas-default-gateway-openshift-default-855f9574cf-8gzjz\" (UID: \"6462b9bb-1018-4029-922d-eb71527fc301\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:44.993981 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.993945 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn7m4\" (UniqueName: \"kubernetes.io/projected/d14046b2-dd71-42a3-b239-0160088faa0a-kube-api-access-vn7m4\") pod \"maas-default-gateway-openshift-default-855f9574cf-vkx59\" (UID: \"d14046b2-dd71-42a3-b239-0160088faa0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:44.994890 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:44.994868 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pksrm\" (UniqueName: \"kubernetes.io/projected/ab375b16-1371-437f-91b4-d5f8580e804d-kube-api-access-pksrm\") pod \"maas-default-gateway-openshift-default-855f9574cf-wtlgc\" (UID: \"ab375b16-1371-437f-91b4-d5f8580e804d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:45.076464 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:45.076430 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:45.104739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:45.104717 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:45.118712 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:45.118680 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:45.357364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:45.357338 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz"] Nov 03 20:42:45.358782 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:42:45.358748 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6462b9bb_1018_4029_922d_eb71527fc301.slice/crio-3c998f56b2c2461184cd397094478aa9de537bafea3872425709f135db5cda2b WatchSource:0}: Error finding container 3c998f56b2c2461184cd397094478aa9de537bafea3872425709f135db5cda2b: Status 404 returned error can't find the container with id 3c998f56b2c2461184cd397094478aa9de537bafea3872425709f135db5cda2b Nov 03 20:42:45.360951 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:45.360916 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:42:45.361030 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:45.360990 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:42:45.361030 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:45.361018 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:42:45.401079 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:45.401054 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc"] Nov 03 20:42:45.401600 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:42:45.401575 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab375b16_1371_437f_91b4_d5f8580e804d.slice/crio-01e453735ef6cdbc32c1f42967251094b6dfb748fea4ad07b5a334d55e3ae5e0 WatchSource:0}: Error finding container 01e453735ef6cdbc32c1f42967251094b6dfb748fea4ad07b5a334d55e3ae5e0: Status 404 returned error can't find the container with id 01e453735ef6cdbc32c1f42967251094b6dfb748fea4ad07b5a334d55e3ae5e0 Nov 03 20:42:45.403893 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:45.403858 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:42:45.403961 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:45.403943 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:42:45.404009 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:45.403992 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:42:45.418351 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:45.418324 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59"] Nov 03 20:42:45.424038 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:45.424002 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:42:45.424112 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:45.424070 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:42:45.424112 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:45.424104 5059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 20:42:46.124684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:46.124330 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" event={"ID":"6462b9bb-1018-4029-922d-eb71527fc301","Type":"ContainerStarted","Data":"4006369c2247a347a4b7058ae79c76f7dee7b0dc2a3c780e28bd3ab1c2f95fe2"} Nov 03 20:42:46.124684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:46.124407 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" event={"ID":"6462b9bb-1018-4029-922d-eb71527fc301","Type":"ContainerStarted","Data":"3c998f56b2c2461184cd397094478aa9de537bafea3872425709f135db5cda2b"} Nov 03 20:42:46.135411 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:46.134707 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" event={"ID":"d14046b2-dd71-42a3-b239-0160088faa0a","Type":"ContainerStarted","Data":"94ed4b5e057933009e6f466178474c0f8d10d23fe8646df2f742b1202a7a885f"} Nov 03 20:42:46.135411 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:46.134788 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" event={"ID":"d14046b2-dd71-42a3-b239-0160088faa0a","Type":"ContainerStarted","Data":"31c879e2a13dc16963651286defce8dc4f1507592db43f9b6df9317028335ae8"} Nov 03 20:42:46.137538 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:46.137504 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" event={"ID":"ab375b16-1371-437f-91b4-d5f8580e804d","Type":"ContainerStarted","Data":"3f4b044b2f2e7533bcf1ba49767347e0cd237560246bd5e04e7928915b670f80"} Nov 03 20:42:46.137850 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:46.137814 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" event={"ID":"ab375b16-1371-437f-91b4-d5f8580e804d","Type":"ContainerStarted","Data":"01e453735ef6cdbc32c1f42967251094b6dfb748fea4ad07b5a334d55e3ae5e0"} Nov 03 20:42:46.155181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:46.155126 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" podStartSLOduration=2.155113038 podStartE2EDuration="2.155113038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:46.152284135 +0000 UTC m=+3960.410081389" watchObservedRunningTime="2026-11-03 20:42:46.155113038 +0000 UTC m=+3960.412910294" Nov 03 20:42:46.179686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:46.178301 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" podStartSLOduration=2.178281564 podStartE2EDuration="2.178281564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:46.175215616 +0000 UTC m=+3960.433012877" watchObservedRunningTime="2026-11-03 20:42:46.178281564 +0000 UTC m=+3960.436078822" Nov 03 20:42:46.230689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:46.229097 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" podStartSLOduration=2.229073933 podStartE2EDuration="2.229073933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:46.21821948 +0000 UTC m=+3960.476016739" watchObservedRunningTime="2026-11-03 20:42:46.229073933 +0000 UTC m=+3960.486871191" Nov 03 20:42:47.078573 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:47.078532 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:47.078925 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:47.078902 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8gzjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.190:15021/healthz/ready\": dial tcp 10.132.1.190:15021: connect: connection refused" start-of-body= Nov 03 20:42:47.079104 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:47.079078 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" podUID="6462b9bb-1018-4029-922d-eb71527fc301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.190:15021/healthz/ready\": dial tcp 10.132.1.190:15021: connect: connection refused" Nov 03 20:42:47.108903 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:47.108825 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:47.116834 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:47.116803 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wtlgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.191:15021/healthz/ready\": dial tcp 10.132.1.191:15021: connect: connection refused" start-of-body= Nov 03 20:42:47.117082 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:47.117047 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" podUID="ab375b16-1371-437f-91b4-d5f8580e804d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.191:15021/healthz/ready\": dial tcp 10.132.1.191:15021: connect: connection refused" Nov 03 20:42:47.121823 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:47.121801 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:47.126139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:47.126116 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vkx59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.192:15021/healthz/ready\": dial tcp 10.132.1.192:15021: connect: connection refused" start-of-body= Nov 03 20:42:47.126629 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:47.126604 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" podUID="d14046b2-dd71-42a3-b239-0160088faa0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.192:15021/healthz/ready\": dial tcp 10.132.1.192:15021: connect: connection refused" Nov 03 20:42:47.733689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:47.731749 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55c765799f-vxzm4"] Nov 03 20:42:47.733689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:47.732004 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-55c765799f-vxzm4" podUID="0a6e3c07-e75c-4bee-9699-5a421ecc0005" containerName="manager" containerID="cri-o://20740c5fbcd9c0c0a9545d0fa3015464f1dcd65e643687bd7193d6b503467e83" gracePeriod=30 Nov 03 20:42:48.086872 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:48.079081 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8gzjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.190:15021/healthz/ready\": dial tcp 10.132.1.190:15021: connect: connection refused" start-of-body= Nov 03 20:42:48.086872 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:48.079147 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" podUID="6462b9bb-1018-4029-922d-eb71527fc301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.190:15021/healthz/ready\": dial tcp 10.132.1.190:15021: connect: connection refused" Nov 03 20:42:48.119686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:48.115102 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wtlgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.191:15021/healthz/ready\": dial tcp 10.132.1.191:15021: connect: connection refused" start-of-body= Nov 03 20:42:48.119686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:48.115166 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" podUID="ab375b16-1371-437f-91b4-d5f8580e804d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.191:15021/healthz/ready\": dial tcp 10.132.1.191:15021: connect: connection refused" Nov 03 20:42:48.131689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:48.129687 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vkx59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.192:15021/healthz/ready\": dial tcp 10.132.1.192:15021: connect: connection refused" start-of-body= Nov 03 20:42:48.131689 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:48.129750 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" podUID="d14046b2-dd71-42a3-b239-0160088faa0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.192:15021/healthz/ready\": dial tcp 10.132.1.192:15021: connect: connection refused" Nov 03 20:42:48.149758 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:48.149718 5059 generic.go:356] "Generic (PLEG): container finished" podID="0a6e3c07-e75c-4bee-9699-5a421ecc0005" containerID="20740c5fbcd9c0c0a9545d0fa3015464f1dcd65e643687bd7193d6b503467e83" exitCode=0 Nov 03 20:42:48.150102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:48.149818 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55c765799f-vxzm4" event={"ID":"0a6e3c07-e75c-4bee-9699-5a421ecc0005","Type":"ContainerDied","Data":"20740c5fbcd9c0c0a9545d0fa3015464f1dcd65e643687bd7193d6b503467e83"} Nov 03 20:42:48.348060 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:48.347994 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55c765799f-vxzm4" Nov 03 20:42:48.449248 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:48.449189 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a6e3c07-e75c-4bee-9699-5a421ecc0005-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a6e3c07-e75c-4bee-9699-5a421ecc0005-metrics-certs\") pod \"0a6e3c07-e75c-4bee-9699-5a421ecc0005\" (UID: \"0a6e3c07-e75c-4bee-9699-5a421ecc0005\") " Nov 03 20:42:48.449453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:48.449299 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a6e3c07-e75c-4bee-9699-5a421ecc0005-webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a6e3c07-e75c-4bee-9699-5a421ecc0005-webhook-cert\") pod \"0a6e3c07-e75c-4bee-9699-5a421ecc0005\" (UID: \"0a6e3c07-e75c-4bee-9699-5a421ecc0005\") " Nov 03 20:42:48.449453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:48.449433 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a6e3c07-e75c-4bee-9699-5a421ecc0005-kube-api-access-hwt4c\" (UniqueName: \"kubernetes.io/projected/0a6e3c07-e75c-4bee-9699-5a421ecc0005-kube-api-access-hwt4c\") pod \"0a6e3c07-e75c-4bee-9699-5a421ecc0005\" (UID: \"0a6e3c07-e75c-4bee-9699-5a421ecc0005\") " Nov 03 20:42:48.451561 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:48.451529 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a6e3c07-e75c-4bee-9699-5a421ecc0005-metrics-certs" pod "0a6e3c07-e75c-4bee-9699-5a421ecc0005" (UID: "0a6e3c07-e75c-4bee-9699-5a421ecc0005"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:42:48.451774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:48.451713 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a6e3c07-e75c-4bee-9699-5a421ecc0005-webhook-cert" pod "0a6e3c07-e75c-4bee-9699-5a421ecc0005" (UID: "0a6e3c07-e75c-4bee-9699-5a421ecc0005"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:42:48.452254 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:48.452230 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a6e3c07-e75c-4bee-9699-5a421ecc0005-kube-api-access-hwt4c" pod "0a6e3c07-e75c-4bee-9699-5a421ecc0005" (UID: "0a6e3c07-e75c-4bee-9699-5a421ecc0005"). InnerVolumeSpecName "kube-api-access-hwt4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:42:48.550508 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:48.550477 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwt4c\" (UniqueName: \"kubernetes.io/projected/0a6e3c07-e75c-4bee-9699-5a421ecc0005-kube-api-access-hwt4c\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:48.550717 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:48.550702 5059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a6e3c07-e75c-4bee-9699-5a421ecc0005-metrics-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:48.550795 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:48.550779 5059 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a6e3c07-e75c-4bee-9699-5a421ecc0005-webhook-cert\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:49.080695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:49.077095 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8gzjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.190:15021/healthz/ready\": dial tcp 10.132.1.190:15021: connect: connection refused" start-of-body= Nov 03 20:42:49.080695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:49.077161 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" podUID="6462b9bb-1018-4029-922d-eb71527fc301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.190:15021/healthz/ready\": dial tcp 10.132.1.190:15021: connect: connection refused" Nov 03 20:42:49.109138 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:49.109098 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wtlgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.191:15021/healthz/ready\": dial tcp 10.132.1.191:15021: connect: connection refused" start-of-body= Nov 03 20:42:49.109439 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:49.109395 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" podUID="ab375b16-1371-437f-91b4-d5f8580e804d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.191:15021/healthz/ready\": dial tcp 10.132.1.191:15021: connect: connection refused" Nov 03 20:42:49.123786 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:49.123754 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vkx59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.192:15021/healthz/ready\": dial tcp 10.132.1.192:15021: connect: connection refused" start-of-body= Nov 03 20:42:49.124039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:49.124011 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" podUID="d14046b2-dd71-42a3-b239-0160088faa0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.192:15021/healthz/ready\": dial tcp 10.132.1.192:15021: connect: connection refused" Nov 03 20:42:49.156266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:49.156237 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55c765799f-vxzm4" Nov 03 20:42:49.156813 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:49.156792 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55c765799f-vxzm4" event={"ID":"0a6e3c07-e75c-4bee-9699-5a421ecc0005","Type":"ContainerDied","Data":"f7af9c96c8137ebb1fa3f569a77a95892549c9249f5ae11ad624a13c6c8792f9"} Nov 03 20:42:49.156964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:49.156953 5059 scope.go:122] "RemoveContainer" containerID="20740c5fbcd9c0c0a9545d0fa3015464f1dcd65e643687bd7193d6b503467e83" Nov 03 20:42:49.251908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:49.251874 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55c765799f-vxzm4"] Nov 03 20:42:49.255610 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:49.255586 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-55c765799f-vxzm4"] Nov 03 20:42:49.856014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:49.855977 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a6e3c07-e75c-4bee-9699-5a421ecc0005" path="/var/lib/kubelet/pods/0a6e3c07-e75c-4bee-9699-5a421ecc0005/volumes" Nov 03 20:42:50.078139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.078093 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8gzjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.190:15021/healthz/ready\": dial tcp 10.132.1.190:15021: connect: connection refused" start-of-body= Nov 03 20:42:50.078461 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.078429 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" podUID="6462b9bb-1018-4029-922d-eb71527fc301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.190:15021/healthz/ready\": dial tcp 10.132.1.190:15021: connect: connection refused" Nov 03 20:42:50.109475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.108996 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wtlgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.191:15021/healthz/ready\": dial tcp 10.132.1.191:15021: connect: connection refused" start-of-body= Nov 03 20:42:50.109475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.109076 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" podUID="ab375b16-1371-437f-91b4-d5f8580e804d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.191:15021/healthz/ready\": dial tcp 10.132.1.191:15021: connect: connection refused" Nov 03 20:42:50.121235 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.121131 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vkx59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.192:15021/healthz/ready\": dial tcp 10.132.1.192:15021: connect: connection refused" start-of-body= Nov 03 20:42:50.121235 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.121181 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" podUID="d14046b2-dd71-42a3-b239-0160088faa0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.192:15021/healthz/ready\": dial tcp 10.132.1.192:15021: connect: connection refused" Nov 03 20:42:50.448981 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.448894 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55c765799f-5qgk6"] Nov 03 20:42:50.449351 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.449264 5059 cpu_manager.go:414] "containerMap: removing container" podUID="0a6e3c07-e75c-4bee-9699-5a421ecc0005" containerName="manager" Nov 03 20:42:50.449351 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.449277 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a6e3c07-e75c-4bee-9699-5a421ecc0005" containerName="manager" Nov 03 20:42:50.449351 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.449348 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a6e3c07-e75c-4bee-9699-5a421ecc0005" containerName="manager" Nov 03 20:42:50.454337 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.454315 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55c765799f-5qgk6" Nov 03 20:42:50.477749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.463037 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xc6tf\"" Nov 03 20:42:50.477749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.471100 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 20:42:50.477749 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.471248 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 20:42:50.538714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.521927 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55c765799f-5qgk6"] Nov 03 20:42:50.612692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.611346 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a3ca718-8b83-4650-b941-3e29e6b80e00-metrics-certs\") pod \"maas-controller-55c765799f-5qgk6\" (UID: \"5a3ca718-8b83-4650-b941-3e29e6b80e00\") " pod="opendatahub/maas-controller-55c765799f-5qgk6" Nov 03 20:42:50.612692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.611439 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a3ca718-8b83-4650-b941-3e29e6b80e00-webhook-cert\") pod \"maas-controller-55c765799f-5qgk6\" (UID: \"5a3ca718-8b83-4650-b941-3e29e6b80e00\") " pod="opendatahub/maas-controller-55c765799f-5qgk6" Nov 03 20:42:50.612692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.611474 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7wfb\" (UniqueName: \"kubernetes.io/projected/5a3ca718-8b83-4650-b941-3e29e6b80e00-kube-api-access-w7wfb\") pod \"maas-controller-55c765799f-5qgk6\" (UID: \"5a3ca718-8b83-4650-b941-3e29e6b80e00\") " pod="opendatahub/maas-controller-55c765799f-5qgk6" Nov 03 20:42:50.618697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.613686 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 20:42:50.713746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.713388 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a3ca718-8b83-4650-b941-3e29e6b80e00-metrics-certs\") pod \"maas-controller-55c765799f-5qgk6\" (UID: \"5a3ca718-8b83-4650-b941-3e29e6b80e00\") " pod="opendatahub/maas-controller-55c765799f-5qgk6" Nov 03 20:42:50.713746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.713494 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a3ca718-8b83-4650-b941-3e29e6b80e00-webhook-cert\") pod \"maas-controller-55c765799f-5qgk6\" (UID: \"5a3ca718-8b83-4650-b941-3e29e6b80e00\") " pod="opendatahub/maas-controller-55c765799f-5qgk6" Nov 03 20:42:50.713746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.713539 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7wfb\" (UniqueName: \"kubernetes.io/projected/5a3ca718-8b83-4650-b941-3e29e6b80e00-kube-api-access-w7wfb\") pod \"maas-controller-55c765799f-5qgk6\" (UID: \"5a3ca718-8b83-4650-b941-3e29e6b80e00\") " pod="opendatahub/maas-controller-55c765799f-5qgk6" Nov 03 20:42:50.720684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.717532 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a3ca718-8b83-4650-b941-3e29e6b80e00-metrics-certs\") pod \"maas-controller-55c765799f-5qgk6\" (UID: \"5a3ca718-8b83-4650-b941-3e29e6b80e00\") " pod="opendatahub/maas-controller-55c765799f-5qgk6" Nov 03 20:42:50.732249 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.731358 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a3ca718-8b83-4650-b941-3e29e6b80e00-webhook-cert\") pod \"maas-controller-55c765799f-5qgk6\" (UID: \"5a3ca718-8b83-4650-b941-3e29e6b80e00\") " pod="opendatahub/maas-controller-55c765799f-5qgk6" Nov 03 20:42:50.739346 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.739299 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7wfb\" (UniqueName: \"kubernetes.io/projected/5a3ca718-8b83-4650-b941-3e29e6b80e00-kube-api-access-w7wfb\") pod \"maas-controller-55c765799f-5qgk6\" (UID: \"5a3ca718-8b83-4650-b941-3e29e6b80e00\") " pod="opendatahub/maas-controller-55c765799f-5qgk6" Nov 03 20:42:50.852189 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:50.852147 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55c765799f-5qgk6" Nov 03 20:42:51.082686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:51.079080 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8gzjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.190:15021/healthz/ready\": dial tcp 10.132.1.190:15021: connect: connection refused" start-of-body= Nov 03 20:42:51.082686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:51.079146 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" podUID="6462b9bb-1018-4029-922d-eb71527fc301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.190:15021/healthz/ready\": dial tcp 10.132.1.190:15021: connect: connection refused" Nov 03 20:42:51.114700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:51.106995 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wtlgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.191:15021/healthz/ready\": dial tcp 10.132.1.191:15021: connect: connection refused" start-of-body= Nov 03 20:42:51.114700 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:51.107054 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" podUID="ab375b16-1371-437f-91b4-d5f8580e804d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.191:15021/healthz/ready\": dial tcp 10.132.1.191:15021: connect: connection refused" Nov 03 20:42:51.138691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:51.134767 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vkx59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.192:15021/healthz/ready\": dial tcp 10.132.1.192:15021: connect: connection refused" start-of-body= Nov 03 20:42:51.138691 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:51.134846 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" podUID="d14046b2-dd71-42a3-b239-0160088faa0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.192:15021/healthz/ready\": dial tcp 10.132.1.192:15021: connect: connection refused" Nov 03 20:42:51.332686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:51.319267 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55c765799f-5qgk6"] Nov 03 20:42:51.338067 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:42:51.338028 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3ca718_8b83_4650_b941_3e29e6b80e00.slice/crio-b35221a5acea6cdf4a4700e55fb3e65858e1b2adf72956e9500fbf34145012b3 WatchSource:0}: Error finding container b35221a5acea6cdf4a4700e55fb3e65858e1b2adf72956e9500fbf34145012b3: Status 404 returned error can't find the container with id b35221a5acea6cdf4a4700e55fb3e65858e1b2adf72956e9500fbf34145012b3 Nov 03 20:42:52.077585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:52.077548 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8gzjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.190:15021/healthz/ready\": dial tcp 10.132.1.190:15021: connect: connection refused" start-of-body= Nov 03 20:42:52.078103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:52.077605 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" podUID="6462b9bb-1018-4029-922d-eb71527fc301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.190:15021/healthz/ready\": dial tcp 10.132.1.190:15021: connect: connection refused" Nov 03 20:42:52.105916 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:52.105877 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wtlgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.191:15021/healthz/ready\": dial tcp 10.132.1.191:15021: connect: connection refused" start-of-body= Nov 03 20:42:52.106046 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:52.105947 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" podUID="ab375b16-1371-437f-91b4-d5f8580e804d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.191:15021/healthz/ready\": dial tcp 10.132.1.191:15021: connect: connection refused" Nov 03 20:42:52.119622 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:52.119595 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vkx59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.192:15021/healthz/ready\": dial tcp 10.132.1.192:15021: connect: connection refused" start-of-body= Nov 03 20:42:52.119723 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:52.119645 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" podUID="d14046b2-dd71-42a3-b239-0160088faa0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.192:15021/healthz/ready\": dial tcp 10.132.1.192:15021: connect: connection refused" Nov 03 20:42:52.235814 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:52.235770 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55c765799f-5qgk6" event={"ID":"5a3ca718-8b83-4650-b941-3e29e6b80e00","Type":"ContainerStarted","Data":"7ba8cab2cf4107a59f0b10dfce9c5cd21d68f4ad420de76819dfb7c8fd5175f5"} Nov 03 20:42:52.235983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:52.235843 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55c765799f-5qgk6" event={"ID":"5a3ca718-8b83-4650-b941-3e29e6b80e00","Type":"ContainerStarted","Data":"b35221a5acea6cdf4a4700e55fb3e65858e1b2adf72956e9500fbf34145012b3"} Nov 03 20:42:52.236092 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:52.236063 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55c765799f-5qgk6" Nov 03 20:42:52.265093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:52.265036 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-55c765799f-5qgk6" podStartSLOduration=1.8071103979999998 podStartE2EDuration="2.265018058s" totalImagesPullingTime="457.90766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:52.261997631 +0000 UTC m=+3966.519794888" watchObservedRunningTime="2026-11-03 20:42:52.265018058 +0000 UTC m=+3966.522815316" Nov 03 20:42:53.077375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:53.077336 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8gzjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.190:15021/healthz/ready\": dial tcp 10.132.1.190:15021: connect: connection refused" start-of-body= Nov 03 20:42:53.077570 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:53.077407 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" podUID="6462b9bb-1018-4029-922d-eb71527fc301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.190:15021/healthz/ready\": dial tcp 10.132.1.190:15021: connect: connection refused" Nov 03 20:42:53.105610 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:53.105562 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wtlgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.191:15021/healthz/ready\": dial tcp 10.132.1.191:15021: connect: connection refused" start-of-body= Nov 03 20:42:53.106139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:53.105636 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" podUID="ab375b16-1371-437f-91b4-d5f8580e804d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.191:15021/healthz/ready\": dial tcp 10.132.1.191:15021: connect: connection refused" Nov 03 20:42:53.120084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:53.120048 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vkx59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.192:15021/healthz/ready\": dial tcp 10.132.1.192:15021: connect: connection refused" start-of-body= Nov 03 20:42:53.120260 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:53.120113 5059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" podUID="d14046b2-dd71-42a3-b239-0160088faa0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.192:15021/healthz/ready\": dial tcp 10.132.1.192:15021: connect: connection refused" Nov 03 20:42:54.234889 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:54.234847 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:54.235423 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:54.235395 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:54.236333 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:54.236308 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" Nov 03 20:42:54.270295 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:54.270260 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:54.270697 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:54.270642 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:54.271354 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:54.271336 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" Nov 03 20:42:54.303940 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:54.303913 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:54.304232 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:54.304204 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:54.305022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:54.304993 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" Nov 03 20:42:56.032375 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:56.032336 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f20dd15-mw284"] Nov 03 20:42:56.037312 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:56.037282 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f20dd15-mw284"] Nov 03 20:42:57.855218 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:42:57.855184 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4116cd80-c522-4b8b-9b3e-6229592443e1" path="/var/lib/kubelet/pods/4116cd80-c522-4b8b-9b3e-6229592443e1/volumes" Nov 03 20:43:03.265494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:03.265459 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55c765799f-5qgk6" Nov 03 20:43:06.023269 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:06.023228 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4df82f-f6w99"] Nov 03 20:43:06.026487 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:06.026455 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4df82f-f6w99"] Nov 03 20:43:07.855682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:07.855628 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85d9aa74-8169-4e28-9811-c3befaa47f6d" path="/var/lib/kubelet/pods/85d9aa74-8169-4e28-9811-c3befaa47f6d/volumes" Nov 03 20:43:10.603619 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:43:10.603579 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:43:13.576160 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:13.576122 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e821e4c8-z89bf"] Nov 03 20:43:13.595960 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:13.595921 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e821e4c8-z89bf"] Nov 03 20:43:13.596234 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:13.596198 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e821e4c8-z89bf" Nov 03 20:43:13.603571 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:13.602821 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:43:13.609021 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:13.607868 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\"" Nov 03 20:43:13.799686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:13.792355 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056c992a-b13c-404b-b437-9d9cbf7c0ebe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e821e4c8-z89bf\" (UID: \"056c992a-b13c-404b-b437-9d9cbf7c0ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e821e4c8-z89bf" Nov 03 20:43:13.799996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:13.799965 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7chxn\" (UniqueName: \"kubernetes.io/projected/056c992a-b13c-404b-b437-9d9cbf7c0ebe-kube-api-access-7chxn\") pod \"maas-api-revoke-sub-e821e4c8-z89bf\" (UID: \"056c992a-b13c-404b-b437-9d9cbf7c0ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e821e4c8-z89bf" Nov 03 20:43:13.831061 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:13.830840 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056c992a-b13c-404b-b437-9d9cbf7c0ebe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e821e4c8-z89bf\" (UID: \"056c992a-b13c-404b-b437-9d9cbf7c0ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e821e4c8-z89bf" Nov 03 20:43:13.933915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:13.932135 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056c992a-b13c-404b-b437-9d9cbf7c0ebe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e821e4c8-z89bf\" (UID: \"056c992a-b13c-404b-b437-9d9cbf7c0ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e821e4c8-z89bf" Nov 03 20:43:13.933915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:13.932268 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056c992a-b13c-404b-b437-9d9cbf7c0ebe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e821e4c8-z89bf\" (UID: \"056c992a-b13c-404b-b437-9d9cbf7c0ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e821e4c8-z89bf" Nov 03 20:43:13.933915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:13.932318 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7chxn\" (UniqueName: \"kubernetes.io/projected/056c992a-b13c-404b-b437-9d9cbf7c0ebe-kube-api-access-7chxn\") pod \"maas-api-revoke-sub-e821e4c8-z89bf\" (UID: \"056c992a-b13c-404b-b437-9d9cbf7c0ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e821e4c8-z89bf" Nov 03 20:43:13.933915 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:13.933461 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056c992a-b13c-404b-b437-9d9cbf7c0ebe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e821e4c8-z89bf\" (UID: \"056c992a-b13c-404b-b437-9d9cbf7c0ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e821e4c8-z89bf" Nov 03 20:43:13.952319 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:13.951274 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056c992a-b13c-404b-b437-9d9cbf7c0ebe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e821e4c8-z89bf\" (UID: \"056c992a-b13c-404b-b437-9d9cbf7c0ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e821e4c8-z89bf" Nov 03 20:43:13.961369 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:13.961339 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7chxn\" (UniqueName: \"kubernetes.io/projected/056c992a-b13c-404b-b437-9d9cbf7c0ebe-kube-api-access-7chxn\") pod \"maas-api-revoke-sub-e821e4c8-z89bf\" (UID: \"056c992a-b13c-404b-b437-9d9cbf7c0ebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e821e4c8-z89bf" Nov 03 20:43:14.253738 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:14.252797 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e821e4c8-z89bf" Nov 03 20:43:14.984093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:14.983974 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e821e4c8-z89bf"] Nov 03 20:43:15.415368 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:15.415328 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e821e4c8-z89bf" event={"ID":"056c992a-b13c-404b-b437-9d9cbf7c0ebe","Type":"ContainerStarted","Data":"81ce95d4024846914da7f3797c4d0ea9d48be58eaed69dc4c17d4843483de52e"} Nov 03 20:43:15.415544 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:15.415419 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e821e4c8-z89bf" event={"ID":"056c992a-b13c-404b-b437-9d9cbf7c0ebe","Type":"ContainerStarted","Data":"8f7faf30171a065fa93f89eff9b26312dd71cc4ab73f28d6da7fd436fc42fb02"} Nov 03 20:43:15.435568 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:15.435507 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e821e4c8-z89bf" podStartSLOduration=2.435489461 podStartE2EDuration="2.435489461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:15.433475138 +0000 UTC m=+3989.691272395" watchObservedRunningTime="2026-11-03 20:43:15.435489461 +0000 UTC m=+3989.693286719" Nov 03 20:43:16.425203 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:16.425160 5059 generic.go:356] "Generic (PLEG): container finished" podID="056c992a-b13c-404b-b437-9d9cbf7c0ebe" containerID="81ce95d4024846914da7f3797c4d0ea9d48be58eaed69dc4c17d4843483de52e" exitCode=0 Nov 03 20:43:16.425645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:16.425256 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e821e4c8-z89bf" event={"ID":"056c992a-b13c-404b-b437-9d9cbf7c0ebe","Type":"ContainerDied","Data":"81ce95d4024846914da7f3797c4d0ea9d48be58eaed69dc4c17d4843483de52e"} Nov 03 20:43:17.745849 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:17.745820 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e821e4c8-z89bf" Nov 03 20:43:17.836683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:17.836109 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/056c992a-b13c-404b-b437-9d9cbf7c0ebe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056c992a-b13c-404b-b437-9d9cbf7c0ebe-maas-api-xks-ca\") pod \"056c992a-b13c-404b-b437-9d9cbf7c0ebe\" (UID: \"056c992a-b13c-404b-b437-9d9cbf7c0ebe\") " Nov 03 20:43:17.836683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:17.836177 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/056c992a-b13c-404b-b437-9d9cbf7c0ebe-kube-api-access-7chxn\" (UniqueName: \"kubernetes.io/projected/056c992a-b13c-404b-b437-9d9cbf7c0ebe-kube-api-access-7chxn\") pod \"056c992a-b13c-404b-b437-9d9cbf7c0ebe\" (UID: \"056c992a-b13c-404b-b437-9d9cbf7c0ebe\") " Nov 03 20:43:17.836683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:17.836253 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/056c992a-b13c-404b-b437-9d9cbf7c0ebe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056c992a-b13c-404b-b437-9d9cbf7c0ebe-maas-api-openshift-ca\") pod \"056c992a-b13c-404b-b437-9d9cbf7c0ebe\" (UID: \"056c992a-b13c-404b-b437-9d9cbf7c0ebe\") " Nov 03 20:43:17.837011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:17.836969 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056c992a-b13c-404b-b437-9d9cbf7c0ebe-maas-api-openshift-ca" pod "056c992a-b13c-404b-b437-9d9cbf7c0ebe" (UID: "056c992a-b13c-404b-b437-9d9cbf7c0ebe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:43:17.840084 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:17.839804 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056c992a-b13c-404b-b437-9d9cbf7c0ebe-kube-api-access-7chxn" pod "056c992a-b13c-404b-b437-9d9cbf7c0ebe" (UID: "056c992a-b13c-404b-b437-9d9cbf7c0ebe"). InnerVolumeSpecName "kube-api-access-7chxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:43:17.841249 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:17.841204 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056c992a-b13c-404b-b437-9d9cbf7c0ebe-maas-api-xks-ca" pod "056c992a-b13c-404b-b437-9d9cbf7c0ebe" (UID: "056c992a-b13c-404b-b437-9d9cbf7c0ebe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:43:17.938399 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:17.938171 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056c992a-b13c-404b-b437-9d9cbf7c0ebe-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:17.938399 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:17.938204 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7chxn\" (UniqueName: \"kubernetes.io/projected/056c992a-b13c-404b-b437-9d9cbf7c0ebe-kube-api-access-7chxn\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:17.938399 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:17.938219 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056c992a-b13c-404b-b437-9d9cbf7c0ebe-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:18.036059 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:18.035970 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afbce252-5p2lc"] Nov 03 20:43:18.039157 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:18.039134 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afbce252-5p2lc"] Nov 03 20:43:18.441281 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:18.441242 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e821e4c8-z89bf" Nov 03 20:43:18.441493 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:18.441285 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e821e4c8-z89bf" event={"ID":"056c992a-b13c-404b-b437-9d9cbf7c0ebe","Type":"ContainerDied","Data":"8f7faf30171a065fa93f89eff9b26312dd71cc4ab73f28d6da7fd436fc42fb02"} Nov 03 20:43:18.441493 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:18.441350 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f7faf30171a065fa93f89eff9b26312dd71cc4ab73f28d6da7fd436fc42fb02" Nov 03 20:43:19.854507 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:19.854454 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63e8799b-adbe-467e-9d24-e023836ea491" path="/var/lib/kubelet/pods/63e8799b-adbe-467e-9d24-e023836ea491/volumes" Nov 03 20:43:24.023074 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:24.022971 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef84e730-nwk7x"] Nov 03 20:43:24.025997 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:24.025967 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef84e730-nwk7x"] Nov 03 20:43:25.855053 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:25.855019 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9838d42f-c011-4397-b083-22844a12a935" path="/var/lib/kubelet/pods/9838d42f-c011-4397-b083-22844a12a935/volumes" Nov 03 20:43:25.991607 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:25.991577 5059 scope.go:122] "RemoveContainer" containerID="ed7c56678aaead66b76d36977ee5bbcff501f82edcc806f55e8ece2a056414d6" Nov 03 20:43:26.003008 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:26.002636 5059 scope.go:122] "RemoveContainer" containerID="d803c02bb6490bea1157b932cae2178c788af8b08840683537bc731d34499fcc" Nov 03 20:43:26.010767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:26.010745 5059 scope.go:122] "RemoveContainer" containerID="ffc657b6b1ccc18cb02ede62e23f821eed2b5cedb21fa8048d678ad100ea5dae" Nov 03 20:43:29.037224 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:29.037185 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1cffd0a-jw76s"] Nov 03 20:43:29.037811 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:29.037789 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1cffd0a-jw76s"] Nov 03 20:43:29.855543 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:29.855508 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94f78637-cb8a-476f-a6cb-40db782c9796" path="/var/lib/kubelet/pods/94f78637-cb8a-476f-a6cb-40db782c9796/volumes" Nov 03 20:43:30.972490 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:30.972443 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-2998j"] Nov 03 20:43:30.972908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:30.972861 5059 cpu_manager.go:414] "containerMap: removing container" podUID="056c992a-b13c-404b-b437-9d9cbf7c0ebe" containerName="revoke-keys" Nov 03 20:43:30.972908 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:30.972877 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="056c992a-b13c-404b-b437-9d9cbf7c0ebe" containerName="revoke-keys" Nov 03 20:43:30.972996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:30.972941 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="056c992a-b13c-404b-b437-9d9cbf7c0ebe" containerName="revoke-keys" Nov 03 20:43:30.977820 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:30.977800 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-2998j" Nov 03 20:43:30.983284 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:30.983255 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:43:30.983753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:30.983299 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\"" Nov 03 20:43:30.985225 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:30.985201 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-2998j"] Nov 03 20:43:31.032146 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:31.032119 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844f167a-92da-4121-a0c3-6d99f4ee8c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-2998j\" (UID: \"844f167a-92da-4121-a0c3-6d99f4ee8c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-2998j" Nov 03 20:43:31.032288 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:31.032237 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844f167a-92da-4121-a0c3-6d99f4ee8c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-2998j\" (UID: \"844f167a-92da-4121-a0c3-6d99f4ee8c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-2998j" Nov 03 20:43:31.032288 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:31.032263 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc27c\" (UniqueName: \"kubernetes.io/projected/844f167a-92da-4121-a0c3-6d99f4ee8c23-kube-api-access-rc27c\") pod \"maas-api-revoke-sub-f904bff9-2998j\" (UID: \"844f167a-92da-4121-a0c3-6d99f4ee8c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-2998j" Nov 03 20:43:31.132892 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:31.132862 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844f167a-92da-4121-a0c3-6d99f4ee8c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-2998j\" (UID: \"844f167a-92da-4121-a0c3-6d99f4ee8c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-2998j" Nov 03 20:43:31.133034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:31.132973 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844f167a-92da-4121-a0c3-6d99f4ee8c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-2998j\" (UID: \"844f167a-92da-4121-a0c3-6d99f4ee8c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-2998j" Nov 03 20:43:31.133034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:31.132995 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc27c\" (UniqueName: \"kubernetes.io/projected/844f167a-92da-4121-a0c3-6d99f4ee8c23-kube-api-access-rc27c\") pod \"maas-api-revoke-sub-f904bff9-2998j\" (UID: \"844f167a-92da-4121-a0c3-6d99f4ee8c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-2998j" Nov 03 20:43:31.133682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:31.133640 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844f167a-92da-4121-a0c3-6d99f4ee8c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-2998j\" (UID: \"844f167a-92da-4121-a0c3-6d99f4ee8c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-2998j" Nov 03 20:43:31.135339 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:31.135306 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844f167a-92da-4121-a0c3-6d99f4ee8c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-2998j\" (UID: \"844f167a-92da-4121-a0c3-6d99f4ee8c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-2998j" Nov 03 20:43:31.145861 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:31.145838 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc27c\" (UniqueName: \"kubernetes.io/projected/844f167a-92da-4121-a0c3-6d99f4ee8c23-kube-api-access-rc27c\") pod \"maas-api-revoke-sub-f904bff9-2998j\" (UID: \"844f167a-92da-4121-a0c3-6d99f4ee8c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-2998j" Nov 03 20:43:31.294730 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:31.294649 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-2998j" Nov 03 20:43:31.569970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:31.569925 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-2998j"] Nov 03 20:43:31.572111 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:43:31.572083 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844f167a_92da_4121_a0c3_6d99f4ee8c23.slice/crio-4e3f8f69baa0cd5d06a42e749f450fecab45faa17b37a74ec91cc48b0d8b1328 WatchSource:0}: Error finding container 4e3f8f69baa0cd5d06a42e749f450fecab45faa17b37a74ec91cc48b0d8b1328: Status 404 returned error can't find the container with id 4e3f8f69baa0cd5d06a42e749f450fecab45faa17b37a74ec91cc48b0d8b1328 Nov 03 20:43:31.622853 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:31.622823 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-2998j" event={"ID":"844f167a-92da-4121-a0c3-6d99f4ee8c23","Type":"ContainerStarted","Data":"4e3f8f69baa0cd5d06a42e749f450fecab45faa17b37a74ec91cc48b0d8b1328"} Nov 03 20:43:32.628743 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:32.628641 5059 generic.go:356] "Generic (PLEG): container finished" podID="844f167a-92da-4121-a0c3-6d99f4ee8c23" containerID="dd3d2b164a369e76a0cf0c7ff20a16c679cba81dafafe97daf8ffc6a18885b40" exitCode=0 Nov 03 20:43:32.629099 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:32.628730 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-2998j" event={"ID":"844f167a-92da-4121-a0c3-6d99f4ee8c23","Type":"ContainerDied","Data":"dd3d2b164a369e76a0cf0c7ff20a16c679cba81dafafe97daf8ffc6a18885b40"} Nov 03 20:43:33.759215 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:33.759184 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-2998j" Nov 03 20:43:33.856388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:33.856360 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/844f167a-92da-4121-a0c3-6d99f4ee8c23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844f167a-92da-4121-a0c3-6d99f4ee8c23-maas-api-openshift-ca\") pod \"844f167a-92da-4121-a0c3-6d99f4ee8c23\" (UID: \"844f167a-92da-4121-a0c3-6d99f4ee8c23\") " Nov 03 20:43:33.856551 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:33.856464 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/844f167a-92da-4121-a0c3-6d99f4ee8c23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844f167a-92da-4121-a0c3-6d99f4ee8c23-maas-api-xks-ca\") pod \"844f167a-92da-4121-a0c3-6d99f4ee8c23\" (UID: \"844f167a-92da-4121-a0c3-6d99f4ee8c23\") " Nov 03 20:43:33.856551 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:33.856511 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844f167a-92da-4121-a0c3-6d99f4ee8c23-kube-api-access-rc27c\" (UniqueName: \"kubernetes.io/projected/844f167a-92da-4121-a0c3-6d99f4ee8c23-kube-api-access-rc27c\") pod \"844f167a-92da-4121-a0c3-6d99f4ee8c23\" (UID: \"844f167a-92da-4121-a0c3-6d99f4ee8c23\") " Nov 03 20:43:33.856745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:33.856721 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/844f167a-92da-4121-a0c3-6d99f4ee8c23-maas-api-openshift-ca" pod "844f167a-92da-4121-a0c3-6d99f4ee8c23" (UID: "844f167a-92da-4121-a0c3-6d99f4ee8c23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:43:33.858602 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:33.858575 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/844f167a-92da-4121-a0c3-6d99f4ee8c23-maas-api-xks-ca" pod "844f167a-92da-4121-a0c3-6d99f4ee8c23" (UID: "844f167a-92da-4121-a0c3-6d99f4ee8c23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:43:33.858763 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:33.858616 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844f167a-92da-4121-a0c3-6d99f4ee8c23-kube-api-access-rc27c" pod "844f167a-92da-4121-a0c3-6d99f4ee8c23" (UID: "844f167a-92da-4121-a0c3-6d99f4ee8c23"). InnerVolumeSpecName "kube-api-access-rc27c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:43:33.958098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:33.958030 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844f167a-92da-4121-a0c3-6d99f4ee8c23-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:33.958098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:33.958068 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc27c\" (UniqueName: \"kubernetes.io/projected/844f167a-92da-4121-a0c3-6d99f4ee8c23-kube-api-access-rc27c\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:33.958098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:33.958079 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844f167a-92da-4121-a0c3-6d99f4ee8c23-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:34.110852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:34.110820 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91eb51b2-ffrxf"] Nov 03 20:43:34.113974 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:34.113943 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91eb51b2-ffrxf"] Nov 03 20:43:34.639370 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:34.639338 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-2998j" event={"ID":"844f167a-92da-4121-a0c3-6d99f4ee8c23","Type":"ContainerDied","Data":"4e3f8f69baa0cd5d06a42e749f450fecab45faa17b37a74ec91cc48b0d8b1328"} Nov 03 20:43:34.639547 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:34.639387 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e3f8f69baa0cd5d06a42e749f450fecab45faa17b37a74ec91cc48b0d8b1328" Nov 03 20:43:34.639547 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:34.639350 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-2998j" Nov 03 20:43:35.854837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:35.854797 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="698bcd39-e520-42bc-90d2-04a26edd1226" path="/var/lib/kubelet/pods/698bcd39-e520-42bc-90d2-04a26edd1226/volumes" Nov 03 20:43:40.022636 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:40.022599 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a2057e-ttfc8"] Nov 03 20:43:40.025271 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:40.025245 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a2057e-ttfc8"] Nov 03 20:43:40.604007 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:43:40.603968 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:43:41.856369 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:41.856332 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d16ebeb-8fdc-4f51-9e3e-5242e112979a" path="/var/lib/kubelet/pods/9d16ebeb-8fdc-4f51-9e3e-5242e112979a/volumes" Nov 03 20:43:45.041123 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:45.041074 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6e55f39-pk65m"] Nov 03 20:43:45.057047 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:45.057010 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6e55f39-pk65m"] Nov 03 20:43:45.889015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:45.888976 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="218d1b82-84c1-4985-a260-20625bae7705" path="/var/lib/kubelet/pods/218d1b82-84c1-4985-a260-20625bae7705/volumes" Nov 03 20:43:50.025792 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:50.025740 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7e090c0-g6dld"] Nov 03 20:43:50.028485 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:50.028456 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7e090c0-g6dld"] Nov 03 20:43:51.893687 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:51.886777 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d66618e-ab4a-47f8-bfc6-1fcafe20214d" path="/var/lib/kubelet/pods/2d66618e-ab4a-47f8-bfc6-1fcafe20214d/volumes" Nov 03 20:43:53.387591 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:53.387552 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f9lpd"] Nov 03 20:43:53.388097 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:53.388075 5059 cpu_manager.go:414] "containerMap: removing container" podUID="844f167a-92da-4121-a0c3-6d99f4ee8c23" containerName="revoke-keys" Nov 03 20:43:53.388196 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:53.388100 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844f167a-92da-4121-a0c3-6d99f4ee8c23" containerName="revoke-keys" Nov 03 20:43:53.388253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:53.388244 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="844f167a-92da-4121-a0c3-6d99f4ee8c23" containerName="revoke-keys" Nov 03 20:43:53.391307 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:53.391285 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f9lpd" Nov 03 20:43:53.393955 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:53.393930 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/162b6a47-7cf0-4da2-9fde-fb272ab94617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-f9lpd\" (UID: \"162b6a47-7cf0-4da2-9fde-fb272ab94617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f9lpd" Nov 03 20:43:53.394054 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:53.393974 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/162b6a47-7cf0-4da2-9fde-fb272ab94617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-f9lpd\" (UID: \"162b6a47-7cf0-4da2-9fde-fb272ab94617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f9lpd" Nov 03 20:43:53.394117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:53.394066 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fglp9\" (UniqueName: \"kubernetes.io/projected/162b6a47-7cf0-4da2-9fde-fb272ab94617-kube-api-access-fglp9\") pod \"maas-api-revoke-sub-f03629db-f9lpd\" (UID: \"162b6a47-7cf0-4da2-9fde-fb272ab94617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f9lpd" Nov 03 20:43:53.398806 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:53.398785 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:43:53.398930 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:53.398915 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\"" Nov 03 20:43:53.401219 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:53.401192 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f9lpd"] Nov 03 20:43:53.494852 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:53.494813 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/162b6a47-7cf0-4da2-9fde-fb272ab94617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-f9lpd\" (UID: \"162b6a47-7cf0-4da2-9fde-fb272ab94617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f9lpd" Nov 03 20:43:53.495027 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:53.494873 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/162b6a47-7cf0-4da2-9fde-fb272ab94617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-f9lpd\" (UID: \"162b6a47-7cf0-4da2-9fde-fb272ab94617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f9lpd" Nov 03 20:43:53.495027 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:53.494921 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fglp9\" (UniqueName: \"kubernetes.io/projected/162b6a47-7cf0-4da2-9fde-fb272ab94617-kube-api-access-fglp9\") pod \"maas-api-revoke-sub-f03629db-f9lpd\" (UID: \"162b6a47-7cf0-4da2-9fde-fb272ab94617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f9lpd" Nov 03 20:43:53.495540 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:53.495511 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/162b6a47-7cf0-4da2-9fde-fb272ab94617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-f9lpd\" (UID: \"162b6a47-7cf0-4da2-9fde-fb272ab94617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f9lpd" Nov 03 20:43:53.497175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:53.497146 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/162b6a47-7cf0-4da2-9fde-fb272ab94617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-f9lpd\" (UID: \"162b6a47-7cf0-4da2-9fde-fb272ab94617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f9lpd" Nov 03 20:43:53.507860 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:53.507838 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fglp9\" (UniqueName: \"kubernetes.io/projected/162b6a47-7cf0-4da2-9fde-fb272ab94617-kube-api-access-fglp9\") pod \"maas-api-revoke-sub-f03629db-f9lpd\" (UID: \"162b6a47-7cf0-4da2-9fde-fb272ab94617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f9lpd" Nov 03 20:43:53.707127 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:53.707066 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f9lpd" Nov 03 20:43:53.984679 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:53.984638 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f9lpd"] Nov 03 20:43:53.986500 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:43:53.986469 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162b6a47_7cf0_4da2_9fde_fb272ab94617.slice/crio-f65cca04590f303ab58f1c323c7b084cc536c64cc66708f9604ba294b005d8f5 WatchSource:0}: Error finding container f65cca04590f303ab58f1c323c7b084cc536c64cc66708f9604ba294b005d8f5: Status 404 returned error can't find the container with id f65cca04590f303ab58f1c323c7b084cc536c64cc66708f9604ba294b005d8f5 Nov 03 20:43:54.789534 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:54.789506 5059 generic.go:356] "Generic (PLEG): container finished" podID="162b6a47-7cf0-4da2-9fde-fb272ab94617" containerID="4d839897d659379659a10012daf2228029da65594f6b94b6360513da4f69bddb" exitCode=0 Nov 03 20:43:54.789847 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:54.789588 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f9lpd" event={"ID":"162b6a47-7cf0-4da2-9fde-fb272ab94617","Type":"ContainerDied","Data":"4d839897d659379659a10012daf2228029da65594f6b94b6360513da4f69bddb"} Nov 03 20:43:54.789847 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:54.789648 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f9lpd" event={"ID":"162b6a47-7cf0-4da2-9fde-fb272ab94617","Type":"ContainerStarted","Data":"f65cca04590f303ab58f1c323c7b084cc536c64cc66708f9604ba294b005d8f5"} Nov 03 20:43:55.021456 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:55.021375 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a0ef22-4nk2n"] Nov 03 20:43:55.026275 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:55.026252 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a0ef22-4nk2n"] Nov 03 20:43:55.856122 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:55.856090 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f4a0ec2-017b-4272-a8e6-a95122b3e845" path="/var/lib/kubelet/pods/0f4a0ec2-017b-4272-a8e6-a95122b3e845/volumes" Nov 03 20:43:55.921995 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:55.921972 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f9lpd" Nov 03 20:43:56.016451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:56.016422 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/162b6a47-7cf0-4da2-9fde-fb272ab94617-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/162b6a47-7cf0-4da2-9fde-fb272ab94617-maas-api-openshift-ca\") pod \"162b6a47-7cf0-4da2-9fde-fb272ab94617\" (UID: \"162b6a47-7cf0-4da2-9fde-fb272ab94617\") " Nov 03 20:43:56.016607 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:56.016503 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/162b6a47-7cf0-4da2-9fde-fb272ab94617-kube-api-access-fglp9\" (UniqueName: \"kubernetes.io/projected/162b6a47-7cf0-4da2-9fde-fb272ab94617-kube-api-access-fglp9\") pod \"162b6a47-7cf0-4da2-9fde-fb272ab94617\" (UID: \"162b6a47-7cf0-4da2-9fde-fb272ab94617\") " Nov 03 20:43:56.016607 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:56.016593 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/162b6a47-7cf0-4da2-9fde-fb272ab94617-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/162b6a47-7cf0-4da2-9fde-fb272ab94617-maas-api-xks-ca\") pod \"162b6a47-7cf0-4da2-9fde-fb272ab94617\" (UID: \"162b6a47-7cf0-4da2-9fde-fb272ab94617\") " Nov 03 20:43:56.016882 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:56.016858 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/162b6a47-7cf0-4da2-9fde-fb272ab94617-maas-api-openshift-ca" pod "162b6a47-7cf0-4da2-9fde-fb272ab94617" (UID: "162b6a47-7cf0-4da2-9fde-fb272ab94617"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:43:56.018716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:56.018684 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/162b6a47-7cf0-4da2-9fde-fb272ab94617-maas-api-xks-ca" pod "162b6a47-7cf0-4da2-9fde-fb272ab94617" (UID: "162b6a47-7cf0-4da2-9fde-fb272ab94617"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:43:56.018827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:56.018728 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/162b6a47-7cf0-4da2-9fde-fb272ab94617-kube-api-access-fglp9" pod "162b6a47-7cf0-4da2-9fde-fb272ab94617" (UID: "162b6a47-7cf0-4da2-9fde-fb272ab94617"). InnerVolumeSpecName "kube-api-access-fglp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:43:56.117922 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:56.117894 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/162b6a47-7cf0-4da2-9fde-fb272ab94617-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:56.118034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:56.117921 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/162b6a47-7cf0-4da2-9fde-fb272ab94617-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:56.118034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:56.117933 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fglp9\" (UniqueName: \"kubernetes.io/projected/162b6a47-7cf0-4da2-9fde-fb272ab94617-kube-api-access-fglp9\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:56.800769 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:56.800736 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f9lpd" event={"ID":"162b6a47-7cf0-4da2-9fde-fb272ab94617","Type":"ContainerDied","Data":"f65cca04590f303ab58f1c323c7b084cc536c64cc66708f9604ba294b005d8f5"} Nov 03 20:43:56.800769 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:56.800761 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f9lpd" Nov 03 20:43:56.800769 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:43:56.800786 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f65cca04590f303ab58f1c323c7b084cc536c64cc66708f9604ba294b005d8f5" Nov 03 20:44:10.603142 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:44:10.603101 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:44:18.908164 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:18.908132 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h2qs2"] Nov 03 20:44:18.908583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:18.908540 5059 cpu_manager.go:414] "containerMap: removing container" podUID="162b6a47-7cf0-4da2-9fde-fb272ab94617" containerName="revoke-keys" Nov 03 20:44:18.908583 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:18.908553 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="162b6a47-7cf0-4da2-9fde-fb272ab94617" containerName="revoke-keys" Nov 03 20:44:18.908692 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:18.908615 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="162b6a47-7cf0-4da2-9fde-fb272ab94617" containerName="revoke-keys" Nov 03 20:44:18.912007 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:18.911990 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h2qs2" Nov 03 20:44:18.917439 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:18.917367 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:44:18.920004 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:18.919418 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\"" Nov 03 20:44:18.923238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:18.923193 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h2qs2"] Nov 03 20:44:18.963566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:18.963539 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85hk6\" (UniqueName: \"kubernetes.io/projected/0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19-kube-api-access-85hk6\") pod \"maas-api-revoke-sub-0165d7d2-h2qs2\" (UID: \"0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h2qs2" Nov 03 20:44:18.963703 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:18.963573 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-h2qs2\" (UID: \"0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h2qs2" Nov 03 20:44:18.963703 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:18.963593 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-h2qs2\" (UID: \"0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h2qs2" Nov 03 20:44:19.065269 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:19.065236 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-h2qs2\" (UID: \"0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h2qs2" Nov 03 20:44:19.065402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:19.065277 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-h2qs2\" (UID: \"0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h2qs2" Nov 03 20:44:19.065402 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:19.065366 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85hk6\" (UniqueName: \"kubernetes.io/projected/0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19-kube-api-access-85hk6\") pod \"maas-api-revoke-sub-0165d7d2-h2qs2\" (UID: \"0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h2qs2" Nov 03 20:44:19.066101 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:19.066081 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-h2qs2\" (UID: \"0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h2qs2" Nov 03 20:44:19.067721 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:19.067699 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-h2qs2\" (UID: \"0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h2qs2" Nov 03 20:44:19.077999 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:19.077972 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85hk6\" (UniqueName: \"kubernetes.io/projected/0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19-kube-api-access-85hk6\") pod \"maas-api-revoke-sub-0165d7d2-h2qs2\" (UID: \"0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h2qs2" Nov 03 20:44:19.227628 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:19.227568 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h2qs2" Nov 03 20:44:19.502050 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:19.502017 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h2qs2"] Nov 03 20:44:19.503948 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:44:19.503909 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae6cda1_4bc8_46b6_af16_6ad7cbb8cf19.slice/crio-383e7cd12b44056f15c2d7b27f05cb240020582ad92fa4e18ccb64ddaed1a612 WatchSource:0}: Error finding container 383e7cd12b44056f15c2d7b27f05cb240020582ad92fa4e18ccb64ddaed1a612: Status 404 returned error can't find the container with id 383e7cd12b44056f15c2d7b27f05cb240020582ad92fa4e18ccb64ddaed1a612 Nov 03 20:44:20.007737 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:20.007693 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h2qs2" event={"ID":"0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19","Type":"ContainerStarted","Data":"27bbce1874459ff8a0f12e84bd3996099da61d18b379f82e83ce736a5d2b4a28"} Nov 03 20:44:20.008115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:20.007760 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h2qs2" event={"ID":"0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19","Type":"ContainerStarted","Data":"383e7cd12b44056f15c2d7b27f05cb240020582ad92fa4e18ccb64ddaed1a612"} Nov 03 20:44:20.027723 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:20.027645 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h2qs2" podStartSLOduration=2.02763204 podStartE2EDuration="2.02763204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:20.025964436 +0000 UTC m=+4054.283761693" watchObservedRunningTime="2026-11-03 20:44:20.02763204 +0000 UTC m=+4054.285429297" Nov 03 20:44:21.017739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:21.017701 5059 generic.go:356] "Generic (PLEG): container finished" podID="0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19" containerID="27bbce1874459ff8a0f12e84bd3996099da61d18b379f82e83ce736a5d2b4a28" exitCode=0 Nov 03 20:44:21.018221 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:21.017783 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h2qs2" event={"ID":"0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19","Type":"ContainerDied","Data":"27bbce1874459ff8a0f12e84bd3996099da61d18b379f82e83ce736a5d2b4a28"} Nov 03 20:44:22.152295 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:22.152265 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h2qs2" Nov 03 20:44:22.294076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:22.294005 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19-maas-api-openshift-ca\") pod \"0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19\" (UID: \"0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19\") " Nov 03 20:44:22.294076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:22.294055 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19-maas-api-xks-ca\") pod \"0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19\" (UID: \"0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19\") " Nov 03 20:44:22.294297 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:22.294078 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19-kube-api-access-85hk6\" (UniqueName: \"kubernetes.io/projected/0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19-kube-api-access-85hk6\") pod \"0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19\" (UID: \"0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19\") " Nov 03 20:44:22.294394 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:22.294368 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19-maas-api-openshift-ca" pod "0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19" (UID: "0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:44:22.296073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:22.296050 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19-maas-api-xks-ca" pod "0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19" (UID: "0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:44:22.296189 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:22.296139 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19-kube-api-access-85hk6" pod "0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19" (UID: "0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19"). InnerVolumeSpecName "kube-api-access-85hk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:44:22.395595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:22.395566 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:44:22.395595 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:22.395594 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:44:22.395754 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:22.395610 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85hk6\" (UniqueName: \"kubernetes.io/projected/0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19-kube-api-access-85hk6\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:44:23.029051 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:23.029019 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h2qs2" Nov 03 20:44:23.029051 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:23.029040 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h2qs2" event={"ID":"0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19","Type":"ContainerDied","Data":"383e7cd12b44056f15c2d7b27f05cb240020582ad92fa4e18ccb64ddaed1a612"} Nov 03 20:44:23.029277 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:23.029089 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="383e7cd12b44056f15c2d7b27f05cb240020582ad92fa4e18ccb64ddaed1a612" Nov 03 20:44:26.041096 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:26.041062 5059 scope.go:122] "RemoveContainer" containerID="c90537b876b3c98c55cfbbfdebb402a668ecd00a9ef5329a9c55ef34f54464bc" Nov 03 20:44:26.049767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:26.049741 5059 scope.go:122] "RemoveContainer" containerID="fa1149da688477f85a9f5ad2c8fec97d2e9933c65e460445477663d5bc18db84" Nov 03 20:44:26.057178 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:26.057157 5059 scope.go:122] "RemoveContainer" containerID="a73d1f17cbb151aa3c795ae2e9fb3a6caf3cd036fdad2181a72441d23d729288" Nov 03 20:44:26.064778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:26.064758 5059 scope.go:122] "RemoveContainer" containerID="4dfc1c488968df6a3c6e2090c8ab8f4cda5e9e8aa654dc23b5dd6a2512bfb770" Nov 03 20:44:26.073855 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:26.073833 5059 scope.go:122] "RemoveContainer" containerID="0ed100d4d9dd9a93af0fb92d61429fe51c8e89a379a6a74ce59ae9100b66a5ea" Nov 03 20:44:29.736814 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:29.736777 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnm86"] Nov 03 20:44:29.737213 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:29.737143 5059 cpu_manager.go:414] "containerMap: removing container" podUID="0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19" containerName="revoke-keys" Nov 03 20:44:29.737213 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:29.737155 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19" containerName="revoke-keys" Nov 03 20:44:29.737294 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:29.737219 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19" containerName="revoke-keys" Nov 03 20:44:29.742179 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:29.742147 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnm86" Nov 03 20:44:29.747535 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:29.747512 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:44:29.748061 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:29.748037 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\"" Nov 03 20:44:29.750285 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:29.750257 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnm86"] Nov 03 20:44:29.858108 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:29.858074 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f2df39-c182-4e84-a6bb-c326becd0959-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-mnm86\" (UID: \"71f2df39-c182-4e84-a6bb-c326becd0959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnm86" Nov 03 20:44:29.858281 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:29.858131 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f2df39-c182-4e84-a6bb-c326becd0959-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-mnm86\" (UID: \"71f2df39-c182-4e84-a6bb-c326becd0959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnm86" Nov 03 20:44:29.858327 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:29.858270 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjg6s\" (UniqueName: \"kubernetes.io/projected/71f2df39-c182-4e84-a6bb-c326becd0959-kube-api-access-qjg6s\") pod \"maas-api-revoke-sub-6b19338f-mnm86\" (UID: \"71f2df39-c182-4e84-a6bb-c326becd0959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnm86" Nov 03 20:44:29.959645 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:29.959613 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjg6s\" (UniqueName: \"kubernetes.io/projected/71f2df39-c182-4e84-a6bb-c326becd0959-kube-api-access-qjg6s\") pod \"maas-api-revoke-sub-6b19338f-mnm86\" (UID: \"71f2df39-c182-4e84-a6bb-c326becd0959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnm86" Nov 03 20:44:29.959802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:29.959705 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f2df39-c182-4e84-a6bb-c326becd0959-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-mnm86\" (UID: \"71f2df39-c182-4e84-a6bb-c326becd0959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnm86" Nov 03 20:44:29.959802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:29.959768 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f2df39-c182-4e84-a6bb-c326becd0959-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-mnm86\" (UID: \"71f2df39-c182-4e84-a6bb-c326becd0959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnm86" Nov 03 20:44:29.960446 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:29.960423 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f2df39-c182-4e84-a6bb-c326becd0959-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-mnm86\" (UID: \"71f2df39-c182-4e84-a6bb-c326becd0959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnm86" Nov 03 20:44:29.962036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:29.962015 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f2df39-c182-4e84-a6bb-c326becd0959-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-mnm86\" (UID: \"71f2df39-c182-4e84-a6bb-c326becd0959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnm86" Nov 03 20:44:29.971995 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:29.971975 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjg6s\" (UniqueName: \"kubernetes.io/projected/71f2df39-c182-4e84-a6bb-c326becd0959-kube-api-access-qjg6s\") pod \"maas-api-revoke-sub-6b19338f-mnm86\" (UID: \"71f2df39-c182-4e84-a6bb-c326becd0959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnm86" Nov 03 20:44:30.058115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:30.058031 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnm86" Nov 03 20:44:30.335427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:30.335401 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnm86"] Nov 03 20:44:30.337576 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:44:30.337532 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f2df39_c182_4e84_a6bb_c326becd0959.slice/crio-283ea5f2c40089847cddd86bea3654d44492e6b8ff71eef05111df9f39e1876d WatchSource:0}: Error finding container 283ea5f2c40089847cddd86bea3654d44492e6b8ff71eef05111df9f39e1876d: Status 404 returned error can't find the container with id 283ea5f2c40089847cddd86bea3654d44492e6b8ff71eef05111df9f39e1876d Nov 03 20:44:31.069291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:31.069258 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnm86" event={"ID":"71f2df39-c182-4e84-a6bb-c326becd0959","Type":"ContainerStarted","Data":"4e3da925b77498e1e3766d9acf54f5bc0fac760bd7365a8da895f9a4dee3b286"} Nov 03 20:44:31.069646 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:31.069314 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnm86" event={"ID":"71f2df39-c182-4e84-a6bb-c326becd0959","Type":"ContainerStarted","Data":"283ea5f2c40089847cddd86bea3654d44492e6b8ff71eef05111df9f39e1876d"} Nov 03 20:44:31.087957 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:31.087903 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnm86" podStartSLOduration=2.087890812 podStartE2EDuration="2.087890812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:31.08696156 +0000 UTC m=+4065.344758828" watchObservedRunningTime="2026-11-03 20:44:31.087890812 +0000 UTC m=+4065.345688069" Nov 03 20:44:32.075445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:32.075407 5059 generic.go:356] "Generic (PLEG): container finished" podID="71f2df39-c182-4e84-a6bb-c326becd0959" containerID="4e3da925b77498e1e3766d9acf54f5bc0fac760bd7365a8da895f9a4dee3b286" exitCode=0 Nov 03 20:44:32.075916 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:32.075494 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnm86" event={"ID":"71f2df39-c182-4e84-a6bb-c326becd0959","Type":"ContainerDied","Data":"4e3da925b77498e1e3766d9acf54f5bc0fac760bd7365a8da895f9a4dee3b286"} Nov 03 20:44:33.210353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:33.210321 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnm86" Nov 03 20:44:33.408187 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:33.408082 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71f2df39-c182-4e84-a6bb-c326becd0959-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f2df39-c182-4e84-a6bb-c326becd0959-maas-api-openshift-ca\") pod \"71f2df39-c182-4e84-a6bb-c326becd0959\" (UID: \"71f2df39-c182-4e84-a6bb-c326becd0959\") " Nov 03 20:44:33.408187 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:33.408148 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71f2df39-c182-4e84-a6bb-c326becd0959-kube-api-access-qjg6s\" (UniqueName: \"kubernetes.io/projected/71f2df39-c182-4e84-a6bb-c326becd0959-kube-api-access-qjg6s\") pod \"71f2df39-c182-4e84-a6bb-c326becd0959\" (UID: \"71f2df39-c182-4e84-a6bb-c326becd0959\") " Nov 03 20:44:33.408428 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:33.408316 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71f2df39-c182-4e84-a6bb-c326becd0959-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f2df39-c182-4e84-a6bb-c326becd0959-maas-api-xks-ca\") pod \"71f2df39-c182-4e84-a6bb-c326becd0959\" (UID: \"71f2df39-c182-4e84-a6bb-c326becd0959\") " Nov 03 20:44:33.408511 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:33.408484 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f2df39-c182-4e84-a6bb-c326becd0959-maas-api-openshift-ca" pod "71f2df39-c182-4e84-a6bb-c326becd0959" (UID: "71f2df39-c182-4e84-a6bb-c326becd0959"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:44:33.408638 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:33.408614 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f2df39-c182-4e84-a6bb-c326becd0959-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:44:33.410423 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:33.410402 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71f2df39-c182-4e84-a6bb-c326becd0959-maas-api-xks-ca" pod "71f2df39-c182-4e84-a6bb-c326becd0959" (UID: "71f2df39-c182-4e84-a6bb-c326becd0959"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:44:33.410423 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:33.410414 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f2df39-c182-4e84-a6bb-c326becd0959-kube-api-access-qjg6s" pod "71f2df39-c182-4e84-a6bb-c326becd0959" (UID: "71f2df39-c182-4e84-a6bb-c326becd0959"). InnerVolumeSpecName "kube-api-access-qjg6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:44:33.510103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:33.510070 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f2df39-c182-4e84-a6bb-c326becd0959-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:44:33.510103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:33.510100 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjg6s\" (UniqueName: \"kubernetes.io/projected/71f2df39-c182-4e84-a6bb-c326becd0959-kube-api-access-qjg6s\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:44:34.087442 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:34.087409 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnm86" Nov 03 20:44:34.087613 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:34.087446 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnm86" event={"ID":"71f2df39-c182-4e84-a6bb-c326becd0959","Type":"ContainerDied","Data":"283ea5f2c40089847cddd86bea3654d44492e6b8ff71eef05111df9f39e1876d"} Nov 03 20:44:34.087613 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:34.087495 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="283ea5f2c40089847cddd86bea3654d44492e6b8ff71eef05111df9f39e1876d" Nov 03 20:44:40.604423 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:44:40.604380 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:44:41.024493 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:41.024395 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5162af5-5t4dg"] Nov 03 20:44:41.027972 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:41.027941 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5162af5-5t4dg"] Nov 03 20:44:41.968716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:41.951376 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="429244dc-47a6-43af-967f-91baa92f3bca" path="/var/lib/kubelet/pods/429244dc-47a6-43af-967f-91baa92f3bca/volumes" Nov 03 20:44:43.145582 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.145538 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt"] Nov 03 20:44:43.146677 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.146633 5059 cpu_manager.go:414] "containerMap: removing container" podUID="71f2df39-c182-4e84-a6bb-c326becd0959" containerName="revoke-keys" Nov 03 20:44:43.146848 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.146828 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71f2df39-c182-4e84-a6bb-c326becd0959" containerName="revoke-keys" Nov 03 20:44:43.147098 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.147079 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="71f2df39-c182-4e84-a6bb-c326becd0959" containerName="revoke-keys" Nov 03 20:44:43.150986 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.150966 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt"] Nov 03 20:44:43.151291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.151258 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:44:43.206228 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.206196 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-f44f3e5c-kserve-self-signed-certs\"" Nov 03 20:44:43.316379 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.316334 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-dshm\") pod \"e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt\" (UID: \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\") " pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:44:43.316775 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.316705 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-tmp-dir\") pod \"e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt\" (UID: \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\") " pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:44:43.316963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.316945 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-model-cache\") pod \"e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt\" (UID: \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\") " pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:44:43.317195 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.317176 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-home\") pod \"e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt\" (UID: \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\") " pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:44:43.317579 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.317518 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-tls-certs\") pod \"e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt\" (UID: \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\") " pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:44:43.317781 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.317752 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2mwv\" (UniqueName: \"kubernetes.io/projected/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-kube-api-access-x2mwv\") pod \"e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt\" (UID: \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\") " pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:44:43.398339 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.398255 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2"] Nov 03 20:44:43.420195 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.420157 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-tls-certs\") pod \"e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt\" (UID: \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\") " pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:44:43.420405 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.420385 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2mwv\" (UniqueName: \"kubernetes.io/projected/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-kube-api-access-x2mwv\") pod \"e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt\" (UID: \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\") " pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:44:43.420558 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.420543 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-dshm\") pod \"e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt\" (UID: \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\") " pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:44:43.420675 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.420646 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-tmp-dir\") pod \"e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt\" (UID: \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\") " pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:44:43.420802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.420787 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-model-cache\") pod \"e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt\" (UID: \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\") " pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:44:43.420902 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.420889 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-home\") pod \"e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt\" (UID: \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\") " pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:44:43.421535 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.421508 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-home\") pod \"e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt\" (UID: \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\") " pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:44:43.423495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.423469 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-tmp-dir\") pod \"e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt\" (UID: \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\") " pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:44:43.425678 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.425372 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2" Nov 03 20:44:43.428453 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.425599 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-model-cache\") pod \"e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt\" (UID: \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\") " pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:44:43.432681 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.432098 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-f44f3e5c-epp-sa-dockercfg-q2f5z\"" Nov 03 20:44:43.448376 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.448355 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2"] Nov 03 20:44:43.458304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.458277 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-tls-certs\") pod \"e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt\" (UID: \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\") " pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:44:43.458679 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.458636 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-dshm\") pod \"e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt\" (UID: \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\") " pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:44:43.460983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.460929 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2mwv\" (UniqueName: \"kubernetes.io/projected/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-kube-api-access-x2mwv\") pod \"e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt\" (UID: \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\") " pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:44:43.522397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.522184 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z4gt\" (UniqueName: \"kubernetes.io/projected/75a93f18-b77a-4379-8d7d-15a37a9d8094-kube-api-access-8z4gt\") pod \"e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2\" (UID: \"75a93f18-b77a-4379-8d7d-15a37a9d8094\") " pod="llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2" Nov 03 20:44:43.522397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.522254 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a93f18-b77a-4379-8d7d-15a37a9d8094-tls-certs\") pod \"e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2\" (UID: \"75a93f18-b77a-4379-8d7d-15a37a9d8094\") " pod="llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2" Nov 03 20:44:43.574264 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.574226 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:44:43.631965 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.631172 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z4gt\" (UniqueName: \"kubernetes.io/projected/75a93f18-b77a-4379-8d7d-15a37a9d8094-kube-api-access-8z4gt\") pod \"e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2\" (UID: \"75a93f18-b77a-4379-8d7d-15a37a9d8094\") " pod="llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2" Nov 03 20:44:43.631965 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.631229 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a93f18-b77a-4379-8d7d-15a37a9d8094-tls-certs\") pod \"e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2\" (UID: \"75a93f18-b77a-4379-8d7d-15a37a9d8094\") " pod="llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2" Nov 03 20:44:43.644197 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.642326 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a93f18-b77a-4379-8d7d-15a37a9d8094-tls-certs\") pod \"e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2\" (UID: \"75a93f18-b77a-4379-8d7d-15a37a9d8094\") " pod="llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2" Nov 03 20:44:43.649709 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.649592 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z4gt\" (UniqueName: \"kubernetes.io/projected/75a93f18-b77a-4379-8d7d-15a37a9d8094-kube-api-access-8z4gt\") pod \"e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2\" (UID: \"75a93f18-b77a-4379-8d7d-15a37a9d8094\") " pod="llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2" Nov 03 20:44:43.748262 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.748232 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2" Nov 03 20:44:43.906714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:43.906612 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt"] Nov 03 20:44:43.912841 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:44:43.912812 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod982df7da_1adf_46c9_b249_ec2e6b5b3e3b.slice/crio-dbc6c2335f70f9dfffd5806187ff79501251990bb6a1a092970cbdc0c0a16b97 WatchSource:0}: Error finding container dbc6c2335f70f9dfffd5806187ff79501251990bb6a1a092970cbdc0c0a16b97: Status 404 returned error can't find the container with id dbc6c2335f70f9dfffd5806187ff79501251990bb6a1a092970cbdc0c0a16b97 Nov 03 20:44:44.024598 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:44.024563 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2"] Nov 03 20:44:44.026549 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:44:44.026521 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a93f18_b77a_4379_8d7d_15a37a9d8094.slice/crio-1bd2a7d4691dd7b4fc7212bf03a06619b63512c37a699c9e6d9b4ebd22e0bf78 WatchSource:0}: Error finding container 1bd2a7d4691dd7b4fc7212bf03a06619b63512c37a699c9e6d9b4ebd22e0bf78: Status 404 returned error can't find the container with id 1bd2a7d4691dd7b4fc7212bf03a06619b63512c37a699c9e6d9b4ebd22e0bf78 Nov 03 20:44:44.282035 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:44.281928 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2" event={"ID":"75a93f18-b77a-4379-8d7d-15a37a9d8094","Type":"ContainerStarted","Data":"1bd2a7d4691dd7b4fc7212bf03a06619b63512c37a699c9e6d9b4ebd22e0bf78"} Nov 03 20:44:44.283562 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:44.283529 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" event={"ID":"982df7da-1adf-46c9-b249-ec2e6b5b3e3b","Type":"ContainerStarted","Data":"aab7bab9bf4cdfb73432f58d161a0cb5f87b9bca7b26b57844acddbdfbffdd86"} Nov 03 20:44:44.283716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:44.283566 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" event={"ID":"982df7da-1adf-46c9-b249-ec2e6b5b3e3b","Type":"ContainerStarted","Data":"dbc6c2335f70f9dfffd5806187ff79501251990bb6a1a092970cbdc0c0a16b97"} Nov 03 20:44:44.304773 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:44.304727 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" podStartSLOduration=1.304715077 podStartE2EDuration="1.304715077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:44.302744705 +0000 UTC m=+4078.560541962" watchObservedRunningTime="2026-11-03 20:44:44.304715077 +0000 UTC m=+4078.562512333" Nov 03 20:44:44.574450 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:44.574412 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:44:47.330686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:47.329739 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2" event={"ID":"75a93f18-b77a-4379-8d7d-15a37a9d8094","Type":"ContainerStarted","Data":"a1fde4edddbaf8230512efe9ceeab87733a5e8f11b9a37aa1e36131bcd1468ee"} Nov 03 20:44:47.331217 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:47.330709 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2" Nov 03 20:44:47.358536 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:47.358477 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2" podStartSLOduration=2.080917761 podStartE2EDuration="4.358458638s" totalImagesPullingTime="2.277540877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:47.357063434 +0000 UTC m=+4081.614860694" watchObservedRunningTime="2026-11-03 20:44:47.358458638 +0000 UTC m=+4081.616255895" Nov 03 20:44:51.025419 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:51.025331 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44e73305-xdqsv"] Nov 03 20:44:51.029010 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:51.028980 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44e73305-xdqsv"] Nov 03 20:44:51.854425 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:51.854392 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4624bf15-7e01-4304-801c-a315f5ab38b8" path="/var/lib/kubelet/pods/4624bf15-7e01-4304-801c-a315f5ab38b8/volumes" Nov 03 20:44:53.574925 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:53.574892 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:44:53.587415 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:53.587386 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:44:54.385400 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:44:54.385369 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:45:04.024811 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:04.024768 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82e9ecf7-bk8kt"] Nov 03 20:45:04.026937 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:04.026910 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82e9ecf7-bk8kt"] Nov 03 20:45:05.855608 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:05.855570 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b330e09-cf3a-4278-bc82-5386105badd1" path="/var/lib/kubelet/pods/3b330e09-cf3a-4278-bc82-5386105badd1/volumes" Nov 03 20:45:10.603392 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:45:10.603350 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:45:17.029021 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:17.028989 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de0676fc-bk5xz"] Nov 03 20:45:17.031871 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:17.031845 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de0676fc-bk5xz"] Nov 03 20:45:17.855641 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:17.855607 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75ed2a98-0045-44d3-ac8b-c9c32ea1b658" path="/var/lib/kubelet/pods/75ed2a98-0045-44d3-ac8b-c9c32ea1b658/volumes" Nov 03 20:45:19.352750 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:19.352718 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2" Nov 03 20:45:26.024445 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:26.024382 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70e9c51f-2g5tq"] Nov 03 20:45:26.027885 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:26.027853 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70e9c51f-2g5tq"] Nov 03 20:45:26.124683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:26.107419 5059 scope.go:122] "RemoveContainer" containerID="613590ff856112cf9b392125922ee05136abf2b331b26654dd09d54fa11ca2c7" Nov 03 20:45:26.197193 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:26.196816 5059 scope.go:122] "RemoveContainer" containerID="d69aefb2211ad25e5f23b2f7114fb99097dde9e8d402059bcfd5a7ae5c2dbc94" Nov 03 20:45:26.245701 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:26.245250 5059 scope.go:122] "RemoveContainer" containerID="5c408396e866fe620db1812eac6559783162f40c16b03f771dea78b617cb35bc" Nov 03 20:45:26.287143 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:26.287036 5059 scope.go:122] "RemoveContainer" containerID="41e6f3ac25965c349009bee82654ba6d7bfbd2293c37b04e17673cefb35a9a22" Nov 03 20:45:26.355811 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:26.355792 5059 scope.go:122] "RemoveContainer" containerID="7c2c89a7d43a8afbeca773d9e3555c337c87948f42cef32e61ef1975070cbc13" Nov 03 20:45:28.017572 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:28.017534 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7dae9ac-15d8-4751-87db-2ca6ed359f2e" path="/var/lib/kubelet/pods/c7dae9ac-15d8-4751-87db-2ca6ed359f2e/volumes" Nov 03 20:45:40.604296 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:45:40.604255 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:45:44.023217 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:44.023169 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14917091-d5vtz"] Nov 03 20:45:44.025795 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:44.025771 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14917091-d5vtz"] Nov 03 20:45:44.783523 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:44.783481 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt"] Nov 03 20:45:44.785839 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:44.785801 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2"] Nov 03 20:45:45.855731 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:45.855702 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="998d8105-98ae-451b-8d6f-639a2c61bf12" path="/var/lib/kubelet/pods/998d8105-98ae-451b-8d6f-639a2c61bf12/volumes" Nov 03 20:45:47.376686 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:47.376618 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-5sk9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:45:47.377213 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:47.376714 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" podUID="81fa410e-25d7-4800-a22b-9dea6ccca3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:45:49.353234 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:49.353189 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2" podUID="75a93f18-b77a-4379-8d7d-15a37a9d8094" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:45:59.366695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:59.365749 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2" podUID="75a93f18-b77a-4379-8d7d-15a37a9d8094" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:45:59.818331 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:59.816825 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" podUID="982df7da-1adf-46c9-b249-ec2e6b5b3e3b" containerName="main" containerID="cri-o://aab7bab9bf4cdfb73432f58d161a0cb5f87b9bca7b26b57844acddbdfbffdd86" gracePeriod=45 Nov 03 20:45:59.843967 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:45:59.843904 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2" podUID="75a93f18-b77a-4379-8d7d-15a37a9d8094" containerName="main" containerID="cri-o://a1fde4edddbaf8230512efe9ceeab87733a5e8f11b9a37aa1e36131bcd1468ee" gracePeriod=45 Nov 03 20:46:00.170782 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:00.170731 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cmtzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:46:00.171069 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:00.170966 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" podUID="a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:46:00.388308 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:00.388249 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" podUID="982df7da-1adf-46c9-b249-ec2e6b5b3e3b" containerName="main" probeResult="failure" output="Get \"https://10.132.1.199:8000/ready\": dial tcp 10.132.1.199:8000: connect: connection refused" Nov 03 20:46:00.812786 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:00.812538 5059 generic.go:356] "Generic (PLEG): container finished" podID="982df7da-1adf-46c9-b249-ec2e6b5b3e3b" containerID="aab7bab9bf4cdfb73432f58d161a0cb5f87b9bca7b26b57844acddbdfbffdd86" exitCode=0 Nov 03 20:46:00.812786 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:00.812721 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" event={"ID":"982df7da-1adf-46c9-b249-ec2e6b5b3e3b","Type":"ContainerDied","Data":"aab7bab9bf4cdfb73432f58d161a0cb5f87b9bca7b26b57844acddbdfbffdd86"} Nov 03 20:46:00.845774 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:00.845729 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:46:00.957307 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:00.957214 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-kube-api-access-x2mwv\" (UniqueName: \"kubernetes.io/projected/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-kube-api-access-x2mwv\") pod \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\" (UID: \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\") " Nov 03 20:46:00.957307 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:00.957259 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-tmp-dir\") pod \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\" (UID: \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\") " Nov 03 20:46:00.957307 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:00.957308 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-home\" (UniqueName: \"kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-home\") pod \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\" (UID: \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\") " Nov 03 20:46:00.957596 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:00.957364 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-tls-certs\" (UniqueName: \"kubernetes.io/secret/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-tls-certs\") pod \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\" (UID: \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\") " Nov 03 20:46:00.957596 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:00.957397 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-dshm\") pod \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\" (UID: \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\") " Nov 03 20:46:00.957596 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:00.957439 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-model-cache\") pod \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\" (UID: \"982df7da-1adf-46c9-b249-ec2e6b5b3e3b\") " Nov 03 20:46:00.957784 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:00.957610 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-tmp-dir" pod "982df7da-1adf-46c9-b249-ec2e6b5b3e3b" (UID: "982df7da-1adf-46c9-b249-ec2e6b5b3e3b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:00.957784 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:00.957642 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-home" pod "982df7da-1adf-46c9-b249-ec2e6b5b3e3b" (UID: "982df7da-1adf-46c9-b249-ec2e6b5b3e3b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:00.957901 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:00.957851 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:00.957901 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:00.957872 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:00.957901 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:00.957884 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-model-cache" pod "982df7da-1adf-46c9-b249-ec2e6b5b3e3b" (UID: "982df7da-1adf-46c9-b249-ec2e6b5b3e3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:00.960308 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:00.960269 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-dshm" pod "982df7da-1adf-46c9-b249-ec2e6b5b3e3b" (UID: "982df7da-1adf-46c9-b249-ec2e6b5b3e3b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:00.960734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:00.960711 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-kube-api-access-x2mwv" pod "982df7da-1adf-46c9-b249-ec2e6b5b3e3b" (UID: "982df7da-1adf-46c9-b249-ec2e6b5b3e3b"). InnerVolumeSpecName "kube-api-access-x2mwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:00.961270 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:00.961242 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-tls-certs" pod "982df7da-1adf-46c9-b249-ec2e6b5b3e3b" (UID: "982df7da-1adf-46c9-b249-ec2e6b5b3e3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:46:01.059121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:01.059074 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:01.059121 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:01.059115 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:01.059353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:01.059130 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:01.059353 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:01.059147 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2mwv\" (UniqueName: \"kubernetes.io/projected/982df7da-1adf-46c9-b249-ec2e6b5b3e3b-kube-api-access-x2mwv\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:01.818608 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:01.818575 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" Nov 03 20:46:01.819007 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:01.818621 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt" event={"ID":"982df7da-1adf-46c9-b249-ec2e6b5b3e3b","Type":"ContainerDied","Data":"dbc6c2335f70f9dfffd5806187ff79501251990bb6a1a092970cbdc0c0a16b97"} Nov 03 20:46:01.819007 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:01.818691 5059 scope.go:122] "RemoveContainer" containerID="aab7bab9bf4cdfb73432f58d161a0cb5f87b9bca7b26b57844acddbdfbffdd86" Nov 03 20:46:01.841795 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:01.841770 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt"] Nov 03 20:46:01.846049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:01.846026 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-f44f3e5c-kserve-675c5859f5-4lnmt"] Nov 03 20:46:01.855132 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:01.855108 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="982df7da-1adf-46c9-b249-ec2e6b5b3e3b" path="/var/lib/kubelet/pods/982df7da-1adf-46c9-b249-ec2e6b5b3e3b/volumes" Nov 03 20:46:09.352597 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:09.352557 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2" podUID="75a93f18-b77a-4379-8d7d-15a37a9d8094" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:46:09.353092 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:09.352687 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2" Nov 03 20:46:10.603851 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:46:10.603818 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:46:19.352993 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:19.352951 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2" podUID="75a93f18-b77a-4379-8d7d-15a37a9d8094" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:46:26.542491 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:26.542408 5059 scope.go:122] "RemoveContainer" containerID="73901e37b4ea8f0d28c4915f0e106f668be2fea2c0e7ad02397fa7f5ca74a3a0" Nov 03 20:46:26.551315 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:26.551293 5059 scope.go:122] "RemoveContainer" containerID="339b9db0f75cc3254388d71fefa4fc52189afbbc0916496dc7cc5536d5c39ccd" Nov 03 20:46:26.559037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:26.559012 5059 scope.go:122] "RemoveContainer" containerID="573561366a6d6b3b3f6054e0a9da378208d34e2a4ed82a68de8ff981c80e8f9e" Nov 03 20:46:26.577399 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:26.572563 5059 scope.go:122] "RemoveContainer" containerID="59cacebca1eec8a19ee9119d3b5833976b021ea541538d9597b0a7409ce592e0" Nov 03 20:46:26.602230 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:26.602195 5059 scope.go:122] "RemoveContainer" containerID="759eaa50200c464b2b367e9ff03ae386e5484c3102d75acd506810534e98bfb0" Nov 03 20:46:29.352618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:29.352578 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2" podUID="75a93f18-b77a-4379-8d7d-15a37a9d8094" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:46:30.206539 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:30.206511 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2" Nov 03 20:46:30.320763 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:30.320730 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75a93f18-b77a-4379-8d7d-15a37a9d8094-tls-certs\" (UniqueName: \"kubernetes.io/secret/75a93f18-b77a-4379-8d7d-15a37a9d8094-tls-certs\") pod \"75a93f18-b77a-4379-8d7d-15a37a9d8094\" (UID: \"75a93f18-b77a-4379-8d7d-15a37a9d8094\") " Nov 03 20:46:30.320940 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:30.320926 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a93f18-b77a-4379-8d7d-15a37a9d8094-kube-api-access-8z4gt\" (UniqueName: \"kubernetes.io/projected/75a93f18-b77a-4379-8d7d-15a37a9d8094-kube-api-access-8z4gt\") pod \"75a93f18-b77a-4379-8d7d-15a37a9d8094\" (UID: \"75a93f18-b77a-4379-8d7d-15a37a9d8094\") " Nov 03 20:46:30.322860 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:30.322828 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a93f18-b77a-4379-8d7d-15a37a9d8094-tls-certs" pod "75a93f18-b77a-4379-8d7d-15a37a9d8094" (UID: "75a93f18-b77a-4379-8d7d-15a37a9d8094"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:46:30.322974 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:30.322951 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a93f18-b77a-4379-8d7d-15a37a9d8094-kube-api-access-8z4gt" pod "75a93f18-b77a-4379-8d7d-15a37a9d8094" (UID: "75a93f18-b77a-4379-8d7d-15a37a9d8094"). InnerVolumeSpecName "kube-api-access-8z4gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:30.422521 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:30.422483 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z4gt\" (UniqueName: \"kubernetes.io/projected/75a93f18-b77a-4379-8d7d-15a37a9d8094-kube-api-access-8z4gt\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:30.422521 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:30.422517 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a93f18-b77a-4379-8d7d-15a37a9d8094-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:30.982303 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:30.982265 5059 generic.go:356] "Generic (PLEG): container finished" podID="75a93f18-b77a-4379-8d7d-15a37a9d8094" containerID="a1fde4edddbaf8230512efe9ceeab87733a5e8f11b9a37aa1e36131bcd1468ee" exitCode=0 Nov 03 20:46:30.982503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:30.982311 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2" event={"ID":"75a93f18-b77a-4379-8d7d-15a37a9d8094","Type":"ContainerDied","Data":"a1fde4edddbaf8230512efe9ceeab87733a5e8f11b9a37aa1e36131bcd1468ee"} Nov 03 20:46:30.982503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:30.982361 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2" event={"ID":"75a93f18-b77a-4379-8d7d-15a37a9d8094","Type":"ContainerDied","Data":"1bd2a7d4691dd7b4fc7212bf03a06619b63512c37a699c9e6d9b4ebd22e0bf78"} Nov 03 20:46:30.982503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:30.982372 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2" Nov 03 20:46:30.982503 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:30.982387 5059 scope.go:122] "RemoveContainer" containerID="a1fde4edddbaf8230512efe9ceeab87733a5e8f11b9a37aa1e36131bcd1468ee" Nov 03 20:46:30.993056 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:30.993035 5059 scope.go:122] "RemoveContainer" containerID="a1fde4edddbaf8230512efe9ceeab87733a5e8f11b9a37aa1e36131bcd1468ee" Nov 03 20:46:30.993325 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:46:30.993308 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1fde4edddbaf8230512efe9ceeab87733a5e8f11b9a37aa1e36131bcd1468ee\": container with ID starting with a1fde4edddbaf8230512efe9ceeab87733a5e8f11b9a37aa1e36131bcd1468ee not found: ID does not exist" containerID="a1fde4edddbaf8230512efe9ceeab87733a5e8f11b9a37aa1e36131bcd1468ee" Nov 03 20:46:30.993397 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:30.993334 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1fde4edddbaf8230512efe9ceeab87733a5e8f11b9a37aa1e36131bcd1468ee"} err="failed to get container status \"a1fde4edddbaf8230512efe9ceeab87733a5e8f11b9a37aa1e36131bcd1468ee\": rpc error: code = NotFound desc = could not find container \"a1fde4edddbaf8230512efe9ceeab87733a5e8f11b9a37aa1e36131bcd1468ee\": container with ID starting with a1fde4edddbaf8230512efe9ceeab87733a5e8f11b9a37aa1e36131bcd1468ee not found: ID does not exist" Nov 03 20:46:31.007392 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:31.007364 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2"] Nov 03 20:46:31.022742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:31.022717 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-f44f3e5c-kserve-router-scheduler-79b4db8898-fl6h2"] Nov 03 20:46:31.854801 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:31.854765 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a93f18-b77a-4379-8d7d-15a37a9d8094" path="/var/lib/kubelet/pods/75a93f18-b77a-4379-8d7d-15a37a9d8094/volumes" Nov 03 20:46:35.630200 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.630156 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk"] Nov 03 20:46:35.630788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.630757 5059 cpu_manager.go:414] "containerMap: removing container" podUID="982df7da-1adf-46c9-b249-ec2e6b5b3e3b" containerName="main" Nov 03 20:46:35.630788 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.630784 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="982df7da-1adf-46c9-b249-ec2e6b5b3e3b" containerName="main" Nov 03 20:46:35.630962 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.630796 5059 cpu_manager.go:414] "containerMap: removing container" podUID="75a93f18-b77a-4379-8d7d-15a37a9d8094" containerName="main" Nov 03 20:46:35.630962 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.630802 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a93f18-b77a-4379-8d7d-15a37a9d8094" containerName="main" Nov 03 20:46:35.630962 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.630881 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="982df7da-1adf-46c9-b249-ec2e6b5b3e3b" containerName="main" Nov 03 20:46:35.630962 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.630891 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a93f18-b77a-4379-8d7d-15a37a9d8094" containerName="main" Nov 03 20:46:35.635286 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.635267 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:46:35.644494 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.644467 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk"] Nov 03 20:46:35.645832 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.645807 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-e7bf5b\"/\"openshift-service-ca.crt\"" Nov 03 20:46:35.645951 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.645900 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-e7bf5b\"/\"default-dockercfg-ggxhb\"" Nov 03 20:46:35.646011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.645958 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-e7bf5b\"/\"kube-root-ca.crt\"" Nov 03 20:46:35.646135 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.646118 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-e7bf5b\"/\"simulator-single-main-e7bf5b-kserve-self-signed-certs\"" Nov 03 20:46:35.785812 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.785772 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-tmp-dir\") pod \"simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk\" (UID: \"336b0d67-69a8-4f67-9c43-6026420f378c\") " pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:46:35.786003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.785825 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/336b0d67-69a8-4f67-9c43-6026420f378c-tls-certs\") pod \"simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk\" (UID: \"336b0d67-69a8-4f67-9c43-6026420f378c\") " pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:46:35.786003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.785921 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-home\") pod \"simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk\" (UID: \"336b0d67-69a8-4f67-9c43-6026420f378c\") " pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:46:35.786003 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.785969 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-dshm\") pod \"simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk\" (UID: \"336b0d67-69a8-4f67-9c43-6026420f378c\") " pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:46:35.786123 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.786034 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-model-cache\") pod \"simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk\" (UID: \"336b0d67-69a8-4f67-9c43-6026420f378c\") " pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:46:35.786158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.786131 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g72p\" (UniqueName: \"kubernetes.io/projected/336b0d67-69a8-4f67-9c43-6026420f378c-kube-api-access-6g72p\") pod \"simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk\" (UID: \"336b0d67-69a8-4f67-9c43-6026420f378c\") " pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:46:35.886931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.886854 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g72p\" (UniqueName: \"kubernetes.io/projected/336b0d67-69a8-4f67-9c43-6026420f378c-kube-api-access-6g72p\") pod \"simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk\" (UID: \"336b0d67-69a8-4f67-9c43-6026420f378c\") " pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:46:35.887090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.886932 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-tmp-dir\") pod \"simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk\" (UID: \"336b0d67-69a8-4f67-9c43-6026420f378c\") " pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:46:35.887090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.886990 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/336b0d67-69a8-4f67-9c43-6026420f378c-tls-certs\") pod \"simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk\" (UID: \"336b0d67-69a8-4f67-9c43-6026420f378c\") " pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:46:35.887090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.887028 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-home\") pod \"simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk\" (UID: \"336b0d67-69a8-4f67-9c43-6026420f378c\") " pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:46:35.887090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.887052 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-dshm\") pod \"simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk\" (UID: \"336b0d67-69a8-4f67-9c43-6026420f378c\") " pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:46:35.887090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.887088 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-model-cache\") pod \"simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk\" (UID: \"336b0d67-69a8-4f67-9c43-6026420f378c\") " pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:46:35.887432 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.887404 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-tmp-dir\") pod \"simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk\" (UID: \"336b0d67-69a8-4f67-9c43-6026420f378c\") " pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:46:35.887511 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.887454 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-model-cache\") pod \"simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk\" (UID: \"336b0d67-69a8-4f67-9c43-6026420f378c\") " pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:46:35.887565 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.887533 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-home\") pod \"simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk\" (UID: \"336b0d67-69a8-4f67-9c43-6026420f378c\") " pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:46:35.889400 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.889375 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-dshm\") pod \"simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk\" (UID: \"336b0d67-69a8-4f67-9c43-6026420f378c\") " pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:46:35.889835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.889811 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/336b0d67-69a8-4f67-9c43-6026420f378c-tls-certs\") pod \"simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk\" (UID: \"336b0d67-69a8-4f67-9c43-6026420f378c\") " pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:46:35.908497 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.908464 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g72p\" (UniqueName: \"kubernetes.io/projected/336b0d67-69a8-4f67-9c43-6026420f378c-kube-api-access-6g72p\") pod \"simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk\" (UID: \"336b0d67-69a8-4f67-9c43-6026420f378c\") " pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:46:35.956152 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:35.956119 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:46:36.232458 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:36.232428 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk"] Nov 03 20:46:36.234637 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:46:36.234614 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod336b0d67_69a8_4f67_9c43_6026420f378c.slice/crio-536c6c5cf8bd26ea18a7928d401ad514434e17cf82f1a9f784fd48f3cd68c155 WatchSource:0}: Error finding container 536c6c5cf8bd26ea18a7928d401ad514434e17cf82f1a9f784fd48f3cd68c155: Status 404 returned error can't find the container with id 536c6c5cf8bd26ea18a7928d401ad514434e17cf82f1a9f784fd48f3cd68c155 Nov 03 20:46:37.014161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:37.014122 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" event={"ID":"336b0d67-69a8-4f67-9c43-6026420f378c","Type":"ContainerStarted","Data":"6c33181f593a34d5e283dce935dec29fbb8b7e0522662a73d2921a23dcba19d1"} Nov 03 20:46:37.014541 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:37.014183 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" event={"ID":"336b0d67-69a8-4f67-9c43-6026420f378c","Type":"ContainerStarted","Data":"536c6c5cf8bd26ea18a7928d401ad514434e17cf82f1a9f784fd48f3cd68c155"} Nov 03 20:46:37.035194 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:37.035143 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" podStartSLOduration=2.035129982 podStartE2EDuration="2.035129982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:37.03318442 +0000 UTC m=+4191.290981680" watchObservedRunningTime="2026-11-03 20:46:37.035129982 +0000 UTC m=+4191.292927260" Nov 03 20:46:37.956267 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:37.956228 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:46:40.603887 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:46:40.603850 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:46:45.956335 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:45.956292 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:46:45.969318 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:45.969290 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:46:46.067753 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:46:46.067719 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:47:05.388102 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:05.388059 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b5aa2f9-t7cch"] Nov 03 20:47:05.391610 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:05.391589 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5aa2f9-t7cch" Nov 03 20:47:05.397408 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:05.397369 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:47:05.398833 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:05.398648 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\"" Nov 03 20:47:05.400459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:05.400434 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b5aa2f9-t7cch"] Nov 03 20:47:05.468988 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:05.468963 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b5aa2f9-t7cch\" (UID: \"98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5aa2f9-t7cch" Nov 03 20:47:05.469128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:05.468994 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dplh\" (UniqueName: \"kubernetes.io/projected/98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0-kube-api-access-8dplh\") pod \"maas-api-revoke-sub-2b5aa2f9-t7cch\" (UID: \"98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5aa2f9-t7cch" Nov 03 20:47:05.469128 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:05.469030 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b5aa2f9-t7cch\" (UID: \"98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5aa2f9-t7cch" Nov 03 20:47:05.569996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:05.569966 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b5aa2f9-t7cch\" (UID: \"98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5aa2f9-t7cch" Nov 03 20:47:05.569996 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:05.569996 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dplh\" (UniqueName: \"kubernetes.io/projected/98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0-kube-api-access-8dplh\") pod \"maas-api-revoke-sub-2b5aa2f9-t7cch\" (UID: \"98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5aa2f9-t7cch" Nov 03 20:47:05.570158 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:05.570032 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b5aa2f9-t7cch\" (UID: \"98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5aa2f9-t7cch" Nov 03 20:47:05.570599 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:05.570578 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b5aa2f9-t7cch\" (UID: \"98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5aa2f9-t7cch" Nov 03 20:47:05.572631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:05.572609 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b5aa2f9-t7cch\" (UID: \"98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5aa2f9-t7cch" Nov 03 20:47:05.582276 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:05.582255 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dplh\" (UniqueName: \"kubernetes.io/projected/98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0-kube-api-access-8dplh\") pod \"maas-api-revoke-sub-2b5aa2f9-t7cch\" (UID: \"98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5aa2f9-t7cch" Nov 03 20:47:05.706716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:05.706630 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5aa2f9-t7cch" Nov 03 20:47:05.985769 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:05.985734 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b5aa2f9-t7cch"] Nov 03 20:47:06.148710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:06.148675 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5aa2f9-t7cch" event={"ID":"98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0","Type":"ContainerStarted","Data":"cbcfbf8dac93c1e46240a0c5e610ebce16705f8d44744951ba04b60a664a0088"} Nov 03 20:47:07.154763 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:07.154641 5059 generic.go:356] "Generic (PLEG): container finished" podID="98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0" containerID="5fd33926e9b5dfc72679d58737c0b5f86d0e63e270752aab7411b7e29b73c5f2" exitCode=0 Nov 03 20:47:07.154763 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:07.154712 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5aa2f9-t7cch" event={"ID":"98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0","Type":"ContainerDied","Data":"5fd33926e9b5dfc72679d58737c0b5f86d0e63e270752aab7411b7e29b73c5f2"} Nov 03 20:47:08.291382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:08.291359 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5aa2f9-t7cch" Nov 03 20:47:08.396586 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:08.396551 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0-maas-api-xks-ca\") pod \"98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0\" (UID: \"98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0\") " Nov 03 20:47:08.396778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:08.396619 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0-kube-api-access-8dplh\" (UniqueName: \"kubernetes.io/projected/98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0-kube-api-access-8dplh\") pod \"98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0\" (UID: \"98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0\") " Nov 03 20:47:08.396778 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:08.396682 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0-maas-api-openshift-ca\") pod \"98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0\" (UID: \"98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0\") " Nov 03 20:47:08.397062 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:08.397035 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0-maas-api-openshift-ca" pod "98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0" (UID: "98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:08.398764 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:08.398740 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0-maas-api-xks-ca" pod "98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0" (UID: "98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:47:08.398931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:08.398912 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0-kube-api-access-8dplh" pod "98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0" (UID: "98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0"). InnerVolumeSpecName "kube-api-access-8dplh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:08.497827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:08.497746 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:08.497827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:08.497779 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dplh\" (UniqueName: \"kubernetes.io/projected/98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0-kube-api-access-8dplh\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:08.497827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:08.497790 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:09.166835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:09.166802 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5aa2f9-t7cch" Nov 03 20:47:09.167046 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:09.166846 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5aa2f9-t7cch" event={"ID":"98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0","Type":"ContainerDied","Data":"cbcfbf8dac93c1e46240a0c5e610ebce16705f8d44744951ba04b60a664a0088"} Nov 03 20:47:09.167046 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:09.166899 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbcfbf8dac93c1e46240a0c5e610ebce16705f8d44744951ba04b60a664a0088" Nov 03 20:47:10.606681 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:47:10.606633 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:47:21.329283 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:21.329238 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk"] Nov 03 20:47:26.643850 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:26.643817 5059 scope.go:122] "RemoveContainer" containerID="d721492c32ba0aa19654d57c84d125b4b5495409ea04136336c07ae71e367369" Nov 03 20:47:36.349773 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:36.349729 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" podUID="336b0d67-69a8-4f67-9c43-6026420f378c" containerName="main" containerID="cri-o://6c33181f593a34d5e283dce935dec29fbb8b7e0522662a73d2921a23dcba19d1" gracePeriod=45 Nov 03 20:47:36.597817 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:36.597790 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:47:36.659931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:36.659842 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-home\" (UniqueName: \"kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-home\") pod \"336b0d67-69a8-4f67-9c43-6026420f378c\" (UID: \"336b0d67-69a8-4f67-9c43-6026420f378c\") " Nov 03 20:47:36.659931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:36.659885 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/336b0d67-69a8-4f67-9c43-6026420f378c-tls-certs\" (UniqueName: \"kubernetes.io/secret/336b0d67-69a8-4f67-9c43-6026420f378c-tls-certs\") pod \"336b0d67-69a8-4f67-9c43-6026420f378c\" (UID: \"336b0d67-69a8-4f67-9c43-6026420f378c\") " Nov 03 20:47:36.660130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:36.659992 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-tmp-dir\") pod \"336b0d67-69a8-4f67-9c43-6026420f378c\" (UID: \"336b0d67-69a8-4f67-9c43-6026420f378c\") " Nov 03 20:47:36.660130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:36.660030 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/336b0d67-69a8-4f67-9c43-6026420f378c-kube-api-access-6g72p\" (UniqueName: \"kubernetes.io/projected/336b0d67-69a8-4f67-9c43-6026420f378c-kube-api-access-6g72p\") pod \"336b0d67-69a8-4f67-9c43-6026420f378c\" (UID: \"336b0d67-69a8-4f67-9c43-6026420f378c\") " Nov 03 20:47:36.660130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:36.660064 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-dshm\") pod \"336b0d67-69a8-4f67-9c43-6026420f378c\" (UID: \"336b0d67-69a8-4f67-9c43-6026420f378c\") " Nov 03 20:47:36.660130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:36.660095 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-model-cache\") pod \"336b0d67-69a8-4f67-9c43-6026420f378c\" (UID: \"336b0d67-69a8-4f67-9c43-6026420f378c\") " Nov 03 20:47:36.660355 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:36.660117 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-home" pod "336b0d67-69a8-4f67-9c43-6026420f378c" (UID: "336b0d67-69a8-4f67-9c43-6026420f378c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:36.660355 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:36.660336 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-tmp-dir" pod "336b0d67-69a8-4f67-9c43-6026420f378c" (UID: "336b0d67-69a8-4f67-9c43-6026420f378c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:36.660466 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:36.660430 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-model-cache" pod "336b0d67-69a8-4f67-9c43-6026420f378c" (UID: "336b0d67-69a8-4f67-9c43-6026420f378c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:36.660510 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:36.660498 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:36.660555 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:36.660519 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:36.662358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:36.662324 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-dshm" pod "336b0d67-69a8-4f67-9c43-6026420f378c" (UID: "336b0d67-69a8-4f67-9c43-6026420f378c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:36.662486 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:36.662347 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336b0d67-69a8-4f67-9c43-6026420f378c-kube-api-access-6g72p" pod "336b0d67-69a8-4f67-9c43-6026420f378c" (UID: "336b0d67-69a8-4f67-9c43-6026420f378c"). InnerVolumeSpecName "kube-api-access-6g72p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:36.662486 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:36.662432 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/336b0d67-69a8-4f67-9c43-6026420f378c-tls-certs" pod "336b0d67-69a8-4f67-9c43-6026420f378c" (UID: "336b0d67-69a8-4f67-9c43-6026420f378c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:47:36.761143 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:36.761105 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g72p\" (UniqueName: \"kubernetes.io/projected/336b0d67-69a8-4f67-9c43-6026420f378c-kube-api-access-6g72p\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:36.761143 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:36.761138 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:36.761143 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:36.761149 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/336b0d67-69a8-4f67-9c43-6026420f378c-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:36.761367 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:36.761158 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/336b0d67-69a8-4f67-9c43-6026420f378c-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:37.289293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:37.289254 5059 generic.go:356] "Generic (PLEG): container finished" podID="336b0d67-69a8-4f67-9c43-6026420f378c" containerID="6c33181f593a34d5e283dce935dec29fbb8b7e0522662a73d2921a23dcba19d1" exitCode=0 Nov 03 20:47:37.289463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:37.289304 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" Nov 03 20:47:37.289463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:37.289346 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" event={"ID":"336b0d67-69a8-4f67-9c43-6026420f378c","Type":"ContainerDied","Data":"6c33181f593a34d5e283dce935dec29fbb8b7e0522662a73d2921a23dcba19d1"} Nov 03 20:47:37.289463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:37.289404 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk" event={"ID":"336b0d67-69a8-4f67-9c43-6026420f378c","Type":"ContainerDied","Data":"536c6c5cf8bd26ea18a7928d401ad514434e17cf82f1a9f784fd48f3cd68c155"} Nov 03 20:47:37.289463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:37.289427 5059 scope.go:122] "RemoveContainer" containerID="6c33181f593a34d5e283dce935dec29fbb8b7e0522662a73d2921a23dcba19d1" Nov 03 20:47:37.298269 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:37.298252 5059 scope.go:122] "RemoveContainer" containerID="6c33181f593a34d5e283dce935dec29fbb8b7e0522662a73d2921a23dcba19d1" Nov 03 20:47:37.298550 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:47:37.298532 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c33181f593a34d5e283dce935dec29fbb8b7e0522662a73d2921a23dcba19d1\": container with ID starting with 6c33181f593a34d5e283dce935dec29fbb8b7e0522662a73d2921a23dcba19d1 not found: ID does not exist" containerID="6c33181f593a34d5e283dce935dec29fbb8b7e0522662a73d2921a23dcba19d1" Nov 03 20:47:37.298617 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:37.298564 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c33181f593a34d5e283dce935dec29fbb8b7e0522662a73d2921a23dcba19d1"} err="failed to get container status \"6c33181f593a34d5e283dce935dec29fbb8b7e0522662a73d2921a23dcba19d1\": rpc error: code = NotFound desc = could not find container \"6c33181f593a34d5e283dce935dec29fbb8b7e0522662a73d2921a23dcba19d1\": container with ID starting with 6c33181f593a34d5e283dce935dec29fbb8b7e0522662a73d2921a23dcba19d1 not found: ID does not exist" Nov 03 20:47:37.312597 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:37.312571 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk"] Nov 03 20:47:37.318857 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:37.318832 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-e7bf5b/simulator-single-main-e7bf5b-kserve-79494b6cfc-bgxkk"] Nov 03 20:47:37.856019 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:37.855983 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="336b0d67-69a8-4f67-9c43-6026420f378c" path="/var/lib/kubelet/pods/336b0d67-69a8-4f67-9c43-6026420f378c/volumes" Nov 03 20:47:40.021872 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:40.021834 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czzww"] Nov 03 20:47:40.024360 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:40.024334 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czzww"] Nov 03 20:47:40.603759 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:47:40.603714 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:47:41.855773 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:41.855743 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54e3978e-1396-4270-8a8d-b2075ceadada" path="/var/lib/kubelet/pods/54e3978e-1396-4270-8a8d-b2075ceadada/volumes" Nov 03 20:47:55.910945 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:55.910857 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr"] Nov 03 20:47:55.912167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:55.911629 5059 cpu_manager.go:414] "containerMap: removing container" podUID="336b0d67-69a8-4f67-9c43-6026420f378c" containerName="main" Nov 03 20:47:55.912167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:55.911650 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="336b0d67-69a8-4f67-9c43-6026420f378c" containerName="main" Nov 03 20:47:55.912167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:55.911688 5059 cpu_manager.go:414] "containerMap: removing container" podUID="98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0" containerName="revoke-keys" Nov 03 20:47:55.912167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:55.911696 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0" containerName="revoke-keys" Nov 03 20:47:55.912167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:55.911798 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="336b0d67-69a8-4f67-9c43-6026420f378c" containerName="main" Nov 03 20:47:55.912167 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:55.911813 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0" containerName="revoke-keys" Nov 03 20:47:55.914820 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:55.914798 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds"] Nov 03 20:47:55.914950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:55.914912 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr" Nov 03 20:47:55.917952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:55.917924 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds" Nov 03 20:47:55.921716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:55.921695 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-e7bf5b\"" Nov 03 20:47:55.924646 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:55.924627 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr"] Nov 03 20:47:55.927468 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:55.927415 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-942q2\"" Nov 03 20:47:55.927883 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:55.927650 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-e7bf5b-serving-cert\"" Nov 03 20:47:55.930100 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:55.930079 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds"] Nov 03 20:47:55.935028 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:55.935000 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:47:56.023745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:56.023709 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3782b950-c027-46ee-8a4a-66ea3010adee-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds\" (UID: \"3782b950-c027-46ee-8a4a-66ea3010adee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds" Nov 03 20:47:56.023745 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:56.023748 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3782b950-c027-46ee-8a4a-66ea3010adee-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds\" (UID: \"3782b950-c027-46ee-8a4a-66ea3010adee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds" Nov 03 20:47:56.023948 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:56.023854 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgrvb\" (UniqueName: \"kubernetes.io/projected/3782b950-c027-46ee-8a4a-66ea3010adee-kube-api-access-rgrvb\") pod \"maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds\" (UID: \"3782b950-c027-46ee-8a4a-66ea3010adee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds" Nov 03 20:47:56.023948 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:56.023891 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa817d8-bacf-4fe6-96f7-c73157b4ca3a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr\" (UID: \"baa817d8-bacf-4fe6-96f7-c73157b4ca3a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr" Nov 03 20:47:56.024019 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:56.023964 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3782b950-c027-46ee-8a4a-66ea3010adee-metrics-certs\") pod \"maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds\" (UID: \"3782b950-c027-46ee-8a4a-66ea3010adee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds" Nov 03 20:47:56.024114 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:56.024094 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jc7n\" (UniqueName: \"kubernetes.io/projected/baa817d8-bacf-4fe6-96f7-c73157b4ca3a-kube-api-access-6jc7n\") pod \"payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr\" (UID: \"baa817d8-bacf-4fe6-96f7-c73157b4ca3a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr" Nov 03 20:47:56.125488 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:56.125445 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3782b950-c027-46ee-8a4a-66ea3010adee-metrics-certs\") pod \"maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds\" (UID: \"3782b950-c027-46ee-8a4a-66ea3010adee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds" Nov 03 20:47:56.125727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:56.125497 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jc7n\" (UniqueName: \"kubernetes.io/projected/baa817d8-bacf-4fe6-96f7-c73157b4ca3a-kube-api-access-6jc7n\") pod \"payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr\" (UID: \"baa817d8-bacf-4fe6-96f7-c73157b4ca3a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr" Nov 03 20:47:56.125727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:56.125552 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3782b950-c027-46ee-8a4a-66ea3010adee-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds\" (UID: \"3782b950-c027-46ee-8a4a-66ea3010adee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds" Nov 03 20:47:56.125727 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:56.125570 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3782b950-c027-46ee-8a4a-66ea3010adee-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds\" (UID: \"3782b950-c027-46ee-8a4a-66ea3010adee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds" Nov 03 20:47:56.125925 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:56.125721 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgrvb\" (UniqueName: \"kubernetes.io/projected/3782b950-c027-46ee-8a4a-66ea3010adee-kube-api-access-rgrvb\") pod \"maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds\" (UID: \"3782b950-c027-46ee-8a4a-66ea3010adee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds" Nov 03 20:47:56.125925 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:56.125759 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa817d8-bacf-4fe6-96f7-c73157b4ca3a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr\" (UID: \"baa817d8-bacf-4fe6-96f7-c73157b4ca3a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr" Nov 03 20:47:56.126473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:56.126444 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3782b950-c027-46ee-8a4a-66ea3010adee-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds\" (UID: \"3782b950-c027-46ee-8a4a-66ea3010adee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds" Nov 03 20:47:56.126617 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:56.126599 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa817d8-bacf-4fe6-96f7-c73157b4ca3a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr\" (UID: \"baa817d8-bacf-4fe6-96f7-c73157b4ca3a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr" Nov 03 20:47:56.128148 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:56.128124 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3782b950-c027-46ee-8a4a-66ea3010adee-metrics-certs\") pod \"maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds\" (UID: \"3782b950-c027-46ee-8a4a-66ea3010adee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds" Nov 03 20:47:56.128226 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:56.128207 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3782b950-c027-46ee-8a4a-66ea3010adee-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds\" (UID: \"3782b950-c027-46ee-8a4a-66ea3010adee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds" Nov 03 20:47:56.152708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:56.152677 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgrvb\" (UniqueName: \"kubernetes.io/projected/3782b950-c027-46ee-8a4a-66ea3010adee-kube-api-access-rgrvb\") pod \"maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds\" (UID: \"3782b950-c027-46ee-8a4a-66ea3010adee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds" Nov 03 20:47:56.153363 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:56.153340 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jc7n\" (UniqueName: \"kubernetes.io/projected/baa817d8-bacf-4fe6-96f7-c73157b4ca3a-kube-api-access-6jc7n\") pod \"payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr\" (UID: \"baa817d8-bacf-4fe6-96f7-c73157b4ca3a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr" Nov 03 20:47:56.233580 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:56.233490 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr" Nov 03 20:47:56.241487 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:56.241460 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds" Nov 03 20:47:56.511416 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:56.511388 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr"] Nov 03 20:47:56.513805 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:47:56.513774 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa817d8_bacf_4fe6_96f7_c73157b4ca3a.slice/crio-154460d3aba6b717113c209e33e19e4e1cc895386f89a3fe0364dbc46ae7cf0f WatchSource:0}: Error finding container 154460d3aba6b717113c209e33e19e4e1cc895386f89a3fe0364dbc46ae7cf0f: Status 404 returned error can't find the container with id 154460d3aba6b717113c209e33e19e4e1cc895386f89a3fe0364dbc46ae7cf0f Nov 03 20:47:56.528716 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:56.528685 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds"] Nov 03 20:47:56.532545 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:47:56.530814 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3782b950_c027_46ee_8a4a_66ea3010adee.slice/crio-290143ad88ddbd31ffa67dde6d1211d2461d0b28f765d1a5b933d008d1f8d283 WatchSource:0}: Error finding container 290143ad88ddbd31ffa67dde6d1211d2461d0b28f765d1a5b933d008d1f8d283: Status 404 returned error can't find the container with id 290143ad88ddbd31ffa67dde6d1211d2461d0b28f765d1a5b933d008d1f8d283 Nov 03 20:47:56.532707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:56.532571 5059 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:47:57.388011 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:57.387973 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr" event={"ID":"baa817d8-bacf-4fe6-96f7-c73157b4ca3a","Type":"ContainerStarted","Data":"2b1d4013e0604f32de32ac95e7ad722e4ddfb680bfed1b4f48dc15dce41038bf"} Nov 03 20:47:57.388473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:57.388044 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr" Nov 03 20:47:57.388473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:57.388068 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr" event={"ID":"baa817d8-bacf-4fe6-96f7-c73157b4ca3a","Type":"ContainerStarted","Data":"154460d3aba6b717113c209e33e19e4e1cc895386f89a3fe0364dbc46ae7cf0f"} Nov 03 20:47:57.389618 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:57.389590 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds" event={"ID":"3782b950-c027-46ee-8a4a-66ea3010adee","Type":"ContainerStarted","Data":"8a4428de5f17b81be964aa3a1ae21ca6f0a70bc64c6f0e202e5600f593dab3c3"} Nov 03 20:47:57.389751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:57.389632 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds" event={"ID":"3782b950-c027-46ee-8a4a-66ea3010adee","Type":"ContainerStarted","Data":"290143ad88ddbd31ffa67dde6d1211d2461d0b28f765d1a5b933d008d1f8d283"} Nov 03 20:47:57.389751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:57.389651 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds" Nov 03 20:47:57.406718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:57.406657 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr" podStartSLOduration=2.406643625 podStartE2EDuration="2.406643625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:57.405456366 +0000 UTC m=+4271.663253624" watchObservedRunningTime="2026-11-03 20:47:57.406643625 +0000 UTC m=+4271.664440882" Nov 03 20:47:57.432575 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:47:57.432519 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds" podStartSLOduration=1.919273185 podStartE2EDuration="2.432506196s" totalImagesPullingTime="513.233011ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:57.430951042 +0000 UTC m=+4271.688748301" watchObservedRunningTime="2026-11-03 20:47:57.432506196 +0000 UTC m=+4271.690303452" Nov 03 20:48:03.405186 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:03.405155 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds" Nov 03 20:48:08.402739 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:08.402704 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr" Nov 03 20:48:10.603371 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:48:10.603325 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:48:14.386617 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:14.386580 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr"] Nov 03 20:48:14.913217 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:14.913172 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds"] Nov 03 20:48:14.913442 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:14.913412 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds" podUID="3782b950-c027-46ee-8a4a-66ea3010adee" containerName="maas-api" containerID="cri-o://8a4428de5f17b81be964aa3a1ae21ca6f0a70bc64c6f0e202e5600f593dab3c3" gracePeriod=30 Nov 03 20:48:15.163896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:15.163823 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds" Nov 03 20:48:15.207367 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:15.206852 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3782b950-c027-46ee-8a4a-66ea3010adee-kube-api-access-rgrvb\" (UniqueName: \"kubernetes.io/projected/3782b950-c027-46ee-8a4a-66ea3010adee-kube-api-access-rgrvb\") pod \"3782b950-c027-46ee-8a4a-66ea3010adee\" (UID: \"3782b950-c027-46ee-8a4a-66ea3010adee\") " Nov 03 20:48:15.207367 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:15.206913 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3782b950-c027-46ee-8a4a-66ea3010adee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3782b950-c027-46ee-8a4a-66ea3010adee-maas-api-tls\") pod \"3782b950-c027-46ee-8a4a-66ea3010adee\" (UID: \"3782b950-c027-46ee-8a4a-66ea3010adee\") " Nov 03 20:48:15.207367 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:15.207033 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3782b950-c027-46ee-8a4a-66ea3010adee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3782b950-c027-46ee-8a4a-66ea3010adee-openshift-service-ca\") pod \"3782b950-c027-46ee-8a4a-66ea3010adee\" (UID: \"3782b950-c027-46ee-8a4a-66ea3010adee\") " Nov 03 20:48:15.207367 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:15.207081 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3782b950-c027-46ee-8a4a-66ea3010adee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3782b950-c027-46ee-8a4a-66ea3010adee-metrics-certs\") pod \"3782b950-c027-46ee-8a4a-66ea3010adee\" (UID: \"3782b950-c027-46ee-8a4a-66ea3010adee\") " Nov 03 20:48:15.209647 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:15.209600 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3782b950-c027-46ee-8a4a-66ea3010adee-openshift-service-ca" pod "3782b950-c027-46ee-8a4a-66ea3010adee" (UID: "3782b950-c027-46ee-8a4a-66ea3010adee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:48:15.220034 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:15.219983 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3782b950-c027-46ee-8a4a-66ea3010adee-metrics-certs" pod "3782b950-c027-46ee-8a4a-66ea3010adee" (UID: "3782b950-c027-46ee-8a4a-66ea3010adee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:48:15.222508 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:15.222460 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3782b950-c027-46ee-8a4a-66ea3010adee-kube-api-access-rgrvb" pod "3782b950-c027-46ee-8a4a-66ea3010adee" (UID: "3782b950-c027-46ee-8a4a-66ea3010adee"). InnerVolumeSpecName "kube-api-access-rgrvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:48:15.231366 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:15.231311 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3782b950-c027-46ee-8a4a-66ea3010adee-maas-api-tls" pod "3782b950-c027-46ee-8a4a-66ea3010adee" (UID: "3782b950-c027-46ee-8a4a-66ea3010adee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:48:15.309046 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:15.309009 5059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3782b950-c027-46ee-8a4a-66ea3010adee-metrics-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:15.309046 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:15.309042 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgrvb\" (UniqueName: \"kubernetes.io/projected/3782b950-c027-46ee-8a4a-66ea3010adee-kube-api-access-rgrvb\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:15.309257 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:15.309053 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3782b950-c027-46ee-8a4a-66ea3010adee-maas-api-tls\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:15.309257 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:15.309064 5059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3782b950-c027-46ee-8a4a-66ea3010adee-openshift-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:15.481398 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:15.481308 5059 generic.go:356] "Generic (PLEG): container finished" podID="3782b950-c027-46ee-8a4a-66ea3010adee" containerID="8a4428de5f17b81be964aa3a1ae21ca6f0a70bc64c6f0e202e5600f593dab3c3" exitCode=0 Nov 03 20:48:15.481398 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:15.481357 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds" Nov 03 20:48:15.481927 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:15.481403 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds" event={"ID":"3782b950-c027-46ee-8a4a-66ea3010adee","Type":"ContainerDied","Data":"8a4428de5f17b81be964aa3a1ae21ca6f0a70bc64c6f0e202e5600f593dab3c3"} Nov 03 20:48:15.481927 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:15.481467 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds" event={"ID":"3782b950-c027-46ee-8a4a-66ea3010adee","Type":"ContainerDied","Data":"290143ad88ddbd31ffa67dde6d1211d2461d0b28f765d1a5b933d008d1f8d283"} Nov 03 20:48:15.481927 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:15.481500 5059 scope.go:122] "RemoveContainer" containerID="8a4428de5f17b81be964aa3a1ae21ca6f0a70bc64c6f0e202e5600f593dab3c3" Nov 03 20:48:15.497499 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:15.497476 5059 scope.go:122] "RemoveContainer" containerID="8a4428de5f17b81be964aa3a1ae21ca6f0a70bc64c6f0e202e5600f593dab3c3" Nov 03 20:48:15.497754 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:48:15.497730 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a4428de5f17b81be964aa3a1ae21ca6f0a70bc64c6f0e202e5600f593dab3c3\": container with ID starting with 8a4428de5f17b81be964aa3a1ae21ca6f0a70bc64c6f0e202e5600f593dab3c3 not found: ID does not exist" containerID="8a4428de5f17b81be964aa3a1ae21ca6f0a70bc64c6f0e202e5600f593dab3c3" Nov 03 20:48:15.497844 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:15.497768 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a4428de5f17b81be964aa3a1ae21ca6f0a70bc64c6f0e202e5600f593dab3c3"} err="failed to get container status \"8a4428de5f17b81be964aa3a1ae21ca6f0a70bc64c6f0e202e5600f593dab3c3\": rpc error: code = NotFound desc = could not find container \"8a4428de5f17b81be964aa3a1ae21ca6f0a70bc64c6f0e202e5600f593dab3c3\": container with ID starting with 8a4428de5f17b81be964aa3a1ae21ca6f0a70bc64c6f0e202e5600f593dab3c3 not found: ID does not exist" Nov 03 20:48:15.508870 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:15.508685 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds"] Nov 03 20:48:15.511862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:15.511836 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-vvtds"] Nov 03 20:48:15.856442 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:15.856407 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3782b950-c027-46ee-8a4a-66ea3010adee" path="/var/lib/kubelet/pods/3782b950-c027-46ee-8a4a-66ea3010adee/volumes" Nov 03 20:48:18.026236 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:18.026200 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e821e4c8-z89bf"] Nov 03 20:48:18.029427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:18.029400 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e821e4c8-z89bf"] Nov 03 20:48:19.387334 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.387291 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr" podUID="baa817d8-bacf-4fe6-96f7-c73157b4ca3a" containerName="payload-processing" containerID="cri-o://2b1d4013e0604f32de32ac95e7ad722e4ddfb680bfed1b4f48dc15dce41038bf" gracePeriod=25 Nov 03 20:48:19.486707 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.486654 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn"] Nov 03 20:48:19.487133 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.487107 5059 cpu_manager.go:414] "containerMap: removing container" podUID="3782b950-c027-46ee-8a4a-66ea3010adee" containerName="maas-api" Nov 03 20:48:19.487253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.487137 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3782b950-c027-46ee-8a4a-66ea3010adee" containerName="maas-api" Nov 03 20:48:19.487253 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.487229 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3782b950-c027-46ee-8a4a-66ea3010adee" containerName="maas-api" Nov 03 20:48:19.490304 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.490280 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:48:19.503061 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.501854 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-2414e1ac-kserve-self-signed-certs\"" Nov 03 20:48:19.504855 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.504824 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn"] Nov 03 20:48:19.551046 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.551009 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjbg8\" (UniqueName: \"kubernetes.io/projected/423eb759-cffb-47a4-b071-1213e8f5acf6-kube-api-access-gjbg8\") pod \"e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn\" (UID: \"423eb759-cffb-47a4-b071-1213e8f5acf6\") " pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:48:19.551199 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.551081 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/423eb759-cffb-47a4-b071-1213e8f5acf6-tls-certs\") pod \"e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn\" (UID: \"423eb759-cffb-47a4-b071-1213e8f5acf6\") " pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:48:19.551199 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.551134 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-tmp-dir\") pod \"e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn\" (UID: \"423eb759-cffb-47a4-b071-1213e8f5acf6\") " pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:48:19.551290 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.551218 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-dshm\") pod \"e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn\" (UID: \"423eb759-cffb-47a4-b071-1213e8f5acf6\") " pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:48:19.551345 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.551307 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-model-cache\") pod \"e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn\" (UID: \"423eb759-cffb-47a4-b071-1213e8f5acf6\") " pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:48:19.551414 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.551387 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-home\") pod \"e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn\" (UID: \"423eb759-cffb-47a4-b071-1213e8f5acf6\") " pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:48:19.658780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.657769 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-home\") pod \"e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn\" (UID: \"423eb759-cffb-47a4-b071-1213e8f5acf6\") " pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:48:19.658780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.657888 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjbg8\" (UniqueName: \"kubernetes.io/projected/423eb759-cffb-47a4-b071-1213e8f5acf6-kube-api-access-gjbg8\") pod \"e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn\" (UID: \"423eb759-cffb-47a4-b071-1213e8f5acf6\") " pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:48:19.658780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.657955 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/423eb759-cffb-47a4-b071-1213e8f5acf6-tls-certs\") pod \"e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn\" (UID: \"423eb759-cffb-47a4-b071-1213e8f5acf6\") " pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:48:19.658780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.658000 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-tmp-dir\") pod \"e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn\" (UID: \"423eb759-cffb-47a4-b071-1213e8f5acf6\") " pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:48:19.658780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.658043 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-dshm\") pod \"e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn\" (UID: \"423eb759-cffb-47a4-b071-1213e8f5acf6\") " pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:48:19.658780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.658142 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-model-cache\") pod \"e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn\" (UID: \"423eb759-cffb-47a4-b071-1213e8f5acf6\") " pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:48:19.658780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.658644 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-model-cache\") pod \"e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn\" (UID: \"423eb759-cffb-47a4-b071-1213e8f5acf6\") " pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:48:19.658780 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.658733 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-home\") pod \"e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn\" (UID: \"423eb759-cffb-47a4-b071-1213e8f5acf6\") " pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:48:19.665020 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.659827 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-tmp-dir\") pod \"e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn\" (UID: \"423eb759-cffb-47a4-b071-1213e8f5acf6\") " pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:48:19.675221 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.675161 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-dshm\") pod \"e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn\" (UID: \"423eb759-cffb-47a4-b071-1213e8f5acf6\") " pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:48:19.676229 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.676199 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/423eb759-cffb-47a4-b071-1213e8f5acf6-tls-certs\") pod \"e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn\" (UID: \"423eb759-cffb-47a4-b071-1213e8f5acf6\") " pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:48:19.676888 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.676867 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjbg8\" (UniqueName: \"kubernetes.io/projected/423eb759-cffb-47a4-b071-1213e8f5acf6-kube-api-access-gjbg8\") pod \"e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn\" (UID: \"423eb759-cffb-47a4-b071-1213e8f5acf6\") " pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:48:19.678859 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.678836 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr" Nov 03 20:48:19.758444 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.758416 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baa817d8-bacf-4fe6-96f7-c73157b4ca3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa817d8-bacf-4fe6-96f7-c73157b4ca3a-plugins-config-volume\") pod \"baa817d8-bacf-4fe6-96f7-c73157b4ca3a\" (UID: \"baa817d8-bacf-4fe6-96f7-c73157b4ca3a\") " Nov 03 20:48:19.758603 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.758590 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baa817d8-bacf-4fe6-96f7-c73157b4ca3a-kube-api-access-6jc7n\" (UniqueName: \"kubernetes.io/projected/baa817d8-bacf-4fe6-96f7-c73157b4ca3a-kube-api-access-6jc7n\") pod \"baa817d8-bacf-4fe6-96f7-c73157b4ca3a\" (UID: \"baa817d8-bacf-4fe6-96f7-c73157b4ca3a\") " Nov 03 20:48:19.758805 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.758783 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baa817d8-bacf-4fe6-96f7-c73157b4ca3a-plugins-config-volume" pod "baa817d8-bacf-4fe6-96f7-c73157b4ca3a" (UID: "baa817d8-bacf-4fe6-96f7-c73157b4ca3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:48:19.760495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.760469 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa817d8-bacf-4fe6-96f7-c73157b4ca3a-kube-api-access-6jc7n" pod "baa817d8-bacf-4fe6-96f7-c73157b4ca3a" (UID: "baa817d8-bacf-4fe6-96f7-c73157b4ca3a"). InnerVolumeSpecName "kube-api-access-6jc7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:48:19.816918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.816896 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:48:19.857695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.855986 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="056c992a-b13c-404b-b437-9d9cbf7c0ebe" path="/var/lib/kubelet/pods/056c992a-b13c-404b-b437-9d9cbf7c0ebe/volumes" Nov 03 20:48:19.860164 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.860135 5059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa817d8-bacf-4fe6-96f7-c73157b4ca3a-plugins-config-volume\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:19.860164 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:19.860168 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jc7n\" (UniqueName: \"kubernetes.io/projected/baa817d8-bacf-4fe6-96f7-c73157b4ca3a-kube-api-access-6jc7n\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:20.093030 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:20.092959 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn"] Nov 03 20:48:20.095434 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:48:20.095408 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod423eb759_cffb_47a4_b071_1213e8f5acf6.slice/crio-862ac2ff4a962e16032bdf8946dc0ee356b4bdf5b9dddb1572ccc101d71698ef WatchSource:0}: Error finding container 862ac2ff4a962e16032bdf8946dc0ee356b4bdf5b9dddb1572ccc101d71698ef: Status 404 returned error can't find the container with id 862ac2ff4a962e16032bdf8946dc0ee356b4bdf5b9dddb1572ccc101d71698ef Nov 03 20:48:20.507719 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:20.507684 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" event={"ID":"423eb759-cffb-47a4-b071-1213e8f5acf6","Type":"ContainerStarted","Data":"16b572e66951ee0200b9d0e3adcde83c8d277043aabafba9b2e34f45244ed64d"} Nov 03 20:48:20.508160 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:20.507733 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" event={"ID":"423eb759-cffb-47a4-b071-1213e8f5acf6","Type":"ContainerStarted","Data":"862ac2ff4a962e16032bdf8946dc0ee356b4bdf5b9dddb1572ccc101d71698ef"} Nov 03 20:48:20.508911 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:20.508885 5059 generic.go:356] "Generic (PLEG): container finished" podID="baa817d8-bacf-4fe6-96f7-c73157b4ca3a" containerID="2b1d4013e0604f32de32ac95e7ad722e4ddfb680bfed1b4f48dc15dce41038bf" exitCode=0 Nov 03 20:48:20.508978 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:20.508922 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr" event={"ID":"baa817d8-bacf-4fe6-96f7-c73157b4ca3a","Type":"ContainerDied","Data":"2b1d4013e0604f32de32ac95e7ad722e4ddfb680bfed1b4f48dc15dce41038bf"} Nov 03 20:48:20.508978 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:20.508927 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr" Nov 03 20:48:20.508978 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:20.508959 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr" event={"ID":"baa817d8-bacf-4fe6-96f7-c73157b4ca3a","Type":"ContainerDied","Data":"154460d3aba6b717113c209e33e19e4e1cc895386f89a3fe0364dbc46ae7cf0f"} Nov 03 20:48:20.509086 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:20.508981 5059 scope.go:122] "RemoveContainer" containerID="2b1d4013e0604f32de32ac95e7ad722e4ddfb680bfed1b4f48dc15dce41038bf" Nov 03 20:48:20.517450 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:20.517420 5059 scope.go:122] "RemoveContainer" containerID="2b1d4013e0604f32de32ac95e7ad722e4ddfb680bfed1b4f48dc15dce41038bf" Nov 03 20:48:20.517723 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:48:20.517701 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b1d4013e0604f32de32ac95e7ad722e4ddfb680bfed1b4f48dc15dce41038bf\": container with ID starting with 2b1d4013e0604f32de32ac95e7ad722e4ddfb680bfed1b4f48dc15dce41038bf not found: ID does not exist" containerID="2b1d4013e0604f32de32ac95e7ad722e4ddfb680bfed1b4f48dc15dce41038bf" Nov 03 20:48:20.517789 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:20.517733 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b1d4013e0604f32de32ac95e7ad722e4ddfb680bfed1b4f48dc15dce41038bf"} err="failed to get container status \"2b1d4013e0604f32de32ac95e7ad722e4ddfb680bfed1b4f48dc15dce41038bf\": rpc error: code = NotFound desc = could not find container \"2b1d4013e0604f32de32ac95e7ad722e4ddfb680bfed1b4f48dc15dce41038bf\": container with ID starting with 2b1d4013e0604f32de32ac95e7ad722e4ddfb680bfed1b4f48dc15dce41038bf not found: ID does not exist" Nov 03 20:48:20.528310 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:20.528272 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" podStartSLOduration=1.528259971 podStartE2EDuration="1.528259971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:20.526358398 +0000 UTC m=+4294.784155681" watchObservedRunningTime="2026-11-03 20:48:20.528259971 +0000 UTC m=+4294.786057227" Nov 03 20:48:20.540360 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:20.540337 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr"] Nov 03 20:48:20.544277 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:20.544255 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d747fkdr"] Nov 03 20:48:20.817870 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:20.817797 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:48:21.859609 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:21.859570 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baa817d8-bacf-4fe6-96f7-c73157b4ca3a" path="/var/lib/kubelet/pods/baa817d8-bacf-4fe6-96f7-c73157b4ca3a/volumes" Nov 03 20:48:25.277862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:25.277821 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8gzjz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:48:25.278398 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:25.277913 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" podUID="6462b9bb-1018-4029-922d-eb71527fc301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:48:29.817544 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:29.817511 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:48:29.829837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:29.829813 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:48:30.581370 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:30.581339 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:48:34.022776 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:34.022738 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-2998j"] Nov 03 20:48:34.025463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:34.025437 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-2998j"] Nov 03 20:48:35.855985 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:35.855947 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="844f167a-92da-4121-a0c3-6d99f4ee8c23" path="/var/lib/kubelet/pods/844f167a-92da-4121-a0c3-6d99f4ee8c23/volumes" Nov 03 20:48:36.594224 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.594187 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47"] Nov 03 20:48:36.594566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.594550 5059 cpu_manager.go:414] "containerMap: removing container" podUID="baa817d8-bacf-4fe6-96f7-c73157b4ca3a" containerName="payload-processing" Nov 03 20:48:36.594649 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.594567 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baa817d8-bacf-4fe6-96f7-c73157b4ca3a" containerName="payload-processing" Nov 03 20:48:36.594649 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.594647 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="baa817d8-bacf-4fe6-96f7-c73157b4ca3a" containerName="payload-processing" Nov 03 20:48:36.599426 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.599398 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:48:36.614480 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.614448 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-2414e1ac-kserve-self-signed-certs\"" Nov 03 20:48:36.617263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.617234 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47"] Nov 03 20:48:36.627907 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.627878 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-model-cache\") pod \"e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47\" (UID: \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\") " pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:48:36.628022 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.627937 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-dshm\") pod \"e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47\" (UID: \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\") " pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:48:36.628073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.628023 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-home\") pod \"e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47\" (UID: \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\") " pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:48:36.628073 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.628065 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-tmp-dir\") pod \"e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47\" (UID: \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\") " pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:48:36.628147 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.628122 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf25416d-a4d5-402e-a40d-d3b5f84bb362-tls-certs\") pod \"e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47\" (UID: \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\") " pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:48:36.628190 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.628159 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6ptn\" (UniqueName: \"kubernetes.io/projected/cf25416d-a4d5-402e-a40d-d3b5f84bb362-kube-api-access-f6ptn\") pod \"e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47\" (UID: \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\") " pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:48:36.729103 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.729069 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-home\") pod \"e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47\" (UID: \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\") " pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:48:36.729263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.729118 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-tmp-dir\") pod \"e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47\" (UID: \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\") " pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:48:36.729263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.729164 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf25416d-a4d5-402e-a40d-d3b5f84bb362-tls-certs\") pod \"e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47\" (UID: \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\") " pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:48:36.729263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.729196 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6ptn\" (UniqueName: \"kubernetes.io/projected/cf25416d-a4d5-402e-a40d-d3b5f84bb362-kube-api-access-f6ptn\") pod \"e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47\" (UID: \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\") " pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:48:36.729263 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.729227 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-model-cache\") pod \"e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47\" (UID: \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\") " pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:48:36.729462 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.729354 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-dshm\") pod \"e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47\" (UID: \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\") " pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:48:36.729546 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.729523 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-home\") pod \"e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47\" (UID: \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\") " pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:48:36.729623 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.729601 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-tmp-dir\") pod \"e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47\" (UID: \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\") " pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:48:36.729718 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.729647 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-model-cache\") pod \"e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47\" (UID: \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\") " pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:48:36.732090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.732064 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf25416d-a4d5-402e-a40d-d3b5f84bb362-tls-certs\") pod \"e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47\" (UID: \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\") " pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:48:36.732765 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.732742 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-dshm\") pod \"e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47\" (UID: \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\") " pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:48:36.761593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.761553 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6ptn\" (UniqueName: \"kubernetes.io/projected/cf25416d-a4d5-402e-a40d-d3b5f84bb362-kube-api-access-f6ptn\") pod \"e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47\" (UID: \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\") " pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:48:36.920702 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:36.920599 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:48:37.195924 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:37.195894 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47"] Nov 03 20:48:37.197589 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:48:37.197562 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf25416d_a4d5_402e_a40d_d3b5f84bb362.slice/crio-e3832214496fafa3803c8702172084f567276f2cbc00e7ceba73bf5e2b8a7692 WatchSource:0}: Error finding container e3832214496fafa3803c8702172084f567276f2cbc00e7ceba73bf5e2b8a7692: Status 404 returned error can't find the container with id e3832214496fafa3803c8702172084f567276f2cbc00e7ceba73bf5e2b8a7692 Nov 03 20:48:37.605341 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:37.605308 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" event={"ID":"cf25416d-a4d5-402e-a40d-d3b5f84bb362","Type":"ContainerStarted","Data":"2fb168a9f12a62d66f664c703b9d3761f474f7171240ac0382924895a06ff1df"} Nov 03 20:48:37.605534 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:37.605351 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" event={"ID":"cf25416d-a4d5-402e-a40d-d3b5f84bb362","Type":"ContainerStarted","Data":"e3832214496fafa3803c8702172084f567276f2cbc00e7ceba73bf5e2b8a7692"} Nov 03 20:48:37.626037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:37.625993 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" podStartSLOduration=1.625980095 podStartE2EDuration="1.625980095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:37.623885226 +0000 UTC m=+4311.881682497" watchObservedRunningTime="2026-11-03 20:48:37.625980095 +0000 UTC m=+4311.883777379" Nov 03 20:48:37.920988 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:37.920912 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:48:40.720325 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:48:40.720246 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:48:46.920740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:46.920699 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:48:46.933355 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:46.933323 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:48:47.724291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:47.724257 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:48:56.022798 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:56.022760 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f9lpd"] Nov 03 20:48:56.025380 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:56.025353 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-f9lpd"] Nov 03 20:48:57.855220 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:48:57.855177 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="162b6a47-7cf0-4da2-9fde-fb272ab94617" path="/var/lib/kubelet/pods/162b6a47-7cf0-4da2-9fde-fb272ab94617/volumes" Nov 03 20:49:10.603556 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:49:10.603517 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:49:14.339076 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:14.339038 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn"] Nov 03 20:49:15.778528 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:15.778491 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47"] Nov 03 20:49:23.033585 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:23.033379 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h2qs2"] Nov 03 20:49:23.036746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:23.036718 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-h2qs2"] Nov 03 20:49:23.855535 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:23.855496 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19" path="/var/lib/kubelet/pods/0ae6cda1-4bc8-46b6-af16-6ad7cbb8cf19/volumes" Nov 03 20:49:26.686459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:26.686431 5059 scope.go:122] "RemoveContainer" containerID="81ce95d4024846914da7f3797c4d0ea9d48be58eaed69dc4c17d4843483de52e" Nov 03 20:49:26.694547 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:26.694526 5059 scope.go:122] "RemoveContainer" containerID="0251afacbeae9bf1baa37258c48a970396374ddbbf7fcb5e8c880e8fbdfbd599" Nov 03 20:49:29.375130 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:29.375071 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" podUID="423eb759-cffb-47a4-b071-1213e8f5acf6" containerName="main" containerID="cri-o://16b572e66951ee0200b9d0e3adcde83c8d277043aabafba9b2e34f45244ed64d" gracePeriod=45 Nov 03 20:49:29.574393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:29.574034 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" podUID="423eb759-cffb-47a4-b071-1213e8f5acf6" containerName="main" probeResult="failure" output="Get \"https://10.132.1.205:8000/ready\": dial tcp 10.132.1.205:8000: connect: connection refused" Nov 03 20:49:30.106023 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.105972 5059 generic.go:356] "Generic (PLEG): container finished" podID="423eb759-cffb-47a4-b071-1213e8f5acf6" containerID="16b572e66951ee0200b9d0e3adcde83c8d277043aabafba9b2e34f45244ed64d" exitCode=0 Nov 03 20:49:30.107393 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.106260 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" event={"ID":"423eb759-cffb-47a4-b071-1213e8f5acf6","Type":"ContainerDied","Data":"16b572e66951ee0200b9d0e3adcde83c8d277043aabafba9b2e34f45244ed64d"} Nov 03 20:49:30.161808 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.161754 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cmtzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:49:30.162075 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.162029 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" podUID="a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:49:30.346683 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.329682 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:49:30.439999 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.439960 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/423eb759-cffb-47a4-b071-1213e8f5acf6-kube-api-access-gjbg8\" (UniqueName: \"kubernetes.io/projected/423eb759-cffb-47a4-b071-1213e8f5acf6-kube-api-access-gjbg8\") pod \"423eb759-cffb-47a4-b071-1213e8f5acf6\" (UID: \"423eb759-cffb-47a4-b071-1213e8f5acf6\") " Nov 03 20:49:30.440598 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.440570 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-model-cache\") pod \"423eb759-cffb-47a4-b071-1213e8f5acf6\" (UID: \"423eb759-cffb-47a4-b071-1213e8f5acf6\") " Nov 03 20:49:30.444901 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.444875 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-dshm\") pod \"423eb759-cffb-47a4-b071-1213e8f5acf6\" (UID: \"423eb759-cffb-47a4-b071-1213e8f5acf6\") " Nov 03 20:49:30.445071 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.445051 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-tmp-dir\") pod \"423eb759-cffb-47a4-b071-1213e8f5acf6\" (UID: \"423eb759-cffb-47a4-b071-1213e8f5acf6\") " Nov 03 20:49:30.456923 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.456878 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-home\" (UniqueName: \"kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-home\") pod \"423eb759-cffb-47a4-b071-1213e8f5acf6\" (UID: \"423eb759-cffb-47a4-b071-1213e8f5acf6\") " Nov 03 20:49:30.457117 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.457099 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/423eb759-cffb-47a4-b071-1213e8f5acf6-tls-certs\" (UniqueName: \"kubernetes.io/secret/423eb759-cffb-47a4-b071-1213e8f5acf6-tls-certs\") pod \"423eb759-cffb-47a4-b071-1213e8f5acf6\" (UID: \"423eb759-cffb-47a4-b071-1213e8f5acf6\") " Nov 03 20:49:30.468820 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.468795 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-home" pod "423eb759-cffb-47a4-b071-1213e8f5acf6" (UID: "423eb759-cffb-47a4-b071-1213e8f5acf6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:49:30.486947 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.486912 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-model-cache" pod "423eb759-cffb-47a4-b071-1213e8f5acf6" (UID: "423eb759-cffb-47a4-b071-1213e8f5acf6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:49:30.487740 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.487704 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-tmp-dir" pod "423eb759-cffb-47a4-b071-1213e8f5acf6" (UID: "423eb759-cffb-47a4-b071-1213e8f5acf6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:49:30.538733 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.531509 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-dshm" pod "423eb759-cffb-47a4-b071-1213e8f5acf6" (UID: "423eb759-cffb-47a4-b071-1213e8f5acf6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:49:30.581615 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.537759 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/423eb759-cffb-47a4-b071-1213e8f5acf6-tls-certs" pod "423eb759-cffb-47a4-b071-1213e8f5acf6" (UID: "423eb759-cffb-47a4-b071-1213e8f5acf6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:49:30.581615 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.560613 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:30.581837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.581645 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:30.581837 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.581681 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:30.582441 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.582400 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423eb759-cffb-47a4-b071-1213e8f5acf6-kube-api-access-gjbg8" pod "423eb759-cffb-47a4-b071-1213e8f5acf6" (UID: "423eb759-cffb-47a4-b071-1213e8f5acf6"). InnerVolumeSpecName "kube-api-access-gjbg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:49:30.685308 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.685270 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/423eb759-cffb-47a4-b071-1213e8f5acf6-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:30.685568 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.685546 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjbg8\" (UniqueName: \"kubernetes.io/projected/423eb759-cffb-47a4-b071-1213e8f5acf6-kube-api-access-gjbg8\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:30.685724 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.685705 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/423eb759-cffb-47a4-b071-1213e8f5acf6-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:30.922698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.919046 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" podUID="cf25416d-a4d5-402e-a40d-d3b5f84bb362" containerName="main" containerID="cri-o://2fb168a9f12a62d66f664c703b9d3761f474f7171240ac0382924895a06ff1df" gracePeriod=45 Nov 03 20:49:30.995475 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:30.995432 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" podUID="cf25416d-a4d5-402e-a40d-d3b5f84bb362" containerName="main" probeResult="failure" output="Get \"https://10.132.1.206:8000/ready\": EOF" Nov 03 20:49:31.134801 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.134764 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" event={"ID":"423eb759-cffb-47a4-b071-1213e8f5acf6","Type":"ContainerDied","Data":"862ac2ff4a962e16032bdf8946dc0ee356b4bdf5b9dddb1572ccc101d71698ef"} Nov 03 20:49:31.135010 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.134996 5059 scope.go:122] "RemoveContainer" containerID="16b572e66951ee0200b9d0e3adcde83c8d277043aabafba9b2e34f45244ed64d" Nov 03 20:49:31.135254 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.135241 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn" Nov 03 20:49:31.153875 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.153712 5059 generic.go:356] "Generic (PLEG): container finished" podID="cf25416d-a4d5-402e-a40d-d3b5f84bb362" containerID="2fb168a9f12a62d66f664c703b9d3761f474f7171240ac0382924895a06ff1df" exitCode=0 Nov 03 20:49:31.153875 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.153818 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" event={"ID":"cf25416d-a4d5-402e-a40d-d3b5f84bb362","Type":"ContainerDied","Data":"2fb168a9f12a62d66f664c703b9d3761f474f7171240ac0382924895a06ff1df"} Nov 03 20:49:31.243684 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.239542 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn"] Nov 03 20:49:31.270695 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.247221 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-2414e1ac-kserve-5549978c7-qdrcn"] Nov 03 20:49:31.300689 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:49:31.287953 5059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf25416d_a4d5_402e_a40d_d3b5f84bb362.slice/crio-2fb168a9f12a62d66f664c703b9d3761f474f7171240ac0382924895a06ff1df.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:49:31.585291 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.585251 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:49:31.648451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.641889 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-home\" (UniqueName: \"kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-home\") pod \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\" (UID: \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\") " Nov 03 20:49:31.648451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.641952 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-model-cache\") pod \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\" (UID: \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\") " Nov 03 20:49:31.648451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.642041 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf25416d-a4d5-402e-a40d-d3b5f84bb362-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf25416d-a4d5-402e-a40d-d3b5f84bb362-tls-certs\") pod \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\" (UID: \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\") " Nov 03 20:49:31.648451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.642104 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf25416d-a4d5-402e-a40d-d3b5f84bb362-kube-api-access-f6ptn\" (UniqueName: \"kubernetes.io/projected/cf25416d-a4d5-402e-a40d-d3b5f84bb362-kube-api-access-f6ptn\") pod \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\" (UID: \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\") " Nov 03 20:49:31.648451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.642219 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-dshm\") pod \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\" (UID: \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\") " Nov 03 20:49:31.648451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.642336 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-tmp-dir\") pod \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\" (UID: \"cf25416d-a4d5-402e-a40d-d3b5f84bb362\") " Nov 03 20:49:31.648451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.643110 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-tmp-dir" pod "cf25416d-a4d5-402e-a40d-d3b5f84bb362" (UID: "cf25416d-a4d5-402e-a40d-d3b5f84bb362"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:49:31.648451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.643227 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-home" pod "cf25416d-a4d5-402e-a40d-d3b5f84bb362" (UID: "cf25416d-a4d5-402e-a40d-d3b5f84bb362"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:49:31.648451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.643729 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-model-cache" pod "cf25416d-a4d5-402e-a40d-d3b5f84bb362" (UID: "cf25416d-a4d5-402e-a40d-d3b5f84bb362"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:49:31.648451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.647794 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf25416d-a4d5-402e-a40d-d3b5f84bb362-kube-api-access-f6ptn" pod "cf25416d-a4d5-402e-a40d-d3b5f84bb362" (UID: "cf25416d-a4d5-402e-a40d-d3b5f84bb362"). InnerVolumeSpecName "kube-api-access-f6ptn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:49:31.664599 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.656220 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-dshm" pod "cf25416d-a4d5-402e-a40d-d3b5f84bb362" (UID: "cf25416d-a4d5-402e-a40d-d3b5f84bb362"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:49:31.664599 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.656625 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf25416d-a4d5-402e-a40d-d3b5f84bb362-tls-certs" pod "cf25416d-a4d5-402e-a40d-d3b5f84bb362" (UID: "cf25416d-a4d5-402e-a40d-d3b5f84bb362"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:49:31.744150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.744109 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf25416d-a4d5-402e-a40d-d3b5f84bb362-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:31.744150 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.744145 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6ptn\" (UniqueName: \"kubernetes.io/projected/cf25416d-a4d5-402e-a40d-d3b5f84bb362-kube-api-access-f6ptn\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:31.744370 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.744162 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:31.744370 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.744176 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:31.744370 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.744190 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:31.744370 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.744202 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf25416d-a4d5-402e-a40d-d3b5f84bb362-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:31.858710 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:31.858646 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="423eb759-cffb-47a4-b071-1213e8f5acf6" path="/var/lib/kubelet/pods/423eb759-cffb-47a4-b071-1213e8f5acf6/volumes" Nov 03 20:49:32.163139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:32.163039 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" event={"ID":"cf25416d-a4d5-402e-a40d-d3b5f84bb362","Type":"ContainerDied","Data":"e3832214496fafa3803c8702172084f567276f2cbc00e7ceba73bf5e2b8a7692"} Nov 03 20:49:32.163139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:32.163070 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47" Nov 03 20:49:32.163139 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:32.163104 5059 scope.go:122] "RemoveContainer" containerID="2fb168a9f12a62d66f664c703b9d3761f474f7171240ac0382924895a06ff1df" Nov 03 20:49:32.183984 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:32.183950 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47"] Nov 03 20:49:32.189767 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:32.189738 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-2414e1ac-kserve-599dd54969-zwb47"] Nov 03 20:49:33.871685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:33.870878 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf25416d-a4d5-402e-a40d-d3b5f84bb362" path="/var/lib/kubelet/pods/cf25416d-a4d5-402e-a40d-d3b5f84bb362/volumes" Nov 03 20:49:34.041751 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:34.041711 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnm86"] Nov 03 20:49:34.045682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:34.044805 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mnm86"] Nov 03 20:49:35.131005 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:35.130968 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4edb53-7shxm"] Nov 03 20:49:35.131380 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:35.131363 5059 cpu_manager.go:414] "containerMap: removing container" podUID="cf25416d-a4d5-402e-a40d-d3b5f84bb362" containerName="main" Nov 03 20:49:35.131439 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:35.131383 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf25416d-a4d5-402e-a40d-d3b5f84bb362" containerName="main" Nov 03 20:49:35.131439 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:35.131399 5059 cpu_manager.go:414] "containerMap: removing container" podUID="423eb759-cffb-47a4-b071-1213e8f5acf6" containerName="main" Nov 03 20:49:35.131439 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:35.131407 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="423eb759-cffb-47a4-b071-1213e8f5acf6" containerName="main" Nov 03 20:49:35.131565 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:35.131489 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="423eb759-cffb-47a4-b071-1213e8f5acf6" containerName="main" Nov 03 20:49:35.131565 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:35.131521 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf25416d-a4d5-402e-a40d-d3b5f84bb362" containerName="main" Nov 03 20:49:35.135567 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:35.135548 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4edb53-7shxm" Nov 03 20:49:35.142195 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:35.142149 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:49:35.142752 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:35.142235 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\"" Nov 03 20:49:35.143287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:35.143264 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4edb53-7shxm"] Nov 03 20:49:35.262040 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:35.262007 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a87c4e3-9662-49cc-b496-5b22e07ac9e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e4edb53-7shxm\" (UID: \"5a87c4e3-9662-49cc-b496-5b22e07ac9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4edb53-7shxm" Nov 03 20:49:35.262191 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:35.262076 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr95l\" (UniqueName: \"kubernetes.io/projected/5a87c4e3-9662-49cc-b496-5b22e07ac9e7-kube-api-access-mr95l\") pod \"maas-api-revoke-sub-7e4edb53-7shxm\" (UID: \"5a87c4e3-9662-49cc-b496-5b22e07ac9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4edb53-7shxm" Nov 03 20:49:35.262191 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:35.262137 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a87c4e3-9662-49cc-b496-5b22e07ac9e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e4edb53-7shxm\" (UID: \"5a87c4e3-9662-49cc-b496-5b22e07ac9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4edb53-7shxm" Nov 03 20:49:35.363027 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:35.362995 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr95l\" (UniqueName: \"kubernetes.io/projected/5a87c4e3-9662-49cc-b496-5b22e07ac9e7-kube-api-access-mr95l\") pod \"maas-api-revoke-sub-7e4edb53-7shxm\" (UID: \"5a87c4e3-9662-49cc-b496-5b22e07ac9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4edb53-7shxm" Nov 03 20:49:35.363146 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:35.363043 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a87c4e3-9662-49cc-b496-5b22e07ac9e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e4edb53-7shxm\" (UID: \"5a87c4e3-9662-49cc-b496-5b22e07ac9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4edb53-7shxm" Nov 03 20:49:35.363146 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:35.363111 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a87c4e3-9662-49cc-b496-5b22e07ac9e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e4edb53-7shxm\" (UID: \"5a87c4e3-9662-49cc-b496-5b22e07ac9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4edb53-7shxm" Nov 03 20:49:35.363849 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:35.363821 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a87c4e3-9662-49cc-b496-5b22e07ac9e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e4edb53-7shxm\" (UID: \"5a87c4e3-9662-49cc-b496-5b22e07ac9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4edb53-7shxm" Nov 03 20:49:35.365437 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:35.365410 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a87c4e3-9662-49cc-b496-5b22e07ac9e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e4edb53-7shxm\" (UID: \"5a87c4e3-9662-49cc-b496-5b22e07ac9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4edb53-7shxm" Nov 03 20:49:35.375106 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:35.375083 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr95l\" (UniqueName: \"kubernetes.io/projected/5a87c4e3-9662-49cc-b496-5b22e07ac9e7-kube-api-access-mr95l\") pod \"maas-api-revoke-sub-7e4edb53-7shxm\" (UID: \"5a87c4e3-9662-49cc-b496-5b22e07ac9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4edb53-7shxm" Nov 03 20:49:35.449094 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:35.449043 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4edb53-7shxm" Nov 03 20:49:35.723500 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:35.723471 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4edb53-7shxm"] Nov 03 20:49:35.729227 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:49:35.729193 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a87c4e3_9662_49cc_b496_5b22e07ac9e7.slice/crio-d4163811bc1092ab48ec91f9541b85e0cdd9b13542856b50fe5a2ea323aedeb1 WatchSource:0}: Error finding container d4163811bc1092ab48ec91f9541b85e0cdd9b13542856b50fe5a2ea323aedeb1: Status 404 returned error can't find the container with id d4163811bc1092ab48ec91f9541b85e0cdd9b13542856b50fe5a2ea323aedeb1 Nov 03 20:49:35.856358 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:35.856326 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71f2df39-c182-4e84-a6bb-c326becd0959" path="/var/lib/kubelet/pods/71f2df39-c182-4e84-a6bb-c326becd0959/volumes" Nov 03 20:49:36.214257 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:36.214216 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4edb53-7shxm" event={"ID":"5a87c4e3-9662-49cc-b496-5b22e07ac9e7","Type":"ContainerStarted","Data":"024dd23cb0283d262126302ad4985a1fdd874c89e07ec0ae202a99999adf0311"} Nov 03 20:49:36.214654 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:36.214279 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4edb53-7shxm" event={"ID":"5a87c4e3-9662-49cc-b496-5b22e07ac9e7","Type":"ContainerStarted","Data":"d4163811bc1092ab48ec91f9541b85e0cdd9b13542856b50fe5a2ea323aedeb1"} Nov 03 20:49:36.235055 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:36.235000 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4edb53-7shxm" podStartSLOduration=1.234987005 podStartE2EDuration="1.234987005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:36.233540195 +0000 UTC m=+4370.491337453" watchObservedRunningTime="2026-11-03 20:49:36.234987005 +0000 UTC m=+4370.492784262" Nov 03 20:49:37.219771 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:37.219733 5059 generic.go:356] "Generic (PLEG): container finished" podID="5a87c4e3-9662-49cc-b496-5b22e07ac9e7" containerID="024dd23cb0283d262126302ad4985a1fdd874c89e07ec0ae202a99999adf0311" exitCode=0 Nov 03 20:49:37.220181 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:37.219791 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4edb53-7shxm" event={"ID":"5a87c4e3-9662-49cc-b496-5b22e07ac9e7","Type":"ContainerDied","Data":"024dd23cb0283d262126302ad4985a1fdd874c89e07ec0ae202a99999adf0311"} Nov 03 20:49:38.353238 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:38.353208 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4edb53-7shxm" Nov 03 20:49:38.490918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:38.490824 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a87c4e3-9662-49cc-b496-5b22e07ac9e7-kube-api-access-mr95l\" (UniqueName: \"kubernetes.io/projected/5a87c4e3-9662-49cc-b496-5b22e07ac9e7-kube-api-access-mr95l\") pod \"5a87c4e3-9662-49cc-b496-5b22e07ac9e7\" (UID: \"5a87c4e3-9662-49cc-b496-5b22e07ac9e7\") " Nov 03 20:49:38.490918 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:38.490903 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a87c4e3-9662-49cc-b496-5b22e07ac9e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a87c4e3-9662-49cc-b496-5b22e07ac9e7-maas-api-openshift-ca\") pod \"5a87c4e3-9662-49cc-b496-5b22e07ac9e7\" (UID: \"5a87c4e3-9662-49cc-b496-5b22e07ac9e7\") " Nov 03 20:49:38.491168 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:38.490972 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a87c4e3-9662-49cc-b496-5b22e07ac9e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a87c4e3-9662-49cc-b496-5b22e07ac9e7-maas-api-xks-ca\") pod \"5a87c4e3-9662-49cc-b496-5b22e07ac9e7\" (UID: \"5a87c4e3-9662-49cc-b496-5b22e07ac9e7\") " Nov 03 20:49:38.491313 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:38.491282 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a87c4e3-9662-49cc-b496-5b22e07ac9e7-maas-api-openshift-ca" pod "5a87c4e3-9662-49cc-b496-5b22e07ac9e7" (UID: "5a87c4e3-9662-49cc-b496-5b22e07ac9e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:49:38.493136 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:38.493107 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a87c4e3-9662-49cc-b496-5b22e07ac9e7-maas-api-xks-ca" pod "5a87c4e3-9662-49cc-b496-5b22e07ac9e7" (UID: "5a87c4e3-9662-49cc-b496-5b22e07ac9e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:49:38.493257 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:38.493108 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a87c4e3-9662-49cc-b496-5b22e07ac9e7-kube-api-access-mr95l" pod "5a87c4e3-9662-49cc-b496-5b22e07ac9e7" (UID: "5a87c4e3-9662-49cc-b496-5b22e07ac9e7"). InnerVolumeSpecName "kube-api-access-mr95l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:49:38.592382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:38.592354 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a87c4e3-9662-49cc-b496-5b22e07ac9e7-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:38.592382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:38.592382 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a87c4e3-9662-49cc-b496-5b22e07ac9e7-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:38.592534 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:38.592392 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr95l\" (UniqueName: \"kubernetes.io/projected/5a87c4e3-9662-49cc-b496-5b22e07ac9e7-kube-api-access-mr95l\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:39.229934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:39.229899 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4edb53-7shxm" Nov 03 20:49:39.229934 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:39.229923 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4edb53-7shxm" event={"ID":"5a87c4e3-9662-49cc-b496-5b22e07ac9e7","Type":"ContainerDied","Data":"d4163811bc1092ab48ec91f9541b85e0cdd9b13542856b50fe5a2ea323aedeb1"} Nov 03 20:49:39.230193 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:49:39.229969 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4163811bc1092ab48ec91f9541b85e0cdd9b13542856b50fe5a2ea323aedeb1" Nov 03 20:49:40.602490 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:49:40.602454 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:50:10.603221 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:50:10.603177 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:50:16.618606 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:50:16.618567 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-6dc578d85f-q59ph"] Nov 03 20:50:16.619133 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:50:16.619111 5059 cpu_manager.go:414] "containerMap: removing container" podUID="5a87c4e3-9662-49cc-b496-5b22e07ac9e7" containerName="revoke-keys" Nov 03 20:50:16.619220 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:50:16.619136 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a87c4e3-9662-49cc-b496-5b22e07ac9e7" containerName="revoke-keys" Nov 03 20:50:16.619282 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:50:16.619260 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a87c4e3-9662-49cc-b496-5b22e07ac9e7" containerName="revoke-keys" Nov 03 20:50:16.625057 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:50:16.625035 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6dc578d85f-q59ph" Nov 03 20:50:16.630275 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:50:16.630249 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 20:50:16.630420 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:50:16.630299 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6dc578d85f-q59ph"] Nov 03 20:50:16.632711 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:50:16.632190 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2tlvj\"" Nov 03 20:50:16.734383 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:50:16.734350 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv29s\" (UniqueName: \"kubernetes.io/projected/cee2af62-5ba9-4f27-852b-25fb6b43c945-kube-api-access-zv29s\") pod \"payload-pre-processing-6dc578d85f-q59ph\" (UID: \"cee2af62-5ba9-4f27-852b-25fb6b43c945\") " pod="openshift-ingress/payload-pre-processing-6dc578d85f-q59ph" Nov 03 20:50:16.734539 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:50:16.734406 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee2af62-5ba9-4f27-852b-25fb6b43c945-plugins-config-volume\") pod \"payload-pre-processing-6dc578d85f-q59ph\" (UID: \"cee2af62-5ba9-4f27-852b-25fb6b43c945\") " pod="openshift-ingress/payload-pre-processing-6dc578d85f-q59ph" Nov 03 20:50:16.835411 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:50:16.835368 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee2af62-5ba9-4f27-852b-25fb6b43c945-plugins-config-volume\") pod \"payload-pre-processing-6dc578d85f-q59ph\" (UID: \"cee2af62-5ba9-4f27-852b-25fb6b43c945\") " pod="openshift-ingress/payload-pre-processing-6dc578d85f-q59ph" Nov 03 20:50:16.835593 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:50:16.835493 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv29s\" (UniqueName: \"kubernetes.io/projected/cee2af62-5ba9-4f27-852b-25fb6b43c945-kube-api-access-zv29s\") pod \"payload-pre-processing-6dc578d85f-q59ph\" (UID: \"cee2af62-5ba9-4f27-852b-25fb6b43c945\") " pod="openshift-ingress/payload-pre-processing-6dc578d85f-q59ph" Nov 03 20:50:16.836132 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:50:16.836108 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee2af62-5ba9-4f27-852b-25fb6b43c945-plugins-config-volume\") pod \"payload-pre-processing-6dc578d85f-q59ph\" (UID: \"cee2af62-5ba9-4f27-852b-25fb6b43c945\") " pod="openshift-ingress/payload-pre-processing-6dc578d85f-q59ph" Nov 03 20:50:16.848070 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:50:16.848044 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv29s\" (UniqueName: \"kubernetes.io/projected/cee2af62-5ba9-4f27-852b-25fb6b43c945-kube-api-access-zv29s\") pod \"payload-pre-processing-6dc578d85f-q59ph\" (UID: \"cee2af62-5ba9-4f27-852b-25fb6b43c945\") " pod="openshift-ingress/payload-pre-processing-6dc578d85f-q59ph" Nov 03 20:50:16.947791 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:50:16.947716 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6dc578d85f-q59ph" Nov 03 20:50:17.222463 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:50:17.222437 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6dc578d85f-q59ph"] Nov 03 20:50:17.224835 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:50:17.224798 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee2af62_5ba9_4f27_852b_25fb6b43c945.slice/crio-1b26a7c409b151e1fb8c51e692feac9849c10001ac2b4e05670d88172c08c9fe WatchSource:0}: Error finding container 1b26a7c409b151e1fb8c51e692feac9849c10001ac2b4e05670d88172c08c9fe: Status 404 returned error can't find the container with id 1b26a7c409b151e1fb8c51e692feac9849c10001ac2b4e05670d88172c08c9fe Nov 03 20:50:17.396742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:50:17.396706 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6dc578d85f-q59ph" event={"ID":"cee2af62-5ba9-4f27-852b-25fb6b43c945","Type":"ContainerStarted","Data":"89ed1861e97ab8ab4a525a3660c3f421e2ed1c403532de3a4be7a88b59a64b50"} Nov 03 20:50:17.396891 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:50:17.396764 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6dc578d85f-q59ph" event={"ID":"cee2af62-5ba9-4f27-852b-25fb6b43c945","Type":"ContainerStarted","Data":"1b26a7c409b151e1fb8c51e692feac9849c10001ac2b4e05670d88172c08c9fe"} Nov 03 20:50:17.396891 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:50:17.396871 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-6dc578d85f-q59ph" Nov 03 20:50:17.416205 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:50:17.416155 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-6dc578d85f-q59ph" podStartSLOduration=1.416141621 podStartE2EDuration="1.416141621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:17.415320427 +0000 UTC m=+4411.673117684" watchObservedRunningTime="2026-11-03 20:50:17.416141621 +0000 UTC m=+4411.673938878" Nov 03 20:50:26.716014 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:50:26.715977 5059 scope.go:122] "RemoveContainer" containerID="27bbce1874459ff8a0f12e84bd3996099da61d18b379f82e83ce736a5d2b4a28" Nov 03 20:50:26.723847 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:50:26.723816 5059 scope.go:122] "RemoveContainer" containerID="4d839897d659379659a10012daf2228029da65594f6b94b6360513da4f69bddb" Nov 03 20:50:26.730963 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:50:26.730940 5059 scope.go:122] "RemoveContainer" containerID="dd3d2b164a369e76a0cf0c7ff20a16c679cba81dafafe97daf8ffc6a18885b40" Nov 03 20:50:28.404068 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:50:28.404032 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-6dc578d85f-q59ph" Nov 03 20:50:40.603195 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:50:40.603154 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:51:10.607580 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:51:10.607540 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:51:26.759869 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:51:26.759835 5059 scope.go:122] "RemoveContainer" containerID="4e3da925b77498e1e3766d9acf54f5bc0fac760bd7365a8da895f9a4dee3b286" Nov 03 20:51:40.603784 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:51:40.603744 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:52:09.022078 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:52:09.022040 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b5aa2f9-t7cch"] Nov 03 20:52:09.025459 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:52:09.025436 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b5aa2f9-t7cch"] Nov 03 20:52:09.855794 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:52:09.855761 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0" path="/var/lib/kubelet/pods/98a7afbb-8b0e-4417-bef5-a0d9d31ac7f0/volumes" Nov 03 20:52:10.604039 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:52:10.604003 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:52:40.604138 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:52:40.604052 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:53:10.324833 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:53:10.324788 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wtlgc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:53:10.325506 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:53:10.325470 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" podUID="ab375b16-1371-437f-91b4-d5f8580e804d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:10.764415 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:53:10.764315 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:53:26.799587 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:53:26.799552 5059 scope.go:122] "RemoveContainer" containerID="5fd33926e9b5dfc72679d58737c0b5f86d0e63e270752aab7411b7e29b73c5f2" Nov 03 20:53:40.603128 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:53:40.603089 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:54:10.603891 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:54:10.603850 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:54:39.025371 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:54:39.025324 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4edb53-7shxm"] Nov 03 20:54:39.041364 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:54:39.041331 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4edb53-7shxm"] Nov 03 20:54:40.041531 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:54:40.041484 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a87c4e3-9662-49cc-b496-5b22e07ac9e7" path="/var/lib/kubelet/pods/5a87c4e3-9662-49cc-b496-5b22e07ac9e7/volumes" Nov 03 20:54:40.368247 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:54:40.368170 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vkx59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:54:40.368608 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:54:40.368571 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" podUID="d14046b2-dd71-42a3-b239-0160088faa0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:40.743016 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:54:40.742878 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:54:55.283266 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:54:55.283225 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8gzjz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:54:55.283974 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:54:55.283939 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" podUID="6462b9bb-1018-4029-922d-eb71527fc301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:55.314859 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:54:55.314829 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wtlgc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:54:55.315044 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:54:55.315020 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" podUID="ab375b16-1371-437f-91b4-d5f8580e804d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:55.349773 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:54:55.349745 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vkx59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:54:55.349971 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:54:55.349946 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" podUID="d14046b2-dd71-42a3-b239-0160088faa0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:55:10.603863 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:55:10.603820 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:55:40.344905 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:55:40.344799 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vkx59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:55:40.345734 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:55:40.345697 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vkx59" podUID="d14046b2-dd71-42a3-b239-0160088faa0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:55:40.636899 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:55:40.636799 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:56:10.602383 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:56:10.602341 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:56:26.830015 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:56:26.829980 5059 scope.go:122] "RemoveContainer" containerID="024dd23cb0283d262126302ad4985a1fdd874c89e07ec0ae202a99999adf0311" Nov 03 20:56:40.603558 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:56:40.603510 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:57:10.603111 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:57:10.603073 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:57:15.137333 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:15.136785 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cmtzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:57:15.137333 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:15.136882 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cmtzs" podUID="a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:57:20.625862 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:20.625819 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7"] Nov 03 20:57:20.629573 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:20.629551 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7" Nov 03 20:57:20.645713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:20.641356 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:57:20.645713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:20.641577 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-4ca23e-serving-cert\"" Nov 03 20:57:20.645713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:20.642531 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-942q2\"" Nov 03 20:57:20.645713 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:20.644705 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7"] Nov 03 20:57:20.776746 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:20.776711 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cd52f0-647e-439f-aaa6-d4823c2510d9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7\" (UID: \"b3cd52f0-647e-439f-aaa6-d4823c2510d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7" Nov 03 20:57:20.776920 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:20.776762 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8frvf\" (UniqueName: \"kubernetes.io/projected/b3cd52f0-647e-439f-aaa6-d4823c2510d9-kube-api-access-8frvf\") pod \"maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7\" (UID: \"b3cd52f0-647e-439f-aaa6-d4823c2510d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7" Nov 03 20:57:20.776920 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:20.776836 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3cd52f0-647e-439f-aaa6-d4823c2510d9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7\" (UID: \"b3cd52f0-647e-439f-aaa6-d4823c2510d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7" Nov 03 20:57:20.777021 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:20.776949 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cd52f0-647e-439f-aaa6-d4823c2510d9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7\" (UID: \"b3cd52f0-647e-439f-aaa6-d4823c2510d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7" Nov 03 20:57:20.877988 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:20.877909 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cd52f0-647e-439f-aaa6-d4823c2510d9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7\" (UID: \"b3cd52f0-647e-439f-aaa6-d4823c2510d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7" Nov 03 20:57:20.877988 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:20.877957 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cd52f0-647e-439f-aaa6-d4823c2510d9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7\" (UID: \"b3cd52f0-647e-439f-aaa6-d4823c2510d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7" Nov 03 20:57:20.878171 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:20.877993 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8frvf\" (UniqueName: \"kubernetes.io/projected/b3cd52f0-647e-439f-aaa6-d4823c2510d9-kube-api-access-8frvf\") pod \"maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7\" (UID: \"b3cd52f0-647e-439f-aaa6-d4823c2510d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7" Nov 03 20:57:20.878171 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:20.878025 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3cd52f0-647e-439f-aaa6-d4823c2510d9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7\" (UID: \"b3cd52f0-647e-439f-aaa6-d4823c2510d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7" Nov 03 20:57:20.878825 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:20.878787 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cd52f0-647e-439f-aaa6-d4823c2510d9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7\" (UID: \"b3cd52f0-647e-439f-aaa6-d4823c2510d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7" Nov 03 20:57:20.880474 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:20.880446 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cd52f0-647e-439f-aaa6-d4823c2510d9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7\" (UID: \"b3cd52f0-647e-439f-aaa6-d4823c2510d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7" Nov 03 20:57:20.880596 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:20.880500 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3cd52f0-647e-439f-aaa6-d4823c2510d9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7\" (UID: \"b3cd52f0-647e-439f-aaa6-d4823c2510d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7" Nov 03 20:57:20.891792 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:20.891761 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8frvf\" (UniqueName: \"kubernetes.io/projected/b3cd52f0-647e-439f-aaa6-d4823c2510d9-kube-api-access-8frvf\") pod \"maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7\" (UID: \"b3cd52f0-647e-439f-aaa6-d4823c2510d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7" Nov 03 20:57:20.947080 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:20.947041 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7" Nov 03 20:57:21.225469 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:21.225441 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7"] Nov 03 20:57:21.227639 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:57:21.227603 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3cd52f0_647e_439f_aaa6_d4823c2510d9.slice/crio-08d2d2e563d7918a67b3be14deb2d3683ef75498d1e9bf651cac8ec66cf928a8 WatchSource:0}: Error finding container 08d2d2e563d7918a67b3be14deb2d3683ef75498d1e9bf651cac8ec66cf928a8: Status 404 returned error can't find the container with id 08d2d2e563d7918a67b3be14deb2d3683ef75498d1e9bf651cac8ec66cf928a8 Nov 03 20:57:21.229387 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:21.229367 5059 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:57:21.702035 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:21.702000 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7" event={"ID":"b3cd52f0-647e-439f-aaa6-d4823c2510d9","Type":"ContainerStarted","Data":"08d2d2e563d7918a67b3be14deb2d3683ef75498d1e9bf651cac8ec66cf928a8"} Nov 03 20:57:22.707983 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:22.707942 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7" event={"ID":"b3cd52f0-647e-439f-aaa6-d4823c2510d9","Type":"ContainerStarted","Data":"720e92e5af293176898386b233fc10f3dceb98924a77b6de60ed39049c548ec5"} Nov 03 20:57:22.708394 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:22.708047 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7" Nov 03 20:57:22.728566 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:22.728517 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7" podStartSLOduration=2.224194829 podStartE2EDuration="2.728504865s" totalImagesPullingTime="504.310036ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:22.727179287 +0000 UTC m=+4836.984976549" watchObservedRunningTime="2026-11-03 20:57:22.728504865 +0000 UTC m=+4836.986302122" Nov 03 20:57:28.718631 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:28.718597 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7" Nov 03 20:57:36.467708 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.467649 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg"] Nov 03 20:57:36.471706 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.471679 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:57:36.480175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.480154 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-4ca23e\"/\"kube-root-ca.crt\"" Nov 03 20:57:36.481428 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.481407 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-4ca23e\"/\"default-dockercfg-mt5cq\"" Nov 03 20:57:36.481519 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.481454 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-4ca23e\"/\"openshift-service-ca.crt\"" Nov 03 20:57:36.483596 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.483571 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg"] Nov 03 20:57:36.483726 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.483644 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-4ca23e\"/\"simulator-single-main-4ca23e-kserve-self-signed-certs\"" Nov 03 20:57:36.531232 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.531207 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wthfl\" (UniqueName: \"kubernetes.io/projected/b6c6fc73-6755-4c9d-9b61-a3812cb33771-kube-api-access-wthfl\") pod \"simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg\" (UID: \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\") " pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:57:36.531373 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.531351 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-model-cache\") pod \"simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg\" (UID: \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\") " pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:57:36.531418 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.531391 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-home\") pod \"simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg\" (UID: \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\") " pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:57:36.531418 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.531412 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-tmp-dir\") pod \"simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg\" (UID: \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\") " pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:57:36.531492 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.531429 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-dshm\") pod \"simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg\" (UID: \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\") " pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:57:36.531527 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.531488 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6c6fc73-6755-4c9d-9b61-a3812cb33771-tls-certs\") pod \"simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg\" (UID: \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\") " pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:57:36.633287 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.633253 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-home\") pod \"simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg\" (UID: \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\") " pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:57:36.633544 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.633309 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-tmp-dir\") pod \"simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg\" (UID: \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\") " pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:57:36.633544 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.633336 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-dshm\") pod \"simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg\" (UID: \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\") " pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:57:36.633544 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.633374 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6c6fc73-6755-4c9d-9b61-a3812cb33771-tls-certs\") pod \"simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg\" (UID: \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\") " pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:57:36.633544 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.633412 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wthfl\" (UniqueName: \"kubernetes.io/projected/b6c6fc73-6755-4c9d-9b61-a3812cb33771-kube-api-access-wthfl\") pod \"simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg\" (UID: \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\") " pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:57:36.634116 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.634089 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-model-cache\") pod \"simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg\" (UID: \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\") " pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:57:36.634714 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.634692 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-model-cache\") pod \"simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg\" (UID: \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\") " pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:57:36.635052 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.635026 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-home\") pod \"simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg\" (UID: \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\") " pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:57:36.635145 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.635068 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-tmp-dir\") pod \"simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg\" (UID: \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\") " pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:57:36.638184 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.638139 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6c6fc73-6755-4c9d-9b61-a3812cb33771-tls-certs\") pod \"simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg\" (UID: \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\") " pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:57:36.645216 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.645189 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-dshm\") pod \"simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg\" (UID: \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\") " pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:57:36.648574 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.648547 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wthfl\" (UniqueName: \"kubernetes.io/projected/b6c6fc73-6755-4c9d-9b61-a3812cb33771-kube-api-access-wthfl\") pod \"simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg\" (UID: \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\") " pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:57:36.791644 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:36.791566 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:57:37.067055 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:37.067025 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg"] Nov 03 20:57:37.069675 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:57:37.069628 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6c6fc73_6755_4c9d_9b61_a3812cb33771.slice/crio-5693d354a5a968e27eafd27be2c6454b7a2b3802918762d590ddf5209874f45d WatchSource:0}: Error finding container 5693d354a5a968e27eafd27be2c6454b7a2b3802918762d590ddf5209874f45d: Status 404 returned error can't find the container with id 5693d354a5a968e27eafd27be2c6454b7a2b3802918762d590ddf5209874f45d Nov 03 20:57:37.775049 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:37.775003 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" event={"ID":"b6c6fc73-6755-4c9d-9b61-a3812cb33771","Type":"ContainerStarted","Data":"bce0d1dafef3e9121234d09234f0d5a8d295c478afececf57cfe93fbef967aaf"} Nov 03 20:57:37.775451 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:37.775056 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" event={"ID":"b6c6fc73-6755-4c9d-9b61-a3812cb33771","Type":"ContainerStarted","Data":"5693d354a5a968e27eafd27be2c6454b7a2b3802918762d590ddf5209874f45d"} Nov 03 20:57:37.792090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:37.792064 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:57:37.795563 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:37.795519 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" podStartSLOduration=1.795507464 podStartE2EDuration="1.795507464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:37.794207986 +0000 UTC m=+4852.052005244" watchObservedRunningTime="2026-11-03 20:57:37.795507464 +0000 UTC m=+4852.053304721" Nov 03 20:57:40.603776 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:57:40.603734 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:57:46.792562 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:46.792529 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:57:46.804909 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:46.804882 5059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:57:46.823293 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:57:46.823269 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:58:10.603620 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:58:10.603580 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:58:27.919615 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:27.919532 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg"] Nov 03 20:58:30.626722 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:30.626683 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7"] Nov 03 20:58:30.627222 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:30.626939 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7" podUID="b3cd52f0-647e-439f-aaa6-d4823c2510d9" containerName="maas-api" containerID="cri-o://720e92e5af293176898386b233fc10f3dceb98924a77b6de60ed39049c548ec5" gracePeriod=30 Nov 03 20:58:31.025399 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:31.025357 5059 generic.go:356] "Generic (PLEG): container finished" podID="b3cd52f0-647e-439f-aaa6-d4823c2510d9" containerID="720e92e5af293176898386b233fc10f3dceb98924a77b6de60ed39049c548ec5" exitCode=0 Nov 03 20:58:31.025548 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:31.025405 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7" event={"ID":"b3cd52f0-647e-439f-aaa6-d4823c2510d9","Type":"ContainerDied","Data":"720e92e5af293176898386b233fc10f3dceb98924a77b6de60ed39049c548ec5"} Nov 03 20:58:31.075404 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:31.075380 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7" Nov 03 20:58:31.150457 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:31.150385 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3cd52f0-647e-439f-aaa6-d4823c2510d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cd52f0-647e-439f-aaa6-d4823c2510d9-maas-api-tls\") pod \"b3cd52f0-647e-439f-aaa6-d4823c2510d9\" (UID: \"b3cd52f0-647e-439f-aaa6-d4823c2510d9\") " Nov 03 20:58:31.150457 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:31.150420 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3cd52f0-647e-439f-aaa6-d4823c2510d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cd52f0-647e-439f-aaa6-d4823c2510d9-openshift-service-ca\") pod \"b3cd52f0-647e-439f-aaa6-d4823c2510d9\" (UID: \"b3cd52f0-647e-439f-aaa6-d4823c2510d9\") " Nov 03 20:58:31.150696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:31.150504 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3cd52f0-647e-439f-aaa6-d4823c2510d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3cd52f0-647e-439f-aaa6-d4823c2510d9-metrics-certs\") pod \"b3cd52f0-647e-439f-aaa6-d4823c2510d9\" (UID: \"b3cd52f0-647e-439f-aaa6-d4823c2510d9\") " Nov 03 20:58:31.150696 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:31.150547 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3cd52f0-647e-439f-aaa6-d4823c2510d9-kube-api-access-8frvf\" (UniqueName: \"kubernetes.io/projected/b3cd52f0-647e-439f-aaa6-d4823c2510d9-kube-api-access-8frvf\") pod \"b3cd52f0-647e-439f-aaa6-d4823c2510d9\" (UID: \"b3cd52f0-647e-439f-aaa6-d4823c2510d9\") " Nov 03 20:58:31.150928 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:31.150898 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3cd52f0-647e-439f-aaa6-d4823c2510d9-openshift-service-ca" pod "b3cd52f0-647e-439f-aaa6-d4823c2510d9" (UID: "b3cd52f0-647e-439f-aaa6-d4823c2510d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:31.152573 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:31.152548 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3cd52f0-647e-439f-aaa6-d4823c2510d9-maas-api-tls" pod "b3cd52f0-647e-439f-aaa6-d4823c2510d9" (UID: "b3cd52f0-647e-439f-aaa6-d4823c2510d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:58:31.152715 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:31.152696 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3cd52f0-647e-439f-aaa6-d4823c2510d9-kube-api-access-8frvf" pod "b3cd52f0-647e-439f-aaa6-d4823c2510d9" (UID: "b3cd52f0-647e-439f-aaa6-d4823c2510d9"). InnerVolumeSpecName "kube-api-access-8frvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:31.152793 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:31.152731 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3cd52f0-647e-439f-aaa6-d4823c2510d9-metrics-certs" pod "b3cd52f0-647e-439f-aaa6-d4823c2510d9" (UID: "b3cd52f0-647e-439f-aaa6-d4823c2510d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:58:31.252093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:31.252057 5059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3cd52f0-647e-439f-aaa6-d4823c2510d9-metrics-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:31.252093 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:31.252091 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8frvf\" (UniqueName: \"kubernetes.io/projected/b3cd52f0-647e-439f-aaa6-d4823c2510d9-kube-api-access-8frvf\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:31.252254 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:31.252103 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cd52f0-647e-439f-aaa6-d4823c2510d9-maas-api-tls\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:31.252254 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:31.252113 5059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cd52f0-647e-439f-aaa6-d4823c2510d9-openshift-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:32.031029 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:32.030989 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7" event={"ID":"b3cd52f0-647e-439f-aaa6-d4823c2510d9","Type":"ContainerDied","Data":"08d2d2e563d7918a67b3be14deb2d3683ef75498d1e9bf651cac8ec66cf928a8"} Nov 03 20:58:32.031428 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:32.031049 5059 scope.go:122] "RemoveContainer" containerID="720e92e5af293176898386b233fc10f3dceb98924a77b6de60ed39049c548ec5" Nov 03 20:58:32.031428 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:32.031014 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7" Nov 03 20:58:32.049754 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:32.049730 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7"] Nov 03 20:58:32.054087 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:32.054061 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-nxbm7"] Nov 03 20:58:33.855251 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:33.855216 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3cd52f0-647e-439f-aaa6-d4823c2510d9" path="/var/lib/kubelet/pods/b3cd52f0-647e-439f-aaa6-d4823c2510d9/volumes" Nov 03 20:58:40.603416 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:58:40.603368 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:58:42.938682 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:42.938631 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" podUID="b6c6fc73-6755-4c9d-9b61-a3812cb33771" containerName="main" containerID="cri-o://bce0d1dafef3e9121234d09234f0d5a8d295c478afececf57cfe93fbef967aaf" gracePeriod=45 Nov 03 20:58:43.077952 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:43.077921 5059 generic.go:356] "Generic (PLEG): container finished" podID="b6c6fc73-6755-4c9d-9b61-a3812cb33771" containerID="bce0d1dafef3e9121234d09234f0d5a8d295c478afececf57cfe93fbef967aaf" exitCode=0 Nov 03 20:58:43.078074 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:43.077971 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" event={"ID":"b6c6fc73-6755-4c9d-9b61-a3812cb33771","Type":"ContainerDied","Data":"bce0d1dafef3e9121234d09234f0d5a8d295c478afececf57cfe93fbef967aaf"} Nov 03 20:58:43.191944 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:43.191890 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:58:43.256607 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:43.256576 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6c6fc73-6755-4c9d-9b61-a3812cb33771-kube-api-access-wthfl\" (UniqueName: \"kubernetes.io/projected/b6c6fc73-6755-4c9d-9b61-a3812cb33771-kube-api-access-wthfl\") pod \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\" (UID: \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\") " Nov 03 20:58:43.256800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:43.256632 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-home\" (UniqueName: \"kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-home\") pod \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\" (UID: \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\") " Nov 03 20:58:43.256800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:43.256651 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-tmp-dir\") pod \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\" (UID: \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\") " Nov 03 20:58:43.256800 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:43.256768 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-dshm\") pod \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\" (UID: \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\") " Nov 03 20:58:43.256997 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:43.256806 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6c6fc73-6755-4c9d-9b61-a3812cb33771-tls-certs\" (UniqueName: \"kubernetes.io/secret/b6c6fc73-6755-4c9d-9b61-a3812cb33771-tls-certs\") pod \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\" (UID: \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\") " Nov 03 20:58:43.256997 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:43.256880 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-model-cache\") pod \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\" (UID: \"b6c6fc73-6755-4c9d-9b61-a3812cb33771\") " Nov 03 20:58:43.256997 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:43.256919 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-home" pod "b6c6fc73-6755-4c9d-9b61-a3812cb33771" (UID: "b6c6fc73-6755-4c9d-9b61-a3812cb33771"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:43.257198 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:43.257172 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-tmp-dir" pod "b6c6fc73-6755-4c9d-9b61-a3812cb33771" (UID: "b6c6fc73-6755-4c9d-9b61-a3812cb33771"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:43.257257 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:43.257205 5059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-home\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:43.257355 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:43.257325 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-model-cache" pod "b6c6fc73-6755-4c9d-9b61-a3812cb33771" (UID: "b6c6fc73-6755-4c9d-9b61-a3812cb33771"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:43.259140 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:43.259111 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-dshm" pod "b6c6fc73-6755-4c9d-9b61-a3812cb33771" (UID: "b6c6fc73-6755-4c9d-9b61-a3812cb33771"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:43.259283 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:43.259268 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c6fc73-6755-4c9d-9b61-a3812cb33771-kube-api-access-wthfl" pod "b6c6fc73-6755-4c9d-9b61-a3812cb33771" (UID: "b6c6fc73-6755-4c9d-9b61-a3812cb33771"). InnerVolumeSpecName "kube-api-access-wthfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:43.259335 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:43.259301 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6c6fc73-6755-4c9d-9b61-a3812cb33771-tls-certs" pod "b6c6fc73-6755-4c9d-9b61-a3812cb33771" (UID: "b6c6fc73-6755-4c9d-9b61-a3812cb33771"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:58:43.357989 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:43.357946 5059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-model-cache\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:43.357989 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:43.357986 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wthfl\" (UniqueName: \"kubernetes.io/projected/b6c6fc73-6755-4c9d-9b61-a3812cb33771-kube-api-access-wthfl\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:43.358207 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:43.358001 5059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-tmp-dir\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:43.358207 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:43.358015 5059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6c6fc73-6755-4c9d-9b61-a3812cb33771-dshm\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:43.358207 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:43.358027 5059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6c6fc73-6755-4c9d-9b61-a3812cb33771-tls-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:44.083796 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:44.083759 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" event={"ID":"b6c6fc73-6755-4c9d-9b61-a3812cb33771","Type":"ContainerDied","Data":"5693d354a5a968e27eafd27be2c6454b7a2b3802918762d590ddf5209874f45d"} Nov 03 20:58:44.084312 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:44.083804 5059 scope.go:122] "RemoveContainer" containerID="bce0d1dafef3e9121234d09234f0d5a8d295c478afececf57cfe93fbef967aaf" Nov 03 20:58:44.084312 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:44.083774 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg" Nov 03 20:58:44.101569 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:44.101541 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg"] Nov 03 20:58:44.105816 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:44.105794 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-4ca23e/simulator-single-main-4ca23e-kserve-99f7fc688-xkxlg"] Nov 03 20:58:45.854943 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:58:45.854908 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6c6fc73-6755-4c9d-9b61-a3812cb33771" path="/var/lib/kubelet/pods/b6c6fc73-6755-4c9d-9b61-a3812cb33771/volumes" Nov 03 20:59:02.041509 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.041471 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd"] Nov 03 20:59:02.042036 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.042016 5059 cpu_manager.go:414] "containerMap: removing container" podUID="b3cd52f0-647e-439f-aaa6-d4823c2510d9" containerName="maas-api" Nov 03 20:59:02.042124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.042039 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3cd52f0-647e-439f-aaa6-d4823c2510d9" containerName="maas-api" Nov 03 20:59:02.042124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.042079 5059 cpu_manager.go:414] "containerMap: removing container" podUID="b6c6fc73-6755-4c9d-9b61-a3812cb33771" containerName="main" Nov 03 20:59:02.042124 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.042088 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6c6fc73-6755-4c9d-9b61-a3812cb33771" containerName="main" Nov 03 20:59:02.042297 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.042172 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6c6fc73-6755-4c9d-9b61-a3812cb33771" containerName="main" Nov 03 20:59:02.042297 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.042188 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3cd52f0-647e-439f-aaa6-d4823c2510d9" containerName="maas-api" Nov 03 20:59:02.045878 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.045853 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd" Nov 03 20:59:02.056176 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.056141 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-4ca23e-serving-cert\"" Nov 03 20:59:02.057488 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.057465 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-942q2\"" Nov 03 20:59:02.057565 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.057503 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:59:02.063311 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.063083 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd"] Nov 03 20:59:02.121155 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.121119 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd\" (UID: \"de0866dc-1140-4a5a-9ee9-bba3f731cbfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd" Nov 03 20:59:02.121322 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.121187 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jms9h\" (UniqueName: \"kubernetes.io/projected/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-kube-api-access-jms9h\") pod \"maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd\" (UID: \"de0866dc-1140-4a5a-9ee9-bba3f731cbfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd" Nov 03 20:59:02.121322 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.121279 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd\" (UID: \"de0866dc-1140-4a5a-9ee9-bba3f731cbfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd" Nov 03 20:59:02.121429 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.121414 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd\" (UID: \"de0866dc-1140-4a5a-9ee9-bba3f731cbfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd" Nov 03 20:59:02.222115 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.222075 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd\" (UID: \"de0866dc-1140-4a5a-9ee9-bba3f731cbfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd" Nov 03 20:59:02.222300 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.222126 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jms9h\" (UniqueName: \"kubernetes.io/projected/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-kube-api-access-jms9h\") pod \"maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd\" (UID: \"de0866dc-1140-4a5a-9ee9-bba3f731cbfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd" Nov 03 20:59:02.222300 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.222164 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd\" (UID: \"de0866dc-1140-4a5a-9ee9-bba3f731cbfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd" Nov 03 20:59:02.222300 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.222237 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd\" (UID: \"de0866dc-1140-4a5a-9ee9-bba3f731cbfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd" Nov 03 20:59:02.222896 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.222859 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd\" (UID: \"de0866dc-1140-4a5a-9ee9-bba3f731cbfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd" Nov 03 20:59:02.224698 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.224678 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd\" (UID: \"de0866dc-1140-4a5a-9ee9-bba3f731cbfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd" Nov 03 20:59:02.224802 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.224788 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd\" (UID: \"de0866dc-1140-4a5a-9ee9-bba3f731cbfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd" Nov 03 20:59:02.233950 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.233923 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jms9h\" (UniqueName: \"kubernetes.io/projected/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-kube-api-access-jms9h\") pod \"maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd\" (UID: \"de0866dc-1140-4a5a-9ee9-bba3f731cbfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd" Nov 03 20:59:02.360189 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.360147 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd" Nov 03 20:59:02.636427 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:02.636398 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd"] Nov 03 20:59:02.638937 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:59:02.638908 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde0866dc_1140_4a5a_9ee9_bba3f731cbfa.slice/crio-9196523950a5d4a6421428d4df24ca764e2e93e651e1aa03d98ad3d69117f5d1 WatchSource:0}: Error finding container 9196523950a5d4a6421428d4df24ca764e2e93e651e1aa03d98ad3d69117f5d1: Status 404 returned error can't find the container with id 9196523950a5d4a6421428d4df24ca764e2e93e651e1aa03d98ad3d69117f5d1 Nov 03 20:59:03.163603 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:03.163575 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd" event={"ID":"de0866dc-1140-4a5a-9ee9-bba3f731cbfa","Type":"ContainerStarted","Data":"9196523950a5d4a6421428d4df24ca764e2e93e651e1aa03d98ad3d69117f5d1"} Nov 03 20:59:04.169288 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:04.169239 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd" event={"ID":"de0866dc-1140-4a5a-9ee9-bba3f731cbfa","Type":"ContainerStarted","Data":"4d75d6b6cc8765efb53cb6d659c4c01728721fd3b51a3376c21b20de968ee177"} Nov 03 20:59:04.169794 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:04.169400 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd" Nov 03 20:59:04.190039 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:04.189991 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd" podStartSLOduration=1.7034382240000001 podStartE2EDuration="2.189978651s" totalImagesPullingTime="486.540427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:04.188209421 +0000 UTC m=+4938.446006678" watchObservedRunningTime="2026-11-03 20:59:04.189978651 +0000 UTC m=+4938.447775924" Nov 03 20:59:10.179054 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:10.179019 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd" Nov 03 20:59:10.603585 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:59:10.603541 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 20:59:21.077161 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:21.077115 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd"] Nov 03 20:59:21.077742 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:21.077373 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd" podUID="de0866dc-1140-4a5a-9ee9-bba3f731cbfa" containerName="maas-api" containerID="cri-o://4d75d6b6cc8765efb53cb6d659c4c01728721fd3b51a3376c21b20de968ee177" gracePeriod=30 Nov 03 20:59:21.248827 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:21.248792 5059 generic.go:356] "Generic (PLEG): container finished" podID="de0866dc-1140-4a5a-9ee9-bba3f731cbfa" containerID="4d75d6b6cc8765efb53cb6d659c4c01728721fd3b51a3376c21b20de968ee177" exitCode=0 Nov 03 20:59:21.248999 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:21.248871 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd" event={"ID":"de0866dc-1140-4a5a-9ee9-bba3f731cbfa","Type":"ContainerDied","Data":"4d75d6b6cc8765efb53cb6d659c4c01728721fd3b51a3376c21b20de968ee177"} Nov 03 20:59:21.336299 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:21.336240 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd" Nov 03 20:59:21.423324 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:21.423284 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-metrics-certs\") pod \"de0866dc-1140-4a5a-9ee9-bba3f731cbfa\" (UID: \"de0866dc-1140-4a5a-9ee9-bba3f731cbfa\") " Nov 03 20:59:21.423495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:21.423417 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-openshift-service-ca\") pod \"de0866dc-1140-4a5a-9ee9-bba3f731cbfa\" (UID: \"de0866dc-1140-4a5a-9ee9-bba3f731cbfa\") " Nov 03 20:59:21.423495 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:21.423456 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-maas-api-tls\") pod \"de0866dc-1140-4a5a-9ee9-bba3f731cbfa\" (UID: \"de0866dc-1140-4a5a-9ee9-bba3f731cbfa\") " Nov 03 20:59:21.423622 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:21.423594 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-kube-api-access-jms9h\" (UniqueName: \"kubernetes.io/projected/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-kube-api-access-jms9h\") pod \"de0866dc-1140-4a5a-9ee9-bba3f731cbfa\" (UID: \"de0866dc-1140-4a5a-9ee9-bba3f731cbfa\") " Nov 03 20:59:21.423891 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:21.423853 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-openshift-service-ca" pod "de0866dc-1140-4a5a-9ee9-bba3f731cbfa" (UID: "de0866dc-1140-4a5a-9ee9-bba3f731cbfa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:21.424090 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:21.424066 5059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-openshift-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:21.425964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:21.425803 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-kube-api-access-jms9h" pod "de0866dc-1140-4a5a-9ee9-bba3f731cbfa" (UID: "de0866dc-1140-4a5a-9ee9-bba3f731cbfa"). InnerVolumeSpecName "kube-api-access-jms9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:21.425964 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:21.425910 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-maas-api-tls" pod "de0866dc-1140-4a5a-9ee9-bba3f731cbfa" (UID: "de0866dc-1140-4a5a-9ee9-bba3f731cbfa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:59:21.426144 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:21.426107 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-metrics-certs" pod "de0866dc-1140-4a5a-9ee9-bba3f731cbfa" (UID: "de0866dc-1140-4a5a-9ee9-bba3f731cbfa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:59:21.525440 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:21.525407 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-maas-api-tls\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:21.525440 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:21.525438 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jms9h\" (UniqueName: \"kubernetes.io/projected/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-kube-api-access-jms9h\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:21.525651 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:21.525448 5059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de0866dc-1140-4a5a-9ee9-bba3f731cbfa-metrics-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:22.254231 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:22.254140 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd" event={"ID":"de0866dc-1140-4a5a-9ee9-bba3f731cbfa","Type":"ContainerDied","Data":"9196523950a5d4a6421428d4df24ca764e2e93e651e1aa03d98ad3d69117f5d1"} Nov 03 20:59:22.254231 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:22.254157 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd" Nov 03 20:59:22.254231 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:22.254230 5059 scope.go:122] "RemoveContainer" containerID="4d75d6b6cc8765efb53cb6d659c4c01728721fd3b51a3376c21b20de968ee177" Nov 03 20:59:22.271874 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:22.271846 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd"] Nov 03 20:59:22.278656 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:22.278627 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4ca23e-dd6f87597-kppdd"] Nov 03 20:59:22.992037 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:22.992003 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lx7kz"] Nov 03 20:59:22.992388 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:22.992375 5059 cpu_manager.go:414] "containerMap: removing container" podUID="de0866dc-1140-4a5a-9ee9-bba3f731cbfa" containerName="maas-api" Nov 03 20:59:22.992439 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:22.992390 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de0866dc-1140-4a5a-9ee9-bba3f731cbfa" containerName="maas-api" Nov 03 20:59:22.992473 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:22.992450 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="de0866dc-1140-4a5a-9ee9-bba3f731cbfa" containerName="maas-api" Nov 03 20:59:22.996970 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:22.996952 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lx7kz" Nov 03 20:59:23.003166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:23.002985 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:59:23.003166 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:23.003113 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\"" Nov 03 20:59:23.005042 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:23.004401 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lx7kz"] Nov 03 20:59:23.038038 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:23.038014 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c93ac8-e7b7-4848-8502-1224102f9d75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lx7kz\" (UID: \"f2c93ac8-e7b7-4848-8502-1224102f9d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lx7kz" Nov 03 20:59:23.038141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:23.038061 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvb8b\" (UniqueName: \"kubernetes.io/projected/f2c93ac8-e7b7-4848-8502-1224102f9d75-kube-api-access-zvb8b\") pod \"maas-api-revoke-sub-3d4156f1-lx7kz\" (UID: \"f2c93ac8-e7b7-4848-8502-1224102f9d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lx7kz" Nov 03 20:59:23.038141 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:23.038130 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c93ac8-e7b7-4848-8502-1224102f9d75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lx7kz\" (UID: \"f2c93ac8-e7b7-4848-8502-1224102f9d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lx7kz" Nov 03 20:59:23.139395 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:23.139364 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvb8b\" (UniqueName: \"kubernetes.io/projected/f2c93ac8-e7b7-4848-8502-1224102f9d75-kube-api-access-zvb8b\") pod \"maas-api-revoke-sub-3d4156f1-lx7kz\" (UID: \"f2c93ac8-e7b7-4848-8502-1224102f9d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lx7kz" Nov 03 20:59:23.139520 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:23.139409 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c93ac8-e7b7-4848-8502-1224102f9d75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lx7kz\" (UID: \"f2c93ac8-e7b7-4848-8502-1224102f9d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lx7kz" Nov 03 20:59:23.139520 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:23.139476 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c93ac8-e7b7-4848-8502-1224102f9d75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lx7kz\" (UID: \"f2c93ac8-e7b7-4848-8502-1224102f9d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lx7kz" Nov 03 20:59:23.140072 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:23.140049 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c93ac8-e7b7-4848-8502-1224102f9d75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lx7kz\" (UID: \"f2c93ac8-e7b7-4848-8502-1224102f9d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lx7kz" Nov 03 20:59:23.141916 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:23.141897 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c93ac8-e7b7-4848-8502-1224102f9d75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lx7kz\" (UID: \"f2c93ac8-e7b7-4848-8502-1224102f9d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lx7kz" Nov 03 20:59:23.151522 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:23.151504 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvb8b\" (UniqueName: \"kubernetes.io/projected/f2c93ac8-e7b7-4848-8502-1224102f9d75-kube-api-access-zvb8b\") pod \"maas-api-revoke-sub-3d4156f1-lx7kz\" (UID: \"f2c93ac8-e7b7-4848-8502-1224102f9d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lx7kz" Nov 03 20:59:23.311452 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:23.311387 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lx7kz" Nov 03 20:59:23.583270 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:23.583194 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lx7kz"] Nov 03 20:59:23.585741 ip-10-0-132-83 kubenswrapper[5059]: W1103 20:59:23.585716 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c93ac8_e7b7_4848_8502_1224102f9d75.slice/crio-270bb168458cb10f520001a3c449a7c74f1dfe5970d87d0aa02ade8ccf5b327a WatchSource:0}: Error finding container 270bb168458cb10f520001a3c449a7c74f1dfe5970d87d0aa02ade8ccf5b327a: Status 404 returned error can't find the container with id 270bb168458cb10f520001a3c449a7c74f1dfe5970d87d0aa02ade8ccf5b327a Nov 03 20:59:23.854754 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:23.854699 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de0866dc-1140-4a5a-9ee9-bba3f731cbfa" path="/var/lib/kubelet/pods/de0866dc-1140-4a5a-9ee9-bba3f731cbfa/volumes" Nov 03 20:59:24.264835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:24.264738 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lx7kz" event={"ID":"f2c93ac8-e7b7-4848-8502-1224102f9d75","Type":"ContainerStarted","Data":"cea5842a3f5d5e46a1691dd24c42f5fcdd80f6d128e58c6fe77431d7f7ecd9bd"} Nov 03 20:59:24.264835 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:24.264809 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lx7kz" event={"ID":"f2c93ac8-e7b7-4848-8502-1224102f9d75","Type":"ContainerStarted","Data":"270bb168458cb10f520001a3c449a7c74f1dfe5970d87d0aa02ade8ccf5b327a"} Nov 03 20:59:24.284175 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:24.284127 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lx7kz" podStartSLOduration=2.28411573 podStartE2EDuration="2.28411573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:24.283990856 +0000 UTC m=+4958.541788110" watchObservedRunningTime="2026-11-03 20:59:24.28411573 +0000 UTC m=+4958.541912986" Nov 03 20:59:25.271005 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:25.270961 5059 generic.go:356] "Generic (PLEG): container finished" podID="f2c93ac8-e7b7-4848-8502-1224102f9d75" containerID="cea5842a3f5d5e46a1691dd24c42f5fcdd80f6d128e58c6fe77431d7f7ecd9bd" exitCode=0 Nov 03 20:59:25.271471 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:25.271021 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lx7kz" event={"ID":"f2c93ac8-e7b7-4848-8502-1224102f9d75","Type":"ContainerDied","Data":"cea5842a3f5d5e46a1691dd24c42f5fcdd80f6d128e58c6fe77431d7f7ecd9bd"} Nov 03 20:59:27.454477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:27.454370 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lx7kz" event={"ID":"f2c93ac8-e7b7-4848-8502-1224102f9d75","Type":"ContainerDied","Data":"270bb168458cb10f520001a3c449a7c74f1dfe5970d87d0aa02ade8ccf5b327a"} Nov 03 20:59:27.454477 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:27.454440 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="270bb168458cb10f520001a3c449a7c74f1dfe5970d87d0aa02ade8ccf5b327a" Nov 03 20:59:27.456652 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:27.456627 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lx7kz" Nov 03 20:59:27.629685 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:27.626004 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c93ac8-e7b7-4848-8502-1224102f9d75-kube-api-access-zvb8b\" (UniqueName: \"kubernetes.io/projected/f2c93ac8-e7b7-4848-8502-1224102f9d75-kube-api-access-zvb8b\") pod \"f2c93ac8-e7b7-4848-8502-1224102f9d75\" (UID: \"f2c93ac8-e7b7-4848-8502-1224102f9d75\") " Nov 03 20:59:27.636699 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:27.636650 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c93ac8-e7b7-4848-8502-1224102f9d75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c93ac8-e7b7-4848-8502-1224102f9d75-maas-api-openshift-ca\") pod \"f2c93ac8-e7b7-4848-8502-1224102f9d75\" (UID: \"f2c93ac8-e7b7-4848-8502-1224102f9d75\") " Nov 03 20:59:27.636903 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:27.636882 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2c93ac8-e7b7-4848-8502-1224102f9d75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c93ac8-e7b7-4848-8502-1224102f9d75-maas-api-xks-ca\") pod \"f2c93ac8-e7b7-4848-8502-1224102f9d75\" (UID: \"f2c93ac8-e7b7-4848-8502-1224102f9d75\") " Nov 03 20:59:27.637931 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:27.637891 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c93ac8-e7b7-4848-8502-1224102f9d75-maas-api-openshift-ca" pod "f2c93ac8-e7b7-4848-8502-1224102f9d75" (UID: "f2c93ac8-e7b7-4848-8502-1224102f9d75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:27.642932 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:27.641881 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2c93ac8-e7b7-4848-8502-1224102f9d75-maas-api-xks-ca" pod "f2c93ac8-e7b7-4848-8502-1224102f9d75" (UID: "f2c93ac8-e7b7-4848-8502-1224102f9d75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:59:27.643189 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:27.643150 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c93ac8-e7b7-4848-8502-1224102f9d75-kube-api-access-zvb8b" pod "f2c93ac8-e7b7-4848-8502-1224102f9d75" (UID: "f2c93ac8-e7b7-4848-8502-1224102f9d75"). InnerVolumeSpecName "kube-api-access-zvb8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:27.738116 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:27.738064 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvb8b\" (UniqueName: \"kubernetes.io/projected/f2c93ac8-e7b7-4848-8502-1224102f9d75-kube-api-access-zvb8b\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:27.738116 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:27.738107 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c93ac8-e7b7-4848-8502-1224102f9d75-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:27.738382 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:27.738126 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c93ac8-e7b7-4848-8502-1224102f9d75-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:28.458120 ip-10-0-132-83 kubenswrapper[5059]: I1103 20:59:28.458087 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lx7kz" Nov 03 20:59:40.603493 ip-10-0-132-83 kubenswrapper[5059]: E1103 20:59:40.603457 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 21:00:10.603784 ip-10-0-132-83 kubenswrapper[5059]: E1103 21:00:10.603738 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 21:00:40.604013 ip-10-0-132-83 kubenswrapper[5059]: E1103 21:00:40.603972 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 21:01:07.267453 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:07.267336 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-sqxdz"] Nov 03 21:01:07.269086 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:07.269055 5059 cpu_manager.go:414] "containerMap: removing container" podUID="f2c93ac8-e7b7-4848-8502-1224102f9d75" containerName="revoke-keys" Nov 03 21:01:07.269249 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:07.269231 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c93ac8-e7b7-4848-8502-1224102f9d75" containerName="revoke-keys" Nov 03 21:01:07.269572 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:07.269552 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c93ac8-e7b7-4848-8502-1224102f9d75" containerName="revoke-keys" Nov 03 21:01:07.274838 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:07.274815 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-sqxdz" Nov 03 21:01:07.286382 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:07.285353 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\"" Nov 03 21:01:07.286382 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:07.285848 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:01:07.292767 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:07.292740 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-sqxdz"] Nov 03 21:01:07.372636 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:07.372595 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3a6708a-5a51-4e05-b9ef-3efd43cafb3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-sqxdz\" (UID: \"b3a6708a-5a51-4e05-b9ef-3efd43cafb3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-sqxdz" Nov 03 21:01:07.373032 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:07.373002 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3a6708a-5a51-4e05-b9ef-3efd43cafb3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-sqxdz\" (UID: \"b3a6708a-5a51-4e05-b9ef-3efd43cafb3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-sqxdz" Nov 03 21:01:07.373262 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:07.373242 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p96mg\" (UniqueName: \"kubernetes.io/projected/b3a6708a-5a51-4e05-b9ef-3efd43cafb3b-kube-api-access-p96mg\") pod \"maas-api-revoke-sub-d576e68b-sqxdz\" (UID: \"b3a6708a-5a51-4e05-b9ef-3efd43cafb3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-sqxdz" Nov 03 21:01:07.479192 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:07.479149 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3a6708a-5a51-4e05-b9ef-3efd43cafb3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-sqxdz\" (UID: \"b3a6708a-5a51-4e05-b9ef-3efd43cafb3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-sqxdz" Nov 03 21:01:07.480492 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:07.480466 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3a6708a-5a51-4e05-b9ef-3efd43cafb3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-sqxdz\" (UID: \"b3a6708a-5a51-4e05-b9ef-3efd43cafb3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-sqxdz" Nov 03 21:01:07.481168 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:07.481147 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p96mg\" (UniqueName: \"kubernetes.io/projected/b3a6708a-5a51-4e05-b9ef-3efd43cafb3b-kube-api-access-p96mg\") pod \"maas-api-revoke-sub-d576e68b-sqxdz\" (UID: \"b3a6708a-5a51-4e05-b9ef-3efd43cafb3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-sqxdz" Nov 03 21:01:07.481605 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:07.480279 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3a6708a-5a51-4e05-b9ef-3efd43cafb3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-sqxdz\" (UID: \"b3a6708a-5a51-4e05-b9ef-3efd43cafb3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-sqxdz" Nov 03 21:01:07.498379 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:07.498329 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3a6708a-5a51-4e05-b9ef-3efd43cafb3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-sqxdz\" (UID: \"b3a6708a-5a51-4e05-b9ef-3efd43cafb3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-sqxdz" Nov 03 21:01:07.511498 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:07.511448 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p96mg\" (UniqueName: \"kubernetes.io/projected/b3a6708a-5a51-4e05-b9ef-3efd43cafb3b-kube-api-access-p96mg\") pod \"maas-api-revoke-sub-d576e68b-sqxdz\" (UID: \"b3a6708a-5a51-4e05-b9ef-3efd43cafb3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-sqxdz" Nov 03 21:01:07.650319 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:07.650294 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-sqxdz" Nov 03 21:01:07.955801 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:07.955759 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-sqxdz"] Nov 03 21:01:07.956919 ip-10-0-132-83 kubenswrapper[5059]: W1103 21:01:07.956891 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a6708a_5a51_4e05_b9ef_3efd43cafb3b.slice/crio-b09c9223fa41a1a26b3e92ec8b55fcbe1327aef4d77c1f294db57891b1bae682 WatchSource:0}: Error finding container b09c9223fa41a1a26b3e92ec8b55fcbe1327aef4d77c1f294db57891b1bae682: Status 404 returned error can't find the container with id b09c9223fa41a1a26b3e92ec8b55fcbe1327aef4d77c1f294db57891b1bae682 Nov 03 21:01:08.042051 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:08.042022 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-sqxdz" event={"ID":"b3a6708a-5a51-4e05-b9ef-3efd43cafb3b","Type":"ContainerStarted","Data":"b09c9223fa41a1a26b3e92ec8b55fcbe1327aef4d77c1f294db57891b1bae682"} Nov 03 21:01:09.048425 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:09.048336 5059 generic.go:356] "Generic (PLEG): container finished" podID="b3a6708a-5a51-4e05-b9ef-3efd43cafb3b" containerID="464f3f237d06ad54eafc809137354044479658f7d33d041f1eed8cd13845fd20" exitCode=0 Nov 03 21:01:09.048425 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:09.048417 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-sqxdz" event={"ID":"b3a6708a-5a51-4e05-b9ef-3efd43cafb3b","Type":"ContainerDied","Data":"464f3f237d06ad54eafc809137354044479658f7d33d041f1eed8cd13845fd20"} Nov 03 21:01:10.179627 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:10.179604 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-sqxdz" Nov 03 21:01:10.309772 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:10.309687 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3a6708a-5a51-4e05-b9ef-3efd43cafb3b-kube-api-access-p96mg\" (UniqueName: \"kubernetes.io/projected/b3a6708a-5a51-4e05-b9ef-3efd43cafb3b-kube-api-access-p96mg\") pod \"b3a6708a-5a51-4e05-b9ef-3efd43cafb3b\" (UID: \"b3a6708a-5a51-4e05-b9ef-3efd43cafb3b\") " Nov 03 21:01:10.309940 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:10.309768 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3a6708a-5a51-4e05-b9ef-3efd43cafb3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3a6708a-5a51-4e05-b9ef-3efd43cafb3b-maas-api-openshift-ca\") pod \"b3a6708a-5a51-4e05-b9ef-3efd43cafb3b\" (UID: \"b3a6708a-5a51-4e05-b9ef-3efd43cafb3b\") " Nov 03 21:01:10.309940 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:10.309856 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3a6708a-5a51-4e05-b9ef-3efd43cafb3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3a6708a-5a51-4e05-b9ef-3efd43cafb3b-maas-api-xks-ca\") pod \"b3a6708a-5a51-4e05-b9ef-3efd43cafb3b\" (UID: \"b3a6708a-5a51-4e05-b9ef-3efd43cafb3b\") " Nov 03 21:01:10.310129 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:10.310106 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a6708a-5a51-4e05-b9ef-3efd43cafb3b-maas-api-openshift-ca" pod "b3a6708a-5a51-4e05-b9ef-3efd43cafb3b" (UID: "b3a6708a-5a51-4e05-b9ef-3efd43cafb3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:10.311904 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:10.311874 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3a6708a-5a51-4e05-b9ef-3efd43cafb3b-maas-api-xks-ca" pod "b3a6708a-5a51-4e05-b9ef-3efd43cafb3b" (UID: "b3a6708a-5a51-4e05-b9ef-3efd43cafb3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:01:10.312015 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:10.311929 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3a6708a-5a51-4e05-b9ef-3efd43cafb3b-kube-api-access-p96mg" pod "b3a6708a-5a51-4e05-b9ef-3efd43cafb3b" (UID: "b3a6708a-5a51-4e05-b9ef-3efd43cafb3b"). InnerVolumeSpecName "kube-api-access-p96mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:10.410733 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:10.410696 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3a6708a-5a51-4e05-b9ef-3efd43cafb3b-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:10.410733 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:10.410733 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3a6708a-5a51-4e05-b9ef-3efd43cafb3b-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:10.410733 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:10.410744 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p96mg\" (UniqueName: \"kubernetes.io/projected/b3a6708a-5a51-4e05-b9ef-3efd43cafb3b-kube-api-access-p96mg\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:10.602590 ip-10-0-132-83 kubenswrapper[5059]: E1103 21:01:10.602557 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 21:01:11.058021 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:11.057940 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-sqxdz" Nov 03 21:01:11.058021 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:11.057965 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-sqxdz" event={"ID":"b3a6708a-5a51-4e05-b9ef-3efd43cafb3b","Type":"ContainerDied","Data":"b09c9223fa41a1a26b3e92ec8b55fcbe1327aef4d77c1f294db57891b1bae682"} Nov 03 21:01:11.058021 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:11.058016 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b09c9223fa41a1a26b3e92ec8b55fcbe1327aef4d77c1f294db57891b1bae682" Nov 03 21:01:23.648624 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:23.648503 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhcx7"] Nov 03 21:01:23.649178 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:23.649114 5059 cpu_manager.go:414] "containerMap: removing container" podUID="b3a6708a-5a51-4e05-b9ef-3efd43cafb3b" containerName="revoke-keys" Nov 03 21:01:23.649178 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:23.649154 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3a6708a-5a51-4e05-b9ef-3efd43cafb3b" containerName="revoke-keys" Nov 03 21:01:23.649317 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:23.649259 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3a6708a-5a51-4e05-b9ef-3efd43cafb3b" containerName="revoke-keys" Nov 03 21:01:23.653203 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:23.653174 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhcx7" Nov 03 21:01:23.670751 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:23.666221 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhcx7"] Nov 03 21:01:23.670751 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:23.666378 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\"" Nov 03 21:01:23.670751 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:23.666412 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:01:23.787481 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:23.787440 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c0152d-bca8-4995-b47b-d8aa1762b145-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nhcx7\" (UID: \"d6c0152d-bca8-4995-b47b-d8aa1762b145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhcx7" Nov 03 21:01:23.787687 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:23.787645 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c0152d-bca8-4995-b47b-d8aa1762b145-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nhcx7\" (UID: \"d6c0152d-bca8-4995-b47b-d8aa1762b145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhcx7" Nov 03 21:01:23.787926 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:23.787709 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9mg9\" (UniqueName: \"kubernetes.io/projected/d6c0152d-bca8-4995-b47b-d8aa1762b145-kube-api-access-h9mg9\") pod \"maas-api-revoke-sub-d2bc93bc-nhcx7\" (UID: \"d6c0152d-bca8-4995-b47b-d8aa1762b145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhcx7" Nov 03 21:01:23.888759 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:23.888722 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c0152d-bca8-4995-b47b-d8aa1762b145-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nhcx7\" (UID: \"d6c0152d-bca8-4995-b47b-d8aa1762b145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhcx7" Nov 03 21:01:23.888924 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:23.888763 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9mg9\" (UniqueName: \"kubernetes.io/projected/d6c0152d-bca8-4995-b47b-d8aa1762b145-kube-api-access-h9mg9\") pod \"maas-api-revoke-sub-d2bc93bc-nhcx7\" (UID: \"d6c0152d-bca8-4995-b47b-d8aa1762b145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhcx7" Nov 03 21:01:23.888924 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:23.888815 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c0152d-bca8-4995-b47b-d8aa1762b145-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nhcx7\" (UID: \"d6c0152d-bca8-4995-b47b-d8aa1762b145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhcx7" Nov 03 21:01:23.889644 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:23.889617 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c0152d-bca8-4995-b47b-d8aa1762b145-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nhcx7\" (UID: \"d6c0152d-bca8-4995-b47b-d8aa1762b145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhcx7" Nov 03 21:01:23.891340 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:23.891309 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c0152d-bca8-4995-b47b-d8aa1762b145-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nhcx7\" (UID: \"d6c0152d-bca8-4995-b47b-d8aa1762b145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhcx7" Nov 03 21:01:23.902179 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:23.902118 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9mg9\" (UniqueName: \"kubernetes.io/projected/d6c0152d-bca8-4995-b47b-d8aa1762b145-kube-api-access-h9mg9\") pod \"maas-api-revoke-sub-d2bc93bc-nhcx7\" (UID: \"d6c0152d-bca8-4995-b47b-d8aa1762b145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhcx7" Nov 03 21:01:24.002819 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:24.002792 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhcx7" Nov 03 21:01:24.280925 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:24.280901 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhcx7"] Nov 03 21:01:24.283066 ip-10-0-132-83 kubenswrapper[5059]: W1103 21:01:24.283032 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c0152d_bca8_4995_b47b_d8aa1762b145.slice/crio-5cb0b4dab2dd5ede2024288319ca40854c9fc7f7abcac9a286f7cd194225b0e2 WatchSource:0}: Error finding container 5cb0b4dab2dd5ede2024288319ca40854c9fc7f7abcac9a286f7cd194225b0e2: Status 404 returned error can't find the container with id 5cb0b4dab2dd5ede2024288319ca40854c9fc7f7abcac9a286f7cd194225b0e2 Nov 03 21:01:25.144469 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:25.144437 5059 generic.go:356] "Generic (PLEG): container finished" podID="d6c0152d-bca8-4995-b47b-d8aa1762b145" containerID="f21eaffaecfee3b712cad7fa07a09b1c4a99ab589033c70df0dd9335f944f774" exitCode=0 Nov 03 21:01:25.144845 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:25.144525 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhcx7" event={"ID":"d6c0152d-bca8-4995-b47b-d8aa1762b145","Type":"ContainerDied","Data":"f21eaffaecfee3b712cad7fa07a09b1c4a99ab589033c70df0dd9335f944f774"} Nov 03 21:01:25.144845 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:25.144577 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhcx7" event={"ID":"d6c0152d-bca8-4995-b47b-d8aa1762b145","Type":"ContainerStarted","Data":"5cb0b4dab2dd5ede2024288319ca40854c9fc7f7abcac9a286f7cd194225b0e2"} Nov 03 21:01:26.280540 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:26.280511 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhcx7" Nov 03 21:01:26.417636 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:26.417563 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6c0152d-bca8-4995-b47b-d8aa1762b145-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c0152d-bca8-4995-b47b-d8aa1762b145-maas-api-xks-ca\") pod \"d6c0152d-bca8-4995-b47b-d8aa1762b145\" (UID: \"d6c0152d-bca8-4995-b47b-d8aa1762b145\") " Nov 03 21:01:26.417636 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:26.417621 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c0152d-bca8-4995-b47b-d8aa1762b145-kube-api-access-h9mg9\" (UniqueName: \"kubernetes.io/projected/d6c0152d-bca8-4995-b47b-d8aa1762b145-kube-api-access-h9mg9\") pod \"d6c0152d-bca8-4995-b47b-d8aa1762b145\" (UID: \"d6c0152d-bca8-4995-b47b-d8aa1762b145\") " Nov 03 21:01:26.417861 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:26.417742 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c0152d-bca8-4995-b47b-d8aa1762b145-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c0152d-bca8-4995-b47b-d8aa1762b145-maas-api-openshift-ca\") pod \"d6c0152d-bca8-4995-b47b-d8aa1762b145\" (UID: \"d6c0152d-bca8-4995-b47b-d8aa1762b145\") " Nov 03 21:01:26.418120 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:26.418095 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c0152d-bca8-4995-b47b-d8aa1762b145-maas-api-openshift-ca" pod "d6c0152d-bca8-4995-b47b-d8aa1762b145" (UID: "d6c0152d-bca8-4995-b47b-d8aa1762b145"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:26.419761 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:26.419729 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c0152d-bca8-4995-b47b-d8aa1762b145-kube-api-access-h9mg9" pod "d6c0152d-bca8-4995-b47b-d8aa1762b145" (UID: "d6c0152d-bca8-4995-b47b-d8aa1762b145"). InnerVolumeSpecName "kube-api-access-h9mg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:26.419988 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:26.419962 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6c0152d-bca8-4995-b47b-d8aa1762b145-maas-api-xks-ca" pod "d6c0152d-bca8-4995-b47b-d8aa1762b145" (UID: "d6c0152d-bca8-4995-b47b-d8aa1762b145"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:01:26.518870 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:26.518835 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c0152d-bca8-4995-b47b-d8aa1762b145-maas-api-openshift-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:26.518870 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:26.518873 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c0152d-bca8-4995-b47b-d8aa1762b145-maas-api-xks-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:26.519013 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:26.518883 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9mg9\" (UniqueName: \"kubernetes.io/projected/d6c0152d-bca8-4995-b47b-d8aa1762b145-kube-api-access-h9mg9\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:27.154206 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:27.154172 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhcx7" Nov 03 21:01:27.154396 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:27.154200 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhcx7" event={"ID":"d6c0152d-bca8-4995-b47b-d8aa1762b145","Type":"ContainerDied","Data":"5cb0b4dab2dd5ede2024288319ca40854c9fc7f7abcac9a286f7cd194225b0e2"} Nov 03 21:01:27.154396 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:27.154249 5059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cb0b4dab2dd5ede2024288319ca40854c9fc7f7abcac9a286f7cd194225b0e2" Nov 03 21:01:35.541597 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.541553 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz"] Nov 03 21:01:35.542008 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.541976 5059 cpu_manager.go:414] "containerMap: removing container" podUID="d6c0152d-bca8-4995-b47b-d8aa1762b145" containerName="revoke-keys" Nov 03 21:01:35.542008 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.541991 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c0152d-bca8-4995-b47b-d8aa1762b145" containerName="revoke-keys" Nov 03 21:01:35.542081 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.542062 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c0152d-bca8-4995-b47b-d8aa1762b145" containerName="revoke-keys" Nov 03 21:01:35.544837 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.544819 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz" Nov 03 21:01:35.552830 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.552799 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-942q2\"" Nov 03 21:01:35.552933 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.552889 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 21:01:35.554479 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.554459 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 21:01:35.557534 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.557511 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz"] Nov 03 21:01:35.559701 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.559673 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c4b55fdfd-z6js5"] Nov 03 21:01:35.568833 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.564969 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-z6js5" Nov 03 21:01:35.576311 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.576280 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c4b55fdfd-z6js5"] Nov 03 21:01:35.699895 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.699863 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54c54e73-626a-4cf7-903a-eedece4aa64b-openshift-service-ca\") pod \"maas-api-5c4b55fdfd-z6js5\" (UID: \"54c54e73-626a-4cf7-903a-eedece4aa64b\") " pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-z6js5" Nov 03 21:01:35.700053 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.699904 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-maas-api-tls\") pod \"maas-api-7cd7866f98-hc2mz\" (UID: \"3b296bfc-4ec0-4b16-bc7f-4f47571422d0\") " pod="odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz" Nov 03 21:01:35.700053 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.699931 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-metrics-certs\") pod \"maas-api-7cd7866f98-hc2mz\" (UID: \"3b296bfc-4ec0-4b16-bc7f-4f47571422d0\") " pod="odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz" Nov 03 21:01:35.700136 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.700077 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54c54e73-626a-4cf7-903a-eedece4aa64b-maas-api-tls\") pod \"maas-api-5c4b55fdfd-z6js5\" (UID: \"54c54e73-626a-4cf7-903a-eedece4aa64b\") " pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-z6js5" Nov 03 21:01:35.700136 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.700107 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lchfg\" (UniqueName: \"kubernetes.io/projected/54c54e73-626a-4cf7-903a-eedece4aa64b-kube-api-access-lchfg\") pod \"maas-api-5c4b55fdfd-z6js5\" (UID: \"54c54e73-626a-4cf7-903a-eedece4aa64b\") " pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-z6js5" Nov 03 21:01:35.700136 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.700130 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-openshift-service-ca\") pod \"maas-api-7cd7866f98-hc2mz\" (UID: \"3b296bfc-4ec0-4b16-bc7f-4f47571422d0\") " pod="odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz" Nov 03 21:01:35.700256 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.700243 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6msb\" (UniqueName: \"kubernetes.io/projected/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-kube-api-access-c6msb\") pod \"maas-api-7cd7866f98-hc2mz\" (UID: \"3b296bfc-4ec0-4b16-bc7f-4f47571422d0\") " pod="odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz" Nov 03 21:01:35.700302 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.700270 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54c54e73-626a-4cf7-903a-eedece4aa64b-metrics-certs\") pod \"maas-api-5c4b55fdfd-z6js5\" (UID: \"54c54e73-626a-4cf7-903a-eedece4aa64b\") " pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-z6js5" Nov 03 21:01:35.801467 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.801389 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54c54e73-626a-4cf7-903a-eedece4aa64b-maas-api-tls\") pod \"maas-api-5c4b55fdfd-z6js5\" (UID: \"54c54e73-626a-4cf7-903a-eedece4aa64b\") " pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-z6js5" Nov 03 21:01:35.801467 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.801423 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lchfg\" (UniqueName: \"kubernetes.io/projected/54c54e73-626a-4cf7-903a-eedece4aa64b-kube-api-access-lchfg\") pod \"maas-api-5c4b55fdfd-z6js5\" (UID: \"54c54e73-626a-4cf7-903a-eedece4aa64b\") " pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-z6js5" Nov 03 21:01:35.801467 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.801445 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-openshift-service-ca\") pod \"maas-api-7cd7866f98-hc2mz\" (UID: \"3b296bfc-4ec0-4b16-bc7f-4f47571422d0\") " pod="odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz" Nov 03 21:01:35.801777 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.801484 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6msb\" (UniqueName: \"kubernetes.io/projected/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-kube-api-access-c6msb\") pod \"maas-api-7cd7866f98-hc2mz\" (UID: \"3b296bfc-4ec0-4b16-bc7f-4f47571422d0\") " pod="odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz" Nov 03 21:01:35.801777 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.801505 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54c54e73-626a-4cf7-903a-eedece4aa64b-metrics-certs\") pod \"maas-api-5c4b55fdfd-z6js5\" (UID: \"54c54e73-626a-4cf7-903a-eedece4aa64b\") " pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-z6js5" Nov 03 21:01:35.801777 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.801542 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54c54e73-626a-4cf7-903a-eedece4aa64b-openshift-service-ca\") pod \"maas-api-5c4b55fdfd-z6js5\" (UID: \"54c54e73-626a-4cf7-903a-eedece4aa64b\") " pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-z6js5" Nov 03 21:01:35.801777 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.801567 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-maas-api-tls\") pod \"maas-api-7cd7866f98-hc2mz\" (UID: \"3b296bfc-4ec0-4b16-bc7f-4f47571422d0\") " pod="odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz" Nov 03 21:01:35.801777 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.801591 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-metrics-certs\") pod \"maas-api-7cd7866f98-hc2mz\" (UID: \"3b296bfc-4ec0-4b16-bc7f-4f47571422d0\") " pod="odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz" Nov 03 21:01:35.802295 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.802267 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-openshift-service-ca\") pod \"maas-api-7cd7866f98-hc2mz\" (UID: \"3b296bfc-4ec0-4b16-bc7f-4f47571422d0\") " pod="odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz" Nov 03 21:01:35.802420 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.802385 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54c54e73-626a-4cf7-903a-eedece4aa64b-openshift-service-ca\") pod \"maas-api-5c4b55fdfd-z6js5\" (UID: \"54c54e73-626a-4cf7-903a-eedece4aa64b\") " pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-z6js5" Nov 03 21:01:35.804261 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.804233 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-metrics-certs\") pod \"maas-api-7cd7866f98-hc2mz\" (UID: \"3b296bfc-4ec0-4b16-bc7f-4f47571422d0\") " pod="odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz" Nov 03 21:01:35.804360 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.804246 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54c54e73-626a-4cf7-903a-eedece4aa64b-maas-api-tls\") pod \"maas-api-5c4b55fdfd-z6js5\" (UID: \"54c54e73-626a-4cf7-903a-eedece4aa64b\") " pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-z6js5" Nov 03 21:01:35.804593 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.804559 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-maas-api-tls\") pod \"maas-api-7cd7866f98-hc2mz\" (UID: \"3b296bfc-4ec0-4b16-bc7f-4f47571422d0\") " pod="odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz" Nov 03 21:01:35.804756 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.804738 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54c54e73-626a-4cf7-903a-eedece4aa64b-metrics-certs\") pod \"maas-api-5c4b55fdfd-z6js5\" (UID: \"54c54e73-626a-4cf7-903a-eedece4aa64b\") " pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-z6js5" Nov 03 21:01:35.813450 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.813424 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6msb\" (UniqueName: \"kubernetes.io/projected/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-kube-api-access-c6msb\") pod \"maas-api-7cd7866f98-hc2mz\" (UID: \"3b296bfc-4ec0-4b16-bc7f-4f47571422d0\") " pod="odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz" Nov 03 21:01:35.813560 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.813431 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lchfg\" (UniqueName: \"kubernetes.io/projected/54c54e73-626a-4cf7-903a-eedece4aa64b-kube-api-access-lchfg\") pod \"maas-api-5c4b55fdfd-z6js5\" (UID: \"54c54e73-626a-4cf7-903a-eedece4aa64b\") " pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-z6js5" Nov 03 21:01:35.858160 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.858127 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz" Nov 03 21:01:35.885313 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:35.885287 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-z6js5" Nov 03 21:01:36.139358 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:36.139326 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz"] Nov 03 21:01:36.141506 ip-10-0-132-83 kubenswrapper[5059]: W1103 21:01:36.141471 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b296bfc_4ec0_4b16_bc7f_4f47571422d0.slice/crio-733dcf751602290d0bf7d3794e4ef2ec38391391b711b3817866c4860915a79c WatchSource:0}: Error finding container 733dcf751602290d0bf7d3794e4ef2ec38391391b711b3817866c4860915a79c: Status 404 returned error can't find the container with id 733dcf751602290d0bf7d3794e4ef2ec38391391b711b3817866c4860915a79c Nov 03 21:01:36.173076 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:36.173048 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c4b55fdfd-z6js5"] Nov 03 21:01:36.174704 ip-10-0-132-83 kubenswrapper[5059]: W1103 21:01:36.174676 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c54e73_626a_4cf7_903a_eedece4aa64b.slice/crio-dd3443846924d6e8313400ad52aa655d9235e9bf7d26b599fdde49e3fdb148f6 WatchSource:0}: Error finding container dd3443846924d6e8313400ad52aa655d9235e9bf7d26b599fdde49e3fdb148f6: Status 404 returned error can't find the container with id dd3443846924d6e8313400ad52aa655d9235e9bf7d26b599fdde49e3fdb148f6 Nov 03 21:01:36.199060 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:36.199028 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz" event={"ID":"3b296bfc-4ec0-4b16-bc7f-4f47571422d0","Type":"ContainerStarted","Data":"733dcf751602290d0bf7d3794e4ef2ec38391391b711b3817866c4860915a79c"} Nov 03 21:01:36.200110 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:36.200089 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-z6js5" event={"ID":"54c54e73-626a-4cf7-903a-eedece4aa64b","Type":"ContainerStarted","Data":"dd3443846924d6e8313400ad52aa655d9235e9bf7d26b599fdde49e3fdb148f6"} Nov 03 21:01:37.206112 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:37.206073 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-z6js5" event={"ID":"54c54e73-626a-4cf7-903a-eedece4aa64b","Type":"ContainerStarted","Data":"93e556960bf1b21eef75c8f1a5e9e2e3315ed24afa1d91370f8011c6fa07265f"} Nov 03 21:01:37.206509 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:37.206157 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-z6js5" Nov 03 21:01:37.207579 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:37.207557 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz" event={"ID":"3b296bfc-4ec0-4b16-bc7f-4f47571422d0","Type":"ContainerStarted","Data":"b91ca2cd57f87f48fd6365b955db7c47be46916acdf5241e2a5a19c8bba24b85"} Nov 03 21:01:37.207711 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:37.207695 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz" Nov 03 21:01:37.227778 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:37.227726 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-z6js5" podStartSLOduration=1.745424987 podStartE2EDuration="2.2277138s" totalImagesPullingTime="482.288813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:37.226164564 +0000 UTC m=+5091.483961813" watchObservedRunningTime="2026-11-03 21:01:37.2277138 +0000 UTC m=+5091.485511056" Nov 03 21:01:37.247494 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:37.247448 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz" podStartSLOduration=1.77450541 podStartE2EDuration="2.247436306s" totalImagesPullingTime="472.930896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:37.246802968 +0000 UTC m=+5091.504600226" watchObservedRunningTime="2026-11-03 21:01:37.247436306 +0000 UTC m=+5091.505233563" Nov 03 21:01:40.604060 ip-10-0-132-83 kubenswrapper[5059]: E1103 21:01:40.604018 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 21:01:43.217799 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:43.217765 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz" Nov 03 21:01:43.218239 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:43.218207 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-z6js5" Nov 03 21:01:43.292242 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:43.292203 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz"] Nov 03 21:01:43.292419 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:43.292392 5059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz" podUID="3b296bfc-4ec0-4b16-bc7f-4f47571422d0" containerName="maas-api" containerID="cri-o://b91ca2cd57f87f48fd6365b955db7c47be46916acdf5241e2a5a19c8bba24b85" gracePeriod=30 Nov 03 21:01:43.542313 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:43.542289 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz" Nov 03 21:01:43.574840 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:43.574809 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-metrics-certs\") pod \"3b296bfc-4ec0-4b16-bc7f-4f47571422d0\" (UID: \"3b296bfc-4ec0-4b16-bc7f-4f47571422d0\") " Nov 03 21:01:43.574963 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:43.574871 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-kube-api-access-c6msb\" (UniqueName: \"kubernetes.io/projected/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-kube-api-access-c6msb\") pod \"3b296bfc-4ec0-4b16-bc7f-4f47571422d0\" (UID: \"3b296bfc-4ec0-4b16-bc7f-4f47571422d0\") " Nov 03 21:01:43.575027 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:43.574959 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-openshift-service-ca\") pod \"3b296bfc-4ec0-4b16-bc7f-4f47571422d0\" (UID: \"3b296bfc-4ec0-4b16-bc7f-4f47571422d0\") " Nov 03 21:01:43.575027 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:43.574990 5059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-maas-api-tls\") pod \"3b296bfc-4ec0-4b16-bc7f-4f47571422d0\" (UID: \"3b296bfc-4ec0-4b16-bc7f-4f47571422d0\") " Nov 03 21:01:43.575417 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:43.575388 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-openshift-service-ca" pod "3b296bfc-4ec0-4b16-bc7f-4f47571422d0" (UID: "3b296bfc-4ec0-4b16-bc7f-4f47571422d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:43.577041 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:43.577014 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-kube-api-access-c6msb" pod "3b296bfc-4ec0-4b16-bc7f-4f47571422d0" (UID: "3b296bfc-4ec0-4b16-bc7f-4f47571422d0"). InnerVolumeSpecName "kube-api-access-c6msb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:43.577426 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:43.577403 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-maas-api-tls" pod "3b296bfc-4ec0-4b16-bc7f-4f47571422d0" (UID: "3b296bfc-4ec0-4b16-bc7f-4f47571422d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:01:43.579755 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:43.579736 5059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-metrics-certs" pod "3b296bfc-4ec0-4b16-bc7f-4f47571422d0" (UID: "3b296bfc-4ec0-4b16-bc7f-4f47571422d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:01:43.676124 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:43.676096 5059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-openshift-service-ca\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:43.676124 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:43.676125 5059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-maas-api-tls\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:43.676290 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:43.676135 5059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-metrics-certs\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:43.676290 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:43.676144 5059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6msb\" (UniqueName: \"kubernetes.io/projected/3b296bfc-4ec0-4b16-bc7f-4f47571422d0-kube-api-access-c6msb\") on node \"ip-10-0-132-83.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:44.242291 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:44.242193 5059 generic.go:356] "Generic (PLEG): container finished" podID="3b296bfc-4ec0-4b16-bc7f-4f47571422d0" containerID="b91ca2cd57f87f48fd6365b955db7c47be46916acdf5241e2a5a19c8bba24b85" exitCode=0 Nov 03 21:01:44.242291 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:44.242241 5059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz" Nov 03 21:01:44.242291 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:44.242277 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz" event={"ID":"3b296bfc-4ec0-4b16-bc7f-4f47571422d0","Type":"ContainerDied","Data":"b91ca2cd57f87f48fd6365b955db7c47be46916acdf5241e2a5a19c8bba24b85"} Nov 03 21:01:44.242795 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:44.242331 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz" event={"ID":"3b296bfc-4ec0-4b16-bc7f-4f47571422d0","Type":"ContainerDied","Data":"733dcf751602290d0bf7d3794e4ef2ec38391391b711b3817866c4860915a79c"} Nov 03 21:01:44.242795 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:44.242353 5059 scope.go:122] "RemoveContainer" containerID="b91ca2cd57f87f48fd6365b955db7c47be46916acdf5241e2a5a19c8bba24b85" Nov 03 21:01:44.250940 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:44.250921 5059 scope.go:122] "RemoveContainer" containerID="b91ca2cd57f87f48fd6365b955db7c47be46916acdf5241e2a5a19c8bba24b85" Nov 03 21:01:44.251222 ip-10-0-132-83 kubenswrapper[5059]: E1103 21:01:44.251195 5059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b91ca2cd57f87f48fd6365b955db7c47be46916acdf5241e2a5a19c8bba24b85\": container with ID starting with b91ca2cd57f87f48fd6365b955db7c47be46916acdf5241e2a5a19c8bba24b85 not found: ID does not exist" containerID="b91ca2cd57f87f48fd6365b955db7c47be46916acdf5241e2a5a19c8bba24b85" Nov 03 21:01:44.251322 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:44.251230 5059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b91ca2cd57f87f48fd6365b955db7c47be46916acdf5241e2a5a19c8bba24b85"} err="failed to get container status \"b91ca2cd57f87f48fd6365b955db7c47be46916acdf5241e2a5a19c8bba24b85\": rpc error: code = NotFound desc = could not find container \"b91ca2cd57f87f48fd6365b955db7c47be46916acdf5241e2a5a19c8bba24b85\": container with ID starting with b91ca2cd57f87f48fd6365b955db7c47be46916acdf5241e2a5a19c8bba24b85 not found: ID does not exist" Nov 03 21:01:44.260675 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:44.260640 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz"] Nov 03 21:01:44.264537 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:44.264515 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cd7866f98-hc2mz"] Nov 03 21:01:45.855921 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:01:45.855890 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b296bfc-4ec0-4b16-bc7f-4f47571422d0" path="/var/lib/kubelet/pods/3b296bfc-4ec0-4b16-bc7f-4f47571422d0/volumes" Nov 03 21:02:10.603370 ip-10-0-132-83 kubenswrapper[5059]: E1103 21:02:10.603253 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 21:02:40.320827 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:02:40.320779 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wtlgc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:02:40.321537 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:02:40.321480 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wtlgc" podUID="ab375b16-1371-437f-91b4-d5f8580e804d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:40.711164 ip-10-0-132-83 kubenswrapper[5059]: E1103 21:02:40.711064 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 21:03:10.603037 ip-10-0-132-83 kubenswrapper[5059]: E1103 21:03:10.602995 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 21:03:40.238640 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:03:40.238589 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8gzjz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:03:40.239059 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:03:40.238649 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8gzjz" podUID="6462b9bb-1018-4029-922d-eb71527fc301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:03:40.603895 ip-10-0-132-83 kubenswrapper[5059]: E1103 21:03:40.603859 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 21:04:10.618463 ip-10-0-132-83 kubenswrapper[5059]: E1103 21:04:10.618377 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 21:04:28.025869 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:04:28.025613 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lx7kz"] Nov 03 21:04:28.027896 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:04:28.027869 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lx7kz"] Nov 03 21:04:29.861787 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:04:29.861749 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c93ac8-e7b7-4848-8502-1224102f9d75" path="/var/lib/kubelet/pods/f2c93ac8-e7b7-4848-8502-1224102f9d75/volumes" Nov 03 21:04:40.808904 ip-10-0-132-83 kubenswrapper[5059]: E1103 21:04:40.808860 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 21:05:10.713075 ip-10-0-132-83 kubenswrapper[5059]: E1103 21:05:10.713029 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 21:05:27.011534 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:05:27.011502 5059 scope.go:122] "RemoveContainer" containerID="cea5842a3f5d5e46a1691dd24c42f5fcdd80f6d128e58c6fe77431d7f7ecd9bd" Nov 03 21:05:32.378688 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:05:32.374831 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-5sk9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:05:32.378688 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:05:32.374943 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" podUID="81fa410e-25d7-4800-a22b-9dea6ccca3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:05:40.620741 ip-10-0-132-83 kubenswrapper[5059]: E1103 21:05:40.620695 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 21:06:02.381849 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:06:02.381804 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-5sk9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:06:02.382560 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:06:02.382507 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" podUID="81fa410e-25d7-4800-a22b-9dea6ccca3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:10.637373 ip-10-0-132-83 kubenswrapper[5059]: E1103 21:06:10.637326 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 21:06:11.030289 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:06:11.030191 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-sqxdz"] Nov 03 21:06:11.033209 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:06:11.033167 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-sqxdz"] Nov 03 21:06:11.939688 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:06:11.934234 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3a6708a-5a51-4e05-b9ef-3efd43cafb3b" path="/var/lib/kubelet/pods/b3a6708a-5a51-4e05-b9ef-3efd43cafb3b/volumes" Nov 03 21:06:27.022378 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:06:27.022341 5059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhcx7"] Nov 03 21:06:27.024718 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:06:27.024693 5059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nhcx7"] Nov 03 21:06:27.854653 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:06:27.854615 5059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6c0152d-bca8-4995-b47b-d8aa1762b145" path="/var/lib/kubelet/pods/d6c0152d-bca8-4995-b47b-d8aa1762b145/volumes" Nov 03 21:06:32.380688 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:06:32.379754 5059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-5sk9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:06:32.380688 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:06:32.379886 5059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-5sk9w" podUID="81fa410e-25d7-4800-a22b-9dea6ccca3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:40.604157 ip-10-0-132-83 kubenswrapper[5059]: E1103 21:06:40.604113 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 21:07:10.603534 ip-10-0-132-83 kubenswrapper[5059]: E1103 21:07:10.603429 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 21:07:27.036643 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:07:27.036569 5059 scope.go:122] "RemoveContainer" containerID="f21eaffaecfee3b712cad7fa07a09b1c4a99ab589033c70df0dd9335f944f774" Nov 03 21:07:27.045130 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:07:27.045108 5059 scope.go:122] "RemoveContainer" containerID="464f3f237d06ad54eafc809137354044479658f7d33d041f1eed8cd13845fd20" Nov 03 21:07:40.603351 ip-10-0-132-83 kubenswrapper[5059]: E1103 21:07:40.603312 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 21:07:42.040772 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:07:42.040741 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55c765799f-5qgk6_5a3ca718-8b83-4650-b941-3e29e6b80e00/manager/0.log" Nov 03 21:07:42.453554 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:07:42.453519 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-fzr95_191c7922-07dd-4c9c-89d2-d804f638854d/manager/0.log" Nov 03 21:07:43.787274 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:07:43.787242 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5c4b55fdfd-z6js5_54c54e73-626a-4cf7-903a-eedece4aa64b/maas-api/0.log" Nov 03 21:07:47.022374 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:07:47.022341 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-lz2v2_b4d89c5d-1730-4077-a442-d3924153133f/postgres/0.log" Nov 03 21:07:49.135102 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:07:49.135069 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-v5xhc_d50891bc-16c2-497d-8e00-b79ef492ab6e/manager/0.log" Nov 03 21:07:50.174317 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:07:50.174285 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dcrh4_f44d81a0-55a2-4021-b34f-3400deb30c82/manager/0.log" Nov 03 21:07:51.161546 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:07:51.161515 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c557dng2q_e5585d8d-d156-418c-ae87-012781bc0e8c/istio-proxy/0.log" Nov 03 21:07:51.917090 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:07:51.917064 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-5sk9w_81fa410e-25d7-4800-a22b-9dea6ccca3ea/istio-proxy/0.log" Nov 03 21:07:52.099998 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:07:52.099966 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-8gzjz_6462b9bb-1018-4029-922d-eb71527fc301/istio-proxy/0.log" Nov 03 21:07:52.288592 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:07:52.288530 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-cmtzs_a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08/istio-proxy/0.log" Nov 03 21:07:53.103721 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:07:53.103682 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-vkx59_d14046b2-dd71-42a3-b239-0160088faa0a/istio-proxy/0.log" Nov 03 21:07:53.295587 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:07:53.295560 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-wtlgc_ab375b16-1371-437f-91b4-d5f8580e804d/istio-proxy/0.log" Nov 03 21:07:53.873434 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:07:53.873403 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn_77396fff-1778-4f09-9e81-05e715fc015e/istio-proxy/0.log" Nov 03 21:07:54.061394 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:07:54.061361 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6dc578d85f-q59ph_cee2af62-5ba9-4f27-852b-25fb6b43c945/payload-pre-processing/0.log" Nov 03 21:07:55.747645 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:07:55.747613 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-wz2w4_d098ef77-addb-4077-b6d4-f0ae477409b8/main/0.log" Nov 03 21:07:56.298799 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:07:56.298765 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fx98r_60b2ec36-7e99-46e0-9671-1c500c402e45/main/0.log" Nov 03 21:07:56.483921 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:07:56.483891 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-mnmjc_89d29d9c-a3ca-4ebf-a3c1-ad2df96fb4b5/main/0.log" Nov 03 21:08:10.603554 ip-10-0-132-83 kubenswrapper[5059]: E1103 21:08:10.603515 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 21:08:17.096359 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.096316 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c46p7/must-gather-hfxph"] Nov 03 21:08:17.096825 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.096717 5059 cpu_manager.go:414] "containerMap: removing container" podUID="3b296bfc-4ec0-4b16-bc7f-4f47571422d0" containerName="maas-api" Nov 03 21:08:17.096825 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.096729 5059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b296bfc-4ec0-4b16-bc7f-4f47571422d0" containerName="maas-api" Nov 03 21:08:17.096825 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.096783 5059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b296bfc-4ec0-4b16-bc7f-4f47571422d0" containerName="maas-api" Nov 03 21:08:17.099629 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.099611 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c46p7/must-gather-hfxph" Nov 03 21:08:17.109148 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.109128 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c46p7\"/\"default-dockercfg-dfcj5\"" Nov 03 21:08:17.109148 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.109129 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c46p7\"/\"openshift-service-ca.crt\"" Nov 03 21:08:17.109291 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.109146 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c46p7\"/\"kube-root-ca.crt\"" Nov 03 21:08:17.125284 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.125256 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c46p7/must-gather-hfxph"] Nov 03 21:08:17.212803 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.205844 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/51de09cc-566f-43f0-9f45-72d616dbc216-must-gather-output\") pod \"must-gather-hfxph\" (UID: \"51de09cc-566f-43f0-9f45-72d616dbc216\") " pod="openshift-must-gather-c46p7/must-gather-hfxph" Nov 03 21:08:17.212803 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.206144 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsl69\" (UniqueName: \"kubernetes.io/projected/51de09cc-566f-43f0-9f45-72d616dbc216-kube-api-access-vsl69\") pod \"must-gather-hfxph\" (UID: \"51de09cc-566f-43f0-9f45-72d616dbc216\") " pod="openshift-must-gather-c46p7/must-gather-hfxph" Nov 03 21:08:17.212803 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.211014 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4hm9z/must-gather-t7948"] Nov 03 21:08:17.215905 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.215869 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4hm9z/must-gather-t7948" Nov 03 21:08:17.220308 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.220281 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4hm9z\"/\"openshift-service-ca.crt\"" Nov 03 21:08:17.220548 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.220526 5059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4hm9z\"/\"default-dockercfg-tdccn\"" Nov 03 21:08:17.220630 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.220613 5059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4hm9z\"/\"kube-root-ca.crt\"" Nov 03 21:08:17.221362 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.221345 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4hm9z/must-gather-t7948"] Nov 03 21:08:17.307156 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.307120 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f3c01be7-f303-487f-9997-a17873e128ac-must-gather-output\") pod \"must-gather-t7948\" (UID: \"f3c01be7-f303-487f-9997-a17873e128ac\") " pod="openshift-must-gather-4hm9z/must-gather-t7948" Nov 03 21:08:17.307315 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.307162 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsl69\" (UniqueName: \"kubernetes.io/projected/51de09cc-566f-43f0-9f45-72d616dbc216-kube-api-access-vsl69\") pod \"must-gather-hfxph\" (UID: \"51de09cc-566f-43f0-9f45-72d616dbc216\") " pod="openshift-must-gather-c46p7/must-gather-hfxph" Nov 03 21:08:17.307315 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.307284 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rbjl\" (UniqueName: \"kubernetes.io/projected/f3c01be7-f303-487f-9997-a17873e128ac-kube-api-access-5rbjl\") pod \"must-gather-t7948\" (UID: \"f3c01be7-f303-487f-9997-a17873e128ac\") " pod="openshift-must-gather-4hm9z/must-gather-t7948" Nov 03 21:08:17.307441 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.307367 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/51de09cc-566f-43f0-9f45-72d616dbc216-must-gather-output\") pod \"must-gather-hfxph\" (UID: \"51de09cc-566f-43f0-9f45-72d616dbc216\") " pod="openshift-must-gather-c46p7/must-gather-hfxph" Nov 03 21:08:17.307720 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.307702 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/51de09cc-566f-43f0-9f45-72d616dbc216-must-gather-output\") pod \"must-gather-hfxph\" (UID: \"51de09cc-566f-43f0-9f45-72d616dbc216\") " pod="openshift-must-gather-c46p7/must-gather-hfxph" Nov 03 21:08:17.319375 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.319351 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsl69\" (UniqueName: \"kubernetes.io/projected/51de09cc-566f-43f0-9f45-72d616dbc216-kube-api-access-vsl69\") pod \"must-gather-hfxph\" (UID: \"51de09cc-566f-43f0-9f45-72d616dbc216\") " pod="openshift-must-gather-c46p7/must-gather-hfxph" Nov 03 21:08:17.408047 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.407984 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f3c01be7-f303-487f-9997-a17873e128ac-must-gather-output\") pod \"must-gather-t7948\" (UID: \"f3c01be7-f303-487f-9997-a17873e128ac\") " pod="openshift-must-gather-4hm9z/must-gather-t7948" Nov 03 21:08:17.408047 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.408040 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rbjl\" (UniqueName: \"kubernetes.io/projected/f3c01be7-f303-487f-9997-a17873e128ac-kube-api-access-5rbjl\") pod \"must-gather-t7948\" (UID: \"f3c01be7-f303-487f-9997-a17873e128ac\") " pod="openshift-must-gather-4hm9z/must-gather-t7948" Nov 03 21:08:17.408291 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.408271 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f3c01be7-f303-487f-9997-a17873e128ac-must-gather-output\") pod \"must-gather-t7948\" (UID: \"f3c01be7-f303-487f-9997-a17873e128ac\") " pod="openshift-must-gather-4hm9z/must-gather-t7948" Nov 03 21:08:17.422119 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.422099 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rbjl\" (UniqueName: \"kubernetes.io/projected/f3c01be7-f303-487f-9997-a17873e128ac-kube-api-access-5rbjl\") pod \"must-gather-t7948\" (UID: \"f3c01be7-f303-487f-9997-a17873e128ac\") " pod="openshift-must-gather-4hm9z/must-gather-t7948" Nov 03 21:08:17.427021 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.426995 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c46p7/must-gather-hfxph" Nov 03 21:08:17.531046 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.531017 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4hm9z/must-gather-t7948" Nov 03 21:08:17.705321 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.705294 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c46p7/must-gather-hfxph"] Nov 03 21:08:17.707386 ip-10-0-132-83 kubenswrapper[5059]: W1103 21:08:17.707348 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51de09cc_566f_43f0_9f45_72d616dbc216.slice/crio-bac11fb50483d30f9282ad2f62fca9c45be5204f3f0a68653c28e30d8fc32f28 WatchSource:0}: Error finding container bac11fb50483d30f9282ad2f62fca9c45be5204f3f0a68653c28e30d8fc32f28: Status 404 returned error can't find the container with id bac11fb50483d30f9282ad2f62fca9c45be5204f3f0a68653c28e30d8fc32f28 Nov 03 21:08:17.709207 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.709190 5059 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:08:17.804915 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.804890 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4hm9z/must-gather-t7948"] Nov 03 21:08:17.806571 ip-10-0-132-83 kubenswrapper[5059]: W1103 21:08:17.806546 5059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3c01be7_f303_487f_9997_a17873e128ac.slice/crio-5ce2ff7e018b4357bda6bc79b4a73f97bc3dcc8ad10197d74ce99bbd2ac2f8f5 WatchSource:0}: Error finding container 5ce2ff7e018b4357bda6bc79b4a73f97bc3dcc8ad10197d74ce99bbd2ac2f8f5: Status 404 returned error can't find the container with id 5ce2ff7e018b4357bda6bc79b4a73f97bc3dcc8ad10197d74ce99bbd2ac2f8f5 Nov 03 21:08:17.891237 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.891203 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4hm9z/must-gather-t7948" event={"ID":"f3c01be7-f303-487f-9997-a17873e128ac","Type":"ContainerStarted","Data":"5ce2ff7e018b4357bda6bc79b4a73f97bc3dcc8ad10197d74ce99bbd2ac2f8f5"} Nov 03 21:08:17.892326 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:17.892301 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c46p7/must-gather-hfxph" event={"ID":"51de09cc-566f-43f0-9f45-72d616dbc216","Type":"ContainerStarted","Data":"bac11fb50483d30f9282ad2f62fca9c45be5204f3f0a68653c28e30d8fc32f28"} Nov 03 21:08:18.900657 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:18.900621 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c46p7/must-gather-hfxph" event={"ID":"51de09cc-566f-43f0-9f45-72d616dbc216","Type":"ContainerStarted","Data":"bd30e8819fad59524d1f7cb0c103a30001c54deec3d3ede83edcc05424903ea7"} Nov 03 21:08:19.909982 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:19.909938 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c46p7/must-gather-hfxph" event={"ID":"51de09cc-566f-43f0-9f45-72d616dbc216","Type":"ContainerStarted","Data":"9cb022a674e4610cff8ea5d2f363a3f102c071341a884a78938088013d0c5b7b"} Nov 03 21:08:19.930972 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:19.930912 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-c46p7/must-gather-hfxph" podStartSLOduration=2.184897456 podStartE2EDuration="2.930891566s" totalImagesPullingTime="745.99411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:19.930559168 +0000 UTC m=+5494.188356427" watchObservedRunningTime="2026-11-03 21:08:19.930891566 +0000 UTC m=+5494.188688824" Nov 03 21:08:20.791788 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:20.791750 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wh9tj_116af81a-d3a0-4558-93c4-8bf061b6815f/global-pull-secret-syncer/0.log" Nov 03 21:08:21.024396 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:21.024357 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-q4zkg_4fdbd17a-1e15-4442-b715-aaec1087da11/konnectivity-agent/0.log" Nov 03 21:08:21.160184 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:21.160140 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-83.us-east-2.compute.internal_ea3063cb5f7f736e0060bab60ff36a5e/haproxy/0.log" Nov 03 21:08:27.628686 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:27.627402 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-v5xhc_d50891bc-16c2-497d-8e00-b79ef492ab6e/manager/0.log" Nov 03 21:08:27.926743 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:27.923347 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dcrh4_f44d81a0-55a2-4021-b34f-3400deb30c82/manager/0.log" Nov 03 21:08:29.762142 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:29.762104 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8s8ss_cc202826-dc4c-4f05-b4bc-63fc108188c2/node-exporter/0.log" Nov 03 21:08:29.784991 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:29.784950 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8s8ss_cc202826-dc4c-4f05-b4bc-63fc108188c2/kube-rbac-proxy/0.log" Nov 03 21:08:29.816611 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:29.816574 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8s8ss_cc202826-dc4c-4f05-b4bc-63fc108188c2/init-textfile/0.log" Nov 03 21:08:30.629727 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:30.629685 5059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z"] Nov 03 21:08:30.637011 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:30.636978 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z" Nov 03 21:08:30.643016 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:30.642983 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z"] Nov 03 21:08:30.801552 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:30.801508 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d08b9423-1c99-4408-9164-c4a7bef028e5-lib-modules\") pod \"perf-node-gather-daemonset-mwh2z\" (UID: \"d08b9423-1c99-4408-9164-c4a7bef028e5\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z" Nov 03 21:08:30.802060 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:30.801656 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d08b9423-1c99-4408-9164-c4a7bef028e5-sys\") pod \"perf-node-gather-daemonset-mwh2z\" (UID: \"d08b9423-1c99-4408-9164-c4a7bef028e5\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z" Nov 03 21:08:30.802060 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:30.801739 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxhsm\" (UniqueName: \"kubernetes.io/projected/d08b9423-1c99-4408-9164-c4a7bef028e5-kube-api-access-mxhsm\") pod \"perf-node-gather-daemonset-mwh2z\" (UID: \"d08b9423-1c99-4408-9164-c4a7bef028e5\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z" Nov 03 21:08:30.802060 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:30.801812 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d08b9423-1c99-4408-9164-c4a7bef028e5-podres\") pod \"perf-node-gather-daemonset-mwh2z\" (UID: \"d08b9423-1c99-4408-9164-c4a7bef028e5\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z" Nov 03 21:08:30.802060 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:30.801843 5059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d08b9423-1c99-4408-9164-c4a7bef028e5-proc\") pod \"perf-node-gather-daemonset-mwh2z\" (UID: \"d08b9423-1c99-4408-9164-c4a7bef028e5\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z" Nov 03 21:08:30.903303 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:30.903209 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxhsm\" (UniqueName: \"kubernetes.io/projected/d08b9423-1c99-4408-9164-c4a7bef028e5-kube-api-access-mxhsm\") pod \"perf-node-gather-daemonset-mwh2z\" (UID: \"d08b9423-1c99-4408-9164-c4a7bef028e5\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z" Nov 03 21:08:30.903478 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:30.903301 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d08b9423-1c99-4408-9164-c4a7bef028e5-podres\") pod \"perf-node-gather-daemonset-mwh2z\" (UID: \"d08b9423-1c99-4408-9164-c4a7bef028e5\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z" Nov 03 21:08:30.903478 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:30.903338 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d08b9423-1c99-4408-9164-c4a7bef028e5-proc\") pod \"perf-node-gather-daemonset-mwh2z\" (UID: \"d08b9423-1c99-4408-9164-c4a7bef028e5\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z" Nov 03 21:08:30.903478 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:30.903368 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d08b9423-1c99-4408-9164-c4a7bef028e5-lib-modules\") pod \"perf-node-gather-daemonset-mwh2z\" (UID: \"d08b9423-1c99-4408-9164-c4a7bef028e5\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z" Nov 03 21:08:30.903694 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:30.903486 5059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d08b9423-1c99-4408-9164-c4a7bef028e5-sys\") pod \"perf-node-gather-daemonset-mwh2z\" (UID: \"d08b9423-1c99-4408-9164-c4a7bef028e5\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z" Nov 03 21:08:30.903694 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:30.903620 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d08b9423-1c99-4408-9164-c4a7bef028e5-sys\") pod \"perf-node-gather-daemonset-mwh2z\" (UID: \"d08b9423-1c99-4408-9164-c4a7bef028e5\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z" Nov 03 21:08:30.904031 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:30.904008 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d08b9423-1c99-4408-9164-c4a7bef028e5-podres\") pod \"perf-node-gather-daemonset-mwh2z\" (UID: \"d08b9423-1c99-4408-9164-c4a7bef028e5\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z" Nov 03 21:08:30.904111 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:30.904075 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d08b9423-1c99-4408-9164-c4a7bef028e5-proc\") pod \"perf-node-gather-daemonset-mwh2z\" (UID: \"d08b9423-1c99-4408-9164-c4a7bef028e5\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z" Nov 03 21:08:30.904169 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:30.904158 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d08b9423-1c99-4408-9164-c4a7bef028e5-lib-modules\") pod \"perf-node-gather-daemonset-mwh2z\" (UID: \"d08b9423-1c99-4408-9164-c4a7bef028e5\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z" Nov 03 21:08:30.932475 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:30.932406 5059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxhsm\" (UniqueName: \"kubernetes.io/projected/d08b9423-1c99-4408-9164-c4a7bef028e5-kube-api-access-mxhsm\") pod \"perf-node-gather-daemonset-mwh2z\" (UID: \"d08b9423-1c99-4408-9164-c4a7bef028e5\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z" Nov 03 21:08:30.958462 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:30.958178 5059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z" Nov 03 21:08:31.336589 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:31.336547 5059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z"] Nov 03 21:08:32.026153 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:32.026106 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z" event={"ID":"d08b9423-1c99-4408-9164-c4a7bef028e5","Type":"ContainerStarted","Data":"1ca56e1f0090d7e028923c0da9e25d0d79a1a5f408a4a8e0eb17039dc12547ac"} Nov 03 21:08:32.026722 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:32.026188 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z" event={"ID":"d08b9423-1c99-4408-9164-c4a7bef028e5","Type":"ContainerStarted","Data":"97b1dd62c8498d442132a0d0d544dcedf3b3d948847bcc1853432fd7a4d81cb3"} Nov 03 21:08:32.027729 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:32.027167 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z" Nov 03 21:08:32.050778 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:32.049843 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z" podStartSLOduration=2.049825188 podStartE2EDuration="2.049825188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:32.047548748 +0000 UTC m=+5506.305346005" watchObservedRunningTime="2026-11-03 21:08:32.049825188 +0000 UTC m=+5506.307622447" Nov 03 21:08:36.478582 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:36.478546 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nhf67_419841be-7a31-4cdb-b0b7-9d90a3c376c7/dns/0.log" Nov 03 21:08:36.499245 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:36.499216 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nhf67_419841be-7a31-4cdb-b0b7-9d90a3c376c7/kube-rbac-proxy/0.log" Nov 03 21:08:36.526634 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:36.526598 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4d4ll_94d082d5-1c33-4394-9d93-67645f9b60cd/dns-node-resolver/0.log" Nov 03 21:08:37.348923 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:37.348858 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kzr2l_2398ba99-35d3-4461-b66e-f44ac013b2d2/node-ca/0.log" Nov 03 21:08:38.669843 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:38.669811 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c557dng2q_e5585d8d-d156-418c-ae87-012781bc0e8c/istio-proxy/0.log" Nov 03 21:08:39.056988 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:39.056911 5059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-mwh2z" Nov 03 21:08:39.083611 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:39.083580 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4hm9z/must-gather-t7948" event={"ID":"f3c01be7-f303-487f-9997-a17873e128ac","Type":"ContainerStarted","Data":"5d0e052dd7480177bb2f603887f255c692f4ff2e7f1f858b503db8ac779a12ce"} Nov 03 21:08:39.083799 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:39.083626 5059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4hm9z/must-gather-t7948" event={"ID":"f3c01be7-f303-487f-9997-a17873e128ac","Type":"ContainerStarted","Data":"449dfbd95210f0422b4c84c3c5785916e150a26aa8c84e1f6e76ad48a708b544"} Nov 03 21:08:39.103407 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:39.103358 5059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-4hm9z/must-gather-t7948" podStartSLOduration=1.891975824 podStartE2EDuration="22.103344306s" totalImagesPullingTime="20.211368482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:39.101212861 +0000 UTC m=+5513.359010119" watchObservedRunningTime="2026-11-03 21:08:39.103344306 +0000 UTC m=+5513.361141563" Nov 03 21:08:40.627568 ip-10-0-132-83 kubenswrapper[5059]: E1103 21:08:40.627516 5059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hmnx4" containerName="extractor" Nov 03 21:08:40.886375 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:40.884332 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55c765799f-5qgk6_5a3ca718-8b83-4650-b941-3e29e6b80e00/manager/0.log" Nov 03 21:08:41.023616 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:41.023576 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-5sk9w_81fa410e-25d7-4800-a22b-9dea6ccca3ea/istio-proxy/0.log" Nov 03 21:08:41.070102 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:41.070057 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-8gzjz_6462b9bb-1018-4029-922d-eb71527fc301/istio-proxy/0.log" Nov 03 21:08:41.106061 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:41.106006 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-cmtzs_a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08/istio-proxy/0.log" Nov 03 21:08:41.198102 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:41.198031 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-fzr95_191c7922-07dd-4c9c-89d2-d804f638854d/manager/0.log" Nov 03 21:08:41.255706 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:41.255426 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-vkx59_d14046b2-dd71-42a3-b239-0160088faa0a/istio-proxy/0.log" Nov 03 21:08:41.295094 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:41.295007 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-wtlgc_ab375b16-1371-437f-91b4-d5f8580e804d/istio-proxy/0.log" Nov 03 21:08:41.438809 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:41.437263 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn_77396fff-1778-4f09-9e81-05e715fc015e/istio-proxy/0.log" Nov 03 21:08:41.470330 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:41.470038 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6dc578d85f-q59ph_cee2af62-5ba9-4f27-852b-25fb6b43c945/payload-pre-processing/0.log" Nov 03 21:08:43.144789 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:43.144730 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c557dng2q_e5585d8d-d156-418c-ae87-012781bc0e8c/istio-proxy/0.log" Nov 03 21:08:44.155568 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:44.155530 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55c765799f-5qgk6_5a3ca718-8b83-4650-b941-3e29e6b80e00/manager/0.log" Nov 03 21:08:44.477649 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:44.477563 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-fzr95_191c7922-07dd-4c9c-89d2-d804f638854d/manager/0.log" Nov 03 21:08:45.111794 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:45.111766 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fgdh8_18ffe451-b8eb-41ad-a19d-0390f3e1f8e2/serve-healthcheck-canary/0.log" Nov 03 21:08:45.219920 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:45.219887 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-5sk9w_81fa410e-25d7-4800-a22b-9dea6ccca3ea/istio-proxy/0.log" Nov 03 21:08:45.237789 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:45.237758 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-8gzjz_6462b9bb-1018-4029-922d-eb71527fc301/istio-proxy/0.log" Nov 03 21:08:45.262726 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:45.262295 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-cmtzs_a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08/istio-proxy/0.log" Nov 03 21:08:45.354995 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:45.354954 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-vkx59_d14046b2-dd71-42a3-b239-0160088faa0a/istio-proxy/0.log" Nov 03 21:08:45.376774 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:45.376700 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-wtlgc_ab375b16-1371-437f-91b4-d5f8580e804d/istio-proxy/0.log" Nov 03 21:08:45.435197 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:45.435166 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn_77396fff-1778-4f09-9e81-05e715fc015e/istio-proxy/0.log" Nov 03 21:08:45.452329 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:45.452299 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6dc578d85f-q59ph_cee2af62-5ba9-4f27-852b-25fb6b43c945/payload-pre-processing/0.log" Nov 03 21:08:45.772218 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:45.772141 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-wdls7_40834bff-2db0-4690-9cb9-7b4aacdab3ab/insights-operator/0.log" Nov 03 21:08:45.774784 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:45.774655 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-wdls7_40834bff-2db0-4690-9cb9-7b4aacdab3ab/insights-operator/1.log" Nov 03 21:08:45.889242 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:45.889204 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hmnx4_377ffcf5-b097-495d-b986-e61c58ae8ec1/kube-rbac-proxy/0.log" Nov 03 21:08:45.917593 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:45.917558 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hmnx4_377ffcf5-b097-495d-b986-e61c58ae8ec1/exporter/0.log" Nov 03 21:08:45.945486 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:45.945451 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hmnx4_377ffcf5-b097-495d-b986-e61c58ae8ec1/extractor/0.log" Nov 03 21:08:46.672581 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:46.672550 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55c765799f-5qgk6_5a3ca718-8b83-4650-b941-3e29e6b80e00/manager/0.log" Nov 03 21:08:46.982454 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:46.982375 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-fzr95_191c7922-07dd-4c9c-89d2-d804f638854d/manager/0.log" Nov 03 21:08:48.336997 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:48.336962 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c557dng2q_e5585d8d-d156-418c-ae87-012781bc0e8c/istio-proxy/0.log" Nov 03 21:08:48.994107 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:48.994071 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-8464c976fd-dgdfv_32deeada-f159-4266-8026-9b0a9ffa5eb1/manager/0.log" Nov 03 21:08:49.028549 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:49.028517 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-8464c976fd-dgdfv_32deeada-f159-4266-8026-9b0a9ffa5eb1/copy-manifests/0.log" Nov 03 21:08:50.032839 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:50.032799 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-5sk9w_81fa410e-25d7-4800-a22b-9dea6ccca3ea/istio-proxy/0.log" Nov 03 21:08:50.064801 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:50.064755 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-8gzjz_6462b9bb-1018-4029-922d-eb71527fc301/istio-proxy/0.log" Nov 03 21:08:50.089909 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:50.089880 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-cmtzs_a778c4f2-f0f1-4fb7-8cb4-1e31bae43e08/istio-proxy/0.log" Nov 03 21:08:50.188987 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:50.188950 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-vkx59_d14046b2-dd71-42a3-b239-0160088faa0a/istio-proxy/0.log" Nov 03 21:08:50.209854 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:50.209706 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-wtlgc_ab375b16-1371-437f-91b4-d5f8580e804d/istio-proxy/0.log" Nov 03 21:08:50.282421 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:50.282385 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-9b4747bc-rwnxn_77396fff-1778-4f09-9e81-05e715fc015e/istio-proxy/0.log" Nov 03 21:08:50.301751 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:50.301645 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-6dc578d85f-q59ph_cee2af62-5ba9-4f27-852b-25fb6b43c945/payload-pre-processing/0.log" Nov 03 21:08:50.367879 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:50.367847 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c557dng2q_e5585d8d-d156-418c-ae87-012781bc0e8c/istio-proxy/0.log" Nov 03 21:08:50.542151 ip-10-0-132-83 kubenswrapper[5059]: I1103 21:08:50.542067 5059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-5d9f95b69-x7n2b_d9f37244-d91a-480d-baa1-b1d59335804c/manager/0.log"